Repository navigation
Let HA restart a VM without preferring its last host - #14745
Open
bhouse-nexthop wants to merge 1 commit into
Open
bhouse-nexthop wants to merge 1 commit into
bhouse-nexthop wants to merge 1 commit into
Conversation
The HA restart of a VM first tried the host it last ran on, whenever that host was Up and passed the capacity check. That is usually the host that just stopped it. When the host itself stopped the VM, for example its kernel OOM-killed the QEMU process, the VM went straight back onto the same full host. Under memory overcommit, the capacity check still counts that host as having room. Add the cluster-scoped setting vm.ha.restart.consider.last.host, default true (unchanged behaviour). When it is false, the HA restart sets ConsiderLastHost=false, so the deployment planner and vm.allocation.algorithm choose the host. Signed-off-by: Brad House <bhouse@nexthop.ai>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.22 #14745 +/- ##
============================================
- Coverage 18.32% 18.32% -0.01%
- Complexity 16644 16645 +1
============================================
Files 5939 5939
Lines 537787 537793 +6
Branches 65932 65933 +1
============================================
- Hits 98544 98536 -8
- Misses 428033 428047 +14
Partials 11210 11210
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When an HA-enabled VM stops unexpectedly while its host stays Up, the HA restart first tries the VM's last host, which is usually the host that just stopped it. If that host is Up and passes the capacity check, the VM goes straight back there, and the planner and
vm.allocation.algorithmare never asked.That's right when the VM itself failed. It's wrong when the host stopped it, for example when the host's kernel OOM-killed the QEMU process. Under memory overcommit, the capacity check still counts that host as having room, so the VM is restarted on the same full host.
Changing that unconditionally would surprise anyone who relies on HA restarting a VM in place. So this adds a cluster-scoped setting,
vm.ha.restart.consider.last.host, defaulting totrue, which is today's behaviour. When it'sfalse, the HA restart setsConsiderLastHost=false, so the planner and allocators choose the host. They may still choose the same host when it's the best fit.Fixes #14744
Changes
ConfigKey<Boolean>vm.ha.restart.consider.last.hostinHighAvailabilityManagerImpl: Advanced category, dynamic, cluster scope, defaulttrue. It's registered ingetConfigKeys().restart()builds its params in a newgetRestartParams(workType, clusterId):HaTagwhenha.tagis set, andHaOperationfor HA work, both as before;ConsiderLastHost=falsewhen the setting isfalsefor the cluster of the host the VM ran on.ha.tagset, the last host is never preferred anyway, so the setting has no effect there. The description says so.Not changed here
Types of changes
Feature/Enhancement Scale or Bug Severity
Bug Severity
How Has This Been Tested?
HighAvailabilityManagerImplTest:ConsiderLastHost;falsethey set it tofalse;HaOperationis set only for HA work.vm.allocation.algorithm=firstfitleastconsumed. The lab runs a main-based (24.0) build with this commit cherry-picked, not the 4.22 branch itself; the code changed here is the same on both.kill -9on host 1, as the OOM killer does, and HA restarts it.vm.ha.restart.consider.last.host(cluster)true(default)Last host [...] of VM [...] is UP and has enough capacityfalseHow did you try to break this feature and the system with this change?
falseand no other host that fits, the planner still picks the last host, so the restart doesn't fail where it used to succeed.