Skip to content

Let HA restart a VM without preferring its last host - #14745

Open
bhouse-nexthop wants to merge 1 commit into
apache:4.22from
bhouse-nexthop:ha-restart-skip-last-host
Open

bhouse-nexthop wants to merge 1 commit into
apache:4.22from
bhouse-nexthop:ha-restart-skip-last-host

Conversation

@bhouse-nexthop

Copy link
Copy Markdown
Collaborator

Description

When an HA-enabled VM stops unexpectedly while its host stays Up, the HA restart first tries the VM's last host, which is usually the host that just stopped it. If that host is Up and passes the capacity check, the VM goes straight back there, and the planner and vm.allocation.algorithm are never asked.

That's right when the VM itself failed. It's wrong when the host stopped it, for example when the host's kernel OOM-killed the QEMU process. Under memory overcommit, the capacity check still counts that host as having room, so the VM is restarted on the same full host.

Changing that unconditionally would surprise anyone who relies on HA restarting a VM in place. So this adds a cluster-scoped setting, vm.ha.restart.consider.last.host, defaulting to true, which is today's behaviour. When it's false, the HA restart sets ConsiderLastHost=false, so the planner and allocators choose the host. They may still choose the same host when it's the best fit.

Fixes #14744

Changes

  • New ConfigKey<Boolean> vm.ha.restart.consider.last.host in HighAvailabilityManagerImpl: Advanced category, dynamic, cluster scope, default true. It's registered in getConfigKeys().
  • restart() builds its params in a new getRestartParams(workType, clusterId):
    • HaTag when ha.tag is set, and HaOperation for HA work, both as before;
    • ConsiderLastHost=false when the setting is false for the cluster of the host the VM ran on.
  • With ha.tag set, the last host is never preferred anyway, so the setting has no effect there. The description says so.

Not changed here

  • The default keeps today's behaviour.
  • The last host isn't excluded, only no longer preferred. Excluding it would make the restart fail outright when it's the only host that fits.
  • Host failure isn't affected: a host that isn't Up is never used as the last host.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

  • New unit tests in HighAvailabilityManagerImplTest:
    • by default the params don't set ConsiderLastHost;
    • with the setting false they set it to false;
    • HaOperation is set only for HA work.
  • Lab test: a nested KVM cluster of three hosts, with vm.allocation.algorithm=firstfitleastconsumed. The lab runs a main-based (24.0) build with this commit cherry-picked, not the 4.22 branch itself; the code changed here is the same on both.
    • Setup: an HA-enabled VM and two filler VMs on host 1, the most consumed host. Then the VM's QEMU process is killed with kill -9 on host 1, as the OOM killer does, and HA restarts it.
vm.ha.restart.consider.last.host (cluster) HA restarted the VM on Planner log
true (default) host 1, its last host Last host [...] of VM [...] is UP and has enough capacity
false host 2, the least consumed no last-host line

How did you try to break this feature and the system with this change?

  • With false and no other host that fits, the planner still picks the last host, so the restart doesn't fail where it used to succeed.
  • The value is read per cluster at restart time, so a cluster override applies without a restart.

The HA restart of a VM first tried the host it last ran on, whenever
that host was Up and passed the capacity check. That is usually the
host that just stopped it. When the host itself stopped the VM, for
example its kernel OOM-killed the QEMU process, the VM went straight
back onto the same full host. Under memory overcommit, the capacity
check still counts that host as having room.

Add the cluster-scoped setting vm.ha.restart.consider.last.host,
default true (unchanged behaviour). When it is false, the HA restart
sets ConsiderLastHost=false, so the deployment planner and
vm.allocation.algorithm choose the host.

Signed-off-by: Brad House <bhouse@nexthop.ai>
@codecov

codecov Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.32%. Comparing base (b56bbc2) to head (d9bc393).

Files with missing lines Patch % Lines
...java/com/cloud/ha/HighAvailabilityManagerImpl.java 75.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               4.22   #14745      +/-   ##
============================================
- Coverage     18.32%   18.32%   -0.01%     
- Complexity    16644    16645       +1     
============================================
  Files          5939     5939              
  Lines        537787   537793       +6     
  Branches      65932    65933       +1     
============================================
- Hits          98544    98536       -8     
- Misses       428033   428047      +14     
  Partials      11210    11210              
Flag Coverage Δ
uitests 4.08% <ø> (ø)
unittests 19.41% <75.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant