Repository navigation
Description
problem
When a KVM host is put into maintenance, each VM that is live-migrated off it goes back to its last host whenever that host is Up and passes the capacity check. The planner and the configured vm.allocation.algorithm are skipped for those VMs.
Every migration records the source host as the VM's last host, so during a rolling maintenance of a cluster the VMs pile up on the hosts that were evacuated earlier:
- Host A is put into maintenance. Its VMs migrate to B, C, D... and now have last host = A.
- A is rebooted and comes back Up and Enabled.
- Host B is put into maintenance. Every VM on B whose last host is A goes straight back to A, whatever the allocation algorithm would have chosen.
- Each later host repeats this, so the hosts evacuated first collect most of the VMs.
With memory overcommit, the capacity check lets the last host take far more VMs than it has memory. On a cluster with memoryOvercommitRatio 4, a rolling maintenance of 12 hosts left one host with about 2.3 TiB of VM memory on 1.1 TiB of RAM, while three others had 700+ GiB free each. VMs that used their memory (Kubernetes nodes) were then OOM-killed on that host. In the management server's log for that day, 376 of 447 migrations off hosts in maintenance were placed by the last-host check (Last host [...] of VM [...] is UP and has enough capacity).
Cause:
-
DeploymentPlanningManagerImpl.planDeployment()callsdeployInVmLastHost()before the planner. That returns the VM's last host whenever the host is usable and has capacity, unless the profile setsConsiderLastHost=falseor an HA tag. -
The two paths that evacuate a host both mark their plan as a migration plan (
plan.setMigrationPlan(true)):VirtualMachineManagerImpl.orchestrateMigrateAway(), used by host maintenance and HA;ResourceManagerImpl.migrateAwayVmWithVolumes(), used by host maintenance for VMs on local storage.
But nothing reads
isMigrationPlan(), and neither path setsConsiderLastHost. -
migrate()sets the VM's last host to its source host on every migration (vm.setLastHostId(srcHostId)).
A user-initiated migration without a destination host already avoids this. UserVmManagerImpl.chooseVmMigrationDestination() clears the last host first ("Last host does not have higher priority in vm migration").
versions
4.22 and main. The code involved is the same on both, and older releases have the same flow.
The steps to reproduce the bug
- Use a cluster of three or more KVM hosts, A, B and C, with
vm.allocation.algorithmset to one that spreads VMs, such asfirstfitleastconsumed. - Put A into maintenance and wait until its VMs have migrated to B and C. Cancel maintenance on A.
- Put B into maintenance.
- The VMs on B that came from A in step 2 all migrate back to A, even when C has more free capacity. The management server log shows
Last host [... A ...] of VM [...] is UP and has enough capacityfor each of them.
What to do about it?
For a migration plan, don't try the last host in deployInVmLastHost(), so the planner and allocators choose the destination. This covers both evacuation paths, and the HA planner fallback of migrateAway(). The source host is still excluded as before.
Activity