Skip to content

Host maintenance sends migrated VMs back to their last host, skipping the allocator #14742

Open

Description

@bhouse-nexthop

problem

When a KVM host is put into maintenance, each VM that is live-migrated off it goes back to its last host whenever that host is Up and passes the capacity check. The planner and the configured vm.allocation.algorithm are skipped for those VMs.

Every migration records the source host as the VM's last host, so during a rolling maintenance of a cluster the VMs pile up on the hosts that were evacuated earlier:

  1. Host A is put into maintenance. Its VMs migrate to B, C, D... and now have last host = A.
  2. A is rebooted and comes back Up and Enabled.
  3. Host B is put into maintenance. Every VM on B whose last host is A goes straight back to A, whatever the allocation algorithm would have chosen.
  4. Each later host repeats this, so the hosts evacuated first collect most of the VMs.

With memory overcommit, the capacity check lets the last host take far more VMs than it has memory. On a cluster with memoryOvercommitRatio 4, a rolling maintenance of 12 hosts left one host with about 2.3 TiB of VM memory on 1.1 TiB of RAM, while three others had 700+ GiB free each. VMs that used their memory (Kubernetes nodes) were then OOM-killed on that host. In the management server's log for that day, 376 of 447 migrations off hosts in maintenance were placed by the last-host check (Last host [...] of VM [...] is UP and has enough capacity).

Cause:

  • DeploymentPlanningManagerImpl.planDeployment() calls deployInVmLastHost() before the planner. That returns the VM's last host whenever the host is usable and has capacity, unless the profile sets ConsiderLastHost=false or an HA tag.

  • The two paths that evacuate a host both mark their plan as a migration plan (plan.setMigrationPlan(true)):

    • VirtualMachineManagerImpl.orchestrateMigrateAway(), used by host maintenance and HA;
    • ResourceManagerImpl.migrateAwayVmWithVolumes(), used by host maintenance for VMs on local storage.

    But nothing reads isMigrationPlan(), and neither path sets ConsiderLastHost.

  • migrate() sets the VM's last host to its source host on every migration (vm.setLastHostId(srcHostId)).

A user-initiated migration without a destination host already avoids this. UserVmManagerImpl.chooseVmMigrationDestination() clears the last host first ("Last host does not have higher priority in vm migration").

versions

4.22 and main. The code involved is the same on both, and older releases have the same flow.

The steps to reproduce the bug

  1. Use a cluster of three or more KVM hosts, A, B and C, with vm.allocation.algorithm set to one that spreads VMs, such as firstfitleastconsumed.
  2. Put A into maintenance and wait until its VMs have migrated to B and C. Cancel maintenance on A.
  3. Put B into maintenance.
  4. The VMs on B that came from A in step 2 all migrate back to A, even when C has more free capacity. The management server log shows Last host [... A ...] of VM [...] is UP and has enough capacity for each of them.

What to do about it?

For a migration plan, don't try the last host in deployInVmLastHost(), so the planner and allocators choose the destination. This covers both evacuation paths, and the HA planner fallback of migrateAway(). The source host is still excluded as before.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions