Skip to content

HA restart of a VM its host stopped puts it back on that host #14744

Open

Description

@bhouse-nexthop

problem

When an HA-enabled VM stops unexpectedly while its host stays Up, the HA restart first tries the VM's last host, which is usually the host that just stopped it. If that host is Up and passes the capacity check, the VM goes straight back there. The deployment planner and vm.allocation.algorithm are never consulted.

That's the right choice when the VM itself failed, for example a guest crash. It's the wrong one when the host stopped the VM, for example when the host's kernel OOM-killed the QEMU process because the host's VMs together used more memory than it has. With memory overcommit, the capacity check counts allocated memory times the overcommit ratio, so the host that killed the VM still "has capacity". The VM is restarted on the same full host, where it, or one of its neighbours, can be killed again.

We hit this on KVM with memoryOvercommitRatio 4: Kubernetes worker VMs killed by the host's OOM killer were restarted by HA on the hosts that had killed them.

Cause:

  • HighAvailabilityManagerImpl.restart() starts the VM with HaTag (when ha.tag is set) and HaOperation, but never sets VirtualMachineProfile.Param.ConsiderLastHost.
  • DeploymentPlanningManagerImpl.deployInVmLastHost() therefore tries the last host before the planner.
  • A successful start sets the VM's last host to the host it starts on, and a stop the VM didn't ask for doesn't change it, so the last host is the host the VM was stopped on (or, after a migration, the migration's source host).

When the host itself fails, this doesn't arise: a host that isn't Up is never used as the last host.

versions

4.22 and main. The code involved is the same on both.

The steps to reproduce the bug

  1. Use a KVM cluster with two or more hosts, and an HA-enabled VM running on host A.
  2. Kill the VM's QEMU process on A (kill -9 <pid>), the same thing the OOM killer does.
  3. CloudStack reports the VM stopped out of band and schedules an HA restart.
  4. The VM is restarted on A, even with vm.allocation.algorithm set to an algorithm that would choose another host. The management server log shows Last host [Host {... A ...}] of VM [VM instance {...}] is UP and has enough capacity.

Reproduced on a three-host lab cluster with vm.allocation.algorithm=firstfitleastconsumed, the VM's host A being the most consumed: HA restarted it on A, with an empty host available.

What to do about it?

Changing the default would surprise people who rely on HA restarting a VM in place, so make it a setting. Add a cluster-scoped vm.ha.restart.consider.last.host, defaulting to true (today's behaviour). When it is false, the HA restart sets ConsiderLastHost=false, so the planner and allocators choose the host. The planner may still choose the same host when it is the best fit.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions