High Availability (HA) Features

The High Availability function is a function that automatically “restarts” (called a failover) a virtual machine on another normally functioning host when a physical host fails. Therefore, it is necessary to make arrangements in advance within the virtual machine’s guest OS so that the system can be automatically restored.

What are the conditions for using HA?

To use HA with the HRPC KVM version, the following conditions must be met:

1. A virtual data center consisting of three or more identically configured nodes.
2. Must be HCI type or 3-tier type, as shared storage (Ceph, ESF with ES) is required.
3. There must be sufficient spare capacity (especially memory) on the equipment. It is important that at least one node’s worth of memory is free in total.
   *For example, in the case of a virtual data center consisting of three nodes, only the memory of two nodes should be used in total.
4. Ensure that backups of virtual machines are taken
   We strongly recommend that you sign a contract with Backup Storage or similar and set up regular backups of your virtual machines.

How to enable HA function?

If you want to enable the HA function, first select the datacenter in the left pane, open HA in the middle pane, and click Group.

Click the Create button in the right pane, create an HA group ID, and check the hosts related to HA. When creating virtual machines on hosts related to HA, you must prepare at least one physical host’s worth of spare memory. If you are participating in HA with a total of four HRPCs equipped with 768GB of memory, the remaining three machines will need 768GB of free memory so that the other devices can continue to operate even if one HRPC experiences a physical failure. If you do not check the HA group, it will be ignored as a HA target even if it is in the same data center.

If a host is deliberately removed from the HA group, and the HA group setting “restricted” is checked, virtual machines belonging to that HA group cannot be started on the removed host.

If nofailback is checked, when a failed physical node reboots and returns, the virtual machines will not be restored but will continue to run at the destination. If nofailback is not checked, they will be returned to the original node by live migration, allowing for strict management of the placement of node hosts and virtual machines. However, if the physical node becomes unstable and is repeatedly rebooted, HA will be activated again and the virtual machines will be repeatedly restarted. For this reason, we recommend checking nofailback.

Also, while live migration is a feature that rarely fails, there is a momentary internal outage that can cause problems in real-time software.

How do I enable HA for a virtual machine?

Proxmox HA will not be applicable unless the virtual machine is under HA management.

To place a virtual machine under HA management, select the virtual machine and select “Manage HA” from the “More▼” pull-down menu in the upper right. Select the ID of the HA group you configured above from the group, set the desired state to started, and then click the Add button.

This virtual machine is now under HA control.

The scope of HA operation varies depending on the hypervisor. In VMware, when HA is enabled in a virtual data center, it affects all virtual machines in the virtual data center, while in XenOrchestra, it only affects the virtual machines in the pool that have HA enabled and have “restart” set.

Relationship between maintenance restart of nodes (hosts) in an HA group and virtual machines

When a node that is part of an HA group is rebooted for maintenance (not due to an unexpected failure), the behavior of the virtual machines will be different depending on whether they are under HA management or not.

Virtual machines under HA management	Live migration to another machine
Virtual machines not managed by HA	It shuts down automatically and remains shut down after reboot

column

Limitations of HA function

HA functions have limitations, and there are problems that can occur when using HA.

1. Unexpected restart of a virtual machine
2. Virtual machine data corruption

Simply put, HA is the ability to start a virtual machine from another node when a node fails, but this is how it works from a God’s eye perspective, and the actual software logic has to be implemented completely differently.

First, it is not possible to strictly define what constitutes a “failure” in a specific node among all three nodes (i.e. node A, node B, and node C). HA is achieved by having each node work in coordination with the others.

When node A loses communication with node B, it is not clear whether a failure has occurred in node B, or whether communication has simply been lost between node A and node B. If node A can communicate with node C from its perspective, it is conceivable that a failure may have occurred in node B.

However, if the data of the virtual machines hosted on Node B is constantly being updated on the shared storage, Node B may be completely isolated.

In the above example, if nodes A and C are communicating, there is a high possibility that node B has failed. Therefore, nodes A and C will attempt to boot the virtual machines hosted on node B themselves.

However, node B may not have failed, but may just be isolated. If a node determines that it is isolated, it will immediately shut down the VMs it hosts in preparation for nodes A and C to potentially start up, which is what causes the unexpected reboot mentioned above.

So what happens if node B doesn’t shut down?

If the same virtual machine is running in two places, node B and another node, the virtual disk of the virtual machine will be corrupted in an instant, which will lead to the destruction of the data of the virtual machine mentioned above.

So your choice is either reboot or destroy your data.

HA needs to coordinate between each node to determine whether it’s down, has simply lost communication, or is slow.

If the above two failures are mutually exclusive, what you want to avoid is data corruption, so shut down immediately.

This decision is made at intervals, and Proxmox does it every 2 minutes. If there is no response for 2 minutes, it is determined that “the host has failed.” Therefore, at the very least, downtime will be 2 minutes + the time it takes for the virtual machine to boot (in reality, it will take longer).

However, if the node speed slows down, memory is swapped out, memory recovery by the ballooning driver causes temporary slowdown, or storage writing slows down, the two minutes can pass quickly and shutdown may not be possible in time. Because what must be protected is data, the system automatically uses a WatchDog Timer or similar, and private clouds run on hardware equipped with a device that can automatically restart the kernel even if the host kernel stops responding.