To ensure complete system redundancy, the Avi Load Balancer Controller must be highly available. To provide high availability (HA) for the Controller, add two additional Controller nodes to create a three-node Controller cluster.
HA of the Controller requires three separate Controller instances configured as a 3-node cluster. Start with a single-node Controller deployment and use the following steps to add two additional Controller nodes to form a 3-node cluster.
If the cluster is already deployed and you want to modify its node membership or dismantle the cluster, see Changing NSX Advanced Load Balancer Controller Cluster Configuration.
Prerequisites for Cluster Deployment
A Controller cluster can have three nodes, one leader node, and two follower nodes.
- Leader node
-
The leader can be any single node with configuration or without configuration.
The leader can have SEs connected.
The node must have a static IP address.
Using DHCP can cause issues when nodes reboot and their IP addresses change.
The current release does not support the use of hostnames for cluster configuration.
- Follower nodes
-
An Avi Load Balancer Controller cluster can have 3 nodes, 1 leader node and 2 follower nodes.
Follower nodes must have only factory-default configuration.
In AWS environments, follower nodes must have an initial password configured. For more information, see Changes for Cluster Set-up for AWS Deployments topic in the VMware Avi Load BalancerInstallation Guide.
In all other environments, follower nodes must be using the default initial admin credentials; do not run the initial setup wizard to set an admin password.
Follower nodes are expected to be running the same Avi Load Balancer base+patch version as the leader.
Follower typically is a VM or container created from the the Avi Load Balancer Controller installation package.
Each follower node must have a static IP address.
Cluster Compatibility and Upgrade Requirements
The following requirements are enforced for upgrade (brownfield) and greenfield deployments:
All Controller nodes in a cluster must be homogeneous in terms of memory, CPU, and disk configurations.
All Controller nodes in a cluster need to be in the same software base and patch versions.
All Controller nodes in a cluster must meet a minimum requirement of 6 cores and 32 GB of memory. Due to this requirement, Controllers which are in the Essentials sizing cannot be launched.
Controllers will fail to upgrade if they are in the Essentials flavor (less than 6 cores or 32 GB of memory).
If FIPS mode is enforced, all Controller nodes in a cluster must also be FIPS-enabled.
A sample error message displayed when upgrade fails due to insufficient resources controller trying to upgrade:
[admin:12345]: >show upgrade status detail filter controller|
| message |UC::[Thu Apr 18 07:27:32 2024] |
| |Error while running task: |
| |WaitForAllNodesAfterReboot |
| |{"Summary": ["Insufficient resources. |
| |Essentials Controller is deprecated. |
| |Minimum memory and CPU supported are 32GB |
| |and 6 cores, respectively."], |
| |"Detail": {"1 92.163.0.122": |
| |"[ CPU: 8, Memory: 23.47, Disk: 125.19, |
| |Major Version: {30.2.1 9000}, |
| |Patch Version: , Fips mode: false ]"}} |
| |Traceback (most recent call last): |
| | File "/opt/avi/python/lib/avi/upgrade/ |
| |upgrade_tasks.py", line 264, in start |
| | self.run() |
| | File "/opt/avi/python/lib/avi/upgrade/ |
| |upgrade_tasks.py", line 639, in run |
| | raise RuntimeError(result) |
| |RuntimeError: {"Summary": |
| |Essentials Controller is deprecated. |
| |Minimum memory and CPU supported are 32GB |
| |and 6 cores, respectively."], |
| |"Detail": {"192.163.0.122": |
| |"[ CPU: 8, Memory: 23.47,Disk: 125.19, |
| |Major Version: {30.2.1 9000}, |
| |Patch Version: , Fips mode: false ]"}} |
+---------------+-------------------------------------------+
Node Compatibility Checks
Node compatibility checks are enforced when the cluster is formed as well as when a node tries to rejoin a cluster after reboot.
Performing on-the-fly changes to memory or CPU without rebooting controller nodes and not adhering to homogeneity, may result in the loss of cluster quorum and subsequent failures.
Changing or increasing resources one by one on Controllers of a cluster will fail as the cluster cannot be formed with nodes having different resources. The recommended workflow is to schedule a maintenance window before altering resources on the Controllers, shutting down all the Controller nodes, resizing them, bringing them back up, and allowing them to rejoin the cluster.
[admin:1234]: > show cluster detail
| local_node_states[1] | |
| node_ip | 100.65.9.133 |
| state | CLUSTIFY_CHECK_FAILURE |
| reason | {'Summary': ['Cluster nodes need to be |
| |homogeneous regarding memory, CPU, disk, |
| |version, patch version, and FIPS mode.'] |
| |'Detail': {'100.65.9.132': '[ CPU: 5, |
| |Memory: 31.35, Disk: 125.19, |
| |Major Version:{30.2.1 9044},Patch Version:|
| |, Fips mode: false ]', '100.65.9.133': |
| |'[ CPU: 8, Memory: 31.35, Disk: 125.19, |
| |Major Version: {30.2.1 9044}, |
| | Fips mode: false ]'}} |
| local_node_states[2] | |
| node_ip | 100.65.9.131 |
| state | CLUSTIFY_CHECK_FAILURE |
| reason | {'Summary': ['Cluster nodes need to be |
| |homogeneous regarding memory, CPU, disk, |
| |version, patch version, and FIPS mode.'] |
| |'Detail': {'100.65.9.132': '[ CPU: 5, |
| |Memory: 31.35, Disk: 125.19, MajorVersion:|
| |{30.2.1 9044}, Patch Version: , Fips |
| |mode: false ]', '100.65.9.133': ' |
| |'[ CPU: 8, Memory: 31.35, Disk: 125.19, |
| |Major Version: {30.2.1 9044}, |
| | Fips mode: false ]'}} |
| local_node_states[3] | |
| node_ip |100.65.9.132 |
| state |CLUSTIFY_CHECK_FAILURE |
| reason | {'Summary': ['Insufficient resources. |
| | Essentials Controller is deprecated. |
| | Minimum memory and CPU supported are |
| | 32GB and 6 cores, respectively.'], |
| | 'Detail': {'1 00.65.9.132': '[ CPU: 5, |
| | Memory: 31.35, Disk: 125.19, |
| | Major Version: {30.2.1 9044}, |
| | Patch Version: , Fips mode: false ]'}} |
+----------------------------+------------------------------------------+
Caveats
The current release does not support the use of hostnames for cluster configuration.
