The controllers in your environment are upgraded at the cluster level. If an upgrade is available for the NSX Controller cluster, an upgrade link appears next to the primary NSX Manager in thepanel.
Upgrade the controllers during a maintenance window.
The NSX Controller upgrade causes an upgrade file to be downloaded to each controller node. The controllers are upgraded one at a time. While an upgrade is in progress, the Upgrade Available link is not clickable, and API calls to upgrade the controller cluster are blocked until the upgrade is complete.
When the controllers are deleted, any associated DRS anti-affinity rules are deleted. You must create new anti-affinity rules in vCenter to prevent the new controller VMs from residing on the same host.
If you deploy new controllers before the controller upgrade has started, they are deployed as the old version. If you deploy new controllers after the controller upgrade has started, they are deployed as the new version. Controller nodes must be the same version to join a cluster.
Ensure that all the controllers are in the normal state. Upgrading is not possible when one or more of the controllers are in the disconnected state. To reconnect a disconnected controller, try resetting the controller virtual appliance. In the Hosts and Clusters view, right-click the controller and select Power > Reset. For more information about troubleshooting the NSX Controller cluster, see "NSX Controller Cluster Failures" in the NSX Troubleshooting Guide.
A valid NSX Controller cluster contains three controller nodes. Log in to the three controller nodes and run the show control-cluster status command.
controller-node# show control-cluster status Type Status Since -------------------------------------------------------------------------------- Join status: Join complete 05/04 02:36:03 Majority status: Connected to cluster majority 05/19 23:57:23 Restart status: This controller can be safely restarted 05/19 23:57:12 Cluster ID: ff3ebaeb-de68-4455-a3ca-4824e31863a8 Node UUID: ff3ebaeb-de68-4455-a3ca-4824e31863a8 Role Configured status Active status -------------------------------------------------------------------------------- api_provider enabled activated persistence_server enabled activated switch_manager enabled activated logical_manager enabled activated directory_server enabled activated
- For Join status, verify that the controller node is reporting Join Complete.
- For Majority status, verify that the controller is connected to the cluster majority.
- For Cluster ID, all the controller nodes in a cluster have the same cluster ID.
- For Configured status and Active status, verify that the all the controller roles have the status of enabled and activated.
- Make sure that you understand the operational impact of the NSX Controller upgrade while the upgrade is in progress. See Operational Impacts of NSX Upgrades.
- The NSX Controller cluster must contain three controller nodes. If it has fewer than three, you must add additional nodes before starting the upgrade. See "Deploy NSX Controller Cluster" in the NSX Installation Guide for steps to add controller nodes.
- Click Upgrade Available in the Controller Cluster Status column.
The controllers in your environment are upgraded and rebooted one at a time. After you initiate the upgrade, the system downloads the upgrade file, upgrades each controller, reboots each controller, and updates the upgrade status of each controller.
- Monitor the progress of the upgrade.
- You can view the cluster upgrade progress in the Controller Cluster Status column in .
- You can view the upgrade progress of each individual controller node in the Upgrade Status column in .
When the upgrade is complete, the Software Version column in the NSX Controller nodes section displays 6.4.buildNumber for each controller. Rerun the show control-cluster status command to make sure that the controllers can create a majority. If the NSX Controller cluster majority is not achieved, review the controller and NSX Manager logs.
After upgrading controllers, one or more controller nodes may be assigned a new controller ID. This behavior is expected and depends on when the secondary NSX Manager polls the nodes.
The average upgrade time for each upgrade is 6–8 minutes. If the upgrade does not finish within the timeout period (30 minutes), the Upgrade Status column displays Failed. Click Upgrade Available in the NSX Manager section again to resume the upgrade process from the point where it stopped.
If network problems are preventing a successful upgrade within the 30-minute timeout period, work with VMware Support to diagnose and resolve any underlying problems.
If the controller upgrade fails, check connectivity between the controllers and the NSX Manager.
There is a scenario in which the first controller upgrades successfully, and the second controller does not. Assuming you have three controllers in a cluster, the first controller is successfully upgraded to the new version, and the second controller is being upgraded. If the upgrade of the second controller fails, the second controller might be left in a disconnected state. At the same time, the first and third controllers now have two different versions (one upgraded, one not) and are therefore unable to form a majority. At this point, the upgrade cannot be relaunched. To work around this scenario, create another controller. The newly created controller has the newer NSX version (matching controller one) and can form a majority with controller one. At this point, you can relaunch the upgrade procedure. See "Redeploy an NSX Controller" in the NSX Troubleshooting Guide for instructions on creating another controller.