The controllers in your environment are upgraded at the cluster level. If an upgrade is available for the NSX Controller cluster, an upgrade link appears next to the primary NSX Manager in thepanel.
It is recommended that you upgrade the controllers during a maintenance window.
The NSX Controller upgrade causes an upgrade file to be downloaded to each controller node. The controllers are upgrade one at a time. While an upgrade is in progress, the Upgrade Available link is not clickable, and API calls to upgrade the controller cluster are blocked until the upgrade is complete.
If you deploy new controllers before the existing controllers are upgraded, they are deployed as the old version. Controller nodes must be the same version to join a cluster.
Ensure that all of the controllers are in the normal state. Upgrading is not possible when one or more of the controllers are in the disconnected state. To reconnect a disconnected controller, try resetting the controller virtual appliance. In the Hosts and Clusters view, right-click the controller and select Power > Reset.
A valid NSX Controller cluster contains three controller nodes. Log in to the three controller nodes and run the show controller-cluster status command.
controller-node# show control-cluster status Type Status Since -------------------------------------------------------------------------------- Join status: Join complete 05/04 02:36:03 Majority status: Connected to cluster majority 05/19 23:57:23 Restart status: This controller can be safely restarted 05/19 23:57:12 Cluster ID: ff3ebaeb-de68-4455-a3ca-4824e31863a8 Node UUID: ff3ebaeb-de68-4455-a3ca-4824e31863a8 Role Configured status Active status -------------------------------------------------------------------------------- api_provider enabled activated persistence_server enabled activated switch_manager enabled activated logical_manager enabled activated directory_server enabled activated
For Join status, verify the controller node is reporting Join Complete.
For Majority status, verify the controller is connected to the cluster majority.
For Cluster ID, all the controller nodes in a cluster should have the same cluster ID.
For Configured status and Active status, verify that the all the controller roles are enabled and activated.
Make sure that you understand the operational impact of the NSX Controller upgrade while the upgrade is in progress. See Operational Impacts of NSX Upgrades.
The controllers in your environment are upgraded and rebooted one at a time. After you initiate the upgrade, the system downloads the upgrade file, upgrades each controller, reboots each controller, and updates the upgrade status of each controller. The following fields display controller status:
The Controller Cluster Status column in the NSX Manager section displays the upgrade status of the cluster. When the upgrade begins, the status says Downloading upgrade file. When the upgrade file has been downloaded on all controllers in the cluster, the status changes to In progress. After all the controllers in the cluster have been upgraded, the status displayed is Complete, and then this column is no longer displayed.
The Status column in the NSX Controller nodes section displays the status of each controller, which is Normal to begin with. When the controller services are shut down and the controller is rebooted, the status changes to Disconnected. After the upgrade for that controller is complete, the status is Normal again.
The Upgrade Status column in the NSX Controller nodes section displays the upgrade status for each controller. The status displays Downloading upgrade file to begin with, then displays Upgrade in progress, and then Rebooting. After the controller is upgraded, the status displays Upgraded.
When the upgrade is complete, the Software Version column in the NSX Controller nodes section displays 6.2.buildNumber for each controller. Rerun the show controller-cluster status command to make sure the controllers are able to create a majority. If the NSX Controller cluster majority is not re-formed review controller and NSX Manager logs.
After upgrading controllers, one or more controller nodes may be assigned a new controller ID. This behavior is expected and depends on when the secondary NSX Manager polls the nodes.
The average upgrade time for each upgrade is 6-8 minutes. If the upgrade does not complete within the timeout period (30 minutes), the Upgrade Status column displays Failed. Click Upgrade Available in the NSX Manager section again to resume the upgrade process from the point where it stopped.
If network issues are preventing a successful upgrade within the 30-minute timeout period, you may need to configure a longer timeout period. Work with VMware Support to diagnose and resolve any underlying issues and, if needed, configure a longer timeout period.
If the controller upgrade fails, check connectivity between the controllers and the NSX Manager.
There is a scenario in which the first controller upgrades successfully, and the second controller does not. Assuming you have three controllers in a cluster, the first controller is successfully upgraded to the new version, and the second controller is being upgraded. If the upgrade of the second controller fails, the second controller might be left in a disconnected state. At the same time, the first and third controllers now have two different versions (one upgraded, one not) and are therefore unable to form a majority. At this point, the upgrade cannot be relaunched. To work around this scenario, create another controller. The newly created controller will be of the older version (matching controller three) and will therefore form a majority with controller three. At this point, you can relaunch the upgrade procedure.