In case of an NSX Controller failure, you may still have two controllers that are working. The cluster majority is maintained, and the control plane continues to function. Even so, it is important to delete all three controllers and add new ones, so as to maintain a fully functional three-node cluster.
We recommend deleting the controller cluster when one or more of the controllers encounter catastrophic, unrecoverable errors or when one or more of the controller VMs become inaccessible and cannot be fixed.
We recommend deleting all controllers in such a case, even if some of the controllers seem healthy. The recommended process is to create a new controller cluster and use the Update Controller State mechanism on the NSX Manager to synchronize the state to the controllers.
- Login to vSphere Web Client.
- From Networking & Security, click Installation > Management.
- In the NSX Controller nodes section, click each controller and take screen shots/print-screens of the details screens or write down the configuration information for later reference.
- In the NSX Controller nodes section, delete all three of them by selecting each one and clicking the Delete Node (x) icon.
When there are no controllers in the system, the hosts are operating in what is called "headless" mode. New VMs or vMotioned VMs will have networking issues until new controllers are deployed and the synchronization is completed.
- Deploy three new NSX Controller nodes by clicking the Add Node (+) icon.
- In the Add Controller dialog box, select the datacenter on which you are adding the nodes, and configure the controller settings.
- Select the appropriate cluster.
- Select a Host in the cluster and storage.
- Select the distributed port-group.
- Select the IP pool from which IP addresses are to be assigned to the node.
- Click OK, wait for installation to complete, and ensure all nodes have a status of Normal.
- Resynchronize the controller state by clicking Actions > Update Controller State.
Update Controller State pushes the current VXLAN and Distributed Logical Router configuration (including Universal Objects in a Cross-VC NSX deployment) from NSX Manager to the Controller Cluster.