NSX Controllers are deployed by NSX Manager in OVA format. Having a Controller cluster provides high availability.

Deploying Controllers requires that NSX Manager, vCenter Server, and ESXi hosts have DNS and NTP configured.

A static IP pool must be used to assign IP addresses to each Controller.

It is recommended that you implement DRS anti-affinity rules to keep NSX Controllers on separate hosts.

You must deploy three NSX Controllers.

Common Issues with Controllers

During the deployment of NSX Controllers, the typical issues that can be encountered are as follows:

  • NSX Dashboard displaying issue with the connectivity status. The show control-cluster status command is the recommended command to view whether a controller has joined a control cluster. You need to run this on each controller to find out the overall cluster status.

    controller # show control-cluster status
    Type                Status                                       Since
    --------------------------------------------------------------------------------
    Join status:        Join complete                                10/17 18:16:58
    Majority status:    Connected to cluster majority                10/17 18:16:46
    Restart status:     This controller can be safely restarted      10/17 18:16:51
    Cluster ID:         af2e9dec-19b9-4530-8e68-944188584268
    Node UUID:          af2e9dec-19b9-4530-8e68-944188584268
    Role                Configured status   Active status
    --------------------------------------------------------------------------------
    api_provider        enabled             activated
    persistence_server  enabled             activated
    switch_manager      enabled             activated
    logical_manager     enabled             activated
    dht_node            enabled             activated

    Note:

    When you see controller node is disconnected, do NOT use join cluster or force join command. This command is not designed to join node to cluster. Doing this, cluster might enter in to a totally uncertain state.

    Cluster startup nodes are just a hint to the cluster members on where to look when the members start up. Do not be alarmed if this list contains cluster members no longer in service. This will not impact cluster functionality.

    All cluster members should have the same cluster ID. If they do not, then the cluster is in a broken status and you should work with VMware technical support to repair it.

    • The show control-cluster startup-nodes command was not designed to display all nodes currently in the cluster. Instead, it shows which other controller nodes are used by this node to bootstrap membership into the cluster when the controller process restarts. Accordingly, the command output may show some nodes which are shut down or have otherwise been pruned from the cluster.

    • In addition, the show control-cluster network ipsec status command allows to inspect the Internet Protocol Security (IPsec) state. If you see that controllers are unable to communicate between themselves for a few minutes to hours, run the cat /var/log/syslog | egrep "sending DPD request|IKE_SA" command and see if the log messages indicate absence of traffic. You can also run the ipsec statusall | egrep "bytes_i|bytes_o" command and verify that there are no two IPsec tunnels established. Provide the output of these commands and the controller logs when reporting a suspected control cluster issue to your VMware technical support representative.

  • NSX Controller running slowly. This might be caused by insufficient resources. To detect issues with NSX Controller system requirements, run the request system compatibility-report command.

    nsx-controller # request system compatibility-report
    Testing: Number of CPUs. Done.
    Testing: Aggregate CPU speed. Done.
    Testing: Memory. Done.
    Testing: Management NIC speed. Done.
    Testing: NTP configured. Done.
    Testing: /var disk partition size. Done.
    Testing: /var disk speed. Done.
    Testing: pserver-log disk size. Done.
    Testing: pserver-log disk speed. Done.
    Testing: pserver-data disk size. Done.
    Testing: pserver-data disk speed. Done.
    Testing: logging disk size. Done.
    Testing: logging disk speed. Done.
    
                                    Detected      Supported       Required
    Number of CPUs                         2             NO            >=8
    Aggregate CPU speed              5.6 GHz             NO           >=13
    Memory                          1.835 GB             NO           >=63
    Management NIC speed          10000 Mb/s            YES         >=1000
    NTP configured                        No             NO            Yes
    /var disk partition size            - GB             NO          >=128
    /var disk speed                   - MB/s             NO           >=40
    pserver-log disk size               - GB             NO          >=128
    pserver-log disk speed            - MB/s             NO           >=40
    pserver-data disk size              - GB             NO          >=128
    pserver-data disk speed           - MB/s             NO           >=40
    logging disk size                   - GB             NO          >=128
    logging disk speed                - MB/s             NO           >=40
    
  • IP connectivity issues between the NSX Manager and the NSX controllers. This is generally caused by physical network connectivity issues or a firewall blocking communication.

  • Insufficient resources such as storage available on vSphere to host the Controllers. Viewing the vCenter events and tasks log during Controller deployment can identify such issues.

  • A misbehaving "rogue" Controller or an upgraded Controllers in the Disconnected state.

  • DNS on ESXi hosts and NSX Manager have not been configured properly.

  • NTP on ESXi hosts and NSX Manager are not in sync.

  • When newly connected VMs have no network access, this is likely caused by a control-plane issue. Check the Controller status.

    Also try running the esxcli network vswitch dvs vmware vxlan network list --vds-name <name> command on ESXi hosts to check the control-plane status. Note that the Controller connection is down.

  • Running the show log manager follow NSX Manager CLI command can identify any other reasons for a failure to deploy controllers.

For more information on troubleshooting controllers, refer to NSX Controller.