NSX prepares the vSphere Distributed Switch that you select for VXLAN by creating a distributed virtual port group for the VTEP VMkernel NICs.

The teaming policy, load balancing method, MTU, and VLAN ID of the VTEPs are chosen during VXLAN configuration. The teaming and load balancing methods must match the configuration of the DVS selected for VXLAN.

The MTU must be set to be at least 1600 and not less than what is already configured on the DVS.

The number of VTEPs created depends on the teaming policy selected and the DVS configuration.

Common Issues During VXLAN Preparation

VXLAN preparation can fail for several reasons:
  • Teaming method chosen for VXLAN does not match what can be supported by the DVS. To review supported methods, see the NSX Network Virtualization Design Guide at https://communities.vmware.com/docs/DOC-27683.
  • Incorrect VLAN ID is chosen for the VTEPs.
  • DHCP selected to assign VTEP IP addresses, but no DHCP server is available.
  • A VMkernel NIC is missing. Resolve the error as described in VXLAN VMkernel NIC Out Of Sync.
  • A VMkernel NIC has a bad IP address. Resolve the error as described in https://kb.vmware.com/kb/2137025.
  • Incorrect MTU setting is chosen for the VTEPs. You should investigate if there is an MTU mismatch as described later in this topic.
  • Incorrect VXLAN gateway is chosen. You should investigate if there is an error while configuring the VXLAN gateway as described later in this topic.

Important Port Numbers

The VXLAN UDP port is used for UDP encapsulation. Prior to NSX 6.2.3, the default VXLAN port number was 8472. In NSX 6.2.3 the default VXLAN port number changed to 4789 for new installations. In NSX 6.2 and later installations that use a hardware VTEP, you must use VXLAN port number 4789. For information on changing the VXLAN port configuration, see "Change VXLAN Port" in the NSX Administration Guide.

Control plane status displays as disabled if the host does not have any active VMs which need a controller connection

Use the show logical-switch commands to view VXLAN details on the host. For details, refer to NSX Command Line Interface Reference.

The show logical-switch host hostID verbose command will display status of control plane as disabled if the host has not been populated with any VMs which require a connection to the controller cluster for forwarding table information. 

Network count:  18
VXLAN network:  32003
Multicast IP:   0.0.0.0
Control plane:  Disabled  <<========
MAC entry count:        0
ARP entry count:        0
Port count:     1

Error while configuring VXLAN gateway

When configuring VXLAN using a static IP pool at Networking & Security > Installation and Upgrade> Host Preparation > Configure VXLAN and the configuration fails to set an IP pool gateway on the VTEP, the VXLAN configuration status enters the Error (RED) state for the host cluster. The error message is “VXLAN Gateway cannot be set on host” and the error status is “VXLAN_GATEWAY_SETUP_FAILURE”.

In the REST API call, GET https://<nsxmgr-ip>/api/2.0/nwfabric/status?resource=<cluster-moid>, the status of VXLAN is as follows: 

<nwFabricFeatureStatus>
<featureId>com.vmware.vshield.nsxmgr.vxlan</featureId>
  <featureVersion>5.5</featureVersion>
  <updateAvailable>false</updateAvailable>
  <status>RED</status>
  <message>VXLAN Gateway cannot be set on host</message>
  <installed>true</installed>
  <enabled>true</enabled>
  <errorStatus>VXLAN_GATEWAY_SETUP_FAILURE</errorStatus>
</nwFabricFeatureStatus>

Workaround: To fix the error, there are two options.

  • Option 1: Remove VXLAN configuration for the host cluster, fix the underlying gateway setup in the IP pool by making sure the gateway is properly configured and reachable, and then reconfigure VXLAN for the host cluster.
  • Option 2: Perform the following steps.
    1. Fix the underlying gateway setup in the IP pool by making sure the gateway is properly configured and reachable.
    2. Put the host (or hosts) into maintenance mode to ensure no VM traffic is active on the host.
    3. Delete the VXLAN VTEPs from the host.
    4. Take the host out of maintenance mode. Taking hosts out of maintenance mode triggers the VXLAN VTEP creation process on NSX Manager. NSX Manager will try to re-create the required VTEPs on the host.

Investigate an MTU mismatch

  • Run the following command to verify if the MTU is configured to 1600 or above:

    ping ++netstack=vxlan -d -s 1572 -I <vmkx hostname_or_IP>

    where vmkx  is the ID of your VMkernel port and  hostname_or_IP  is the IP or hostname of the VMkernel port. 

    This allows you to check the validity of all uplinks. If you are working in a multi-VTEP environment, you can validate all uplinks by running the ping command from each possible VTEP VMkernel source/destination interface to validate all the paths.

  • Check the physical infrastructure. Many times issue gets resolved by a configuration change to the physical infrastructure.
  • Determine whether the issue is confined to a single logical switch, or other logical switches are also affected. Verify if the issue affects all the logical switches.

For more information about the MTU check, see "Verify the NSX Working State" in the NSX Upgrade Guide.