Traffic Performance Issues Related to NSX Edge VMs in Multiple NSX-T Data Center Environment

In a multiple NSX-T Data Center environment, only the NSX Manager which deployed NSX Edge VM can use it for routing and Inter TEP communication. None of the other NSX Managers registered to the same vCenter Server can use it for routing and Inter TEP communication. The other NSX Manager instances consider the NSX Edge VM as a regular VM. This scenario can cause traffic performance issues on the NSX Edge VM.

Problem

In a multiple NSX-T Data Center scenario, you have the following configuration:

• NSX Manager-1 and NSX Manager-2 are registered to the same vCenter Server (compute manager).
• NSX Manager-1 deployed the NSX Edge VM.
• NSX Manager-2 prepared the ESXi host.
• From vSphere Web Client, you perform vMotion of NSX Edge VM to an ESXi host prepared by NSX Manager-2. NSX Manager-2 did not deploy the NSX Edge VM.
• NSX Manager-1 does not recognize NSX Edge as an inventory VM. So, NSX Manager-1 does not apply any DFW rules on it.

After moving the NSX Edge VM to the new ESXi host:

• NSX Manager-2 categorizes NSX Edge as a regular VM and not as an NSX Edge VM. If there are any DFW rules configured, NSX Manager-2 applies any DFW rules on the NSX Edge VM.

  See a sample output,

  https://<NSX Manager-2>/api/v1/fabric/virtual-machines
  {
              “host_id”: “59ac4c38-56b1-4b82-a131-dd9ad119f53d”,
              “source”: {
                  “target_id”: “59ac4c38-56b1-4b82-a131-dd9ad119f53d”,
                  “target_display_name”: “10.172.17.133”,
                  “target_type”: “HostNode”,
                  “is_valid”: true
              },
             …..
              “type”: “REGULAR”,
              “guest_info”: {
                  “os_name”: “Ubuntu Linux (64-bit)“,
                  “computer_name”: “vm”
              },
              “resource_type”: “VirtualMachine”,
              “display_name”: “mgr2_edge1",
              “_last_sync_time”: 1663802733277
          },
  

Cause

As NSX Manager-2 exclude list does not filter out NSX Edge VM, it is considered as a regular VM and not as a NSX Edge VM. So, DFW rules or any third-party firewall rules configured for workloads are applied to the NSX Edge VM too. This scenario might cause traffic disruption.

Solution

1. Log in to the vCenter Server, https://vCenter-Server-IP.
2. As NSX Manager-2 considers the NSX Edge VM as a regular VM, create an NS Group "Edge-VMs-From-Other-Managers" and add the Edge VMs to the NS Group.
3. To identify Edge VMs from NSX Manager-1 that must be added to Exclude lists on NSX Manager-2, follow these steps:
   1. Use display_name that you get after call the following API, https://<NSX Manager-1>/api/v1/transport-nodes?node_types=EdgeNode.
   2. Match the name with the display_name in API response from NSX Manager-2, https://<NSX Manager-2>/api/v1/fabric/virtual-machines.
4. Add NS Group "Edge-VMs-From-Other-Managers" to the DFW Exclusion List and SI Exclusion Lists on NSX Manager-2.
5. Verify and exclude NSX Edge VM from third party firewalls.
6. If Edge VM Id changes, update the NS Group.
7. Before you delete the Edge VM, remove the entry from the Exclude lists.
   Note: Inter TEP communication on the NSX Edge VM is not supported on NSX Manager-2.