The NSX routing subsystem is enabled by multiple components.

  • NSX Manager
  • Cluster of Controllers
  • ESXi host modules (kernel and UWA)
  • DLR Control VMs
  • ESGs

NSX Manager

NSX Manager provides the following functions relevant to NSX routing:
  • Acts as a centralized management plane, providing the unified API access point for all NSX management operations
  • Installs the Distributed Routing Kernel Module and User World Agents on hosts to prepare them for NSX functions
  • Creates/destroys DLRs and DLR LIFs
  • Deploys/deletes DLR Control VM and ESG via vCenter
  • Configures the Controller Cluster via a REST API and hosts via a message bus:
    • Provides host Control Plane agents with the IP addresses of Controllers
    • Generates and distributes to hosts and controllers the certificates to secure control plane communications
  • Configures ESGs and DLR Control VMs via the message bus
    • Note that ESGs can be deployed on unprepared hosts, in which case VIX will be used in lieu of the message bus

Cluster of Controllers

NSX distributed routing requires Controllers, clustered for scale and availability, which provide the following functions:
  • Support VXLAN and distributed routing control plane
  • Provide CLI interface for statistics and runtime states
  • Elect a master controller node for each DLR instance
    • Master node receives routing information from the DLR Control VM and distributes it to the hosts
    • Sends LIF table to the hosts
    • Keeps track of the host where DLR Control VM resides
    • Selects designated instance for VLAN LIFs and communicates this information to hosts; monitors DI host via control plane keepalives (timeout is 30 seconds, and detection time can be 20-40 seconds), sends hosts an update if the selected DI host disappears

ESXi host modules

NSX routing directly utilizes two User World Agents (UWA) and a routing kernel module and also relies on the VXLAN kernel module for VXLAN connectivity.

Here is a summary of what each of these components does:
  • Control Plane Agent (netcpa) is a TCP (SSL) client that communicates with the Controller using the control plane protocol. It might connect to multiple controllers. netcpa communicates with the Message Bus Client (vsfwd) to retrieve control plane related information from NSX Manager.
  • netcpa packaging and deployment:
    • The agent is packaged into the VXLAN VIB (vSphere installation bundle)
    • Installed by NSX Manager via EAM (ESX Agency Manager) during host preparation
    • Runs as a service daemon on ESXi netcpa
    • Can be started / stopped / queried via its startup script /etc/init.d/netcpad
    • Can be restarted remotely via Networking and Security UI Installation -> Host Preparation -> Installation Status, on individual hosts or on a whole cluster
  • DLR Kernel Module (vdrb) integrates with DVS to enable L3 forwarding
    • Configured by netcpa
    • Installed as part of the VXLAN VIB deployment
    • Connects to DVS via the special trunk called “vdrPort," which supports both VLANs and VXLANs
    • Holds information about DLR instances, with per-instance:
      • LIF and Route tables
      • host-local ARP cache
  • Message Bus Client (vsfwd) is used by netcpa, ESGs, and DLR Control VMs to communicate with the NSX Manager
    • vsfwd obtains NSX Manager’s IP address from /UserVars/RmqIpAddress set by vCenter via vpxa/hosd and logs into the Message Bus server using per-host credentials stored in other /UserVars/Rmq* variables
  • netcpa running on an ESXi host relies on vsfwd to do the following:
    • Obtain host’s control plane SSL private key and certificate from NSX Manager. These are then stored in /etc/vmware/ssl/rui-for-netcpa.*
    • Get IP addresses and SSL thumbprints of Controllers from NSX Manager. These are then stored in /etc/vmware/netcpa/config-by-vsm.xml.
    • Create and delete DLR instances on its host on instruction from NSX Manager
  • Packaging and deployment
    • Same as netcpa, it’s a part of the VXLAN VIB
    • Runs as a service daemon on ESXi vsfwd
    • Can be started / stopped / queried via its startup script /etc/init.d/ vShield-Stateful-Firewall
  • ESGs and DLR Control VMs use VMCI channel to vsfwd to receive configuration from NSX Manager

DLR Control VMs and ESGs

  • DLR Control VM is a “route processor” for its DLR instance
    • Has a “placeholder” or a “real vNIC” interfaces for each DLR LIF, along with IP configuration
    • Can run one of two available dynamic routing protocol (BGP or OSPF) and/or use static routes
    • Requires at least one “Uplink” LIF to be able to run OSPF or BGP
    • Computes forwarding table from directly connected (LIF) subnets, static, and dynamic routes, and sends it via its VMCI link to netcpa to the DLR instance’s master Controller
    • Supports HA in Active/Standby VM pair configuration
  • ESG is a self-contained router in a VM
    • Completely independent from the NSX DLR routing subsystem (no NSX control plane integration)
    • Typically used as an upstream gateway for one or more DLRs
    • Supports more than one concurrently running dynamic routing protocol