vSphere ESX Agent Manager deploys vSphere installation bundles (VIBs) onto ESXi hosts.

The deployment on hosts requires that DNS be configured on the hosts, vCenter Server, and NSX Manager. Deployment does not require an ESXi host reboot, but any update or removal of VIBs requires an ESXi host reboot.

VIBs are hosted on NSX Manager and are also available as a zip file.

The file can be accessed from https://<NSX-Manager-IP>/bin/vdn/nwfabric.properties. The downloadable zip file differs based on NSX and ESXi version. For example, in NSX 6.4.0, vSphere 6.0 hosts use the file https://<NSX-Manager-IP>/bin/vdn/vibs-6.4.0/6.0-buildNumber/vxlan.zip.

# 6.0 VDN EAM Info
VDN_VIB_PATH.1=/bin/vdn/vibs-6.4.0/6.0-7263674/vxlan.zip
VDN_VIB_VERSION.1=7263674
VDN_HOST_PRODUCT_LINE.1=embeddedEsx
VDN_HOST_VERSION.1=6.0.*

# 6.5 VDN EAM Info
VDN_VIB_PATH.2=/bin/vdn/vibs-6.4.0/6.5-7263674/vxlan.zip
VDN_VIB_VERSION.2=7263674
VDN_HOST_PRODUCT_LINE.2=embeddedEsx
VDN_HOST_VERSION.2=6.5.*

# 6.6 VDN EAM Info
VDN_VIB_PATH.3=/bin/vdn/vibs-6.4.0/6.6-7263674/vxlan.zip
VDN_VIB_VERSION.3=7263674
VDN_HOST_PRODUCT_LINE.3=embeddedEsx
VDN_HOST_VERSION.3=6.6.*

# 6.6.2 VDN EAM Info for SDDC
VDN_VIB_PATH.4=/bin/vdn/vibs-6.4.0-sddc/6.6-7263674/vxlan.zip
VDN_VIB_VERSION.4=7263674
VDN_HOST_PRODUCT_LINE.4=embeddedEsx.sddc
VDN_HOST_VERSION.4=6.6.2

# Single Version associated with all the VIBs pointed by above VDN_VIB_PATH(s)
VDN_VIB_VERSION=6.4.0.7269135

# Legacy vib location. Used by code to discover avaialble legacy vibs.
LEGACY_VDN_VIB_PATH_FS=/common/em/components/vdn/vibs/legacy/

The VIBs installed on a host depends on the NSX and ESXi versions:

ESXi version NSX version VIBs installed
6.0 or later 6.3.2 or earlier
  • esx-vsip
  • esx-vxlan
6.0 or later 6.3.3 or later
  • esx-nsxv
You can view the installed VIBs using the esxcli software vib list command.
[root@esx-01a:~] esxcli software vib list | grep -e vsip -e vxlan
esx-vsip                       6.0.0-0.0.XXXXXXX                     VMware  VMwareCertified   2016-04-20
esx-vxlan                      6.0.0-0.0.XXXXXXX                     VMware  VMwareCertified   2016-04-20
or
esxcli software vib list | grep nsxv
esx-nsxv                       6.0.0-0.0.XXXXXXX                     VMware  VMwareCertified   2017-08-11

Common Issues During Host Preparation

During the preparation of hosts typical kinds of issues that can be encountered are as follows:
  • EAM fails to deploy VIBs.
    • Might be due to incorrect configured DNS on hosts.
    • Might be due to a firewall blocking required ports between ESXi, NSX Manager, and vCenter Server.

    Most of the issues are resolved by clicking the Resolve option. Refer to Installation Status Is Not Ready.

  • A previous VIB of an older version is already installed. This requires user intervention to reboot hosts.
  • NSX Manager and vCenter Server experience communication issues. The Host Preparation tab in the Networking and Security Plug-in not showing all hosts properly:
    • Check if vCenter Server can enumerate all hosts and clusters.

If problem is not fixed with the Resolve option, refer to Problem Not Fixed With the Resolve Option.

Host Preparation (VIBs) Troubleshooting

  • Check communication channel health for the host. See Checking Communication Channel Health.
  • Check vSphere ESX Agent Manager for errors.

    vCenter home > Administration > vCenter Server Extensions > vSphere ESX Agent Manager.

    On vSphere ESX Agent Manager, check the status of agencies that are prefixed with “VCNS160”. If an agency has a bad status, select the agency and view its issues.

    Agency status is displayed as Alert. The issue is that Agent VIB module is not installed.

  • On the host that is having an issue, run the tail /var/log/esxupdate.log command.

    Log file shows the errors that are detected on the host.

Host Preparation (UWA) Troubleshooting

NSX Manager configures two user world agents on all hosts in a cluster:
  • Messaging bus UWA (vsfwd)
  • Control plane UWA (netcpa)

""

In rare cases, the installation of the VIBs succeeds but for some reason one or both of the user world agents is not functioning correctly. This could manifest itself as:
  • The firewall showing a bad status.

    ""

  • The control plane between hypervisors and the controllers being down. Check NSX Manager System Events. Refer to NSX Logging and System Events.

    System Events page displays an event message when the messaging infrastructure on the host is down.

If more than one ESXi host is affected, check the status of message bus service on NSX Manager Appliance web UI under the Summary tab. If RabbitMQ is stopped, restart it.

Summary page on the NSX Manager Appliance UI displays the status of the RabbitMQ component.

If the message bus service is active on NSX Manager:
  • Check the messaging bus user world agent status on the hosts by running the /etc/init.d/vShield-Stateful-Firewall status command on ESXi hosts.
    [root@esx-01a:~] /etc/init.d/vShield-Stateful-Firewall status
    vShield-Stateful-Firewall is running
    
  • Check the message bus user world logs on hosts at /var/log/vsfwd.log.
  • Run the esxcfg-advcfg -l | grep Rmq command on ESXi hosts to show all Rmq variables. There should be 16 Rmq variables.
    [root@esx-01a:~] esxcfg-advcfg -l | grep Rmq
    /UserVars/RmqIpAddress [String] : Connection info for RMQ Broker
    /UserVars/RmqUsername [String] : RMQ Broker Username
    /UserVars/RmqPassword [String] : RMQ Broker Password
    /UserVars/RmqVHost [String] : RMQ Broker VHost
    /UserVars/RmqVsmRequestQueue [String] : RMQ Broker VSM Request Queue
    /UserVars/RmqPort [String] : RMQ Broker Port
    /UserVars/RmqVsmExchange [String] : RMQ Broker VSM Exchange
    /UserVars/RmqClientPeerName [String] : RMQ Broker Client Peer Name
    /UserVars/RmqHostId [String] : RMQ Broker Client HostId
    /UserVars/RmqHostVer [String] : RMQ Broker Client HostVer
    /UserVars/RmqClientId [String] : RMQ Broker Client Id
    /UserVars/RmqClientToken [String] : RMQ Broker Client Token
    /UserVars/RmqClientRequestQueue [String] : RMQ Broker Client Request Queue
    /UserVars/RmqClientResponseQueue [String] : RMQ Broker Client Response Queue
    /UserVars/RmqClientExchange [String] : RMQ Broker Client Exchange
    /UserVars/RmqSslCertSha1ThumbprintBase64 [String] : RMQ Broker Server Certificate base64 Encoded Sha1 Hash
    
  • Run the esxcfg-advcfg -g /UserVars/RmqIpAddress command on ESXi hosts. The output should display the NSX Manager IP address.
    [root@esx-01a:~] esxcfg-advcfg -g /UserVars/RmqIpAddress
    Value of RmqIpAddress is 192.168.110.15
  • Run the esxcli network ip connection list | grep 5671 command on ESXi hosts to check for active messaging bus connection.
    [root@esx-01a:~] esxcli network ip connection list | grep 5671
    tcp         0       0  192.168.110.51:29969            192.168.110.15:5671   ESTABLISHED     35505  newreno  vsfwd
    tcp         0       0  192.168.110.51:29968            192.168.110.15:5671   ESTABLISHED     35505  newreno  vsfwd
    

For problems related to control plane agent, refer to Control Plane Agent (netcpa) Issues.