How Do I Troubleshoot the Connection of a VMware Cloud Director Instance to a VMware Cloud on AWS SDDC

After you associate your VMware Cloud Director instance with a VMware Cloud on AWS SDDC in VMware Cloud Director service, you might be unable to perform various tasks.

Problem

After you associate a VMware Cloud Director instance to a VMware Cloud on AWS SDDC, vCenter Server instances might show as disconnected. When you attempt to refresh or reconnect, this results in a failure with a networking exception, such as SocketTimeoutException.

You cannot add, edit or delete networking resources, such as network pools, external networks or edge gateways.

In the vcd-debug-container.log, you see entries similar to

java.lang.RuntimeException: Cannot execute request - Cannot execute request -
      com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: (ByteArrayInputStream); line: 1, column: 2] - Json processing error. - Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: UNKNOWN; line: 1, column: 2] 

.

When VMware Cloud Director makes API calls to NSX, this results in Received Response: 503.

Cause

This happens because the VMware Cloud Director instance cannot communicate to either vCenter Server or NSX via proxy. The communication between VMware Cloud Director and the proxy VM that you deployed during the instance association to the VMware Cloud on AWS SDDC is successful, but the connection between the proxy VM and either vCenter Server or NSX fails.

Prerequisites

1. Locate the proxy VM in the resource pool of the vCenter Server UI. The name of the VM starts with cds-proxy.
2. Reach out to your VMware representative to get the root password for the proxy VM.

Solution

• Verify that the proxy VM is assigned a valid IPv4 address.
  1. Navigate to the proxy VM in the resource pool of the vCenter Server UI and check if the VM has a valid IPv4 address.
  2. If the VM does not have a valid IPv4 address, choose one of the following.
     • Click the Networks tab and select a network with a valid DHCP profile.
     • Log in to the OS of the VM and manually assign an IPv4 address to it by running the following commands.

       sudo ifconfig ens160 IPv4-address netmask mask

       sudo route add default gw gateway-address ens160

• Verify that the IPv4 address of the VM is present in the CDS Proxy Management Group and CDS Proxy Compute VM inventory groups of the VMware Cloud on AWS Console, as well as in the CDS Proxy Rule - xxx NAT rule.
  1. Log in to the VMware Cloud on AWS Console at https://vmc.vmware.com.
  2. Click your SDDC card and then click Networking & Security.
  3. Under Inventory, click Groups.
  4. Click Management groups, then CDS Proxy Management Group.
  5. Click View Members and, in the IP Addresses tab, verify that the IP address of the proxy VM is present.
  6. Click Compute Groups and then click CDS Proxy Compute VM.
  7. In the IP Addresses tab, verify that the IPv4 address of the proxy VM is present.
  8. Under Network, click NAT and verify that the IPv4 address of the proxy VM is present in the CDS Proxy Rule - xxx rule.
• Verify that the proxy VM can resolve the internal IP addresses of vCenter Server and NSX Manager.
  1. If necessary, run an GET request to the VMware Cloud on AWS API to retrieve the FQDN of vCenter Server and NSX Manager.

     GET https://vmc.vmware.com/vmc/api/orgs/vmc-organization-ID/sddcs/vmc-SDDC-ID

  2. Log in to the OS of the proxy VM as root and run the following commands.

     nslookup vcenter-server-FQDN.com

     nslookup nsx-manager-FQDN.com

  3. If the lookup command for vCenter Server fails, add a corresponding entry in the /etc/hosts file.
     internal_vc_ip vcenter-server-FQDN.com
  4. If the lookup for NSX Manager FQDN fails, get the NSX FQDN by running nslookup from the internet and add a corresponding entry in /etc/hosts file.
     public_nsx_ip nsx-manager-FQDN.com
• Verify that you can ping vCenter Server and its IP address is resolving.
• Verify that NSX Manager firewall and NAT rules are configured correctly by running a cURL request.
  curl -v nsx-manager-FQDN.com
  If the firewall and NAT rules are configured correctly, the request results in Error 404 Page Not Found message.
• Verify that the firewall rules on the NSX edge gateway allow connectivity between the proxy VM and vCenter Server, ESXi, and VMware Cloud Director.
  1. Log in to the VMware Cloud on AWS Console at https://vmc.vmware.com.
  2. Click your SDDC card.
  3. Click Networking & Security, and then click Gateway Firewall .
  4. Click the Management Gateway tab and verify that both Proxy VC Access and Proxy ESX Access management gateway firewall rules are configured with the correct IP address for the proxy VM.
  5. Click the Compute Gateway tab and verify that both CDS Proxy Compute Firewall inbound and outbound rules allow connectivity between the proxy VM and VMware Cloud Director service.