Configure Direct Connect to a Private Virtual Interface for SDDC Management and Compute Network Traffic

Create a private VIF over your DX to provide direct connectivity between your on-premises network and the SDDC's workloads, ESXi Management, and Management Appliances using their private IPs.

Create one private virtual interface (VIF) for each Direct Connect (DX) circuit you want to attach to your SDDC. Each private VIF establishes a separate BGP session, which can be used in active/standby or active/active (including ECMP) designs or used for private network segments. If you want DX redundancy, attach separate private VIFs provisioned on different DX circuits to the SDDC.

When connecting multiple private VIFs over separate DX circuits to an SDDC for high availability, all the DX circuits must be created in the same AWS account and delivered to different AWS Direct Connect Locations. When you do this, AWS attempts to leverage separate internal network paths for the DX connectivity to provide better redundancy. See High resiliency and Active/Active and Active/Passive Configurations in AWS Direct Connect in the AWS documentation. See VMware Configuration Maximums for limits on the number of network segments advertised to all private VIFs. Route aggregation is supported to provide more flexibility, but all VIFs will have the same networks advertised by the SDDC.

Important:

When you connect a DX private virtual interface or an SDDC Group to an SDDC, all outbound traffic from ESXi hosts to destinations outside the SDDC network is routed over that interface, regardless of other routing configurations in the SDDC. This includes vMotion and vSphere replication traffic. You must ensure that inbound traffic to ESXi hosts is also routed over the same path so that the inbound and outbound traffic paths are symmetrical. See Creating and Managing SDDC Deployment Groups with VMware Transit Connect in the VMware Cloud on AWS Operations Guide for more about VMware Transit Connect and the VMware Managed Transit Gateway (VTGW).

Although routes learned from a route-based VPN are advertised over BGP to other route-based VPNs, an SDDC advertises only its own networks to an SDDC group. It does not advertise routes learned from VPNs. See AWS Direct Connect quotas in the AWS Direct Connect User Guide for detailed information about limits imposed by AWS on Direct Connect, including limits on routes advertised and learned over BGP.

When you create a private VIF in this way, you can attach it to any of your Organization’s SDDCs in the region where you created the VIF. The private VIF must be created in the same region as the DX circuit, and attached to an SDDC in that same region. After you attach it to an SDDC, the VIF cannot be detached or reassigned to another SDDC. Instead, it must be deleted and a new VIF created. Deleting an SDDC deletes any attached VIFs.

Prerequisites

• Ensure that you meet the prerequisites for virtual interfaces as described in Prerequisites for Virtual Interfaces.
• If you want to use route-based VPN as the backup to Direct Connect, you'll also need to set the Use VPN as backup to Direct Connect switch to Enabled as shown in Step 6. Policy-based VPNs cannot be used to back up another connection.

Procedure

1. Log in to VMware Cloud Services at https://vmc.vmware.com.
2. Click Inventory > SDDCs, then pick an SDDC card and click VIEW DETAILS.
3. Click OPEN NSX MANAGER and log in with the NSX Manager Admin User Account shown on the SDDC Settings page. See SDDC Network Administration with NSX Manager.
   You can also use the VMware Cloud Console Networking & Security tab for this workflow.
4. Log in to the AWS Console and complete the Creating a Hosted Private Virtual Interface procedure under Create a Hosted Virtual Interface.
   If you're using a hosted VIF (see Set Up an AWS Direct Connect Connection), work with your AWS Direct Connect Partner to create the VIF in the account shown in the AWS Account ID field of the Direct Connect page, then skip to Step 5 of this procedure. If you are using a dedicated or hosted connection, take these steps first.
   1. For Virtual interface type, choose Private and make up a Virtual interface name.
   2. For the Virtual interface Owner field, select Another AWS account and use the AWS Account ID from the NSX Direct Connect page.
   3. For VLAN, use the value provided by your AWS Direct Connect Partner.
   4. For BGP ASN, use the ASN of the on-premises router where this connection terminates.
      This value must not be the same as the BGP Local ASN shown on the NSX Direct Connect page.
   5. Expand Additional Settings and make the following choices:

      Address family	Select IPV4
      Your router peer ip	Specify the IP address of the on-premises end of this connection (your router), or leave blank to have AWS automatically assign an address that you'll need to configure in your router.
      Amazon router peer ip	Specify the IP address of the AWS end of this connection, or leave blank to have AWS automatically assign an address that you'll need to configure in your router.
      BGP authentication key	Specify a value or leave blank to have AWS generate a key, which you'll need to configure in your router.
      Jumbo MTU (MTU size 9001)	The default MTU for all SDDC networks is 1500 bytes. To enable DX traffic to this private VIF to use a larger MTU, select Enable under Jumbo MTU (MTU size 9001). After the VIF has been created, you'll also need to open the NSX Global Configuration page and set a higher MTU value under Intranet Uplink, as described in Specify the Direct Connect MTU. Enabling this in the connection properties, even if you don't intend to use it right away, makes it easier to take advantage of jumbo frames in SDDC networks when you need them.

   When the interface has been created, the AWS console reports that it is ready for acceptance.
5. Open NSX Manager or the VMC Console Networking & Security tab. Click Direct Connect and accept the virtual interface by clicking ATTACH.
   Before it has been accepted, a new VIF is visible in all SDDCs in your organization. After you accept the VIF, it is no longer visible in any other SDDC.
   It can take up to 10 minutes for the BGP session to become active. When the connection is ready, the State shows as Attached and the BGP Status as Up.
6. (Optional) Configure a route-based VPN as the backup to Direct Connect.
   In the default configuration, traffic on any route advertised over BGP by both DX and a route-based VPN uses the VPN by default. To have a route advertised by both DX and VPN use DX by default and failover to the VPN when DX is unavailable click Direct Connect and set the Use VPN as backup to Direct Connect switch to Enabled.

   Note: This configuration requires a route-based VPN. You cannot use a policy-based VPN as a backup to Direct Connect. In an SDDC that is a member of an SDDC group, traffic over a route that is advertised by both the DX private VIF and the group's VMware Managed Transit Gateway ( VTGW) will be routed over the VTGW.
   The system requires a minute or so to update your routing preference. When the operation completes, routes advertised by both DX and VPN default to the DX connection, using the VPN only when DX is unavailable. Equivalent routes advertised by both DX and VPN prioritize the VPN connection.

Results

A list of Advertised BGP Routes and Learned BGP Routes is displayed as the routes are learned and advertised. Click the refresh icon to refresh these lists. All routed subnets in the SDDC are advertised as BGP routes, along with this subset of management network subnets:

• Subnet 1 includes routes used by ESXi host vmks and router interfaces.
• Subnet 2 includes routes used for Multi-AZ support and AWS integration.
• Subnet 3 includes management VMs.

Disconnected and extended networks are not advertised. Networks attached to custom T1s are not advertised. If route filtering is enabled then networks attached to the default CGW are also not advertised.

Any route aggregations defined and applied to the DX will be advertised as defined. (See Aggregate and Filter Routes to Uplinks).

The actual CIDR blocks advertised to the private VIFs depend on your management subnet CIDR block. The following table shows the CIDR blocks for these routes in an SDDC that uses the default management network CIDR of 10.2.0.0 in block sizes /16, /20, and /22.

Table 1. Advertised Routes for 10.2.0.0 Default MGW CIDR

MGW CIDR	Subnet 1	Subnet 2	Subnet 3
10.2.0.0/23	10.2.0.0/24	10.2.1.0/26	10.2.1.128/25
10.2.0.0/20	10.2.0.0/21	10.2.8.0/23	10.2.12.0/22
10.2.0.0/16	10.2.0.0/17	10.2.128.0/19	10.2.192.0/18

What to do next

Ensure the on-premises vMotion interfaces are configured to use Direct Connect. See Configure vMotion Interfaces for Use with Direct Connect.