For all three Enterprise Edge types, a minimum of four VLANs are recommended. vSphere Distributed Switch (VDS) should be used for simplicity and benefits, while NSX-T is not recommended due to its footprint. VLANs can be collapsed if there is minimal infrastructure. IP prefixes must be unique and routable for TKG Management Cluster and edge applications.

VLAN requirements and L2 Design

All three Enterprise Edge types will share the common VLAN and layer 2 networking designs. vSphere Distributed Switch (VDS) is recommended to simplify the management of VLANs in the vSphere clusters and leverage many of its benefits. NSX-T is not recommended for the edge due to footprint concerns. A minimum of four VLANs are suggested per the illustrated diagram, in which MGMT and VSAN networks can be collapsed into one VLAN. Similarly, TKG WKL and DATA networks can optionally be collapsed into one VLAN if there is minimal infrastructure at the edge. The IP prefixes assigned to the VLANs must be unique to the edge site and routable from the data center. This is required so the workload clusters can be managed by the TKG Management Cluster, and edge applications can be monitored from the DC.

Figure 1. Edge Network VLANs

It’s recommended to implement at least four VLANs to segregate traffic flows and minimize lateral movement in case any device or container is compromised. Further, the use of VLANs also simplifies firewall and security service insertion when necessary – security inspections can be introduced at the layer boundaries when traffic has to be routed.

The table below summarizes the requirements for the four VLAN networks. The TKG workload clusters will be placed in the ‘TKG WL’ network and the VMs placed in the ‘DATA’ network. The DATA network is also where the external load balancer and firewall should be placed.

Table 1. Port group requirements for edges

VLAN

Purpose

DHCP Required

Notes

MGMT

Management VLAN for vmk0 and IPMI interfaces, and Infrastructure VMs

No

VSAN/VMOTION

Connect VMkernel interface for vSAN and vMotion

No

Not required for single node

TKG WKL

Network for TKG Workload master node and worker nodes

Yes

Reserve a fixed range of 10 IP addresses for VIP’s used for TKG workload cluster control plane

DATA

Network for virtual machines and other infrastructure

No

Inbound client traffic to applications should traverse this VLAN if external load balancers are used instead of Kube-vip or other in-cluster LB
For the networks deploying TKG workload clusters, DHCP service is required either through the local DHCP server or DHCP relay – the control plane (master) and worker nodes both require DHCP for IP addressing. A special consideration to be planned for is the use of a virtual IP address (VIP) by the master node(s). The VIP is utilized by the workload cluster’s control plane even if only one master node is deployed. The control plane nodes are load-balanced by Kube-vip and the VIP is controlled by a container running on each control plane node. The VIP is a static IP address chosen by the administrator that should be part of the network subnet. It’s recommended to reserve minimally 10 IPs for the master node and applications that may need Kubernetes Services of type LoadBalancer.  However, this will ultimately be driven by your application needs.

Edge Network Routing

The edge networks should be routable to the data center without NAT for Enterprise Edge deployments. The network subnets should be unique at each edge site and not overlap with other edges. It is assumed there are routers and firewalls upstream from the Enterprise Edge providing network connectivity back to the data center and other edge sites. It’s suggested that all network routing between edge sites is propagated dynamically.

Optionally, administrators may deploy integrated VMware SD-WAN virtual appliances in ESXi to provide network connectivity within the enterprise and to the cloud. This option provides reliable and secure connectivity to the data center over public Internet links, which is valuable for remote edge locations with poor WAN links. The SD-WAN Edge can also be deployed upstream from the Enterprise Edge as a hardware appliance. The options are covered in the detailed designs.

Firewall Ports and Protocol Requirements

The table below outlines the networking protocols and ports used between Enterprise Edge TKG workload cluster and data center components.

Table 2. TKG protocol and port requirements

Source

Destination

Protocol:Port

Description

TKG Edge Nodes

DNS Server, NTP Server

UDP:53, UDP:123

DNS Service, Time Synchronization

TKG Edge Nodes

DHCP Server

UDP: 67, 68

Allows hosts to get DHCP addresses

TKG Edge Nodes

vCenter IP

TCP:443

Allows components to access vCenter to create VMs and Storage Volumes

TKG Edge Nodes

Harbor Registry

TCP:443

Allows components to retrieve container images. This registry can be a local or a public image registry (projects.registry.vmware.com)

TKG Management Cluster

TKG Edge Nodes

TCP:6443

For management cluster to configure workload cluster

TKG Edge Nodes

TKG Management Cluster

TCP:6443

Allow workload cluster to register with management cluster

TKG Edge Nodes

TKG Management Cluster

TCP:31234

Allow Pinniped concierge on workload cluster to access Pinniped supervisor on management cluster

TKG Edge Nodes

NSX ALB Controllers

TCP:443

Allow Avi Kubernetes Operator (AKO) and AKO Operator (AKOO) access to Avi Controller

NSX ALB Controllers

vCenter and ESXi Hosts

TCP:443

Allow NSX ALB to discover vCenter objects and deploy SEs as required

TKG Edge Nodes

Tanzu Mission Control service

TCP:443 (*.tmc.cloud.vmware.com)

Tanzu Mission Control communications