For all three Enterprise Edge types, a minimum of four VLANs are recommended. vSphere Distributed Switch (VDS) should be used for simplicity and benefits, while NSX-T is not recommended due to its footprint. VLANs can be collapsed if there is minimal infrastructure. IP prefixes must be unique and routable for TKG Management Cluster and edge applications.
VLAN requirements and L2 Design
All three Enterprise Edge types will share the common VLAN and layer 2 networking designs. vSphere Distributed Switch (VDS) is recommended to simplify the management of VLANs in the vSphere clusters and leverage many of its benefits. NSX-T is not recommended for the edge due to footprint concerns. A minimum of four VLANs are suggested per the illustrated diagram, in which MGMT and VSAN networks can be collapsed into one VLAN. Similarly, TKG WKL and DATA networks can optionally be collapsed into one VLAN if there is minimal infrastructure at the edge. The IP prefixes assigned to the VLANs must be unique to the edge site and routable from the data center. This is required so the workload clusters can be managed by the TKG Management Cluster, and edge applications can be monitored from the DC.
It’s recommended to implement at least four VLANs to segregate traffic flows and minimize lateral movement in case any device or container is compromised. Further, the use of VLANs also simplifies firewall and security service insertion when necessary – security inspections can be introduced at the layer boundaries when traffic has to be routed.
The table below summarizes the requirements for the four VLAN networks. The TKG workload clusters will be placed in the ‘TKG WL’ network and the VMs placed in the ‘DATA’ network. The DATA network is also where the external load balancer and firewall should be placed.
VLAN |
Purpose |
DHCP Required |
Notes |
---|---|---|---|
MGMT |
Management VLAN for vmk0 and IPMI interfaces, and Infrastructure VMs |
No |
|
VSAN/VMOTION |
Connect VMkernel interface for vSAN and vMotion |
No |
Not required for single node |
TKG WKL |
Network for TKG Workload master node and worker nodes |
Yes |
Reserve a fixed range of 10 IP addresses for VIP’s used for TKG workload cluster control plane |
DATA |
Network for virtual machines and other infrastructure |
No |
Inbound client traffic to applications should traverse this VLAN if external load balancers are used instead of Kube-vip or other in-cluster LB |
Edge Network Routing
The edge networks should be routable to the data center without NAT for Enterprise Edge deployments. The network subnets should be unique at each edge site and not overlap with other edges. It is assumed there are routers and firewalls upstream from the Enterprise Edge providing network connectivity back to the data center and other edge sites. It’s suggested that all network routing between edge sites is propagated dynamically.
Optionally, administrators may deploy integrated VMware SD-WAN virtual appliances in ESXi to provide network connectivity within the enterprise and to the cloud. This option provides reliable and secure connectivity to the data center over public Internet links, which is valuable for remote edge locations with poor WAN links. The SD-WAN Edge can also be deployed upstream from the Enterprise Edge as a hardware appliance. The options are covered in the detailed designs.
Firewall Ports and Protocol Requirements
The table below outlines the networking protocols and ports used between Enterprise Edge TKG workload cluster and data center components.
Source |
Destination |
Protocol:Port |
Description |
---|---|---|---|
TKG Edge Nodes |
DNS Server, NTP Server |
UDP:53, UDP:123 |
DNS Service, Time Synchronization |
TKG Edge Nodes |
DHCP Server |
UDP: 67, 68 |
Allows hosts to get DHCP addresses |
TKG Edge Nodes |
vCenter IP |
TCP:443 |
Allows components to access vCenter to create VMs and Storage Volumes |
TKG Edge Nodes |
Harbor Registry |
TCP:443 |
Allows components to retrieve container images. This registry can be a local or a public image registry (projects.registry.vmware.com) |
TKG Management Cluster |
TKG Edge Nodes |
TCP:6443 |
For management cluster to configure workload cluster |
TKG Edge Nodes |
TKG Management Cluster |
TCP:6443 |
Allow workload cluster to register with management cluster |
TKG Edge Nodes |
TKG Management Cluster |
TCP:31234 |
Allow Pinniped concierge on workload cluster to access Pinniped supervisor on management cluster |
TKG Edge Nodes |
NSX ALB Controllers |
TCP:443 |
Allow Avi Kubernetes Operator (AKO) and AKO Operator (AKOO) access to Avi Controller |
NSX ALB Controllers |
vCenter and ESXi Hosts |
TCP:443 |
Allow NSX ALB to discover vCenter objects and deploy SEs as required |
TKG Edge Nodes |
Tanzu Mission Control service |
TCP:443 (*.tmc.cloud.vmware.com) |
Tanzu Mission Control communications |