Tanzu Kubernetes Cluster Design

Tanzu Kubernetes clusters are deployed in the compute workload domains.

Telco Cloud Platform consumes resources from the compute workload domain. Resource pools provide guaranteed resource availability to workloads. Resource pools are elastic; more resources can be added as the resource pool capacity grows. A resource pool can map to a single vSphere cluster or stretch across multiple vSphere clusters. The stretch feature is not available without the use of a VIM such as VMware Cloud Director. Each Kubernetes cluster can be mapped to a Resource Pool. A resource pool can be dedicated to a Kubernetes cluster or shared across multiple clusters.

VMware Telco Cloud Automation now supports v2 workload clusters. v2 workload clusters support additional functionalities such as cross-vCenter deployments and the add-on framework that allows the deployment of additional PaaS-type components into clusters. For more information v2 workloads, see the Telco Cloud Automation documentation.

Table 1. Recommended Resource Workload Domain Design
Design Recommendation	Design Justification	Design Implication
Map the Tanzu Kubernetes Clusters to the vSphere Resource Pool in the compute workload domain.	Enables resource guarantee and resource isolation.	During resource contention, workloads can be starved for resources and experience performance degradation. Note: You must proactively perform monitoring and capacity management and add the capacity before the contention occurs.
Create dedicated DHCP IP subnet pools for the Tanzu Kubernetes Cluster Management network. Ensure a portion of this subnet can be statically allocated for the kube-vip address.	Simplifies the IP address assignment to Kubernetes clusters. Use static reservations to reserve IP addresses in the DHCP pool to long-lived devices. Post DHCP assignment, reserve DHCP addresses for the control-plane nodes.	DHCP servers must be monitored for availability. Ensure that the IP address scopes are not overlapping and the static allocation for kube-vip is in the same subnet as the DHCP allocated addresses.
Place the Tanzu Kubernetes cluster management network on a virtual network that is routable to the management network for vSphere, Harbor, and Airgap servers.	Provides connectivity to vSphere, NSX, and supporting infrastructure. Simplifies network design and reduces the complexity of network security and troubleshooting.	Increases the network address management overhead Increased security configuration to allow traffic between the resource and management domains
Deploy or convert existing workload clusters into v2 clusters.	Provides a framework that supports the deployment of additional PaaS components to the cluster Provides access to advanced cluster topologies	Requires adaptation of any automation process that is currently used to build v1 clusters

When you allocate resource pools to Kubernetes clusters, consider the following guidelines:

Enable 1:1 Kubernetes Cluster to Resource Pool mapping for data plane intensive workloads.
- Reduced resource contention can lead to better performance.
- Better resource isolation and resource guarantees and reporting.
Enable N:1 Kubernetes Cluster to Resource Pool mapping for control plane workloads where resources and infrastructure requirements are shared.
- Efficient use of the server resources
- High workload density
Use vRealize Operation Manager to provide recommendations on the required resource by analyzing performance statistics.
Consider the total number of ESXi hosts and Kubernetes cluster limits.

Management and Workload Kubernetes Clusters

A Kubernetes cluster in Telco Cloud Platform consists of etcd and the Kubernetes control and data planes.

Etcd: Etcd must run in the stacked model, where etcd is co-located with the control plane nodes. An odd number of cluster members to establish a quorum. A 3-node cluster tolerates the loss of a single member, while a 5-node cluster tolerates the loss of two members. In a stacked mode deployment, etcd availability determines the number of Kubernetes Control nodes.

Control Plane node: The Kubernetes control plane must run in redundant mode to avoid a single point of failure. To improve API availability, the kube-vip service is placed in front of the Control Plane nodes. The load balancer must perform health checks to ensure the availability of the API server. The following table lists the HA characteristics of the Control node components:


Component	Availability
API Server	Active/Active
Kube-controller-manager	Active/Passive
Kube-scheduler	Active/Passive

Important:

Do not place CNF workloads on the control plane nodes.

Worker Node

5G workloads are classified based on their performance. Generic workloads such as web services, lightweight databases, monitoring dashboards, and so on are supported adequately using standard configurations on Kubernetes nodes. In addition to the recommendations outlined in the Telco Cloud Platform 5G Edition Performance Tuning Guide, the data plane workload performance can benefit from further tuning in the following areas:

NUMA Topology
CPU Core Affinity
Huge Pages

NUMA Topology: When deploying Kubernetes worker nodes that host high data bandwidth applications, ensure that the processor, memory, and vNIC are vertically aligned and remain within a single NUMA boundary.

The topology manager is a new component in the Kubelet and provides NUMA awareness to Kubernetes at the pod admission time. The topology manager figures out the best locality of resources by pulling topology hints from the Device Manager and the CPU manager. Pods are then placed based on the topology information to ensure optimal performance.

Note:

Topology Manager is optional, if the NUMA placement best practices are followed during the Kubernetes cluster creation.

CPU Core Affinity: CPU pinning can be achieved in different ways. Kubernetes built-in CPU manager is the most common. The CPU manager implementation is based on cpuset. When a VM host initializes, host CPU resources are assigned to a shared CPU pool. All non-exclusive CPU containers run on the CPUs in the shared pool. When the Kubelet creates a container requesting a guaranteed CPU, CPUs for that container are removed from the shared pool and assigned exclusively for the life cycle of the container. When a container with exclusive CPUs is terminated, its CPUs are added back to the shared CPU pool.

The CPU manager includes the following two policies:

None: Default policy. The kubelet uses CFS quota to enforce pod CPU limits. The workload can move between different CPU cores depending on the load on the Pod and the available capacity on the worker node.

Static: With the static policy enabled, the CPU request results in the container getting allocated the whole CPU and no other container can schedule on that CPU.

Note:

For data plane intensive workloads, the CPU manager policy must be set to static to guarantee an exclusive CPU core on the worker node.

CPU Manager for Kubernetes (CMK) is another tool used by selective CNF vendors to assign the core and NUMA affinity for data plane workloads. Unlike the built-in CPU manager, CMK is not bundled with Kubernetes binaries and it requires separate download and installation. CMK must be used over the built-in CPU manager if required by the CNF vendor.

Huge Pages: For Telco workloads, the default huge page size can be 2 MB or 1 GB. To report its huge page capacity, the worker node determines the supported huge page sizes by parsing the /sys/kernel/mm/hugepages/hugepages-{size}kB directory on the host. Huge pages must be set to pre-allocated for maximum performance. Pre-allocated huge pages reduce the amount of available memory on a worker node. A node can only pre-allocate huge pages for the default size. The Transport Huge Pages must not be enabled.

Container workloads requiring huge pages use hugepages-<hugepagesize> in the Pod specification. As of Kubernetes 1.18, multiple huge page sizes are supported per Pod. Huge Pages allocation occurs at the pod level.

Recommended tuning details:


Design Recommendation	Design Justification	Design Implication
Use three control nodes per Kubernetes cluster to ensure full redundancy	3-node cluster tolerates the loss of a single member	Each Control node requires CPU and memory resources. CPU/Memory overhead is high for small Kubernetes cluster sizes.
Limit a single Kubernetes Cluster to a single Linux distribution (PhotonOS)and version, to reduce operational overhead.	Photon OS is an open-source, minimal Linux container host that is optimized for cloud native applications, cloud platforms, and VMware infrastructure. Photon OS is the default Linux Distribution for Tanzu Kubernetes Grid deployment through Telco Cloud Automation. The operating system you use must be consistent across all nodes in the cluster.	Cross-cluster communication requires extra configuration and setup.
Install and activate the NTP clock synchronization service with custom NTP servers.	Kubernetes and its components rely on the system clock to track events, logs, states, and so on.	None
Ensure that Swap is not enabled on the Kubernetes Cluster Nodes.	Swap causes a decrease in the overall performance of the cluster.	None
Vertically align Processor, memory, and vNIC and keep them within a single NUMA boundary for data plane intensive workloads.	High packet throughput can be maintained for data transfer across vNICs within the same NUMA zone than in different NUMA zones. Latency_sensitivity must be enabled for best-effort NUMA placement.	Requires an extra configuration step on the vCenter to ensure NUMA alignment. Note: This is not required for generic workloads such as web services, lightweight databases, monitoring dashboards, and so on.
Set the CPU manager policy to `static` for data plane intensive workloads.	When the CPU manager is used for CPU affinity, the static mode is required to guarantee exclusive CPU cores on the worker node for data-intensive workloads.	Requires an extra configuration step for CPU Manager through NodeConfig Operator. Note: This is not required for generic workloads such as web services, lightweight databases, monitoring dashboards, and so on.
When enabling the static CPU manager policy, set aside sufficient CPU resources for the kubelet operation.	The kubelet requires a CPU reservation to ensure that the shared CPU pool is not exhausted under load. The amount of CPU to be reserved depends on the pod density per node.	Requires an extra configuration step for CPU Manager. Less CPU reservation can impact the Kubernetes cluster stability. Note: This is not required for generic workloads such as web services, lightweight databases, monitoring dashboards, and so on.
Enable huge page allocation at boot time.	Huge pages reduce the TLB miss. Huge page allocation at boot time prevents memory from becoming unavailable later due to fragmentation. Update VM setting for Worker Nodes 1G Hugepage. Enable IOMMUs to protect system memory between I/O devices.	Pre-allocated huge pages reduce the amount of available memory on a worker node. Requires an extra configuration step in the worker node VM GRUB configuration. Enabling huge pages requires a VM reboot. Note: This is not required for generic workloads such as web services, lightweight databases, monitoring dashboards, and so on.
Set the default huge page size to 1 GB. Set the overcommit size to 0.	For 64-bit applications, use 1 GB huge pages if the platform supports them. Overcommit size defaults to 0, no actions required.	For 1 GB pages, the huge page memory cannot be reserved after the system boot. Note: This is not required for generic workloads such as web services, lightweight databases, monitoring dashboards, and so on.
Mount the file system type hugetlbfs on the root file system.	The file system of type hugetlbfs is required by the mmap system call. Create an entry in fstab so the mount point persists after a reboot.	Perform an extra configuration step in the worker node VM configuration. Note: This is not required for generic workloads such as web services, lightweight databases, monitoring dashboards, and so on.