VMConfig and NodeConfig Operator plays a critical role in ensuring that the Tanzu Standard for Telco Kubernetes cluster is configured based on the 5G workload requirements.

VMConfig and NodeConfig Operators are Kubernetes operators developed by VMware to handle the Kubernetes node and OS customization. DPDK binding, Kernel upgrade, OS module installation, and so on can be customized using the NodeConfig Operator. VM-specific operations such as vNIC mapping, Network PortGroup assignment, vCPU pinning, and host memory reservation are handed by the VM Operator.

Based on Infrastructure parameters defined in the CNF CSAR file, TCA pushes the node config's intended state to the NodeConfig and VM Operator.

NodeConfig Operator:

  • Node Profile describes the intent that node-config operator is going to realize. Node profile is stored as a Kubernetes ConfigMap.

  • NodeConfig Daemon is a daemonset running per node to realize the node profile config passed down to the NodeConfig Daemon as ConfigMap.

  • NodeConfig Operator is a k8s operator that handles the node OS configuration, performance tuning, OS upgrade. NodeOperator monitors config update events and forwards events to backend Daemon plugins. There are multiple Daemon plugins. Each plugin is responsible for a specific type of event such as Tuning, Package updates, SR-IOV device management, and so on. After each plugin processes the update events, node labels are used to filter out a set of nodes to receive the node profile.

VMConfig Operator:

  • VMConfig Operator is a K8s operator that handles VM configurations for TKG cluster as the CAPV/CAPI extension. VMConfig Operator runs in the TKG management cluster.

  • VMConfig Operator consists of

    • VM Controller: Monitors VMConfig and CAPI/CAPV events and configures K8s worker nodes on the target workload cluster.

    • ESXInfoController: Responsible for hardware capabilities discovery on an ESXi host.

TCA is the single source of truth for both VMConfig and NodeConfig operators. Based on the infrastructure requirements defined in the network function catalog TOSCA YAML (CSAR file), TCA generates a node profile that describes the intended node config the operator is going to realize. The NodeConfig operator runs as K8s Daemonsets on Kubernetes nodes and configures the worker node to realize the desired states specified by TCA.

Figure 1. NodeConfig Operator Design
NodeConfig Operator Design

VMConfig and NodeConfig Operator Workflow

VMConfig and NodeConfig Operator workflows are triggered by the deployment of the CNF function. Network Functions from various vendors have NF-specific requirements on the TKG infrastructure. To ensure that the Tanzu Kubernetes cluster is equipped with required tuning, the TOSCA descriptor is extended to include the NF's infrastructure requirements. Infrastructure requirements are captured as custom extensions in the descriptor file part of the TOSCA YAML (CSAR). The NFs are uploaded as a catalog offering based on the ETSI-compliant CSAR package.

After the NF is consumed from the TCA catalog and a placement decision is made, TCA generates a VM and node host profile based on the TOSCA descriptor and pushes the profile to the VMConfig and NodeConfig Operator.

NodeConfig Operator processes the node profile and updates the corresponding configMap for NodeConfig Damonset to realize.

Design Recommendation

Design Justification

Design Implication

Do not update node profile or cluster ConfigMap outside of TCA.

  • VMConfig and NodeConfig Operators are part of TCA.

  • Manual changes to the NodeConfig and VMConfig operators are not supported.

  • ConfigMap is maintained by TCA only.

None