Deploy Multus on Workload Clusters

Multus CNI is a Container Network Interface (CNI) plugin for Kubernetes that lets you attach multiple network interfaces to a single pod and associate each with a different address range.

This topic explains how to install the Multus package onto a workload cluster deployed by a standalone management cluster and use it to create pods with multiple network interfaces. For example, Antrea or Calico as the primary CNI, and a secondary interface such as macvlan or ipvlan, or SR-IOV or DPDK devices for hardware or accelerated interfaces.

Binaries for macvlan and ipvlan are already installed in the workload cluster node template.

Note
Multus CNI is not supported for workload clusters deployed by a vSphere with Tanzu Supervisor.

Prerequisites

A bootstrap machine with the following installed:
- Tanzu CLI, Tanzu CLI plugins, and kubectl, as described in Install the Tanzu CLI and Other Tools for Use with a vSphere with Tanzu Supervisor or Install the Tanzu CLI and Other Tools for Use with Standalone Management Clusters.
- Carvel tools, as described in Install the Carvel Tools.
- yq v4.5 or later.
A Tanzu Kubernetes Grid management cluster and workload cluster running on vSphere, Amazon Web Services (AWS), or Azure.
- Multus CNI requires workload cluster worker nodes of size large or extra-large, as described in Predefined Node Sizes.

Install the Multus CNI Package

Note
Once the Multus CNI is installed in a cluster, it should not be deleted. See Deleting Multus Unsupported below.

To install the Multus CNI package on a workload cluster and configure the cluster to use it:

If the cluster does not have a package repository with the Multus CNI package installed, such as the tanzu-standard repository, install one:

Note
If you are targeting a plan-based cluster (legacy), skip this step. For plan-based clusters, the tanzu-standard package repository is automatically enabled in every cluster, in the tanzu-package-repo-global namespace.
```
tanzu package repository add PACKAGE-REPO-NAME --url PACKAGE-REPO-ENDPOINT --namespace tkg-system
```
Where:
- PACKAGE-REPO-NAME is the name of the package repository, such as tanzu-standard or the name of a private image registry configured with ADDITIONAL_IMAGE_REGISTRY variables.
- PACKAGE-REPO-ENDPOINT is the URL of the package repository.
  - For this release, the tanzu-standard URL is projects.registry.vmware.com/tkg/packages/standard/repo:v2.2.0. See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.
(Optional) To configure Multus:
1. Create a configuration file that retrieves the Multus parameters and deploys it as a Daemonset.
```
tanzu package available get multus-cni.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
```
  Where PACKAGE-VERSION is the version of the Multus package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, multus-data-values.yaml.
  
  See entrypoint.sh parameters in the Multus CNI repository for information about the settings for the configuration file.
2. Run the tanzu package available list command to list the available versions of the Multus package, for example:
```
tanzu package available list multus-cni.tanzu.vmware.com -A
 NAME                        VERSION              RELEASED-AT                   NAMESPACE
 multus-cni.tanzu.vmware.com 3.7.1+vmware.1-tkg.1 2021-06-04 18:00:00 +0000 UTC tanzu-package-repo-global
 multus-cni.tanzu.vmware.com 3.7.1+vmware.2-tkg.1 2021-06-04 18:00:00 +0000 UTC tanzu-package-repo-global
 multus-cni.tanzu.vmware.com 3.7.1+vmware.2-tkg.2 2021-06-04 18:00:00 +0000 UTC tanzu-package-repo-global
```
  Note
  Make sure that your custom image registry can be reached if you are operating in a network-restricted environment.
3. Run the tanzu package available get command with --values-schema to see which field values can be set:
```
tanzu package available get multus-cni.tanzu.vmware.com/VERSION --values-schema -o FORMAT
```
  Where: - VERSION is a version listed in the tanzu package available list output - FORMAT is either yaml or json
4. Populate the multus-data-values.yaml configuration file with your desired field values.
Remove all comments from the multus-data-values.yaml file:
```
yq -i eval '... comments=""' multus-data-values.yaml
```
Run tanzu package install to install the package.
```
tanzu package install multus-cni --package multus-cni.tanzu.vmware.com --version AVAILABLE-PACKAGE-VERSION --values-file multus-data-values.yaml --namespace TARGET-NAMESPACE
```
Where:
- TARGET-NAMESPACE is the namespace in which you want to install the Multus package. For example, the my-packages or tanzu-cli-managed-packages namespace.
  - If the --namespace flag is not specified, the Tanzu CLI installs the package in the default namespace.
  - The specified namespace must already exist, for example from running kubectl create namespace my-packages.
- AVAILABLE-PACKAGE-VERSION is the version that you retrieved above, for example 3.8.0+vmware.3-tkg.1.
Run tanzu package installed get to check the status of the installed package.
```
tanzu package installed get multus-cni --namespace NAMESPACE
```

Create a custom resource definition (CRD) for NetworkAttachmentDefinition that defines the CNI configuration for network interfaces to be used by Multus CNI.

Create a CRD specification. For example, this multus-cni-crd.yaml specifies a NetworkAttachmentDefinition named macvlan-conf that configures a macvlan CNI:

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
    "cniVersion": "0.3.0",
    "type": "macvlan",
    "master": "ens5",
    "mode": "bridge",
    "ipam": {
      "type": "host-local",
      "subnet": "192.168.1.0/24",
      "rangeStart": "192.168.1.200",
      "rangeEnd": "192.168.1.216",
      "routes": [
        { "dst": "0.0.0.0/0" }
        ],
      "gateway": "192.168.1.1"
    }
  }'

Create the resource; for example kubectl create -f multus-cni-crd.yaml

Create a pod with the annotation k8s.v1.cni.cncf.io/networks, which takes a comma-delimited list of the names of NetworkAttachmentDefinition custom resource.
1. Create the pod specification, for example my-multi-cni-pod.yaml:
```
apiVersion: v1
kind: Pod
metadata:
  name: sample-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: sample-pod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: harbor-repo.vmware.com/dockerhub-proxy-cache/library/alpine
```
2. Create the pod; for example kubectl create -f my-multi-cni-crd.yaml creates the pod sample-pod.

Once the pod is created, it will have three network interfaces:

lo the loopback interface
eth0 the default pod network managed by Antrea or Calico CNI
net1 the new interface created via the annotation k8s.v1.cni.cncf.io/networks: macvlan-conf.

Note
The default network gets the name eth0 and additional network pod interfaces get the name as net1, net2, and so on.

Validating Multus

Run kubectl describe pod on the pod, and confirm that the annotation k8s.v1.cni.cncf.io/network-status lists all network interfaces. For example:

$ kubectl describe pod sample-pod

Name:         sample-pod
Namespace:    default
Priority:     0
Node:         tcecluster-md-0-6476897f75-rl9vt/10.170.109.225
Start Time:   Thu, 27 May 2021 15:31:20 +0000
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "100.96.1.80"
                    ],
                    "mac": "66:39:dc:63:50:a3",
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/macvlan-conf",
                    "interface": "net1",
                    "ips": [
                        "192.168.1.201"
                    ],
                    "mac": "02:77:cb:a0:60:e3",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: macvlan-conf

Then run kubectl exec sample-pod -- ip a show dev net1 to check if the target interface is up and running with IP listed in annotations above.

Deleting Multus Unsupported

Once the Multus CNI is installed in a cluster, it should not be deleted.

Deleting Multus does not uninstall the the Multus configuration file /etc/cni/net.d/00-multus.conf from the CNI scripts directory, which causes issues such as:

Failure to create new pods. This is a known issue; see Issue #461 in the Multus repository.
Failure to delete pods created with before Multus is deleted. Pods remain stuck with status Terminating with errors in kubelet log.