Deploy Multus on Workload Clusters

Multus CNI is a Container Network Interface (CNI) plugin for Kubernetes that lets you attach multiple network interfaces to a single pod and associate each with a different address range.

This topic explains how to install the Multus package onto a workload cluster deployed by a standalone management cluster and use it to create pods with multiple network interfaces. For example, Antrea or Calico as the primary CNI, and a secondary interface such as macvlan or ipvlan, or SR-IOV or DPDK devices for hardware or accelerated interfaces.

Binaries for macvlan and ipvlan are already installed in the workload cluster node template.

Note

Multus CNI is not supported for workload clusters deployed by a vSphere with Tanzu Supervisor.

Prerequisites

Install the Multus CNI Package

Note

Once the Multus CNI is installed in a cluster, it should not be deleted. See Deleting Multus Unsupported below.

To install the Multus CNI package on a workload cluster and configure the cluster to use it:

  1. If the cluster does not have a package repository with the Multus CNI package installed, such as the tanzu-standard repository, install one:

    tanzu package repository add PACKAGE-REPO-NAME --url PACKAGE-REPO-ENDPOINT --namespace tkg-system
    

    Where:

    • PACKAGE-REPO-NAME is the name of the package repository, such as tanzu-standard or the name of a private image registry configured with ADDITIONAL_IMAGE_REGISTRY variables.
    • PACKAGE-REPO-ENDPOINT is the URL of the package repository.

      • For this release, the tanzu-standard URL is projects.registry.vmware.com/tkg/packages/standard/repo:v2023.10.16. See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.
  2. (Optional) To configure Multus:

    1. Create a configuration file that retrieves the Multus parameters and deploys it as a Daemonset.

      tanzu package available get multus-cni.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
      

      Where PACKAGE-VERSION is the version of the Multus package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, multus-data-values.yaml.

      See entrypoint.sh parameters in the Multus CNI repository for information about the settings for the configuration file.

    2. Run the tanzu package available list command to list the available versions of the Multus package, for example:

      tanzu package available list multus-cni.tanzu.vmware.com -A
       NAME                        VERSION              RELEASED-AT                   NAMESPACE
       multus-cni.tanzu.vmware.com 4.0.1+vmware.1-tkg.1 2021-06-04 18:00:00 +0000 UTC tanzu-package-repo-global
       multus-cni.tanzu.vmware.com 3.8.0+vmware.1-tkg.1 2021-06-04 18:00:00 +0000 UTC tanzu-package-repo-global
       multus-cni.tanzu.vmware.com 3.7.1+vmware.2-tkg.2 2021-06-04 18:00:00 +0000 UTC tanzu-package-repo-global
      
      Note

      Make sure that your custom image registry can be reached if you are operating in a network-restricted environment.

    3. Run the tanzu package available get command with --values-schema to see which field values can be set:

      tanzu package available get multus-cni.tanzu.vmware.com/VERSION --values-schema -o FORMAT
      

      Where: - VERSION is a version listed in the tanzu package available list output - FORMAT is either yaml or json

    4. Populate the multus-data-values.yaml configuration file with your desired field values.

  3. Remove all comments from the multus-data-values.yaml file:

    yq -i eval '... comments=""' multus-data-values.yaml
    
  4. Run tanzu package install to install the package.

    tanzu package install multus-cni --package multus-cni.tanzu.vmware.com --version AVAILABLE-PACKAGE-VERSION --values-file multus-data-values.yaml --namespace TARGET-NAMESPACE
    

    Where:

    • TARGET-NAMESPACE is the namespace in which you want to install the Multus package. For example, the my-packages or tanzu-cli-managed-packages namespace.

      • If the --namespace flag is not specified, the Tanzu CLI installs the package in the default namespace.
      • The specified namespace must already exist, for example from running kubectl create namespace my-packages.
    • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above, for example 4.0.1+vmware.1-tkg.1.
  5. Run tanzu package installed get to check the status of the installed package.

    tanzu package installed get multus-cni --namespace NAMESPACE
    
  6. Create a custom resource definition (CRD) for NetworkAttachmentDefinition that defines the CNI configuration for network interfaces to be used by Multus CNI.

    1. Create a CRD specification. For example, this multus-cni-crd.yaml specifies a NetworkAttachmentDefinition named macvlan-conf that configures a macvlan CNI:

      ---
      apiVersion: "k8s.cni.cncf.io/v1"
      kind: NetworkAttachmentDefinition
      metadata:
        name: macvlan-conf
      spec:
        config: '{
          "cniVersion": "0.3.0",
          "type": "macvlan",
          "master": "ens5",
          "mode": "bridge",
          "ipam": {
            "type": "host-local",
            "subnet": "192.168.1.0/24",
            "rangeStart": "192.168.1.200",
            "rangeEnd": "192.168.1.216",
            "routes": [
              { "dst": "0.0.0.0/0" }
              ],
            "gateway": "192.168.1.1"
          }
        }'
      
    2. Create the resource; for example kubectl create -f multus-cni-crd.yaml

  7. Create a pod with the annotation k8s.v1.cni.cncf.io/networks, which takes a comma-delimited list of the names of NetworkAttachmentDefinition custom resource.

    1. Create the pod specification, for example my-multi-cni-pod.yaml:

      apiVersion: v1
      kind: Pod
      metadata:
        name: sample-pod
        annotations:
          k8s.v1.cni.cncf.io/networks: macvlan-conf
      spec:
        securityContext:
          runAsUser: 1000
          runAsGroup: 3000
          fsGroup: 2000
          seccompProfile:
            type: RuntimeDefault
          runAsNonRoot: true
        containers:
        - name: sample-pod
          image: harbor-repo.vmware.com/dockerhub-proxy-cache/library/busybox:1.28
          command: [ "sh", "-c", "sleep 1h" ]
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
      
    2. Create the pod; for example kubectl create -f my-multi-cni-crd.yaml creates the pod sample-pod.

Once the pod is created, it will have three network interfaces:

  • lo the loopback interface
  • eth0 the default pod network managed by Antrea or Calico CNI
  • net1 the new interface created via the annotation k8s.v1.cni.cncf.io/networks: macvlan-conf.
Note

The default network gets the name eth0 and additional network pod interfaces get the name as net1, net2, and so on.

Validating Multus

Run kubectl describe pod on the pod, and confirm that the annotation k8s.v1.cni.cncf.io/network-status lists all network interfaces. For example:

$ kubectl describe pod sample-pod

Name:         sample-pod
Namespace:    default
Priority:     0
Node:         tcecluster-md-0-6476897f75-rl9vt/10.170.109.225
Start Time:   Thu, 27 May 2021 15:31:20 +0000
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "100.96.1.80"
                    ],
                    "mac": "66:39:dc:63:50:a3",
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/macvlan-conf",
                    "interface": "net1",
                    "ips": [
                        "192.168.1.201"
                    ],
                    "mac": "02:77:cb:a0:60:e3",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: macvlan-conf

Then run kubectl exec sample-pod -- ip a show dev net1 to check if the target interface is up and running with IP listed in annotations above.

Deleting Multus Unsupported

Once the Multus CNI is installed in a cluster, it should not be deleted.

Deleting Multus does not uninstall the the Multus configuration file /etc/cni/net.d/00-multus.conf from the CNI scripts directory, which causes issues such as:

  • Failure to create new pods. This is a known issue; see Issue #461 in the Multus repository.
  • Failure to delete pods created with before Multus is deleted. Pods remain stuck with status Terminating with errors in kubelet log.
check-circle-line exclamation-circle-line close-line
Scroll to top icon