RSS
 

Deploy Multus on Workload Clusters

Multus CNI is a Container Network Interface (CNI) plugin for Kubernetes that lets you attach multiple network interfaces to a single pod and associate each with a different address range.

This topic explains how to install the Multus package onto a workload cluster deployed by a standalone management cluster and use it to create pods with multiple network interfaces. For example, Antrea or Calico as the primary CNI, and a secondary interface such as macvlan or ipvlan, or SR-IOV or DPDK devices for hardware or accelerated interfaces.

Binaries for macvlan and ipvlan are already installed in the workload cluster node template.

Note

Multus CNI is not supported for workload clusters deployed by a vSphere with Tanzu Supervisor.

Prerequisites

Install the Multus CNI Package

Note

Once the Multus CNI is installed in a cluster, it should not be deleted. See Deleting Multus Unsupported below.

To install the Multus CNI package on a workload cluster and configure the cluster to use it:

  1. If the cluster does not already have the standard package repository installed, install it:

    Note

    If you are targeting a plan-based cluster (legacy), skip this step. For plan-based clusters, the tanzu-standard package repository is automatically enabled in every cluster, in the tanzu-package-repo-global namespace.

    tanzu package repository add tanzu-standard --url PACKAGE-REPOSITORY-ENDPOINT --namespace tkg-system

    Where PACKAGE-REPOSITORY-ENDPOINT is the URL of the standard package repository. For this release, the URL is projects.registry.vmware.com/tkg/packages/standard/repo:v2.1.1.

    See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.

  2. (Optional) To configure Multus:

    1. Create a configuration file that retrieves the Multus parameters and deploys it as a Daemonset.

      tanzu package available get multus-cni.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH

      Where PACKAGE-VERSION is the version of the Multus package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, multus-data-values.yaml.

      See entrypoint.sh parameters in the Multus CNI repository for information about the settings for the configuration file.

    2. Run the tanzu package available list command to list the available versions of the Multus package, for example:

      tanzu package available list multus-cni.tanzu.vmware.com -A
 NAME                        VERSION              RELEASED-AT                   NAMESPACE
 multus-cni.tanzu.vmware.com 3.7.1+vmware.1-tkg.1 2021-06-04 18:00:00 +0000 UTC tanzu-package-repo-global
 multus-cni.tanzu.vmware.com 3.7.1+vmware.2-tkg.1 2021-06-04 18:00:00 +0000 UTC tanzu-package-repo-global
 multus-cni.tanzu.vmware.com 3.7.1+vmware.2-tkg.2 2021-06-04 18:00:00 +0000 UTC tanzu-package-repo-global
      Note

      Make sure that your custom image registry can be reached if you are operating in a network-restricted environment.

    3. Run the tanzu package available get command with --values-schema to see which field values can be set:

      tanzu package available get multus-cni.tanzu.vmware.com/VERSION --values-schema -o FORMAT

      Where: - VERSION is a version listed in the tanzu package available list output - FORMAT is either yaml or json

    4. Populate the multus-data-values.yaml configuration file with your desired field values.

  3. Remove all comments from the multus-data-values.yaml file:

    yq -i eval '... comments=""' multus-data-values.yaml

  4. Run tanzu package install to install the package.

    tanzu package install multus-cni --package multus-cni.tanzu.vmware.com --version AVAILABLE-PACKAGE-VERSION --values-file multus-data-values.yaml --namespace TARGET-NAMESPACE

    Where:

    • TARGET-NAMESPACE is the namespace in which you want to install the Multus package. For example, the my-packages or tanzu-cli-managed-packages namespace.

      • If the --namespace flag is not specified, the Tanzu CLI installs the package in the default namespace.
      • The specified namespace must already exist, for example from running kubectl create namespace my-packages.
    • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above, for example 3.8.0+vmware.1-tkg.1.

  5. Run tanzu package installed get to check the status of the installed package.

    tanzu package installed get multus-cni --namespace NAMESPACE

  6. Create a custom resource definition (CRD) for NetworkAttachmentDefinition that defines the CNI configuration for network interfaces to be used by Multus CNI.

    1. Create a CRD specification. For example, this multus-cni-crd.yaml specifies a NetworkAttachmentDefinition named macvlan-conf that configures a macvlan CNI:

      ---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
    "cniVersion": "0.3.0",
    "type": "macvlan",
    "master": "ens5",
    "mode": "bridge",
    "ipam": {
      "type": "host-local",
      "subnet": "192.168.1.0/24",
      "rangeStart": "192.168.1.200",
      "rangeEnd": "192.168.1.216",
      "routes": [
        { "dst": "0.0.0.0/0" }
        ],
      "gateway": "192.168.1.1"
    }
  }'

    2. Create the resource; for example kubectl create -f multus-cni-crd.yaml

  7. Create a pod with the annotation k8s.v1.cni.cncf.io/networks, which takes a comma-delimited list of the names of NetworkAttachmentDefinition custom resource.

    1. Create the pod specification, for example my-multi-cni-pod.yaml:

      apiVersion: v1
kind: Pod
metadata:
  name: sample-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: sample-pod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: harbor-repo.vmware.com/dockerhub-proxy-cache/library/alpine

    2. Create the pod; for example kubectl create -f my-multi-cni-crd.yaml creates the pod sample-pod.

Once the pod is created, it will have three network interfaces:

  • lo the loopback interface
  • eth0 the default pod network managed by Antrea or Calico CNI
  • net1 the new interface created via the annotation k8s.v1.cni.cncf.io/networks: macvlan-conf.
Note

The default network gets the name eth0 and additional network pod interfaces get the name as net1, net2, and so on.

Validating Multus

Run kubectl describe pod on the pod, and confirm that the annotation k8s.v1.cni.cncf.io/network-status lists all network interfaces. For example:

$ kubectl describe pod sample-pod

Name:         sample-pod
Namespace:    default
Priority:     0
Node:         tcecluster-md-0-6476897f75-rl9vt/10.170.109.225
Start Time:   Thu, 27 May 2021 15:31:20 +0000
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "100.96.1.80"
                    ],
                    "mac": "66:39:dc:63:50:a3",
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/macvlan-conf",
                    "interface": "net1",
                    "ips": [
                        "192.168.1.201"
                    ],
                    "mac": "02:77:cb:a0:60:e3",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: macvlan-conf

Then run kubectl exec sample-pod -- ip a show dev net1 to check if the target interface is up and running with IP listed in annotations above.

Deleting Multus Unsupported

Once the Multus CNI is installed in a cluster, it should not be deleted.

Deleting Multus does not uninstall the the Multus configuration file /etc/cni/net.d/00-multus.conf from the CNI scripts directory, which prevents the cluster from creating new pods.

This is a known issue; see Issue #461 in the Multus repository.

check-circle-line exclamation-circle-line close-line
Scroll to top icon