VMware Container Networking with Antrea 1.8.0 | Oct 19 2023| Build ob-22590067

Check for additions and updates to these release notes.

What's New

  • VMware Container Networking with Antrea v1.8.0 is based off the Antrea v1.13.1 open-source release.

  • Antrea/NSX Integration capabilities available when networkPolicyOnly mode is enabled in Antrea.

  • Antrea L7 NetworkPolicy support for TLS protocol.

  • Graduate Antrea Egress HA on AWS to beta.

  • Pod Name (together with Pod IP) added in Antrea Network Policy logs (np.log).

  • Tier, Egress, ClusterGroup/Group, AntreaClusterNetworkPolicy, AntreaNativeNetworkPolicy, Traceflow APIs promoted to v1beta1.

  • Bootstrap command option to antreansxcli to create cert, PI user and generate bootstrap config.

    antreansxctl bootstrap --help for more details.

  • Wavefront integration for Antrea metrics visibility.

  • Containerd support for Windows OS.

  • Container images on VMware distribution Harbor:

    • Antrea images:

      • projects.registry.vmware.com/antreainterworking/antrea-advanced-debian:v1.13.1_vmware.1

      • projects.registry.vmware.com/antreainterworking/antrea-standard-debian:v1.13.1_vmware.1

      • projects.registry.vmware.com/antreainterworking/antrea-ubi:v1.13.1_vmware.1

    • Antrea multi-cluster controller images:

      • projects.registry.vmware.com/antreainterworking/antrea-mc-controller-debian:v1.13.1_vmware.1

      • projects.registry.vmware.com/antreainterworking/antrea-mc-controller-ubi:v1.13.1_vmware.1

    • Antrea flow-aggregator images:

      • projects.registry.vmware.com/antreainterworking/flow-aggregator-debian:v1.13.1_vmware.1

      • projects.registry.vmware.com/antreainterworking/flow-aggregator-ubi:v1.13.1_vmware.1

  • Antrea IDPS images:

    • IDPS controller and agent

      • projects.registry.vmware.com/antreainterworking/idps-debian:v1.13.1_vmware.1

      • projects.registry.vmware.com/antreainterworking/idps-ubi:v1.13.1_vmware.1

    • Suricata

      • projects.registry.vmware.com/antreainterworking/suricata:v1.13.1_vmware.1

  • Operator image:

    • projects.registry.vmware.com/antreainterworking/antrea-operator:v1.13.1_vmware.1

  • Antrea-NSX images:

    • projects.registry.vmware.com/antreainterworking/interworking-debian:0.13.0_vmware.1

    • projects.registry.vmware.com/antreainterworking/interworking-ubuntu:0.13.0_vmware.1

    • projects.registry.vmware.com/antreainterworking/interworking-photon:0.13.0_vmware.1

    • projects.registry.vmware.com/antreainterworking/interworking-ubi:0.13.0_vmware.1

Note:

  1. UBI images can only run on RHEL 8 or newer OSes with nftables kernel module (nf_tables) loaded.

  2. Photon images can only run on Photon OS or OSes with iptables legacy kernel module (ip_tables) loaded.

Compatibility Testing Matrix

K8S Distribution

K8S Versions

OS

Encapsulation

K8s

1.27.x, 1.26.x, 1.25.x, 1.24.x

Ubuntu, RedHat, Photon, Windows

Geneve, NoEncap, Hybrid

AWS EKS

1.27

Amazon Linux 2

Policy Only Mode

Azure AKS, AKS Engine

1.26

Ubuntu 22.04

Policy Only Mode

GKE (Google Kubernetes Engine)

1.27

Ubuntu 22.04, Windows

NoEncap, Policy Only Mode

RHEL

RHEL 7.9 onwards

RHEL

Geneve, NoEncap, Hybrid

OpenShift (*)

4.13, 4.12, 4.11, 4.10

RHCOS and RHEL

Geneve, NoEncap, Hybrid

Rancher (*)

2.7

Ubuntu 22.04

Geneve, NoEncap, Hybrid

NSX

3.2.x, 4.0.x, 4.1.x

(*) Antrea CNI and Antrea Operator are supported for the OpenShift and Rancher versions listed in the compatibility matrix.

Change Logs:

Known Issues

  • Switch Between IPSec and non-IPSec on the Live Cluster

    Antrea-agent does not update the exiting Pod interface MTU after switching to IPSec and switching back to non-IPSec encryption mode. This would cause Pod traffic interrupt.

    Deleting all Pods (except hostNetwork Pods) and re-creating the Pods will trigger Antrea-agent creating new veth interfaces with the correct MTU.

  • Container Network Traffic Throughput Drops to Zero on Buggy Physical NIC

    Antrea enables Geneve tunnel checksum offload by default. However, sometimes the container networking traffic throughput drops to nearly zero. In packet capture we see that TCP 3-way handshake is successful but the first data packet in MTU size gets wrong checksum and it's dropped in the receiver side. This can happen when the K8s node VMs are running on overlay network and the underlay network cannot correctly process checksum offloading in double encapsulation scenario, or the physical NIC has a bug in checksum offloading.

    We introduced the following ConfigMap antrea-agent-tweaker in antrea.yml to allow disabling tunnel checksum offloading.

    apiVersion: v1
data:
  antrea-agent-tweaker.conf: |-
    # Enable disableUdpTunnelOffload will disable udp tunnel offloading feature on kubernetes node's default interface.
    # By default, no actions will be taken.
    disableUdpTunnelOffload: false
kind: ConfigMap
metadata:
  labels:
    app: antrea
  name: antrea-agent-tweaker
  namespace: kube-system

    This is only for Linux. You can use kubectl to edit the live ConfigMap on K8s API to disable tunnel checksum offload, then restart all Antrea agents (usually run the command kubectl delete pod -l component=antrea-agent -n kube-system) to make the option effective. You can also edit this ConfigMap in antrea.yml before deploying Antrea. We suggest to set disableUdpTunnelOffload: true only if you hit a tunnel checksum offloading issue. Note that disabling tunnel checksum offloading reduces the networking throughput by almost 50%.

  • Container Overlay Packets Dropped by ESX vSwitch

    NSX vSwitch (VDL2 module) drops GENEVE and VXLAN packets generated by VM when the packets go to the same VLAN (transport VLAN) as the NSX VTEP vmk NICs. This is a security protection for NSX transport VLAN, and the protection cannot be disabled. This causes ESX vSwitch dropping VM overlay packets when the VM shares the same vSwitch as NSX transport node and breaks Kubernetes container network connectivity.

    The issue will happen in the follow conditions:

    • The vSphere DVS is used by NSX for TN node switch.

    • A VLAN port group is created on the above DVS.

      • The VLAN is the same as NSX overlay transport VLAN.

        • Sometimes the above port group is not managed by NSX. It’s created from VC. It coexists with NSX port groups (NSX Segments) on the same DVS.

        • Sometimes the above port group is an NSX VLAN segment.

    • Kubernetes VMs are deployed on the above VLAN port group.

    • Kubernetes container networking is configured to use GENEVE or VXLAN tunnel.

    Note:

    In the vCenter web interface, VLAN ID of ESX vmk50 is usually shown as 0, this is not the NSX transport VLAN. You can check the transport VLAN used by NSX by running the following command on ESX.

    # net-vdl2 -l | grep -i transportTransport VLAN ID: 0

    Alternative Workaround 1

    Change the VLAN ID of DVPG used by K8s VMs to be different to NSX transport VLAN. If it's an NSX VLAN segment, change the VLAN ID of the segment.

    Alternative Workaround 2

    Use a dedicated VLAN ID for transport VLAN when NSX TN is configured on the ESX. This is usually set in NSX "System" -> "Profies" -> "Uplink Profiles". Find out the Uplink Profile referenced by the Transport Node Profie of your TNs, then change the "Transport VLAN" value in the Uplink Profile.

    Alternative Workaround 3

    Use a different DVS or standard switch for K8s VMs.

check-circle-line exclamation-circle-line close-line
Scroll to top icon