VMware Container Networking with Antrea 1.10.0 | October 16, 2024| Build ob-24325090

Check for additions and updates to these release notes.

What's New

  • VMware Container Networking with Antrea v1.10.0 is based off the Antrea v2.1.0 open-source release.

  • Add Antrea ODS runbook support, which provides an automated tool that abstracts debugging workflows and facilitates the procedure for root cause analysis (Alpha).

  • Introduce RBAC Egress feature to provides fine-grained control over network traffic based on entitlement and entitlement bindings.

  • Introduce EgressSeparateSubnet feature to allows users to allocate Egress IPs from a different subnet from the default Node subnet.

  • Antrea-NSX integration:

    • Add new sub-commands to the antreansxctl command-line tool for managing NSX ChildSegment.

    • Add a context file for antreansxctl. The context file allows user to set NSX credentials via $HOME/.antreansxctl, and switch between contexts targeting different NSX deployments.

  • Container images on Broadcom Artifactory are below.

    • Antrea images:

      • projects.packages.broadcom.com/antreainterworking/antrea-standard-controller-debian:v2.1.0_vmware.3

      • projects.packages.broadcom.com/antreainterworking/antrea-standard-agent-debian:v2.1.0_vmware.3

      • projects.packages.broadcom.com/antreainterworking/antrea-advanced-controller-debian:v2.1.0_vmware.3

      • projects.packages.broadcom.com/antreainterworking/antrea-advanced-agent-debian:v2.1.0_vmware.3

      • projects.packages.broadcom.com/antreainterworking/antrea-controller-ubi:v2.1.0_vmware.3

      • projects.packages.broadcom.com/antreainterworking/antrea-agent-ubi:v2.1.0_vmware.3

    • Antrea multi-cluster controller images:

      • projects.packages.broadcom.com/antreainterworking/antrea-mc-controller-debian:v2.1.0_vmware.3

      • projects.packages.broadcom.com/antreainterworking/antrea-mc-controller-ubi:v2.1.0_vmware.3

    • Antrea flow-aggregator images:

      • projects.packages.broadcom.com/antreainterworking/flow-aggregator-ubi:v2.1.0_vmware.3

      • projects.packages.broadcom.com/antreainterworking/flow-aggregator-debian:v2.1.0_vmware.3

    • Antrea IDPS images:

      • IDPS controller and agent

        • projects.packages.broadcom.com/antreainterworking/idps-debian:v2.1.0_vmware.3

        • projects.packages.broadcom.com/antreainterworking/idps-ubi:v2.1.0_vmware.3

      • Suricata

        • projects.packages.broadcom.com/antreainterworking/suricata:v2.1.0_vmware.3

    • Antrea ODS image:

      • projects.packages.broadcom.com/antreainterworking/antrea-ods-debian:v2.1.0_vmware.3

    • Operator image:

      • projects.packages.broadcom.com/antreainterworking/antrea-operator:v2.1.0_vmware.3

    • Antrea-NSX images:

      • projects.packages.broadcom.com/antreainterworking/interworking-debian:1.1.0_vmware.1

      • projects.packages.broadcom.com/antreainterworking/interworking-ubuntu:1.1.0_vmware.1

      • projects.packages.broadcom.com/antreainterworking/interworking-photon:1.1.0_vmware.1

      • projects.packages.broadcom.com/antreainterworking/interworking-ubi:1.1.0_vmware.1

Note:

  1. UBI images can only run on RHEL 8 or newer OSes with nftables kernel module (nf_tables) loaded.

  2. Photon images can only run on Photon OS or OSes with iptables legacy kernel module (ip_tables) loaded.

Compatibility Testing Matrix

K8S Distribution

K8S Versions

OS

Encapsulation

K8s

1.31.x, 1.30.x, 1.29.x, 1.28.x, 1.27.x

Ubuntu, RedHat, Photon, Windows

Geneve, NoEncap, Hybrid

AWS EKS

1.27

Amazon Linux 2

Policy Only Mode

Azure AKS, AKS Engine

1.26

Ubuntu 22.04

Policy Only Mode

GKE (Google Kubernetes Engine)

1.27

Ubuntu 22.04, Windows

NoEncap, Policy Only Mode

RHEL

RHEL 8 onwards

RHEL

Geneve, NoEncap, Hybrid

OpenShift (*)

4.15, 4.14, 4.13

RHCOS and RHEL

Geneve, NoEncap, Hybrid

Rancher (*)

2.7

Ubuntu 22.04

Geneve, NoEncap, Hybrid

NSX

4.0.x, 4.1.x, 4.2.0

(*) Antrea CNI and Antrea Operator are supported for the OpenShift and Rancher versions listed in the compatibility matrix.

Change Logs:

Known Issues

  • Antrea Groups with IP Addresses Do Not Work in NSX 4.1.2 and 4.1.2.1

    If the customer creates an Antrea Group in NSX 4.1.2 and 4.1.2.1, and define IP address members in the group, the IP addresses are not delivered to Antrea-NSX adapters. If the group is used in Antrea Policy rule sources or destinations, the IP addresses are not effective for filtering traffic. This bug is fixed in NSX 4.1.2.3 and later NSX versions.

    Customers should avoid using NSX 4.1.2 and 4.1.2.1. Customer can upgrade NSX to version 4.1.2.3 (build number 23382408), or greater versions.

  • Switch Between IPSec and non-IPSec on the Live Cluster

    Antrea-agent does not update the exiting Pod interface MTU after switching to IPSec and switching back to non-IPSec encryption mode. This would cause Pod traffic interrupt.

    Deleting all Pods (except hostNetwork Pods) and re-creating the Pods will trigger Antrea-agent creating new veth interfaces with the correct MTU.

  • Container Network Traffic Throughput Drops to Zero on Buggy Physical NIC

    Antrea enables Geneve tunnel checksum offload by default. However, sometimes the container networking traffic throughput drops to nearly zero. In packet capture we see that TCP 3-way handshake is successful but the first data packet in MTU size gets wrong checksum and it's dropped in the receiver side. This can happen when the K8s node VMs are running on overlay network and the underlay network cannot correctly process checksum offloading in double encapsulation scenario, or the physical NIC has a bug in checksum offloading.

    We introduced the following ConfigMap antrea-agent-tweaker in antrea.yml to allow disabling tunnel checksum offloading.

    apiVersion: v1
data:
  antrea-agent-tweaker.conf: |-
    # Enable disableUdpTunnelOffload will disable udp tunnel offloading feature on kubernetes node's default interface.
    # By default, no actions will be taken.
    disableUdpTunnelOffload: false
kind: ConfigMap
metadata:
  labels:
    app: antrea
  name: antrea-agent-tweaker
  namespace: kube-system

    This is only for Linux. You can use kubectl to edit the live ConfigMap on K8s API to disable tunnel checksum offload, then restart all Antrea agents (usually run the command kubectl delete pod -l component=antrea-agent -n kube-system) to make the option effective. You can also edit this ConfigMap in antrea.yml before deploying Antrea. We suggest to set disableUdpTunnelOffload: true only if you hit a tunnel checksum offloading issue. Note that disabling tunnel checksum offloading reduces the networking throughput by almost 50%.

  • Container Overlay Packets Dropped by ESX vSwitch

    NSX vSwitch (VDL2 module) drops GENEVE and VXLAN packets generated by VM when the packets go to the same VLAN (transport VLAN) as the NSX VTEP vmk NICs. This is a security protection for NSX transport VLAN, and the protection cannot be disabled. This causes ESX vSwitch dropping VM overlay packets when the VM shares the same vSwitch as NSX transport node and breaks Kubernetes container network connectivity.

    The issue will happen in the follow conditions:

    • The vSphere DVS is used by NSX for TN node switch.

    • A VLAN port group is created on the above DVS.

      • The VLAN is the same as NSX overlay transport VLAN.

        • Sometimes the above port group is not managed by NSX. It’s created from VC. It coexists with NSX port groups (NSX Segments) on the same DVS.

        • Sometimes the above port group is an NSX VLAN segment.

    • Kubernetes VMs are deployed on the above VLAN port group.

    • Kubernetes container networking is configured to use GENEVE or VXLAN tunnel.

    Note:

    In the vCenter web interface, VLAN ID of ESX vmk50 is usually shown as 0, this is not the NSX transport VLAN. You can check the transport VLAN used by NSX by running the following command on ESX.

    # net-vdl2 -l | grep -i transportTransport VLAN ID: 0

    Alternative Workaround 1

    Change the VLAN ID of DVPG used by K8s VMs to be different to NSX transport VLAN. If it's an NSX VLAN segment, change the VLAN ID of the segment.

    Alternative Workaround 2

    Use a dedicated VLAN ID for transport VLAN when NSX TN is configured on the ESX. This is usually set in NSX "System" -> "Profies" -> "Uplink Profiles". Find out the Uplink Profile referenced by the Transport Node Profie of your TNs, then change the "Transport VLAN" value in the Uplink Profile.

    Alternative Workaround 3

    Use a different DVS or standard switch for K8s VMs.

check-circle-line exclamation-circle-line close-line
Scroll to top icon