VMware Container Networking with Antrea 1.9.0 Release Notes

VMware Container Networking with Antrea 1.9.0 \| April 2, 2024\| Build ob-23566566 Check for additions and updates to these release notes.

VMware Container Networking with Antrea 1.9.0 | April 2, 2024| Build ob-23566566

Check for additions and updates to these release notes.

What's New

VMware Container Networking with Antrea v1.9.0 is based off the Antrea v1.15.0 open-source release.
Support Egress using IPs from a subnet that is different from the default Node subnet (Alpha).
Enhanced Antrea NetworkPolicy Visibility in Traceflow Observations with NetworkPolicy rule name.
Introduce NodeNetworkPolicy feature to provide ability to apply ClusterNetworkPolicy to K8s worker Nodes (Alpha).
The Antrea-NSX "register" component is changed from a K8s Job a K8s Deployment. Customers need to delete the old "register" Job manually before upgrading Antrea-NSX.
Container images on VMware distribution Harbor are below.
- Antrea images:
  - projects.registry.vmware.com/antreainterworking/antrea-standard-controller-debian:v1.15.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-standard-agent-debian:v1.15.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-advanced-controller-debian:v1.15.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-advanced-agent-debian:v1.15.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-controller-ubi:v1.15.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-agent-ubi:v1.15.0_vmware.2
- Antrea multi-cluster controller images:
  - projects.registry.vmware.com/antreainterworking/antrea-mc-controller-debian:v1.15.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-mc-controller-ubi:v1.15.0_vmware.2
- Antrea flow-aggregator images:
  - projects.registry.vmware.com/antreainterworking/flow-aggregator-ubi:v1.15.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/flow-aggregator-debian:v1.15.0_vmware.2
- Antrea IDPS images:
  - IDPS controller and agent
    - projects.registry.vmware.com/antreainterworking/idps-debian:v1.15.0_vmware.2
    - projects.registry.vmware.com/antreainterworking/idps-ubi:v1.15.0_vmware.2
  - Suricata
    - projects.registry.vmware.com/antreainterworking/suricata:v1.15.0_vmware.2
- Operator image:
  - projects.registry.vmware.com/antreainterworking/antrea-operator:v1.15.0_vmware.2
- Antrea-NSX images:
  - projects.registry.vmware.com/antreainterworking/interworking-debian:0.15.0_vmware.1
  - projects.registry.vmware.com/antreainterworking/interworking-ubuntu:0.15.0_vmware.1
  - projects.registry.vmware.com/antreainterworking/interworking-photon:0.15.0_vmware.1
  - projects.registry.vmware.com/antreainterworking/interworking-ubi:0.15.0_vmware.1

Note:

UBI images can only run on RHEL 8 or newer OSes with nftables kernel module (nf_tables) loaded.
Photon images can only run on Photon OS or OSes with iptables legacy kernel module (ip_tables) loaded.

Compatibility Testing Matrix

K8S Distribution	K8S Versions	OS	Encapsulation
K8s	1.30.x, 1.29.x, 1.28.x, 1.27.x	Ubuntu, RedHat, Photon, Windows	Geneve, NoEncap, Hybrid
AWS EKS	1.27	Amazon Linux 2	Policy Only Mode
Azure AKS, AKS Engine	1.26	Ubuntu 22.04	Policy Only Mode
GKE (Google Kubernetes Engine)	1.27	Ubuntu 22.04, Windows	NoEncap, Policy Only Mode
RHEL	RHEL 8 onwards	RHEL	Geneve, NoEncap, Hybrid
OpenShift (*)	4.15, 4.14, 4.13	RHCOS and RHEL	Geneve, NoEncap, Hybrid
Rancher (*)	2.7	Ubuntu 22.04	Geneve, NoEncap, Hybrid
NSX	4.0.x, 4.1.x, 4.2.0

(*) Antrea CNI and Antrea Operator are supported for the OpenShift and Rancher versions listed in the compatibility matrix.

Change Logs:

Includes all changes from OSS 1.15.0 and 1.14.2

Known Issues

Antrea Groups with IP Addresses Do Not Work in NSX 4.1.2 and 4.1.2.1

If the customer creates an Antrea Group in NSX 4.1.2 and 4.1.2.1, and define IP address members in the group, the IP addresses are not delivered to Antrea-NSX adapters. If the group is used in Antrea Policy rule sources or destinations, the IP addresses are not effective for filtering traffic. This bug is fixed in NSX 4.1.2.3 and later NSX versions.

Customers should avoid using NSX 4.1.2 and 4.1.2.1. Customer can upgrade NSX to version 4.1.2.3 (build number 23382408), or greater versions.
Switch Between IPSec and non-IPSec on the Live Cluster

Antrea-agent does not update the exiting Pod interface MTU after switching to IPSec and switching back to non-IPSec encryption mode. This would cause Pod traffic interrupt.

Deleting all Pods (except hostNetwork Pods) and re-creating the Pods will trigger Antrea-agent creating new veth interfaces with the correct MTU.
Container Network Traffic Throughput Drops to Zero on Buggy Physical NIC

Antrea enables Geneve tunnel checksum offload by default. However, sometimes the container networking traffic throughput drops to nearly zero. In packet capture we see that TCP 3-way handshake is successful but the first data packet in MTU size gets wrong checksum and it's dropped in the receiver side. This can happen when the K8s node VMs are running on overlay network and the underlay network cannot correctly process checksum offloading in double encapsulation scenario, or the physical NIC has a bug in checksum offloading.
We introduced the following ConfigMap antrea-agent-tweaker in antrea.yml to allow disabling tunnel checksum offloading.
```
apiVersion: v1
data:
  antrea-agent-tweaker.conf: |-
    # Enable disableUdpTunnelOffload will disable udp tunnel offloading feature on kubernetes node's default interface.
    # By default, no actions will be taken.
    disableUdpTunnelOffload: false
kind: ConfigMap
metadata:
  labels:
    app: antrea
  name: antrea-agent-tweaker
  namespace: kube-system
```
This is only for Linux. You can use kubectl to edit the live ConfigMap on K8s API to disable tunnel checksum offload, then restart all Antrea agents (usually run the command kubectl delete pod -l component=antrea-agent -n kube-system) to make the option effective. You can also edit this ConfigMap in antrea.yml before deploying Antrea. We suggest to set disableUdpTunnelOffload: true only if you hit a tunnel checksum offloading issue. Note that disabling tunnel checksum offloading reduces the networking throughput by almost 50%.
Container Overlay Packets Dropped by ESX vSwitch
NSX vSwitch (VDL2 module) drops GENEVE and VXLAN packets generated by VM when the packets go to the same VLAN (transport VLAN) as the NSX VTEP vmk NICs. This is a security protection for NSX transport VLAN, and the protection cannot be disabled. This causes ESX vSwitch dropping VM overlay packets when the VM shares the same vSwitch as NSX transport node and breaks Kubernetes container network connectivity.

The issue will happen in the follow conditions:
- The vSphere DVS is used by NSX for TN node switch.
- A VLAN port group is created on the above DVS.
  - The VLAN is the same as NSX overlay transport VLAN.
    
    Sometimes the above port group is not managed by NSX. It’s created from VC. It coexists with NSX port groups (NSX Segments) on the same DVS.
    
    Sometimes the above port group is an NSX VLAN segment.
- Kubernetes VMs are deployed on the above VLAN port group.
- Kubernetes container networking is configured to use GENEVE or VXLAN tunnel.
Note:
In the vCenter web interface, VLAN ID of ESX vmk50 is usually shown as 0, this is not the NSX transport VLAN. You can check the transport VLAN used by NSX by running the following command on ESX.
```
# net-vdl2 -l | grep -i transportTransport VLAN ID: 0
```
Alternative Workaround 1

Change the VLAN ID of DVPG used by K8s VMs to be different to NSX transport VLAN. If it's an NSX VLAN segment, change the VLAN ID of the segment.

Alternative Workaround 2

Use a dedicated VLAN ID for transport VLAN when NSX TN is configured on the ESX. This is usually set in NSX "System" -> "Profies" -> "Uplink Profiles". Find out the Uplink Profile referenced by the Transport Node Profie of your TNs, then change the "Transport VLAN" value in the Uplink Profile.

Alternative Workaround 3

Use a different DVS or standard switch for K8s VMs.