VMware Container Networking with Antrea 1.7.0 Release Notes

VMware Container Networking with Antrea 1.7.0 \| 24 May 2023 \| Build 21789974 Check for additions and updates to these release notes.

VMware Container Networking with Antrea 1.7.0 | 24 May 2023 | Build 21789974

Check for additions and updates to these release notes.

What's New

VMware Container Networking with Antrea v1.7.0 is based off the Antrea v1.11.1 open-source release.
Multiple Egress IP support allowing to have Egress failover from one Availability Zone to another.
Support for Antrea to NSX Integration over IPv6 Management address for NSX.
Support for non-Enterprise Admin role while registering Antrea cluster to NSX.
Support for Network Policy enforcement on Multicast traffic and Network Policy statistics for Multicast traffic. Multicast support is a beta feature in this release.
Added support for OpenEuler K8s platform
Command line cleanup utility 'antreansxctl' that can be used for cleaning up Antrea data from NSX. This can be used when the cluster has been destroyed before the deregisterjob.yaml has run. Sample usage:
- List all registered clusters including running clusters and left-over clusters
  
  antreansxctl cluster-list --nsx-ip string --user string --password string
- View cluster registration and inventory information
  
  antreansxctl cluster-get --cluster-name string --nsx-ip string --user string --password string
  
  antreansxctl cluster-get --cluster-id string --nsx-ip string --user string --password string
- Cleanup cluster registration and inventory
  
  antreansxctl cluster-cleanup --cluster-name string --nsx-ip string --user string --password string
  
  antreansxctl cluster-cleanup --cluster-id string --nsx-ip string --user string --password string
- Cleanup all cluster registration and inventories (Use with caution. This will deregister all Antrea clusters)
  
  antreansxctl cluster-cleanup --delete-all --nsx-ip string --user string --password string

Container images on VMware distribution Harbor:
- Antrea images:
  - projects.registry.vmware.com/antreainterworking/antrea-advanced-debian:v1.11.1_vmware.3
  - projects.registry.vmware.com/antreainterworking/antrea-standard-debian:v1.11.1_vmware.3
  - projects.registry.vmware.com/antreainterworking/antrea-ubi:v1.11.1_vmware.3
- Antrea multi-cluster controller images:
  - projects.registry.vmware.com/antreainterworking/antrea-mc-controller-debian:v1.11.1_vmware.3
  - projects.registry.vmware.com/antreainterworking/antrea-mc-controller-ubi:v1.11.1_vmware.3
- Antrea flow-aggregator images:
  - projects.registry.vmware.com/antreainterworking/flow-aggregator-debian:v1.11.1_vmware.3
  - projects.registry.vmware.com/antreainterworking/flow-aggregator-ubi:v1.11.1_vmware.3
Antrea IDPS images:
- IDPS controller and agent
  - projects.registry.vmware.com/antreainterworking/idps-debian:v1.11.1_vmware.3
  - projects.registry.vmware.com/antreainterworking/idps-ubi:v1.11.1_vmware.3
- Suricata
  - projects.registry.vmware.com/antreainterworking/suricata:v1.11.1_vmware.3
Operator image:
- projects.registry.vmware.com/antreainterworking/antrea-operator:v1.11.1_vmware.3
Antrea-NSX images:
- projects.registry.vmware.com/antreainterworking/interworking-debian:0.11.0
- projects.registry.vmware.com/antreainterworking/interworking-ubi:0.11.0

Note:

UBI images can only run on RHEL 8 or newer OSes with nftables kernel module (nf_tables) loaded.
Photon images can only run on Photon OS or OSes with iptables legacy kernel module (ip_tables) loaded.

Compatibility Testing Matrix

K8S Distribution	K8S Versions	OS	Encapsulation
K8s	1.23.x, 1.24.x, 1.25.x, 1.26.x, 1.27.x	Ubuntu 22.04, Ubuntu 18.04, PhotonOS 3, Debian 10, openEuler 22.03-LTS	Geneve, NoEncap, Hybrid
AWS EKS	1.24	Amazon Linux 2	Policy Only Mode
Azure AKS, AKS Engine	1.25	Ubuntu 18.04	Policy Only Mode
GKE (Google Kubernetes Engine)	1.26	Ubuntu 18.04, Windows	NoEncap, Policy Only Mode
RHEL	RHEL 7.9 onwards	RHEL	Geneve, NoEncap, Hybrid
OpenShift (*)	OCP 4.10, 4.11, 4.12	RHCOS and RHEL	Geneve, NoEncap, Hybrid
Rancher (*)	2.7		Geneve, NoEncap, Hybrid
NSX	3.2.x, 4.0.x, 4.1.x

(*) Antrea CNI and Antrea Operator are supported for the OpenShift and Rancher versions listed in the compability matrix.

Change Logs:

Includes all changes from OSS 1.10.0, OSS 1.11.0 and OSS 1.11.1

Known Issues

Switch Between IPSec and non-IPSec on the Live Cluster

Antrea-agent does not update the exiting Pod interface MTU after switching to IPSec and switching back to non-IPSec encryption mode. This would cause Pod traffic interrupt.

Deleting all Pods (except hostNetwork Pods) and re-creating the Pods will trigger Antrea-agent creating new veth interfaces with the correct MTU.
Container Network Traffic Throughput Drops to Zero on Buggy Physical NIC

Antrea enables Geneve tunnel checksum offload by default. However, sometimes the container networking traffic throughput drops to nearly zero. In packet capture we see that TCP 3-way handshake is successful but the first data packet in MTU size gets wrong checksum and it's dropped in the receiver side. This can happen when the K8s node VMs are running on overlay network and the underlay network cannot correctly process checksum offloading in double encapsulation scenario, or the physical NIC has a bug in checksum offloading.
We introduced the following ConfigMap antrea-agent-tweaker in antrea.yml to allow disabling tunnel checksum offloading.
```
apiVersion: v1
data:
  antrea-agent-tweaker.conf: |-
    # Enable disableUdpTunnelOffload will disable udp tunnel offloading feature on kubernetes node's default interface.
    # By default, no actions will be taken.
    disableUdpTunnelOffload: false
kind: ConfigMap
metadata:
  labels:
    app: antrea
  name: antrea-agent-tweaker
  namespace: kube-system
```
This is only for Linux. You can use kubectl to edit the live ConfigMap on K8s API to disable tunnel checksum offload, then restart all Antrea agents (usually run the command kubectl delete pod -l component=antrea-agent -n kube-system) to make the option effective. You can also edit this ConfigMap in antrea.yml before deploying Antrea. We suggest to set disableUdpTunnelOffload: true only if you hit a tunnel checksum offloading issue. Note that disabling tunnel checksum offloading reduces the networking throughput by almost 50%.
Container Overlay Packets Dropped by ESX vSwitch
NSX vSwitch (VDL2 module) drops GENEVE and VXLAN packets generated by VM when the packets go to the same VLAN (transport VLAN) as the NSX VTEP vmk NICs. This is a security protection for NSX transport VLAN, and the protection cannot be disabled. This causes ESX vSwitch dropping VM overlay packets when the VM shares the same vSwitch as NSX transport node and breaks Kubernetes container network connectivity.

The issue will happen in the follow conditions:
- The vSphere DVS is used by NSX for TN node switch.
- A VLAN port group is created on the above DVS.
  - The VLAN is the same as NSX overlay transport VLAN.
    
    Sometimes the above port group is not managed by NSX. It’s created from VC. It coexists with NSX port groups (NSX Segments) on the same DVS.
    
    Sometimes the above port group is an NSX VLAN segment.
- Kubernetes VMs are deployed on the above VLAN port group.
- Kubernetes container networking is configured to use GENEVE or VXLAN tunnel.
Note:
In the vCenter web interface, VLAN ID of ESX vmk50 is usually shown as 0, this is not the NSX transport VLAN. You can check the transport VLAN used by NSX by running the following command on ESX.
```
# net-vdl2 -l | grep -i transportTransport VLAN ID: 0
```
Alternative Workaround 1

Change the VLAN ID of DVPG used by K8s VMs to be different to NSX transport VLAN. If it's an NSX VLAN segment, change the VLAN ID of the segment.

Alternative Workaround 2

Use a dedicated VLAN ID for transport VLAN when NSX TN is configured on the ESX. This is usually set in NSX "System" -> "Profies" -> "Uplink Profiles". Find out the Uplink Profile referenced by the Transport Node Profie of your TNs, then change the "Transport VLAN" value in the Uplink Profile.

Alternative Workaround 3

Use a different DVS or standard switch for K8s VMs.