VMware Container Networking with Antrea 1.6.0 Release Notes

VMware Container Networking with Antrea 1.6.0 \| 07 FEB 2023 \| Build 21138057 Check for additions and updates to these release notes.

VMware Container Networking with Antrea 1.6.0 | 07 FEB 2023 | Build 21138057

Check for additions and updates to these release notes.

What's New

Antrea Container Networking with Antrea 1.6.0 is based off the Antrea v1.9.0 open-source release
Antrea to NSX Integration improvements
- Create firewall rules matching both NSX objects and Kubernetes objects
- Ability to create rules matching ingress and egress traffic from/to Kubernetes clusters
- Allow including Kubernetes inventory objects like Antrea Egress, IPPool, Kubernetes Node, Service, Ingress, Gateway in NSX generic groups
IPsec support in Antrea Operator
Dualstack (IPv4/IPv6) support with UPI installation on virtual machine nodes
Enhanced Features
- IDS Enhancements
  - Introduction of new deviceType for Antrea IDS for use with NTICS APIs. New deviceType includes:
    - Antrea-Standalone
    - Antrea-Tkg
    - Antrea-Tkgi
    - Antrea-Openshift
- Tanzu Enhancements
  - Introducetion of AntreaConfig to customize Antrea configuration
Container images on VMware distribution Harbor:
- Antrea images:
  - projects.registry.vmware.com/antreainterworking/antrea-standard-debian:v1.9.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-advanced-debian:v1.9.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-ubi:v1.9.0_vmware.2
- Antrea multi-cluster controller images:
  - projects.registry.vmware.com/antreainterworking/antrea-mc-controller-debian:v1.9.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-mc-controller-ubi:v1.9.0_vmware.2
- Antrea flow-aggregator images:
  - projects.registry.vmware.com/antreainterworking/flow-aggregator-debian:v1.9.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/flow-aggregator-ubi:v1.9.0_vmware.2
Antrea IDPS images:
- IDPS controller and agent
  - projects.registry.vmware.com/antreainterworking/idps-debian:v1.9.0_vmware.2
  - projects.registry.vmware.com/antreainterworking/idps-ubi:v1.9.0_vmware.2
- Suricata
  - projects.registry.vmware.com/antreainterworking/suricata:v1.9.0_vmware.2
Operator image:
- projects.registry.vmware.com/antreainterworking/antrea-operator:v1.9.0_vmware.2
Antrea-NSX images:
- projects.registry.vmware.com/antreainterworking/interworking-debian:0.9.0
- Projects.registry.vmware.com/antreainterworking/interworking-ubi:0.9.0

Note:

UBI images can only run on RHEL 8 or newer OSes with nftables kernel module (nf_tables) loaded
Photon images can only run on Photon OS or OSes with iptables legacy kernel module (ip_tables) loaded

Compatibility Testing Matrix

K8S Distribution	K8S Versions	OS	Encapsulation
K8s	1.22.x, 1.23.x, 1.24.x, 1.25.x	Ubuntu 18.04, PhotonOS 3, Debian 10	Geneve, NoEncap, Hybrid
AWS EKS	1.21	Amazon Linux 2	Policy Only Mode
Azure AKS, AKS Engine	1.21	Ubuntu 18.04	Policy Only Mode
GKE (Google Kubernetes Engine)	1.22	Ubuntu 18.04, Windows	NoEncap, Policy Only Mode
RHEL	RHEL 7.9 onwards	RHEL	Geneve, NoEncap, Hybrid
OpenShift(*)	OCP 4.9, 4.10, 4.11	RHCOS and RHEL	Geneve, NoEncap, Hybrid
NSX	3.2.x, 4.0.x, 4.1.0

(*) Antrea CNI and Antrea Operator are supported for the OpenShift versions listed in the compability matrix

Change Logs:

Includes all changes from OSS 1.9.0, and OSS 1.8.0

Known Issues

Container Overlay Packets Dropped by ESX vSwitch
NSX vSwitch (VDL2 module) drops GENEVE and VXLAN packets generated by VM when the packets go to the same VLAN (transport VLAN) as the NSX VTEP vmk NICs. This is a security protection for NSX transport VLAN, and the protection cannot be disabled. This causes ESX vSwitch dropping VM overlay packets when the VM shares the same vSwitch as NSX transport node and breaks Kubernetes container network connectivity.

The issue will happen in the follow conditions:
- The vSphere DVS is used by NSX for TN node switch.
- A VLAN port group is created on the above DVS.
  - The VLAN is the same as NSX overlay transport VLAN.
    
    Sometimes the above port group is not managed by NSX. It’s created from VC. It coexists with NSX port groups (NSX Segments) on the same DVS.
    
    Sometimes the above port group is an NSX VLAN segment.
- Kubernetes VMs are deployed on the above VLAN port group.
- Kubernetes container networking is configured to use GENEVE or VXLAN tunnel.
Note:
In the vCenter web interface, VLAN ID of ESX vmk50 is usually shown as 0, this is not the NSX transport VLAN. You can check the transport VLAN used by NSX by running the following command on ESX.
```
# net-vdl2 -l | grep -i transportTransport VLAN ID: 0
```
Alternative Workaround 1

Change the VLAN ID of DVPG used by K8s VMs to be different to NSX transport VLAN. If it's an NSX VLAN segment, change the VLAN ID of the segment.

Alternative Workaround 2

Use a dedicated VLAN ID for transport VLAN when NSX TN is configured on the ESX. This is usually set in NSX "System" -> "Profies" -> "Uplink Profiles". Find out the Uplink Profile referenced by the Transport Node Profie of your TNs, then change the "Transport VLAN" value in the Uplink Profile.

Alternative Workaround 3

Use a different DVS or standard switch for K8s VMs.
Container Network Traffic Throughput Drops to Zero on Buggy Physical NIC

Antrea enables Geneve tunnel checksum offload by default. However, sometimes the container networking traffic throughput drops to nearly zero. In packet capture we see that TCP 3-way handshake is successful but the first data packet in MTU size gets wrong checksum and it's dropped in the receiver side. This can happen when the K8s node VMs are running on overlay network and the underlay network cannot correctly process checksum offloading in double encapsulation scenario, or the physical NIC has a bug in checksum offloading.
We introduced the following ConfigMap antrea-agent-tweaker in antrea.yml to allow disabling tunnel checksum offloading.
```
apiVersion: v1
data:
  antrea-agent-tweaker.conf: |-
    # Enable disableUdpTunnelOffload will disable udp tunnel offloading feature on kubernetes node's default interface.
    # By default, no actions will be taken.
    disableUdpTunnelOffload: false
kind: ConfigMap
metadata:
  labels:
    app: antrea
  name: antrea-agent-tweaker
  namespace: kube-system
```
This is only for Linux. You can use kubectl to edit the live ConfigMap on K8s API to disable tunnel checksum offload, then restart all Antrea agents (usually run the command kubectl delete pod -l component=antrea-agent -n kube-system) to make the option effective. You can also edit this ConfigMap in antrea.yml before deploying Antrea. We suggest to set disableUdpTunnelOffload: true only if you hit a tunnel checksum offloading issue. Note that disabling tunnel checksum offloading reduces the networking throughput by almost 50%.