| VMware Container Networking with Antrea. December 21, 2021 | 19065941 Check for additions and updates to these release notes. |
What's New
- Antrea to NSX-T integration
- K8s Objects like Namespaces, Pods, Services & K8s and Antrea Network policies are collected and reported to NSX-T inventory
- Antrea Cluster Network Policies can be created in the NSX-T Distributed Firewall UI and deployed into Antrea enabled K8s clusters as Antrea Cluster Network policies
- Antrea Traceflow and log collection can be triggered from the NSX-T UI
- Antrea cluster health status is reported to NSX-T for customers' attentions.
- Details about this new integration including prerequisites, version requirements, and deployment steps can be found in the NSX-T 3.2.0 documentation
- The integration deploys NSX-T adapters as Pod to the managed K8s cluster. The container image for the NSX-T adapters can be pulled from VMware distribution Harbor. To use the image from VMware distribution Harbor, edit the deployment manifest and replace all container images with one of the following URLs.
- projects.registry.vmware.com/antreainterworking/interworking-debian:0.2.0
- projects.registry.vmware.com/antreainterworking/interworking-ubuntu:0.2.0
- projects.registry.vmware.com/antreainterworking/interworking-photon:0.2.0
- Antrea OpenShift support and Antrea K8s Operator
- This release adds the support for OpenShift through the Antrea K8s Operator. For supported versions, please review the Compatibility Testing Matrix bellow
- The Operator image is available on VMware distribution Harbor. Customer can pull the image from below URL.
- projects.registry.vmware.com/antreainterworking/antrea-operator:v1.2.3_vmware
- The Antrea images can be pulled from the below URLs.
- projects.registry.vmware.com/antreainterworking/antrea-standard-debian:v1.2.3_vmware.3
- projects.registry.vmware.com/antreainterworking/antrea-advanced-debian:v1.2.3_vmware.3
- projects.registry.vmware.com/antreainterworking/antrea-ubi:v1.2.3_vmware.3
- Checksum offloading can now be globally disabled through a config parameter. This can be used if deployments experience issues caused e.g. by faulty NIC drivers. Refer to known issues section for details.
Compatibility Testing Matrix
| K8S Distribution | K8S Versions | OS | Encapsulation |
|---|---|---|---|
| K8s | 1.18, 1.19, 1.20, 1.21 | Ubuntu 18.04, PhotonOS 3, Debian10, Photon OS3 | Geneve, NoEncap, Hybrid, |
| AWS EKS | 1.17 | Amazon Linux 2 | Policy Only Mode |
| Azure AKS, AKS Engine | 1.18 | Ubuntu 18.04 | Policy Only Mode |
| GKE(Google Kubernetes Engine) | 1.18 | Ubuntu 18.04, Windows | NoEncap, Policy Only Mode |
| OpenShift | OCP 4.7 & 4.8 | RHCOS and RHEL | Geneve, NoEncap, Hybrid |
Change Logs 1.2.3- 2021-09-24
Includes all the changes from 1.2.0 and 1.2.1
Includes all the changes from 1.1.
Includes all the changes from 1.0.
Changed
- Support returning partial supportbundle results when some Nodes fail to respond. (#2788, @hangyan)
- Remove restriction that only GRE tunnels can be used when enabling IPsec: VXLAN can also be used, and so can Geneve (if the Linux kernel version for the Nodes is recent enough). (#2764, @luolanzone)
- Reduce memory usage of antctl when collecting supportbundle. (#2821, @tnqn)
Fixed
- Fix nil pointer error when collecting a supportbundle on a Node for which the antrea-agent container image does not include "iproute2"; this does not affect the standard antrea/antrea-ubuntu container image. (#2789, @liu4480)
- When creating an IPsec OVS tunnel port to a remote Node, handle the case where the port already exists but with a stale config graciously: delete the existing port first, then recreate it. (#2765, @luolanzone)
- Fix panic in the Antrea Controller when it processes ClusterGroups that are used by multiple ClusterNetworkPolicies. (#2768, @tnqn)
- Fix nil pointer error when antrea-agent updates OpenFlow priorities of Antrea-native policies without Service ports. (#2758, @wenyingd)
- Fix Pod-to-Service access on Windows when the Endpoints are not non-hostNetwork Pods (e.g. the
kubernetesService). (#2702, @wenyingd) [Windows] - Fix container network interface MTU configuration error when using containerd as the runtime on Windows. (#2773, @wenyingd) [Windows]
Known Issues
The known issues are grouped as follows.
Known Issues
- Container Network Traffic Throughput Drops to Zero on Buggy Physical NIC
Antrea enables Geneve tunnel checksum offload by default. However, sometimes the container networking traffic throughput drop to nearly zero. In packet capture we see that TCP 3-way handshake is successful but the first data packet in MTU size gets wrong checksum and it's dropped in the receiver side. This can happen when the K8s node VMs are running on overlay network and the underlay network cannot correctly process checksum offloading in double encapsulation scenario, or the physical NIC has bug in checksum offloading.
We introduced the following ConfigMap
antrea-agent-tweaker-g56hc6fh8tinantrea.ymlto allow disabling tunnel checksum offloading.apiVersion: v1 data: antrea-agent-tweaker.conf: |- # Enable disableUdpTunnelOffload will disable udp tunnel offloading feature on kubernetes node's default interface. # By default, no actions will be taken. disableUdpTunnelOffload: false kind: ConfigMap metadata: labels: app: antrea name: antrea-agent-tweaker-g56hc6fh8t namespace: kube-systemThis is only for Linux. You can use kubectl to edit the live ConfigMap on K8s API to disable tunnel checksum offload, then restart all Antrea agents (usually run the command
kubectl delete pod -l component=antrea-agent -n kube-system) to make the option effective. You can also edit this ConfigMap inantrea.ymlbefore deploying Antrea.
We suggest not to set it to true only if you hit tunnel checksum offloading issue. Disabling tunnel checksum offloading drops container networking throughput by about 50%. - Realization of Network Policies takes longer after upgrade to 1.3.0-1.2.2 with large number (10,000+) of Antrea-Network Policy Custom Resources.
When Antrea is upgraded to 1.3.0-1.2.2 from an older release (1.2.0-0.13.x or older), Antrea controller automatically mirrors Antrea Network Policy to new API group 'antrea.io'. So when a large number of ANP CRs are present, the Antrea Network policies are enforced slowly as the ANP CR is being mirrored to the new API. This is observed with more than 10,000+ ANPs.
Workaround: Manually mirror old API group to new API group before upgrade
- Create the new CRDs of tiers.crd.antrea.io, networkpolicies.crd.antrea.io, clusternetworkpolicies.crd.antrea.io, clustergroups.crd.antrea.io. The CRDs can be fetched from the manifest yaml that you will use for upgrade.
- Dump the resources of the deprecated CRDs (with *.antrea.tanzu.vmware.com suffix), create the same resources with new CRDs (with *.antrea.io suffix):
kubectl get tiers.security.antrea.tanzu.vmware.com -o yaml -A | sed "s/security.antrea.tanzu.vmware.com/crd.antrea.io/g" | kubectl apply -f - kubectl get networkpolicies.security.antrea.tanzu.vmware.com -o yaml -A | sed "s/security.antrea.tanzu.vmware.com/crd.antrea.io/g" | kubectl apply -f - kubectl get clusternetworkpolicies.security.antrea.tanzu.vmware.com -o yaml -A | sed "s/security.antrea.tanzu.vmware.com/crd.antrea.io/g" | kubectl apply -f - kubectl get clustergroups.core.antrea.tanzu.vmware.com -o yaml -A | sed "s/core.antrea.tanzu.vmware.com/crd.antrea.io/g" | kubectl apply -f -Note, if there are no resources for a CRD, the command will get an error output "error: no objects passed to apply", which can be ignored.
