times

VMware Container Networking with Antrea. April 7, 2022 | 19586604

Check for additions and updates to these release notes.

What's New

Antrea Container Networking with Antrea 1.4.0 is based off the Antrea v1.5.2 open-source release
Graduated Antrea Egress to Beta
- Antrea Egress was graduated to Beta and is now enabled by default in the Antrea configuration
- Antrea Egress allows users to define the SNAT IP used for Pod traffic egressing the cluster. It can be applied e.g., down to an individual Pod level or to all Pods in a Namespace.
- Learn more about Antrea Egress here: https://antrea.io/docs/v1.5.2/docs/egress/
Graduated NodePortLocal to Beta
- NodePortLocal was graduated to Beta and is now enabled by default in the Antrea configuration
- NodePortLocal allows external Load Balancers like the NSX Advanced Load Balancer to send traffic directly to Pods, even though the Node IP address is used as the destination of the traffic
- Learn more about NodePortLocal here: https://antrea.io/docs/v1.5.2/docs/node-port-local/
Graduated Antrea ProxyAll to Beta
- Antrea ProxyAll was graduated to Beta
- Support for proxying all Service traffic by Antrea Proxy, including NodePort, LoadBalancer, and ClusterIP traffic. Therefore, running kube-proxy is no longer required.
- The feature works for both Linux and Windows
- If kube-proxy is removed, the kubeAPIServerOverride configuration parameter for the Antrea Agent must be set to access kube-apiserver directly
- Antrea ProxyAll is not enabled by default, as it requires K8s distribution to be deployed with the kubeAPIServerOverride configuration parameter. K8s distribution need to be configured to set this parameter and enable Antrea ProxyAll.
Antrea to NSX-T Integration automated through the operator
- The Antrea to NSX-T Integration that was released with Antrea Container Networking with Antrea Version 1.3.1 can now automatically by deployed and configured through the Antrea operator
IPv6 Enhancements
- dual-stack support for noEncap mode
- IPv6 support with Antrea Egress
Added Experimental features (Disabled by default in Antrea Config)
- Antrea flexible IPAM & Node IPAM
  - Antrea flexible IPAM allows namespaces & pods to use specific IP ranges when using noEncap mode
  - With flexible IPAM, a StatefulSet Pod’s IP will be kept after Pod restarts, when the IP is allocated from the annotated IPPool
  - Antrea Node IPAM replaces the basic K8s Node IPAM with native Antrea IPAM capabilities
  - Learn more about Antrea IPAM capabilities here: https://antrea.io/docs/v1.5.2/docs/antrea-ipam/
- Cross K8s cluster federation
  - Allows users to export and import Services and Endpoints across multiple clusters within a ClusterSet, and enables inter-cluster Service communication in the ClusterSet
  - Learn more about Cross K8s cluster federation here:
    - https://github.com/antrea-io/antrea/blob/v1.5.0/docs/multicluster/architecture.md
    - https://github.com/antrea-io/antrea/blob/v1.5.0/docs/multicluster/getting-started.md
- Multicast support
  - Adds support for Multicast sources and destinations deployed as Pods in Antrea enabled K8s clusters
  - Uses the underlying network for Multicast transport - Currently, there is no support for Multicast Routing over the Overlay
  - Only supported in noEncap mode and only with IPv4
- Native Secondary Interface support
  - Antrea can now create secondary network interfaces for Pods using SR-IOV VFs on bare metal Nodes
- ServiceExternalIP (Native Service Type LB)
  - Adds support for allocating external IPs for Services of type LoadBalancer from an ExternalIPPool
Container images on VMware distribution Harbor:
- Antrea images:
  - projects.registry.vmware.com/antreainterworking/antrea-standard-debian:v1.5.2_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-advanced-debian:v1.5.2_vmware.2
  - projects.registry.vmware.com/antreainterworking/antrea-ubi:v1.5.2_vmware.2
- Operator image:
  - projects.registry.vmware.com/antreainterworking/antrea-operator:v1.5.2_vmware
- Antrea-NSX images:
  - projects.registry.vmware.com/antreainterworking/interworking-debian:0.5.0
  - projects.registry.vmware.com/antreainterworking/interworking-ubuntu:0.5.0
  - projects.registry.vmware.com/antreainterworking/interworking-photon:0.5.0
  - projects.registry.vmware.com/antreainterworking/interworking-ubi:0.5.0

Compatibility Testing Matrix

K8S Distribution	K8S Versions	OS	Encapsulation
K8s	1.21, 1.22, 1.23	Ubuntu 18.04, PhotonOS 3, Debian 10	Geneve, NoEncap, Hybrid,
AWS EKS	1.21	Amazon Linux 2	Policy Only Mode
Azure AKS, AKS Engine	1.21	Ubuntu 18.04	Policy Only Mode
GKE(Google Kubernetes Engine)	1.22	Ubuntu 18.04, Windows	NoEncap, Policy Only Mode
OpenShift	OCP 4.7, 4.8, 4.9, 4.10	RHCOS and RHEL	Geneve, NoEncap, Hybrid
NSX	3.2.0, 3.2.1, 4.0.0.1

Change Logs 1.5.2 - 2021-03-21

Changes from OSS 1.5.2, includes all changes from OSS 1.5.0 and OSS 1.5.1

Changed

Use iptables-wrapper in Antrea container. Now antrea-agent can work with distros that lack the iptables kernel module of "legacy" mode (ip_tables). (#3308, @antoninbas)
Reduce permissions of Antrea ServiceAccount for updating annotations. (#3408, @tnqn)
Make LoadBalancer IP proxying configurable for AntreaProxy to support scenarios in which it is desirable to send Pod-to-ExternalIP traffic to the external LoadBalancer. (#3130, @antoninbas)
Add startTime to the Traceflow Status to avoid issues caused by clock skew. (#2952, @antoninbas)
Add reason field in antctl traceflow command output. (#3175, @Jexf)
Validate serviceCIDR configuration only if AntreaProxy is disabled. (#2936, @wenyingd)
Improve configuration parameter validation for NodeIPAM. (#3009, @tnqn)
More comprehensive validation for Antrea-native policies. (#3104 #3109, @GraysonWu @tnqn)
Update Antrea Octant plugin to support Octant 0.24 and to use the Dashboard client to perform CRUD operations on Antrea CRDs. (#2951, @antoninbas)
Omit hostNetwork Pods when computing members of ClusterGroup and AddressGroup. (#3080, @Dyanngg)
Support for using an env parameter ALLOW_NO_ENCAP_WITHOUT_ANTREA_PROXY to allow running Antrea in noEncap mode without AntreaProxy. (#3116, @Jexf @WenzelZ)
Move throughput calculation for network flow visibility from logstash to flow-aggregator. (#2692, @heanlan)
Add Go version information to full version string for Antrea binaries. (#3182, @antoninbas)
Improve kind-setup.sh script and Kind documentation. (#2937, @antoninbas)
Enable Go benchmark tests in CI. (#3004, @wenqiq)
Upgrade Windows OVS version to 2.15.2 to pick up some recent patches. (#2996, @lzhecheng) [Windows]
Remove HNSEndpoint only if infra container fails to create. (#2976, @lzhecheng) [Windows]
Use OVS Port externalIDs instead of HNSEndpoint to cache the externalIDS when using containerd as the runtime on Windows. (#2931, @wenyingd) [Windows]
Reduce network downtime when starting antrea-agent on Windows Node by using Windows management virtual network adapter as OVS internal port. (#3067, @wenyingd) [Windows]

Fixed

Fix NetworkPolicy may not be enforced correctly after restarting a Node. (#3467, @tnqn)
Fix antrea-agent crash caused by interface detection in AKS/EKS with NetworkPolicyOnly mode. (#3219, @wenyingd)
Fix locally generated packets from Node net namespace might be SNATed mistakenly when Egress is enabled. (#3430, @tnqn)
Fix NodePort/LoadBalancer Service cannot be accessed when externalTrafficPolicy changed from Cluster to Local with proxyAll enabled. (#3330, @hongliangl)
Fix initial egress connections from Pods may go out with node IP rather than Egress IP. (#3378, @tnqn)
Fix NodePort Service access when an Egress selects the same Pod as the NodePort Service. (#3397, @hongliangl)
Fix ipBlock referenced in nested ClusterGroup not processed correctly. (#3405, @Dyanngg)
Fix error handling of the "Reject" action of Antrea-native policies when determining if the packet belongs to Service traffic. (#3010, @GraysonWu)
Make the "Reject" action of Antrea-native policies work in AntreaIPAM mode. (#3003, @GraysonWu)
Set ClusterGroup with child groups to groupMembersComputed after all its child groups are created and processed. (#3030, @Dyanngg)
Fix status report of Antrea-native policies with multiple rules that have different AppliedTo. (#3074, @tnqn)
Fix typos and improve the example YAML in antrea-network-policy doc. (#3079, #3092, #3108 @antoninbas @Jexf @tnqn)
Fix duplicated attempts to delete unreferenced AddressGroups when deleting Antrea-native policies. (#3136, @Jexf)
Add retry to update NetworkPolicy status to avoid error logs. (#3134, @Jexf)
Fix NetworkPolicy resources dump for Agent's supportbundle. (#3083, @antoninbas)
Use go 1.17 to build release assets. (#3007, @antoninbas)
Restore the gateway route automatically configured by kernel when configuring IP address if it is missing. (#2835, @antoninbas)
Fix incorrect parameter used to check if a container is the infra container, which caused errors when reattaching HNS Endpoint. (#3089, @XinShuYang) [Windows]
Fix gateway interface MTU configuration error on Windows. (#3043, @[lzhecheng]) [Windows]
Fix initialization error of antrea-agent on Windows by specifying hostname explicitly in VMSwitch commands. (#3169, @XinShuYang) [Windows]

Changes from 1.4.0

Changed

Remove chmod for OVSDB file from start_ovs, as the permissions are set correctly by OVS 2.15.1. (#2803, @antoninbas)
Reduce memory usage of antctl when collecting supportbundle. (#2813, @tnqn)
Do not perform SNAT for egress traffic to Kubernetes Node IPs. (#2762, @leonstack)
Send gratuitous ARP for EgressIP via the transport interface, as opposed to the interface with Node IP (if they are different). (#2845, @Jexf)
Ignore hostNetwork Pods selected by Egress, as they are not supported. (#2851, @Jexf)
Avoid duplicate processing of Egress. (#2884, @Jexf)
Ignore the IPs of kube-ipvs0 for Egress as they cannot be used for SNAT. (#2930, @Jexf)
Change flow exporter export expiry mechanism to priority queue based, to reduce CPU usage and memory footprint. (#2360, @heanlan)
Make Pod labels optional in the flow records. By default, they will not be included in the flow records. Use the recordContents.podLabels configuration parameter for the Flow Aggregator to include them. (#2739, @yanjunz97)
Wait for AntreaProxy to be ready before accessing any K8s Service if antreaProxy.proxyAll is enabled, to avoid connection issues on Agent startup. (#2858, @tnqn)
Update OVS pipeline documentation to include information about AntreaProxy. (#2725, @hongliangl)
Remove offensive words from scripts and documentation. (#2799, @xiaoxiaobaba)
Use readable names for OpenFlow tables. (#2585, @wenyingd)
Improve the OpenAPI schema for CRDs to validate the matchExpressions field. (#2887, @wenqiq)
Fail fast if the source Pod for non-live-traffic Traceflow is invalid. (#2736, @gran-vmv)
Use the RenewIPConfig parameter to indicate whether to renew ipconfig on the host for Clean-AntreaNetwork.ps1. It defaults to false. (#2955, @wenyingd) [Windows]
Add Windows task delay up to 30s to improve job resiliency of Prepare-AntreaAgent.ps1, to avoid a failure in initialization after Windows startup. (#2864, @perithompson) [Windows]

Fixed

Fix nil pointer error when antrea-agent updates OpenFlow priorities of Antrea-native policies without Service ports. (#2730, @wenyingd)
Fix panic in the Antrea Controller when it processes ClusterGroups that are used by multiple ClusterNetworkPolicies. (#2768, @tnqn)
Fix an issue with NodePortLocal when a given Pod port needs to be exposed for both TCP and UDP. (#2903, @antoninbas)
Fix handling of the "Reject" action of Antrea-native policies when the traffic is intended for Services. (#2772, @GraysonWu)
Fix Agent crash when removing the existing NetNat on Windows Nodes. (#2751, @wenyingd) [Windows]
Fix container network interface MTU configuration error when using containerd as the runtime on Windows. (#2778, @wenyingd) [Windows]
Fix path to Prepare-AntreaAgent.ps1 in Windows docs. (#2840, @perithompson) [Windows]
Fix NetNeighbor Powershell error handling. (#2905, @lzhecheng) [Windows]

Changes from 1.3.0

Changed

Remove the restriction that a ClusterGroup must exist before it can be used as a child group to define other ClusterGroups. (#2443, @Dyanngg)
Remove the restriction that a ClusterGroup must exist before it can be used in an Antrea ClusterNetworkPolicy. (#2478, @Dyanngg @abhiraut)
Remove "controlplane.antrea.tanzu.vmware.com/v1beta1" API as per our API deprecation policy. (#2528 #2631, @luolanzone)
Controller responses to ClusterGroup membership queries ("/clustergroupmembers" API) now include the list of IPBlocks when appropriate. (#2577, @Dyanngg @abhiraut)
Install all Endpoint flows belonging to a Service via a single OpenFlow bundle, to reduce flow installation time when the Agent starts. (#2476, @tnqn)
Improve the batch installation of NetworkPolicy rules when the Agent starts: only generate flow operations based on final desired state instead of incrementally. (#2479, @tnqn @Dyanngg)
Use GroupMemberSet.Merge instead of GroupMemberSet.Union to reduce CPU usage and memory footprint in the Agent's policy controller. (#2467, @tnqn)
When checking for the existence of an iptables chain, stop listing all the chains and searching through them; this change reduces the Agent's memory footprint. (#2458, @tnqn)
Tolerate more failures for the Agent's readiness probe, as the Agent may stay disconnected from the Controller for a long time in some scenarios. (#2535, @tnqn)
Remove restriction that only GRE tunnels can be used when enabling IPsec: VXLAN can also be used, and so can Geneve (if the Linux kernel version for the Nodes is recent enough). (#2489, @luolanzone)
Automatically perform deduplication on NetworkPolicy audit logs for denied connections: all duplicate connections received within a 1 second buffer window will be merged and the corresponding log entry will include the connection count. (#2294 #2578, @qiyueyao)
Support returning partial supportbundle results when some Nodes fail to respond. (#2399, @hangyan)
When listing NetworkPolicyStats through the Controller API, return an empty list if the NetworkPolicyStats Feature Gate is disabled, instead of returning an error. (#2386, @PeterEltgroth)
Update OVS version from 2.14.2 to 2.15.1: the new version fixes Geneve tunnel support in the userspace datapath (used for Kind clusters). (#2515, @antoninbas)
Update [go-ipfix] to version v0.5.7 to improve overall performance of the FlowExporter feature, and in particular of the Flow Aggregator component. (#2574, @srikartati @zyiou)
Support pretty-printing for AntreaAgentInfo and AntreaControllerInfo CRDs. (#2572, @antoninbas)
Improve the process of updating the Status of an Egress resource to report the name of the Node to which the Egress IP is assigned. (#2444, @wenqiq)
Change the singular name of the ClusterGroup CRD from "group" to "clustergroup". (#2484, @abhiraut)
Officially-supported Go version is no longer 1.15 but 1.17. (#2609 #2640, @antoninbas)
- There was a notable change in the implementation of the "ParseIP" and "ParseCIDR" functions, but Antrea users should not be affected; refer to this issue
Standardize the process of reserving OVS register ranges and defining constant values for them; OVS registers are used to store per-packet information when required to implement specific features. (#2455, @wenyingd)
Update ELK stack reference configuration to support TCP transport. (#2387, @zyiou)
Update Windows installation instructions. (#2456, @lzheheng)
Update Antrea-native policies documentation to reflect the addition of the "kubernetes.io/metadata.name" in upstream K8s. (#2596, @abhiraut)
Default to containerd as the container runtime in the Vagrant-based test K8s cluster. (#2583, @stanleywbwong)
Update AllowToCoreDNS example in Antrea-native policies documentation. (#2605, @btrieger)
Update actions/setup-go to v2 in all Github workflows. (#2517, @MysteryBlokHed)

Fixed

Fix panic in Agent when calculating the stats for a rule newly added to an existing NetworkPolicy. (#2495, @tnqn)
Fix bug in iptables rule installation for dual-stack clusters: if a rule was already present for one protocol but not the other, its installation may have been skipped. (#2469, @lzhecheng)
Fix deadlock in the Agent's FlowExporter, between the export goroutine and the conntrack polling goroutine. (#2429, @srikartati)
Upgrade OVS version to 2.14.2-antrea.1 for Windows Nodes; this version of OVS is built on top of the upstream 2.14.2 release and also includes a patch to fix TCP checksum computation when the DNAT action is used. (#2549, @lzhecheng) [Windows]
Handle transient iptables-restore failures (caused by xtables lock contention) in the NodePortLocal initialization logic. (#2555, @antoninbas)
Query and check the list of features supported by the OVS datapath during Agent initialization: if any required feature is not supported, the Agent will log an error and crash, instead of continuing to run which makes it hard to troubleshoot such issues. (#2571, @tnqn)
On Linux, wait for the ovs-vswitchd PID file to be ready before running ovs-apptcl commands. (#2695, @tnqn)
Periodically delete stale connections in the Flow Exporter if they cannot be exported (e.g. because the collector is not available), to avoid running out-of-memory. (#2516, @srikartati)
Fix handling of the "reject" packets generated by the Antrea Agent in the OVS pipeline, to avoid infinite looping when traffic between two endpoints is rejected by network policies in both directions. (#2579, @GraysonWu)
Fix Linux kernel version parsing to accommodate for more Linux distributions, in particular RHEL / CentOS. (#2450, @Jexf)
Fix interface naming for IPsec tunnels: based on Node names, the first char could sometimes be a dash, which is not valid. (#2486, @luolanzone)
When creating an IPsec OVS tunnel port to a remote Node, handle the case where the port already exists but with a stale config graciously: delete the existing port first, then recreate it. (#2582, @luolanzone)
Fix the policy information reported by the Flow Exporter when a Baseline Antrea-native policy is applied to the flow. (#2542, @zyiou)
Clean up log files for the Flow Aggregator periodically: prior to this fix, the "--log_file_max_size" and "--log_file_max_num" command-line flags were ignore for the flow-aggregator Pod. (#2522, @srikartati)
Fix missing template ID when sending the first IPFIX flow record from the FlowAggregator. (#2546, @zyiou)
Ensure that the Windows Node name obtained from the environment or from hostname is converted to lower-case. (#2672, @shettyg) [Windows]
Fix Antrea network clean-up script for Windows; in particular remove Hyper-V binding on network adapter used as OVS uplink so that it can recover its IP address correctly. (#2550, @wenyingd) [Windows]
Fix reference Logstash configuration to avoid division by zero in throughput calculation. (#2432, @zyiou)
Fix nil pointer error when collecting a supportbundle on a Node for which the antrea-agent container image does not include "iproute2"; this does not affect the standard antrea/antrea-ubuntu container image. (#2598, @liu4480)

Resolved Issues

Realization of Network Policies takes longer after upgrade to 1.3.0-1.2.2 with large number (10,000+) of Antrea-Network Policy Custom Resources.
When Antrea is upgraded to 1.3.0-1.2.2 from an older release (1.2.0-0.13.x or older), Antrea controller automatically mirrors Antrea Network Policy to new API group 'antrea.io'. So when a large number of ANP CRs are present, the Antrea Network policies are enforced slowly as the ANP CR is being mirrored to the new API. This is observed with more than 10,000+ ANPs.

Workaround: Manually mirror old API group to new API group before upgrade
1. Create the new CRDs of tiers.crd.antrea.io, networkpolicies.crd.antrea.io, clusternetworkpolicies.crd.antrea.io, clustergroups.crd.antrea.io. The CRDs can be fetched from the manifest yaml that you will use for upgrade.
2. Dump the resources of the deprecated CRDs (with *.antrea.tanzu.vmware.com suffix), create the same resources with new CRDs (with *.antrea.io suffix):
```
kubectl get tiers.security.antrea.tanzu.vmware.com -o yaml -A | sed "s/security.antrea.tanzu.vmware.com/crd.antrea.io/g" | kubectl apply -f -
kubectl get networkpolicies.security.antrea.tanzu.vmware.com -o yaml -A | sed "s/security.antrea.tanzu.vmware.com/crd.antrea.io/g" | kubectl apply -f -
kubectl get clusternetworkpolicies.security.antrea.tanzu.vmware.com -o yaml -A | sed "s/security.antrea.tanzu.vmware.com/crd.antrea.io/g" | kubectl apply -f -
kubectl get clustergroups.core.antrea.tanzu.vmware.com -o yaml -A | sed "s/core.antrea.tanzu.vmware.com/crd.antrea.io/g" | kubectl apply -f -
```
Note, if there are no resources for a CRD, the command will get an error output "error: no objects passed to apply", which can be ignored.
Antrea-NSX interworking Pod is in CrashLoopBackoff State and mp-adapter log shows TCP port 10351 is occupied.
The mp-adapter in previous Antrea release listens on a local TCP port 10351 for liveness probe. The port 10351 is also the default port for Antrea-agent Gossip cluster. The Antrea-agents Gossip cluster is mainly for Egress high availability. Starting from this Antrea release, the Egress feature is enabled by default, so the port 10351 is always occupied by Antrea-agent.

This problem is fixed in the current Antrea release. The interworking Pod uses a different port for liveness probe, and the port number is configurable in the ConfigMap. When you upgrades Antrea, we recommend to also upgrade the Antrea-NSX interworking Deployment to use the image and manifest in the same Antrea release.

Known Issues

The known issues are grouped as follows.

Known Issues

Container Network Traffic Throughput Drops to Zero on Buggy Physical NIC
Antrea enables Geneve tunnel checksum offload by default. However, sometimes the container networking traffic throughput drop to nearly zero. In packet capture we see that TCP 3-way handshake is successful but the first data packet in MTU size gets wrong checksum and it's dropped in the receiver side. This can happen when the K8s node VMs are running on overlay network and the underlay network cannot correctly process checksum offloading in double encapsulation scenario, or the physical NIC has bug in checksum offloading.

We introduced the following ConfigMap antrea-agent-tweaker in antrea.yml to allow disabling tunnel checksum offloading.
```
apiVersion: v1
data:
  antrea-agent-tweaker.conf: |-
    # Enable disableUdpTunnelOffload will disable udp tunnel offloading feature on kubernetes node's default interface.
    # By default, no actions will be taken.
    disableUdpTunnelOffload: false
kind: ConfigMap
metadata:
  labels:
    app: antrea
  name: antrea-agent-tweaker-g56hc6fh8t
  namespace: kube-system
```
This is only for Linux. You can use kubectl to edit the live ConfigMap on K8s API to disable tunnel checksum offload, then restart all Antrea agents (usually run the command kubectl delete pod -l component=antrea-agent -n kube-system) to make the option effective. You can also edit this ConfigMap in antrea.yml before deploying Antrea.
We suggest not to set it to true only if you hit tunnel checksum offloading issue. Disabling tunnel checksum offloading drops container networking throughput by about 50%.
Antrea-NSX Integration: /32 IPv4 and /128 IPv6 CIDRs in IP Address Group are Handled Incorrectly
In NSX >= 4.0.0 release, customer can define IP address groups and refer to these groups in Antrea policy. NSX strips the /32 from IPv4 CIDR and /128 from IPv6 CIDR, and sends the stripped IP configurations to NSX adapter in Kubernetes cluster. NSX adapter wrongly handles IP CIDR without prefix as /0.

Customer can define Antrea group and input the IP CIDRs in Antrea group instead of IP address group. IP CIDRs in Antrea group are correctly handled.