VMware Container Networking with Antrea 1.5.0 | 25 AUG 2022 | Build 20340672

Check for additions and updates to these release notes.

What's New

  • Antrea Container Networking with Antrea 1.5.0 is based off the Antrea v1.7.1 open-source release.
  • Tech Preview of IDS/IPS for Containers using Antrea data-plane.
    • Use TrafficControl feature to mirror traffic originating from specific Pods or destined to specific Pods to a remote destination via tunnel.
    • Customers can create IDPSPolicy CRs to select Pods and apply IDS/IPS rules to ingress/egress traffic of the selected Pods.
      • Requires Antrea to NSX integration.
  • Antrea Operator can deploy NSX adapters for Antrea to NSX integration.
  • NetworkPolicy enforced on Multicast traffic and NetworkPolicy statistics for Multicast traffic.
  • Enhanced Features
    • Antrea IPSec supports certificate-based authentication along with proper key rotation capabilities.
    • Support for ICMP traffic in Antrea native policies.
    • Support pre-allocating continuous IPs for StatefulSet in Antrea IPAM.
    • antctl mc subcommand for Antrea Multi-cluster resources.
    • Support for NodePortLocal on Windows workloads.
    • Support for Traceflow on Windows.
    • Flow aggregator was improved to support large scale.
    • Antrea flexible IPAM graduates as beta. The flexible IPAM usage scenario is expanded to include NSX (outer overlay) and TKG workload clusters.
  • Container images on VMware distribution Harbor:
    • Antrea images:
      • projects.registry.vmware.com/antreainterworking/antrea-standard-debian:v1.7.1_vmware.1
      • projects.registry.vmware.com/antreainterworking/antrea-advanced-debian:v1.7.1_vmware.1
      • projects.registry.vmware.com/antreainterworking/antrea-ubi:v1.7.1_vmware.1
    • Antrea multi-cluster controller images:
      • projects.registry.vmware.com/antreainterworking/antrea-mc-controller-debian:v1.7.1_vmware.1
      • projects.registry.vmware.com/antreainterworking/antrea-mc-controller-ubi:v1.7.1_vmware.1
    • Antrea flow-aggregator images:
      • projects.registry.vmware.com/antreainterworking/flow-aggregator-debian:v1.7.1_vmware.1
      • projects.registry.vmware.com/antreainterworking/flow-aggregator-ubi:v1.7.1_vmware.1
    • Antrea IDPS images:
      • IDPS controller and agent
        • projects.registry.vmware.com/antreainterworking/idps-debian:v1.7.1_vmware.1
        • projects.registry.vmware.com/antreainterworking/idps-ubi:v1.7.1_vmware.1
      • Suricata
        • projects.registry.vmware.com/antreainterworking/suricata:v1.7.1_vmware.1
    • Operator image:
      • projects.registry.vmware.com/antreainterworking/antrea-operator:v1.7.1_vmware
    • Antrea-NSX images:
      • projects.registry.vmware.com/antreainterworking/interworking-debian:0.7.0
      • projects.registry.vmware.com/antreainterworking/interworking-ubuntu:0.7.0
      • projects.registry.vmware.com/antreainterworking/interworking-photon:0.7.0
      • projects.registry.vmware.com/antreainterworking/interworking-ubi:0.7.0

Note:

  1. UBI images can only run on RHEL 8 or newer OSes with nftables kernel module (nf_tables) loaded.
  2. Photon images can only run on Photon OS or OSes with iptables legacy kernel module (ip_tables) loaded.

Compatibility Testing Matrix

K8S Distribution K8S Versions OS Encapsulation
K8s 1.21, 1.22, 1.23, 1.24 Ubuntu 18.04, PhotonOS 3, Debian 10 Geneve,  NoEncap, Hybrid
AWS EKS 1.21 Amazon Linux 2 Policy Only Mode
Azure AKS, AKS Engine 1.21 Ubuntu 18.04 Policy Only Mode
GKE (Google Kubernetes Engine) 1.22 Ubuntu 18.04, Windows NoEncap, Policy Only Mode
RHEL RHEL 7.9 onwards RHEL Geneve, NoEncap, Hybrid
OpenShift OCP 4.7, 4.8, 4.9, 4.10 RHCOS and RHEL Geneve, NoEncap, Hybrid
NSX 3.2.0, 3.2.1, 4.0.0.1

Change Logs 1.7.1 - 2022-07-14

Changes from OSS 1.7.1, includes all changes from OSS 1.7.0, OSS 1.6.0 and OSS 1.6.1

Changes from 1.7.1

Fixed

  • Fix FlowExporter memory bloat when export process is dead. (#3994, @wsquan171)
  • Fix Pod-to-external traffic on EKS in policyOnly mode. (#3975, @antoninbas)
  • Use uplink interface name for host interface internal port to support DHCP client. (#3938, @gran-vmv)

Changes from 1.7.0

Added

  • Add TrafficControl feature to control the transmission of Pod traffic; it allows users to mirror or redirect traffic originating from specific Pods or destined for specific Pods to a local network device or a remote destination via a tunnel of various types. (#3644#3580#3487, @tnqn @hongliangl @wenqiq)
    • Refer to this document for more information about this feature.
    • Refer to this cookbook for more information about using this feature to provide network-based intrusion detection service to your Pods.
  • Add support for the IPsec Certificate-based Authentication. (#3778, @xliuxu)
    • Add an Antrea Agent configuration option ipsec.authenticationMode to specify authentication mode. Supported options are "psk" (default) and "cert".
    • Add an Antrea Controller configuration option ipsecCSRSigner.autoApprove to specify the auto-approve policy of Antrea CSR signer for IPsec certificates management. By default, Antrea will auto-approve the CertificateSingingRequest (CSR) if it is verified.
    • Add an Antrea Controller configuration option ipsecCSRSigner.selfSignedCA to specify whether to use auto-generated self-signed CA certificate. By default, Antrea will auto-generate a self-signed CA certificate.
  • Add the following capabilities to Antrea-native policies:
    • Add support for matching ICMP traffic. (#3472, @GraysonWu)
    • Add support for matching multicast and IGMP traffic. (#3660, @liu4480)
    • Add support for rule-level statistics for multicast and IGMP traffic. (#3449, @ceclinux)
  • Add the following capabilities to the Multicast feature:
    • Add antctl get podmulticaststats command to query Pod-level multicast traffic statistics in Agent mode. (#3449, @ceclinux)
    • Add "MulticastGroup" API to query Pods that have joined multicast groups; kubectl get multicastgroups can generate requests and output responses of the API. (#3354#3449, @ceclinux)
    • Add an Antrea Agent configuration option multicast.igmpQueryInterval to specify the interval at which the antrea-agent sends IGMP queries to Pods. (#3819, @liu4480)
  • Add the following capabilities to the Multi-cluster feature:
    • Add the Multi-cluster Gateway functionality which supports routing Multi-cluster Service traffic across clusters through tunnels between the Gateway Nodes. It enables Multi-cluster Service access across clusters, without requiring direct reachability of Pod IPs between clusters. (#3689#3463#3603, @luolanzone)
    • Add a number of antctl mc subcommands for bootstrapping Multi-cluster; refer to the Multi-cluster antct document for more information. (#3474, @hjiajing)
  • Add the following capabilities to secondary network IPAM:
  • Add support for NodePortLocal on Windows. (#3453, @XinShuYang)
  • Add support for Traceflow on Windows. (#3022, @gran-vmv)
  • Add support for containerd to antrea-eks-node-init.yml. (#3840, @antoninbas)
  • Add an Antrea Agent configuration option disableTXChecksumOffload to support cases in which the datapath's TX checksum offloading does not work properly. (#3832, @tnqn)
  • Add support for InternalTrafficPolicy in AntreaProxy. (#2792, @hongliangl)
  • Add the following documentation:

Changed

  • Optimize generic traffic performance by reducing OVS packet recirculation. (#3858, @tnqn)
  • Optimize NodePort traffic performance by reducing OVS packet recirculation. (#3862, @hongliangl)
  • Improve validation for IPPool CRD. (#3570, @jianjuns)
  • Improve validation for egress.to.namespaces.match of AntreaClusterNetworkPolicy rules. (#3727, @qiyueyao)
  • Deprecate the Antrea Agent configuration option multicastInterfaces in favor of multicast.multicastInterfaces. (#3898, @tnqn)
  • Reduce permissions of Antrea Agent ServiceAccount. (#3691, @xliuxu)
  • Create a Secret in the Antrea manifest for the antctl and antrea-agent ServiceAccount as K8s v1.24 no longer creates a token for each ServiceAccount automatically. (#3730, @antoninbas)
  • Implement garbage collector for IP Pools to clean up allocations and reservations for which owner no longer exists. (#3672, @annakhm)
  • Preserve client IP if the selected Endpoint is local regardless of ExternalTrafficPolicy. (#3604, @hongliangl)
  • Add a Helm chart for Antrea and use the Helm templates to generate the standard Antrea YAML manifests. (#3578, @antoninbas)
  • Make "Agent mode" antctl work out-of-the-box on Windows. (#3645, @antoninbas)
  • Truncate SessionAffinity timeout values of Services instead of wrapping around. (#3609, @antoninbas)
  • Move Antrea Windows log dir from C:\k\antrea\logs\ to C:\var\log\antrea\. (#3416, @GraysonWu)
  • Limit max number of data values displayed on Grafana panels. (#3812, @heanlan)
  • Support deploying ClickHouse with Persistent Volume. (#3608, @yanjunz97)
  • Remove support for ELK Flow Collector. (#3738, @heanlan)
  • Improve documentation for Antrea-native policies. (#3512, @Dyanngg)
  • Update OVS version to 2.17.0. (#3591, @antoninbas)

Fixed

  • Fix Egress not working with kube-proxy IPVS strictARP mode. (#3837, @xliuxu)
  • Fix intra-Node Pod traffic bypassing Ingress NetworkPolicies in some scenarios. (#3809, @hongliangl)
  • Fix FQDN policy support for IPv6. (#3869, @tnqn)
  • Fix multicast not working if the AntreaPolicy feature is disabled. (#3807, @liu4480)
  • Fix tolerations for Pods running on control-plane for Kubernetes >= 1.24. (#3731, @xliuxu)
  • Fix DNS resolution error of antrea-agent on AKS by using ClusterFirst dnsPolicy. (#3701, @tnqn)
  • Clean up stale routes installed by AntreaProxy when ProxyAll is disabled. (#3465, @hongliangl)
  • Ensure that Service traffic does not bypass NetworkPolicies when ProxyAll is enabled on Windows. (#3510, @hongliangl)
  • Use IP and MAC to find virtual management adapter to fix Agent crash in some scenarios on Windows. (#3641, @wenyingd)
  • Fix handling of the "reject" packets generated by the Antrea Agent to avoid infinite looping. (#3569, @GraysonWu)
  • Fix export/import of Services with named ports when using the Antrea Multi-cluster feature. (#3561, @luolanzone)
  • Fix Multi-cluster importer not working after leader controller restarts. (#3596, @luolanzone)
  • Fix Endpoint ResourceExports not cleaned up after corresponding Service is deleted. (#3652, @luolanzone)
  • Fix pool CRD format in egress.md and service-loadbalancer.md. (#3885, @jianjuns)
  • Fix infinite looping when Agent tries to delete a non-existing route. (#3827, @hongliangl)
  • Fix race condition in ConntrackConnectionStore and FlowExporter. (#3655, @heanlan)

Changes from 1.6.1

Added

Fixed

  • Clean up stale routes installed by AntreaProxy when ProxyAll is disabled. (#3465, @hongliangl)
  • Fix export/import of Services with named ports when using the Antrea Multi-cluster feature. (#3561, @luolanzone)
  • Fix handling of the "reject" packets generated by the Antrea Agent to avoid infinite looping. (#3569, @GraysonWu)
  • Fix DNS resolution error of Antrea Agent on AKS by using ClusterFirst dnsPolicy. (#3701, @tnqn)
  • Fix tolerations for Pods running on control-plane for Kubernetes >= 1.24. (#3731, @xliuxu)
  • Reduce permissions of Antrea Agent ServiceAccount. (#3691, @xliuxu)
  • [Windows] Ensure that Service traffic does not bypass NetworkPolicies when ProxyAll is enabled. (#3510, @hongliangl)
  • Fix Antrea wildcard FQDN NetworkPolicies not working when NodeLocal DNSCache is enabled. (#3510, @hongliangl)

Changes from 1.6.0

Added

Changed

  • Remove all legacy (*.antrea.tanzu.vmware.com) APIs. (#3299, @antoninbas)
  • Remove Kind-specific manifest and scripts. Antrea now uses OVS kernel datapath for Kind clusters. (#3413, @antoninbas)
  • [Windows] Use uplink MAC as source MAC when transmitting packets to underlay network from Windows Nodes. Therefore, MAC address spoofing configuration like "Forged transmits" in VMware vSphere doesn't need to be enabled. (#3516, @wenyingd)
  • Add an agent config parameter "enableBridgingMode" for enabling flexible IPAM (bridging mode). (#3297#3365, @jianjuns)
  • Use iptables-wrapper in Antrea container to support distros that runs iptables in "nft" mode. (#3276, @antoninbas)
  • Install CNI configuration files after installing CNI binaries to support container runtime cri-o. (#3154, @tnqn)
  • Upgrade packaged Whereabouts version to v0.5.1. (#3511, @antoninbas)
  • Upgrade to go-ipfix v0.5.12. (#3352, @yanjunz97)
  • Upgrade Kustomize from v3.8.8 to v4.4.1 to fix Cronjob patching bugs. (#3402, @yanjunz97)
  • Fail in Agent initialization if GRE tunnel type is used with IPv6. (#3156, @antoninbas)
  • Refactor the OpenFlow pipeline for future extensibility. (#3058, @hongliangl)
  • Validate IP ranges of IPPool for Antrea IPAM. (#2995, @ksamoray)
  • Validate protocol in the CRD schema of Antrea-native policies. (#3342, @KMAnju-2021)
  • Validate labels in the CRD schema of Antrea-native policies and ClusterGroup. (#3331, @GraysonWu)
  • Reduce permissions of Antrea ServiceAccounts. (#3393, @tnqn)
  • Remove --k8s-1.15 flag from hack/generate-manifest.sh. (#3350, @antoninbas)
  • Remove unnecessary CRDs and RBAC rules from Multi-cluster manifest. (#3491, @luolanzone)
  • Update label and image repo of antrea-mc-controller to be consistent with antrea-controller and antrea-agent. (#3266#3466, @luolanzone)
  • Add clusterID annotation to ServiceExport/Import resources. (#3359, @luolanzone)
  • Do not log error when Service for Endpoints is not found to avoid log spam. (#3256, @tnqn)
  • Ignore Services of type ExternalName for NodePortLocal feature. (#3114, @antoninbas)
  • Add powershell command replacement in the Antrea Windows documentation. (#3264, @GraysonWu)

Fixed

  • Add userspace ARP/NDP responders to fix Egress and ServiceExternalIP support for IPv6 clusters. (#3318, @hty690)
  • Fix incorrect results by antctl get networkpolicy when both Pod and Namespace are specified. (#3499, @Dyanngg)
  • Fix IP leak issue when AntreaIPAM is enabled. (#3314, @gran-vmv)
  • Fix error when dumping OVS flows for a NetworkPolicy via antctl get ovsflows. (#3335, @jainpulkit22)
  • Fix IPsec encryption for IPv6 overlays. (#3155, @antoninbas)
  • Add ignored interfaces names when getting interface by IP to fix NetworkPolicyOnly mode in AKE. (#3219, @wenyingd)
  • Fix duplicate IP case for NetworkPolicy. (#3467, @tnqn)
  • Don't delete the routes which are added for the peer IPv6 gateways on Agent startup. (#3336#3490, @Jexf@xliuxu)
  • Fix pkt mark conflict between HostLocalSourceMark and SNATIPMark. (#3430, @tnqn)
  • Unconditionally sync CA cert for Controller webhooks to fix Egress support when AntreaPolicy is disabled. (#3421, @antoninbas)
  • Fix inability to access NodePort in particular cases. (#3371, @hongliangl)
  • Fix ipBlocks referenced in nested ClusterGroup not processed correctly. (#3383, @Dyanngg)
  • Realize Egress for a Pod as soon as its network is created. (#3360, @tnqn)
  • Fix NodePort/LoadBalancer issue when proxyAll is enabled. (#3295, @hongliangl)
  • Do not panic when processing a PacketIn message for a denied connection. (#3447, @antoninbas)
  • Fix CT mark matching without range in flow exporter. (#3348, @hongliangl)
  • [Windows] Enable IP forwarding of the Windows bridge local interface to fix support for Service of type LoadBalancer. (#3137, @hongliangl)

Known Issues

Container Network Traffic Throughput Drops to Zero on Buggy Physical NIC

Antrea enables Geneve tunnel checksum offload by default. However, sometimes the container networking traffic throughput drop to nearly zero. In packet capture we see that TCP 3-way handshake is successful but the first data packet in MTU size gets wrong checksum and it's dropped in the receiver side. This can happen when the K8s node VMs are running on overlay network and the underlay network cannot correctly process checksum offloading in double encapsulation scenario, or the physical NIC has bug in checksum offloading.

Workaround: We introduced the following ConfigMap antrea-agent-tweaker in antrea.yml to allow disabling tunnel checksum offloading.

apiVersion: v1
data:
  antrea-agent-tweaker.conf: |-
    # Enable disableUdpTunnelOffload will disable udp tunnel offloading feature on kubernetes node's default interface.
    # By default, no actions will be taken.
    disableUdpTunnelOffload: false
kind: ConfigMap
metadata:
  labels:
    app: antrea
  name: antrea-agent-tweaker-g56hc6fh8t
  namespace: kube-system

This is only for Linux. You can use kubectl to edit the live ConfigMap on K8s API to disable tunnel checksum offload, then restart all Antrea agents (usually run the command kubectl delete pod -l component=antrea-agent -n kube-system) to make the option effective. You can also edit this ConfigMap in antrea.yml before deploying Antrea.We suggest not to set it to true only if you hit tunnel checksum offloading issue. Disabling tunnel checksum offloading drops container networking throughput by about 50%.

check-circle-line exclamation-circle-line close-line
Scroll to top icon