Configuring a Subnet for Egress IPs

With the EgressSeparateSubnet feature, you can configure a subnet that is different from the Node subnet for allocating Egress IPs.

For more information about Egress and the EgressSeparateSubnet feature in Antrea, see https://antrea.io/docs/v2.1.0/docs/egress.

Note that VLAN is used to isolate the different Egress subnets when you configure this feature.

Overview of the EgressSeparateSubnet Feature

To enable this feature, follow the steps below:

Enable the EgressSeparateSubnet feature in the Antrea config map.
Configure a subnet in NSX for the Egress IPs.
1. Create a segment in NSX with the subnet's CIDR and gateway IP.
2. Create a segment connection binding map for the new segment with the desired VLAN, and attach it to the segment the Nodes are using.
Note that if the nodes in the cluster are attached to multiple segments in NSX, all these segments must be attached to the new segment (created in step 2.a) using the same VLAN.
Create Antrea ExternalIPPool using the allocated CIDR/gateway/VLAN in the Kubernetes cluster where Antrea is running.
Create Antrea Egress using the new ExternalIPPool.

This following diagram shows the topology for this feature:

Prerequisites

NSX 4.2.0 or later.
You must have the permission to create segments and SegmentConnectionBindingMaps on NSX.

Enable the EgressSeparateSubnet Feature

The EgressSeparateSubnet feature is disabled by default in TKG Service clusters. To enable this feature, perform the following actions:

Authenticate with the Supervisor with the following command:
```
kubectl vsphere login --server --vsphere-username
```
Switch context to the target vSphere namespace where you will provision the TKG service cluster.
```
kubectl config use-context vsphere-namespace
```

Create the AntreaConfig.yaml custom resource definition and enable the feature EgressSeparateSubnet . For example,

apiVersion: cni.tanzu.vmware.com/v1alpha1
kind: AntreaConfig
metadata:
 name: -antrea-package #prefix required
 namespace: 
spec:  
  antrea:
    featureGates:
      EgressSeparateSubnet: true

Apply the AntreaConfig custom resource definition.
```
kubectl apply -f AntreaConfig.yaml
```

Note: The above steps are based on TKGS clusters. If a standalone Antrea cluster is deployed, you can enable the EgressSeparateSubnet feature by editing the configmap antrea-config and restarting the antrea-agent. For example,

apiVersion: v1
kind: ConfigMap
metadata:
 name: antrea-config
 namespace: kube-system
data:
 antrea-agent.conf: |
   featureGates:
     EgressSeparateSubnet: true

Configure a Subnet in NSX for Egress IPs

You can configure a subnet for Egress IPs using the antreansxtl command or NSX UI and API.

Method 1 - Using the antreansxtl command to configure a subnet for Egress IPs.

You can use antreansxctl to create a segment and segment connection binding map in one command. For example,

./antreansxctl child-segment-create --nsx-managers=$manager_ip --user=$user --password=$password --cidr="10.10.0.0/24 --gateway="10.10.0.1" --parent="/infra/segments/node-seg" --vlan=201 egress

Note the following:

The antreansxctl tool is provided in the antrea-interworking Pod by default. You can log in to the Pod to execute the command or download the utility from the Antrea-interworking zip file in the Antrea commercial release.
When running the antreansxctl utility, you must log in as an enterprise admin user.
For more information about managing child segments, see CLI Command antreansxctl.

Method 2 - Using the NSX UI and API to configure a subnet for Egress IPs.

The NSX UI does not support creating a segment connection binding map. NSX API must be used. In the examples below, assume the following:

The subnet we want to create is 10.10.0.0/24 and the gateway IP is 10.10.0.1.
The subnet is created with a path /infra/segments/egress .
A segment with a path /infra/segments/node-seg is used by the Nodes in the cluster.
VLAN 201 is used in the ExternalIPPool.

Make the following call to create a segment connection binding map:

curl -k -u $username:$password -H 'Content-Type: application/json' -X PUT https://$manger_ip/policy/api/v1/infra/segments/egress/segment-connection-binding-maps/map1 -d {
   "segment_path": "/infra/segments/node-seg",
   "vlan_traffic_tag": 201
}

Make the following call to list segment connection binding maps:

curl -k -u $username:$password -H 'Content-Type: application/json' -X GET https://$manger_ip/policy/api/v1/infra/segments/egress/segment-connection-binding-maps

Make the following call to delete a segment connection binding map:

curl -k -u $username:$password -H 'Content-Type: application/json' -X DELETE https://$manger_ip/policy/api/v1/infra/segments/egress/segment-connection-binding-maps/map1

Create Antrea ExternalIPPool and Egress CRs

You can create the Antrea ExternalIPPool and Egress resources inside the guest cluster in the Supervisor-based setup. Below is an example of the ExternalIPPool Custom Resource using the separate subnet.

apiVersion: crd.antrea.io/v1beta1
kind: ExternalIPPool
metadata:
 name: prod-external-ip-pool
spec:
 ipRanges:
 - cidr: 10.10.0.0/24
 subnetInfo:
   gateway: 10.10.0.1
   prefixLength: 24
   vlan: 201
 nodeSelector:
   matchLabels:
     network-role: egress-gateway

Below is an example of the Egress Custom Resource.

apiVersion: crd.antrea.io/v1beta1
kind: Egress
metadata:
 name: egress-prod-web
spec:
 appliedTo:
   namespaceSelector:
     matchLabels:
       kubernetes.io/metadata.name: prod
   podSelector:
     matchLabels:
       app: web
 externalIPPool: external-ip-pool

Known Issue

To support the EgressSeparateSubnet feature, VLAN sub-interfaces will be created by Antrea Agents. The rp_filter of VLAN sub-interfaces should be 2, which enables loose mode filtering. In a vanilla Kubernetes cluster, Antrea Agents will set the rp_filter to 2 automatically without user intervention. However, it has been observed that rp_filter update by Antrea has no effect on OpenShift clusters due to a known issue. A workaround is to leverage OpenShift Node Tuning Operator to update the rp_filter for all interfaces on all Egress Nodes:

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: antrea
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Update rp_filter for all
      [sysctl]
      net.ipv4.conf.all.rp_filter=2
    name: openshift-antrea
  recommend:
  - match:
    - label: network-role
      value: egress-gateway
    priority: 10
    profile: openshift-antrea

After you apply the above Tuned CR named antrea in a given OpenShift cluster, the Node Tuning Operator will watch the CR and update net.ipv4.conf.all.rp_filter to 2 for all matched Nodes (for example, all Nodes with a label network-role=egress-gateway). See the OpenShift official documentation about Using the Node Tuning Operator for more information about Tuned CR.