AviInfraSetting provides a way to segregate Layer-4/Layer-7 virtual services to have properties based on different underlying infrastructure components, like Service Engine Group, intended VIP Network, and so on.

Note:

With AKO version 1.11.1, AviInfrasetting is transitioned to version v1beta1. There are no schema changes between version v1alpha1 and v1beta1. AKO 1.11.1 supports both versions v1alpha1 and v1beta1. However, it is recommended to create new CRD objects in v1beta1 and transition the existing objects to this version.

AKO will deprecate v1alpha1 in the upcoming releases.

A sample AviInfraSetting is as shown below:

apiVersion: ako.vmware.com/v1beta1
kind: AviInfraSetting
metadata:
  name: my-infra-setting
spec:
  seGroup:
    name: compact-se-group
  network:
    vipNetworks:
      - networkUUID: dvportgroup-2167-cloud-d4b24fc7-a435-408d-af9f-150229a6fea6f
        cidr: 10.10.10.0/24
        v6cidr: 2002::1234:abcd:ffff:c0a8:101/64
    nodeNetworks:
      - networkName: node-network-10-10-20-0-24
        cidrs:
        - 10.10.20.0/24
    enableRhi: true
    listeners:
      - port: 8081
        enableHTTP2: false
        enableSSL: false
      - port: 6443
        enableSSL: true
    bgpPeerLabels:
      - peer1
      - peer2
  l7Settings:
    shardSize: MEDIUM
  nsxSettings:
    t1lr: /infra/tier1/tier1_974b13d5-9f68-4be8-8149-a48a5686a3ef

AviInfraSetting with Services, Ingress, or Route

AviInfraSetting is a cluster scoped CRD and can be attached to the intended Services, Kubernetes Ingresses and OpenShift Routes as shown below.



Attaching AviInfraSetting to Services

AviInfraSetting resources can be attached to Services using Gateway APIs or simply by using annotations.

  1. Using Gateway API: Gateway APIs provide interfaces to structure Kubernetes service networking. For more information, see Gateway API. AKO supports Gateway APIs through the servicesAPI flag in the values.yaml. For more information on how AKO integrates with Gateway API, see Gateway and Gateway Class. The AviInfraSetting resource can be attached to a Gateway Class object, through the .spec.parametersRef as shown below:

    apiVersion: networking.x-k8s.io/v1alpha1
kind: GatewayClass
metadata:
  name: avi-gateway-class
spec:
  controller: ako.vmware.com/avi-lb
  parametersRef:
    group: ako.vmware.com
    kind: AviInfraSetting
    name: my-infrasetting

  2. Using Annotation: If the servicesAPI flag is not set to True, and AKO is not watching over the Gateway APIs, Services of Type LoadBalancer can specify the AviInfraSetting using an annotation as shown below:

    annotations:
    aviinfrasetting.ako.vmware.com/name: "my-infrasetting"

Attaching AviInfraSetting to Ingress

AviInfraSetting can be applied to Ingress resources, using the IngressClass construct. IngressClass provides a way to configure Controller-specific load balancing parameters and applies these configurations to a set of Ingress objects. AKO supports listening to IngressClass resources in Kubernetes version 1.19+. The AviInfraSetting reference can be provided in the Ingress Class as shown below:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: avi-ingress-class
spec:
  controller: ako.vmware.com/avi-lb
  parameters:
    apiGroup: ako.vmware.com
    kind: AviInfraSetting
    name: my-infrasetting

Attaching AviInfraSetting to Routes

AviInfraSetting with OpenShift Routes, through the annotation shown below:

 annotations:
     aviinfrasetting.ako.vmware.com/name: "my-infrasetting"

Using Namespace Annotation

AviInfraSetting also integrates with all the obove objects - Services/Ingresses/Routes, via the Namespace annotation. By annotating a namespace with an AviInfraSetting CRD, it can be applied to all the services, ingresses and routes in that namespace. AviInfraSetting CRD specified through Ingress/Gateway class or direct annotation on Services and Routes will take priority over the AviInfraSetting CRD specified at the namespace level through the namespace annotation.

  annotations:
    aviinfrasetting.ako.vmware.com/name: "my-infrasetting"

AviInfraSetting

Configure Service Engine Group

The AviInfraSetting CRD can be used to configure Service Engine Groups for virtual services created corresponding to Services/Ingresses/ Routes. Ensure the Service Engine Group is created and configured in the NSX Advanced Load Balancer Controller prior to this CRD creation.

  seGroup:
       name: compact-se-group

AKO tries to configure labels on the SEG specified in the AviInfraSetting resources, which enables static route syncing on the member SEs. AKO configures the labels only when the SEGs do not have any other labels configured already. In case AKO finds the SEG configured with some different labels, the AviInfraSetting resource would be Rejected.

Note:

Once deployed, the member SEs will not reflect any changes related to label additions or updates on the SEGs.

Therefore, label-based static route syncing will not work on already deployed SEs. Ensure that the SEGs have no member SEs deployed before specifying the SEG in the AviInfraSetting resource to properly configure static route syncing.

Configure VIP Networks

The AviInfraSetting CRD can be used to configure VIP networks for virtual services created corresponding to Services/Ingresses/Openshift Routes. Ensure that the network is present in the NSX Advanced Load Balancer Controller prior to this CRD creation.

Starting with AKO version 1.10.3, networkUUID field has been introduced as part of vipNetworks setting to choose appropriate network. User can specify either networkName or networkUUID while specifying the network.

vipNetworks: - networkName: vip-network-10-10-10-0-24 cidr: 10.10.10.0/24

Or

    network:
      vipNetworks:
        - networkUUID: dvportgroup-2167-cloud-d4b24fc7-a435-408d-af9f-150229a6fea6f
          cidr: 10.10.10.0/24
Note:

  • If there are duplicate network entries at the Controller side, AKO recommends to use networkUUID while choosing vipNetwork for new aviinfrasetting object creation.

  • Multiple network names can be added to the CRD (only in case of AWS cloud). The Avi virtualservices flag will acquire a VIP from each of these specified networks. If Avi fails to allocate a VIP due to IP exhaustion, this results in complete failure of entire request. This is same as vip allocation failures in single VIP.

AKO 1.5.1 updates the schema to provide VIP network information in AviInfraSetting CRD. For further details, see Upgrade AKO.

Configure Pool Placement Networks

AviInfraSetting CRD can be used to configure Pool Placement Settings on the AKO created Pools in order for the Service Engines to place the Pools on appropriate interfaces.

With AKO 1.10.3, networkUUID field has been introduced as part of nodeNetworks setting to choose appropriate network. User can specify either networkName or networkUUID while specifying the network.

        network:
          nodeNetworks:
            - networkName: node-network-10-10-20-0-24
              cidrs:
              - 10.10.20.0/24
        network:
          nodeNetworks:
            - networUUID: dvportgroup-4167-cloud-d4b24fc7-a435-408d-af9f-150229a6fea6f
              cidrs:
              - 10.10.20.0/24

As shown in the examples, each unique network can be specified using either networkName or networkUUID with appropriate cidrs.

Note:

If there are duplicate network entries at the Controller side, AKO recommends to use networkUUID while choosing nodeNetworks for new aviinfrasetting object creation.

If two Kubernetes clusters have overlapping Pod CIDRs, the service engine needs to identify the right gateway for each of the overlapping CIDR groups. This is achieved by specifying the right placement network for the pools that helps the Service Engine place the pools appropriately.

Enable/Disable Route Health Injection

AviInfraSetting CRD is used to activate/deactivate Route Health Injection (RHI) on the virtual services created by AKO.

network:
      enableRhi: true

This overrides the global enableRHI flag for the virtual services corresponding to the AviInfraSetting.

Enable/Disable Public IP

The AviInfraSetting CRD can be used to activate/deactivate Public IP on the virtual services created by AKO.

   network:
      enablePublicIP: true
Note:

Activating Public IP is only supported for public clouds.

Configure BGP Peer Labels for BGP Virtual Services

AviInfraSetting CRD can be used to configure BGP Peer labels for BGP virtual services. AKO configures the virtual services with the appropriate peer labels, only when enableRHI is enabled, either during AKO installation through helm chart’s values.yaml or through the AviInfraSetting CRD itself. If not set to true, the AviInfraSetting resource will be marked Rejected.

   bgpPeerLabels:
      - peer1
      - peer2
Use Dedicated VIP for Ingress

AviInfraSetting CRD can be used to allocate a dedicated vip per Ingress FQDN.

    l7Settings:
      shardSize: DEDICATED

For the subset of ingresses, that refer to an ingress class which in turn refers to an AviInfraSetting CRD setting that has shardSize as DEDICATED, will get VIP per Ingress FQDN.

For passthrough routes/ingresses, setting l7Settings:shardSize present in AviInfrasetting CRD overrides setting L7Settings.passthroughShardSize present in values.yaml.

Note:

The value DEDICATED is not supported when AviInfra CRD is applied to the passthrough route/ingress.

Configure IPv6

AviInfraSetting CRD can be used to enable IPv6, IPv4, or both IPv4 and IPv6 VIPs on the virtual services created by AKO.

    network:
      vipNetworks:
        - networkName: vip-network-10-10-10-0-24
          cidr: 10.10.10.0/24
          v6cidr: 2002::1234:abcd:ffff:c0a8:101/64
Configure T1LR for NSX-T Cloud

AviInfraSetting CRD can be used to configure T1LR. For all the Services and Ingresses that refer to an AviInfraSetting CRD with T1LR configured, AKO will use the T1LR defined in the AviInfraSetting while creating objects in NSX Advanced Load Balancer. For the rest of the resources, AKO will use the global T1LR configured in the ako config map. In case of an update in nsxSettings.t1lr, the existing virtual services will continue to use the old t1lr value. The updated t1lr value will only apply to the newly created virtual services.

    nsxSettings:
      t1lr: /infra/tier1/tier1_974b13d5-9f68-4be8-8149-a48a5686a3ef
Note:

AKO sets up routes in the Avi VRF corresponding to the global T1lr defined in the config map. However, for the T1lr defined in the AviInfraSetting CR, AKO will not setup any routes in the Avi Controller. This will create a connectivity issue between the Service Engine and the pool servers when AKO is deployed in ClusterIP mode. To resolve this connectivity issue, you can manually add routes in VRF, associated with the given T1LR. The connectivity issue will be rewsolved when AKO is deployed in NodePort or NPL mode or using NCP as CNI.