AKO is an operator which works as an ingress Controller and performs specific functions in a Kubernetes/OpenShift environment with the NSX Advanced Load Balancer Controller. It remains in sync with the necessary Kubernetes/ OpenShift objects and calls the NSX Advanced Load Balancer Controller APIs to configure the virtual services.

The AKO deployment consists of the following components:

  • The NSX Advanced Load Balancer Controller

  • The Service Engines (SE)

  • The Avi Kubernetes Operator (AKO)

An overview of the AKO deployment is as shown below:



Create a Cloud in NSX Advanced Load Balancer

The NSX Advanced Load Balancer infrastructure cloud will be used to place the virtual services that are created for the Kubernetes/ OpenShift application.

As a prerequisite to create the cloud, it is recommended to have IPAM and DNS profiles configured.

Configure IPAM and DNS Profile

Configure the IPAM profile and select the underlying network and the DNS profile which will be used for ingresses and external services.

To configure the IPAM Profile,

  1. Navigate to Templates > Profiles > IPAM/DNS Profiles.

  2. Edit the IPAM profile as shown below:



    Note:

    Usable network for the virtual services created by the AKO instance must be provided using the fields vipNetworkList|subnetIP|subnetPrefix fields during helm installation. For more information on Helm installation, see Install Helm CLI.

  3. Click Save.

To configure the DNS Profile,

  1. Navigate to Templates > Profiles > IPAM/DNS Profile.

  2. Configure the DNS profile with the Domain Name.



  3. Click Save.

Configure the Cloud

  1. Navigate to Infrastructure > Clouds.

  2. Select the vCenter cloud and click the edit icon.

  3. Under the IPAM/DNS, select the IPAM and DNS profiles created for the north-south apps as shown below:



  4. Under the vCenter/vSphere, select the Data Center and Management Network as shown below:



  5. Under IP Address Management for Management Network, select Enable DHCP.

  6. Click Save.

AKO in NSX-T deployments

Starting with AKO version 1.5.1, AKO supports the NSX-T write access cloud for both NCP and non-NCP CNIs. In NCP CNI, the pods are assumed to be routable from the SE’s backend data network segments. Due to this, AKO disables the static route configuration when the CNI is specified as ncp in the values.yaml. However, if non-ncp CNIs are used, AKO assumes that static routes can be configured on the SEs to reach the pod networks. In order for this scenario to be valid, the SEs backend data network must be configured on the same logical segment on which the Kubernetes/OpenShift cluster is run.

In addition to this, AKO supports both overlay and VLAN backed NSX-T cloud configurations. AKO automatically ascertains if a cloud is configured with overlay segments or is used with VLAN networks.

The VLAN backed NSX-T setup behaves the same as vCenter write access cloud, thus requiring no inputs from you. However, the overlay based NSX-T setups require configuring a logical segment as the backend data network and correspondingly configure the T1 router’s info during bootup of AKO through the Helm values parameter NetworkSettings.nsxtT1LR.

Configure SE Groups and Node Network List

  • SE Groups

    AKO supports SE groups. Using SE groups, all the clusters can now share the same VRF. Each AKO instance mapped to a unique serviceEngineGroupName. This will be used to push the routes on the SE to reach the pods. Each cluster needs a dedicated SE group, which cannot be shared by any other cluster or workload.

    Note:

    If the label is already configured, ensure the cluster name matches with the value.

  • Pre-requisites

    • Ensure that the NSX Advanced Load Balancer Controller is of version 18.2.10 or later.

    • Create SE groups per AKO cluster (out-of-band)

  • Node Network List

    In a vCenter cloud, nodeNetworkList is a list of PG networks that OpenShift/Kubernetes nodes are a part of. Each node has a CIDR range allocated by Kubernetes. For each node network, the list of all CIDRs has to be mentioned in the nodeNetworkList.

    For example, consider the Kubernetes nodes are a part of two PG networks - pg1-net and pg2-net. There are two nodes which belong to pg1-net with CIDRs 10.1.1.0/24 and 10.1.2.0/24. There are three nodes which belong to pg2-net with CIDRs 20.1.1.0/24, 20.1.2.0/24, and 20.1.3.0/24.

    Then nodeNetworkList contains:

    • pg1-net

      • 10.1.1.0/24

      • 10.1.2.0/24

    • pg2-net

      • 20.1.1.0/24

      • 20.1.2.0/24

      • 20.1.3.0/24

    Note:

    The nodeNetworkList is only used in the ClusterIP deployment of AKO and in vCenter cloud and only when disableStaticRouteSync is set to False.

If two Kubernetes clusters have overlapping CIDRs, the SE needs to identify the right gateway for each of the overlapping CIDR groups. This is achieved by specifying the right placement network for the pools that helps the Service Engine place the pools appropriately.

Configure the fields serviceEngineGroupName and nodeNetworkList in the values.yaml file.

Install Helm CLI

Helm is an application manager for OpenShift/Kubernetes. Helm charts are helpful in configuring the application. For more information, see Helm Installation.

AKO can be installed with or without internet access on the cluster.

For information on installing with internet access, see Install AKO for Kubernetes.

For information on installing without internet access, see Install AKO for Kubernetes.

Install AKO for Kubernetes

  1. Create the avi-system namespace:

     kubectl create ns avi-system 
    Note:
    • AKO can run in namespaces other than avi-system. The namespace in which AKO is deployed, is governed by the --namespace flag value provided during Helm install. There are no updates in the setup steps. avi-system has been kept as is in the entire documentation, and must be replaced by the namespace provided during AKO installation.

    • Helm version 3.8 and above will be required to proceed with Helm installation.

  2. Search the available charts for AKO:

       helm show chart oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.11.4
    
       Pulled: projects.registry.vmware.com/ako/helm-charts/ako:1.11.4
       Digest: sha256:xyxyxxyxyx
       apiVersion: v2
       appVersion: 1.11.4
       description: A helm chart for Avi Kubernetes Operator
       name: ako
       type: application
       version: 1.11.4
    
  3. Use the values.yaml from this chart to edit values related to the NSX Advanced Load Balancer configuration. To get the values.yaml for a release, run the following command:

     helm show values oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.11.4 > values.yaml 
  4. Edit the values.yaml file and update the details according to your environment.

  5. Install AKO:

    Multiple AKO instances can be installed in a cluster.

    Note:
    1. Out of multiple AKO instances, only one AKO instance must be Primary.

    2. Each AKO instance must be installed in a different namespace.

    Primary AKO Installation:

    helm install --generate-name oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.11.4 -f /path/to/values.yaml  --set ControllerSettings.controllerHost=<controller IP or Hostname> --set avicredentials.username=<avi-ctrl-username> --set avicredentials.password=<avi-ctrl-password> --set AKOSettings.primaryInstance=true --namespace=avi-system
    

    Secondary AKO Installation:

    helm install --generate-name oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.11.4 -f /path/to/values.yaml  --set ControllerSettings.controllerHost=<controller IP or Hostname> --set avicredentials.username=<avi-ctrl-username> --set avicredentials.password=<avi-ctrl-password> --set AKOSettings.primaryInstance=false --namespace=avi-system
    
  6. Verify the installation:

    helm list -n avi-system
    
    NAME          	NAMESPACE 	REVISION	UPDATED     STATUS  	CHART    	APP VERSION
    ako-1691752136	avi-system	1       	2024-07-02	deployed	ako-1.11.4	1.11.4 
    

Uninstall using Helm

To uninstall AKO using helm,

  1. Run the following:

    helm delete <ako-release-name> -n avi-system 
    Note:

    The ako-release-name is obtained by running the helm list as shown in the previous step.

  2. Run the following:

    kubectl delete ns avi-system 

AKO in OpenShift Cluster

AKO can be used in the in an OpenShift cluster to configure routes and services of type Loadbalancer.

Pre-requisites for Using AKO in OpenShift Cluster

  1. Configure an NSX Advanced Load Balancer Controller with a vCenter cloud and select the IPAM and DNS profiles created for the north-south apps. For more information, see Configure the Cloud.

  2. Ensure the OpenShift version is 4.4 or higher to perform a Helm-based AKO installation.

    Note:

    For OpenShift 4.x releases prior to 4.4 that do not have Helm, AKO needs to be either installed manually or Helm 3 needs to be manually deployed in the OpenShift cluster.

    Ingresses, if created in the OpenShift cluster will not be handled by AKO.

Install AKO for OpenShift

  1. Create the avi-system namespace.

    oc new-project avi-system 
  2. Search for available charts.

    helm show chart oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.11.4
    
    Pulled: projects.registry.vmware.com/ako/helm-charts/ako:1.11.4
    Digest: sha256:xyxyxxyxyx
    apiVersion: v2
    appVersion: 1.11.4
    description: A helm chart for Avi Kubernetes Operator
    name: ako
    type: application
    version: 1.11.4
  3. Edit the values.yaml file and update the details according to your environment.

    helm show values oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.11.4 > values.yaml
    
  4. Install AKO.

    helm install --generate-name oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.11.4 -f /path/to/values.yaml  --set ControllerSettings.controllerHost=<controller IP or Hostname> --set avicredentials.username=<avi-ctrl-username> --set avicredentials.password=<avi-ctrl-password> --namespace=avi-system
    
  5. Verify the installation.

    helm list -n avi-system
    
    NAME          	NAMESPACE 	REVISION	UPDATED     STATUS  	CHART    	APP VERSION
    ako-1691752136	avi-system	1       	2024-01-12	deployed	ako-1.11.4	1.11.4

Installing AKO Offline Using Helm

  1. Pre-requisites for Installation

    Ensure the following prerequisites are met:

    1. The Docker image is downloaded from the My VMware portal.

      Log in to the portal using your My VMware credentials.

    2. A private container registry to upload the AKO Docker image

    3. Helm version 3.0 or higher installed

  2. Installing AKO

    To install AKO offline using Helm,

    1. Extract the .tar file to get the AKO installation directory with the helm and docker images.

       tar -zxvf ako_cpr_sample.tar.gz
       ako/
       ako/install_docs.txt
       ako/ako-1.11.4-docker.tar.gz
       ako/ako-1.11.4-helm.tgz
    2. Change the working directory to this path: cd ako/.

    3. Load the docker image in one of your machines.

      sudo docker load < ako-1.11.4-docker.tar.gz 
    4. Push the docker image to your private registry. For more information, see Docker registry push.

    5. Extract the AKO Helm package. This will create a sub-directory ako/ako which contains the Helm charts for AKO ( ako/chart.yaml crds templates values.yaml).

    6. Update the helm values.yaml with the required AKO configuration (Controller IP/credentials, docker registry information, and so on).

    7. Create the namespace avi-system on the OpenShift/Kubernetes cluster.

        kubectl create namespace avi-system 
    8. Install AKO using the updated helm charts.

       helm install ./ako --generate-name --namespace=avi-system 

Upgrade AKO

AKO is stateless in nature. It can be rebooted/re-created without impacting the state of the existing objects in NSX Advanced Load Balancer if there is no requirement of an update to them. AKO will be upgraded using Helm.

During the upgrade process, a new docker image will be pulled from the configured repository and the AKO pod will be restarted.

On restarting, the AKO pod will re-evaluate the checksums of the existing NSX Advanced Load Balancer objects with the REST layer’s intended object checksums and do the necessary updates.

Note:

Notes for upgrading from versions lower than 1.5 to version 1.11.4. Refer to this section before upgrading from any version of AKO to AKO version 1.5 and higher:

  • Refactored VIP Network Inputs AKO version 1.5.1 deprecates subnetIP and subnetPrefix in the values.yaml (used during Helm installation), and allows specifying the information through the cidr field within vipNetworkList.

  • The existing AviInfraSetting CRDs need an explicit update after applying the updated CRD schema yaml. To update existing AviInfraSettings while performing a Helm upgrade,

    • Ensure that the AviInfraSetting CRDs have the required configuration. If not save it in the yaml files.

    • Upgrade the CRDs to provide vipNetworks in the new format. For more information, see To upgrade AKO using the Helm repository.

    • Updating the CRD schema would remove the currently invalid spec.network configuration in existing AviInfraSettings. Update the AviInfraSetting to follow the new schema as shown above and apply the changed yaml.

    • Proceed with Step 2 of upgrading AKO.

To upgrade AKO using the Helm repository,

  1. Helm does not upgrade the CRDs during a release upgrade. Before you upgrade a release, run the following command to upgrade the CRDs:

     helm template oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.11.4 --include-crds --output-dir <output_dir> 
  2. This will save the helm files to an output directory which will contain the CRDs corresponding to the AKO version. Install CRDs using:

     kubectl apply -f <output_dir>/ako/crds/ 
  3. The release is listed as shown below:

    helm list -n avi-system
    
     NAME          	NAMESPACE 	REVISION	UPDATED                             	    STATUS  	CHART    	APP VERSION
     ako-1593523840	avi-system	1       	2024-01-12 13:44:31.609195757 +0000 UTC	    deployed	ako-1.11.3	1.11.3  
    
  4. Get the values.yaml for the latest AKO version:

    helm show values oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.11.4 > values.yaml
    
  5. Upgrade the helm chart:

    helm upgrade ako-1593523840  oci://projects.registry.vmware.com/ako/helm-charts/ako -f /path/to/values.yaml --version 1.11.4 --set ControllerSettings.controllerHost=<IP or Hostname> --set avicredentials.password=<username> --set avicredentials.username=<username> --namespace=avi-system
    Note:

    In a multiple AKO deployment scenario, all AKO instances must be on the same version.

Upgrading AKO Offline Using Helm

To upgrade AKO without using the online Helm repository,

  1. Follow the steps 1 to 6 from the Installing AKO Offline Using Helm section.

  2. Upgrade CRDs using the following command:

    kubectl apply -f ./ako/crds/
  3. Use the following command:

    helm upgrade <release-name> ./ako -n avi-system

Delete AKO

  1. Edit the configmap used for AKO and set the deleteConfig flag to true if you want to delete the AKO created objects. Else skip to step 2.

     kubectl edit configmap avi-k8s-config -n avi-system  
  2. Delete AKO using the command shown below:

     helm delete $(helm list -n avi-system -q) -n avi-system 
    Note:

    Do not delete the configmap avi-k8s-config manually, unless you are doing a complete Helm uninstall. The AKO pod has to be rebooted if you delete and the avi-k8s-config configmap has to be reconfigured.

Parameters

The following table lists the configurable parameters of the AKO chart and their default values:

Parameter

Description

Default Values

ControllerSettings.controllerVersion

NSX Advanced Load Balancer Controller version

Current Controller version

ControllerSettings.controllerHost

Used to specify the NSX Advanced Load Balancer Controller IP or Hostname

None

ControllerSettings.cloudName

Name of the cloud managed in NSX Advanced Load Balancer

Default-Cloud

ControllerSettings.tenantsPerCluster

Set to true if you want to map each Kubernetes cluster uniquely to a tenant in NSX Advanced Load Balancer

False

ControllerSettings.tenantName

Name of the tenant where all the AKO objects will be created in NSX Advanced Load Balancer

Admin

L7Settings.shardVSSize

Displays shard VS size enum values: LARGE, MEDIUM, SMALL

LARGE

AKOSettings.fullSyncFrequency

Displays the full sync frequency

1800

L7Settings.defaultIngController

AKO is the default ingress controller

True

ControllerSettings.serviceEngineGroupName

Displays the name of the Service Engine Group

Default-Group

NetworkSettings.nodeNetworkList

Displays the list of networks and corresponding CIDR mappings for the Kubernetes nodes

None

AKOSettings.clusterName

Unique identifier for the running AKO instance. AKO identifies objects it created on NSX Advanced Load Balancer Controller using this parameter

Required

NetworkSettings.subnetIP

Subnet IP of the data network

Deprecated

NetworkSettings.subnetPrefix

Subnet Prefix of the data network

Deprecated

NetworkSettings.vipNetworkList

Displays the list of Network Names and Subnet information for VIP network, multiple networks allowed only for AWS Cloud

Required

NetworkSettings.enableRHI

Publish route information to BGP peers

False

NetworkSettings.bgpPeerLabels

Select BGP peers using bgpPeerLabels, for selective virtual service VIP advertisement.

Empty List

NetworkSettings.nsxtT1LR

Unique ID (Note: not its display name) of the T1 Logical Router for Service Engine connectivity. This only applies to the NSX-T cloud.

Empty String

L4Settings.defaultDomain

Used to specify a default sub-domain for L4 LB services

First domainname found in cloud's DNS profile

L4Settings.autoFQDN

Specify the layer 4 FQDN format

default

L7Settings.noPGForSNI

Skip using Pool Groups for SNI children

False

L7Settings.l7ShardingScheme

Sharding scheme enum values: hostname, namespace

hostname

AKOSettings.cniPlugin

Configure the string if your CNI is calico or openshift or ovn-kubernetes. For Cilium CNI, set the string as cilium only when using Cluster Scope mode for IPAM and leave it empty if using Kubernetes Host Scope mode for IPAM.

calico|canal|flannel|openshift|antrea|ncp|ovn-kubernetes|cilium

AKOSettings.enableEvents

enableEvents can be changed dynamically from the configmap

True

AKOSettings.logLevel

logLevel values: INFO, DEBUG, WARN, ERROR. logLevel can be changed dynamically from the configmap

INFO

AKOSettings.primaryInstance

Specify AKO instance is primary or not

True

AKOSettings.deleteConfig

Set to true if user wants to delete AKO created objects from NSX Advanced Load Balancer. The deleteConfig can be changed dynamically from the configmap

False

AKOSettings.disableStaticRouteSync

Deactivate static route syncing if set to true

False

AKOSettings.apiServerPort

Internal port for AKO's API server for the liveness probe of the AKO pod

8080

AKOSettings.blockedNamespaceList

List of Kubernetes/OpenShift namespaces blocked by AKO

Empty List

AKOSettings.istioEnabled

Set to true if user wants to deploy AKO in istio environment (tech preview)

False

AKOSettings.ipFamily

Set to V6 if user wants to deploy AKO with V6 backend (vCenter cloud with calico CNI only) (tech preview)

V4

AKOSettings.useDefaultSecretsOnly

Restricts the secret handling to default secrets present in the namespace where AKO is installed in OpenShift clusters if set to true

False

AKOSettings.layer7Only

Operate AKO as a pure layer 7 ingress Controller

False

AKOSettings.blockedNamespaceList

List of K8s/OpenShift namespaces blocked by AKO

Empty List

avicredentials.username

Enter the NSX Advanced Load Balancer Controller username

None

avicredentials.password

Enter the NSX Advanced Load Balancer Controller password

None

image.repository

Specify docker-registry that has the AKO image

avinetworks/ako

image.pullSecrets

Specify the pull secrets for the secure private container image registry that has the AKO image

Empty List

Note:
  • The field vipNetworkList is mandatory. This is used for allocating VirtualService IP by IPAM Provider module.

  • Each AKO instance mapped to a given NSX Advanced Load Balancer cloud must have a unique clusterName parameter. This would maintain the uniqueness of object naming across Kubernetes clusters.