This section helps you understand the architectural overview and design considerations for AKO deployment.
The AKO translates Kubernetes/ OpenShift objects to Controller APIs.
Architecture
The NSX Advanced Load Balancer Deployment in Kubernetes/ OpenShift for AKO comprises of the following main components:
The NSX Advanced Load Balancer Controller
The Service Engines (SE)
The NSX Advanced Load Balancer Kubernetes Operator (AKO)
The NSX Advanced Load Balancer Controller
The NSX Advanced Load Balancer Controller which is the central component of the NSX Advanced Load Balancer architecture is responsible for the following:
Control plane functionality like the:
Infrastructure orchestration
Centralized management
Analytics dashboard
Integration with the underlying ecosystem for managing the lifecycle of the data plane (Service Engines).
The NSX Advanced Load Balancer Controller does not handle any data plane traffic.
In Kubernetes/ OpenShift environments, the NSX Advanced Load Balancer Controller is deployed outside the Kubernetes/ OpenShift cluster, typically in the native type of the underlying infrastructure. However, it can be deployed anywhere if connectivity and latency requirements are satisfied.
The NSX Advanced Load Balancer Service Engines
The SEs implement data plane services of load balancing. For example, Web Application Firewall, DNS/GSLB, and so on.
In Kubernetes/ OpenShift environments, the SEs are deployed external to the cluster and typically in the native type of the underlying infrastructure.
The Avi Kubernetes Operator (AKO)
AKO is an NSX Advanced Load Balancer pod running in Kubernetes that provides an Ingress controller and NSX Advanced Load Balancer-configuration functionality. AKO remains in sync with the required Kubernetes/ OpenShift objects and calls the NSX Advanced Load Balancer Controller APIs to deploy the Ingresses and Services through the NSX Advanced Load Balancer Service Engines. AKO is deployed as a pod through Helm.
NSX Advanced Load Balancer Cloud Type
The NSX Advanced Load Balancer Controller uses the NSX Advanced Load Balancer Cloud configuration to manage the SEs. This Cloud is usually of the underlying infrastructure type, for example, VMware vCenter Cloud, Azure Cloud, Linux Server Cloud, and so on.
This deployment in Kubernetes/ OpenShift does not use the Kubernetes/ OpenShift cloud types. The integration with Kubernetes/ OpenShift and application-automation functions are handled by AKO and not by the NSX Advanced Load Balancer Controller.
Multiple Kubernetes/ OpenShift Clusters
A single NSX Advanced Load Balancer Cloud can be used for integration with multiple Kubernetes/ OpenShift clusters, with each cluster running its own instance of AKO. Clusters in the clusterIP mode are separated on the data plane through unique SE groups per cluster.
IPAM and DNS
The IPAM and DNS functionality is handled by the NSX Advanced Load Balancer Controller through the NSX Advanced Load Balancer cloud configuration.
For more information on supported IPAM and DNS types per environment, see Service Discovery Using IPAM and DNS topic in the VMware NSX Advanced Load BalancerConfiguration Guide.
Service Engine Groups
AKO supports a separate SE group per Kubernetes/OpenShift cluster. Each cluster will need to be configured with a separate SE group. However, multiple SE groups within the same cluster is not supported. As a best practice, it is recommended to use non-default SE groups for every cluster. SE group per cluster is not a requirement if AKO runs in the nodeport mode.
NSX Advanced Load Balancer SE Placement / Pod Network Reachability
With AKO, the service engines are deployed outside the cluster. To be able to load balance requests directly to the pods, the pod CIDR must be routable from the SE. Depending on the routability of the Pod CNI used in the cluster, AKO can route using the following options:
- NSX Advanced Load Balancer SE Placement/ Pod Network Reachability
-
For CNIs like Canal, Calico, Antrea, Flannel and so on, the pod subnet is not externally routable. In these cases, the CNI assigns a pod CIDR to each node in the Kuberntes cluster. The pods on a node get IP assigned from the CIDR allocated for that node and is routable from within the node. In this scenario, the pod reachability depends on where the SE is placed.
If SE is placed on the same network as the Kubernetes/ OpenShift nodes, you can turn on static route programming in AKO. With this, AKO syncs the pod CIDR for each Kubernetes/ OpenShift node and programs static route on the NSX Advanced Load Balancer Controller for each Pod CIDR with the Kubernetes/ OpenShift node IP as the next hop. Static routing per cluster uses a new label-based routing scheme. No additional user configuration is required for this label-based scheme, however the upgrading AKO will be service impacting, requiring an AKO restart.
- Pods are not externally routable – NodePort
-
In cases where direct load-balancing to the pods is not possible, NodePort based services can be used as the pool members in the NSX Advanced Load Balancer virtual service as end points. For this functionality, configure the services referenced by Ingresses/Routes as type NodePort and set the
configs.serviceType
parameter to enable NodePort based Routes/Ingresses. ThenodeSelectorLabels.key
andnodeSelectorLabels.value
parameters are specified during the AKO installation to select the required Nodes from the cluster for load balancing. The required nodes in the cluster need to be labelled with the configured key and value pair. - Pod Subnet is Routable
-
For CNIs like NSX-T CNI, AWS CNI (in EKS), Azure CNI (in AKS) and so on, the pod subnet is externally routable. In this case no additional configuration is required to allow SEs to reach the Pod IPs. Set Static Route Programming to Off in the AKO configuration. SEs can be placed on any network and will be able to route the pods.
To know more about the CNIs supported, see Compatibility Matrix for AKO.
Deployment Modes
- Single Arm Deployment
-
The deployment in which the virtual IP (VIP) address and the Kubernetes/ OpenShift cluster are in the same network subnet is called a Single Arm Deployment.
- Two-Arm Deployment
-
When the virtual IP (VIP) address and the Kubernetes/ OpenShift cluster are in different network subnets, then the deployment is a Two-Arm deployment.
AKO supports both Single-Arm and Two-Arm deployments with vCenter Cloud in write-access mode.
Annotations
AKO does not support annotations. However, the HTTPRule and HostRule CRDs can be leveraged to customize the NSX Advanced Load Balancer configuration. For more information on what parameters can be tweaked, see Setting up Routing Rules using CRDs.
Multi-Tenancy
AKO supports multi-tenancy. For more information, see Tenancy in AKO.
AKO Support
For information on supportability of features and environments with AKO, see Compatibility Matrix for AKO.