Use the following reference while configuring Add-Ons on your workload cluster.

antrea-tca-addon

Option Description

Network Permissions

This parameter is exclusive to file volumes backed by vSAN file shares and is optional. The set of parameters restricts the network capabilities of all file share volumes that are created for the cluster. If you do not specify the complete set of NetPermissions for a given IP range or completely omit the section, the system uses default values. You can add multiple NetPermission rules, and each section can include the following strings:

  • Permission Name: Defines the name of the NetPermssion rule item.

  • IP Set/Subnet: Defines the IP range or IP subnet to which these restrictions apply. The default value for Ips is *, which means all IPs.

  • Permissions: Defines the permissions level, such as READ_WRITE, READ_ONLY or NO_ACCESS. The default value for Permissions is READ_WRITE for the specified IP range.

  • RootSquash: Defines the security access level for the file share volume. The default for RootSquash is false. It allows root access to all file share volumes that are created within the specified IP range.

Note:

  • NetPermissions only take effect when the file service in the vSAN cluster is enabled to create file share volumes, configure the required file service domains, IP pools, network.

  • Do not use "NO_ACCESS" permissions for IPs "*" or the subnets of the node IPs in the Kubernetes cluster. Otherwise, the volume created with this network permissions cannot be used with the pod.

  • The netPermission rules are honored from top to bottom. Top rules override bottom ones. You can put more general rules below the specific rule items. For example, can use "*" to denote "any other IP addresses not mentioned in above rule items".

  • Only TKG-2.5 (ManagementCluster kubernetes version is 1.28.x) supports the netPermission configuration).
No SNAT Click the No SNAT toggle button to activate Source Network Address Translation (SNAT).

This option is for noEncap traffic mode only. By default, this option is not activated. In the noEncap mode, if the cluster’s POD CIDR is reachable from the external network, then this option can be deactivated.

In the networkPolicyOnly mode, antrea-tca-addon ignores SNAT. However, for the other modes, this option must be activated.

Traffic Encap Mode Determines how traffic is encapsulated. Select one of the following options:
  • encap: Internal Pod traffic is encapsulated and for external Pod network traffic, SNAT is performed.
  • noEncap: Internal Pod traffic is not encapsulated and for external Pod network traffic, SNAT is either performed or not based on the No SNAT setting in the preceding field.
    Note: Underlying network must be capable of supporting Pod traffic across IP subnets.

vsphere-csi

Option

Description

Zone

Zone is the tag category name defined in vCenter Server. Tags belonging to this category are assigned to the host or vSphere cluster objects for marking the storage topology.

Region

Region is the tag category name defined in vCenter Server. Tags belonging to this category are assigned to the Data Center objects for marking the storage topology.

VC Username

Enter a user name for vSphere-CSI.

VC Password

Enter a password for vSphere-CSI.

Storage Class

Enter the storage class name. This storage class is used to provision persistent volumes dynamically. A storage class with this name is created in the Kubernetes cluster.

Is Default

Select True if you want to set the storage class as a default one. Else, select False.

Note: Only one storage class can be set to True between vsphere-csi and nfs-client.

Reclaim Policy

Select whether to delete or retain the add-on during a reclaim event.

Datastore URL

Enter the datastore URL.

Use Storage Policy

Select the required storage policy.

Add New StorageClass

Click this button to add one or more storage classes.

Note: You can add multiple storage classes. However, you can set only one storage class as default between vsphere-csi and nfs-client.

nfs-client

Option

Description

Storage Class

Enter the storage class name. This storage class is used to provision persistent volumes dynamically. A storage class with this name is created in the Kubernetes cluster.

Is Default

To set this storage class as default, select True.

NFS Server Address

For an IPv4 cluster, enter the IPv4 address or FQDN of the NFS Server. For an IPv6 cluster, enter the FQDN.

Path

Enter server IP address and mount path of the NFS client. Ensure that the NFS server is reachable from the cluster. The mount path must also be accessible to read and write.

Add New StorageClass

Click this button to add one or more storage classes.

Note: You can add multiple storage classes. However, you can set only one storage class as default between vsphere-csi and nfs-client.

harbor

If a Harbor has already been registered, click Select Registered Harbor and select the appropriate Harbor from the list. Otherwise, click Add New Harbor and provide the following details:

Option

Description

URL

Enter the Harbor URL.

Username

Enter the Harbor user name.

Password

Enter the Harbor password.

multus

Caution:

Do NOT delete multus add-on once it is provisioned, as this might prevent creating or deleting pods on the workload cluster. See multus-cni known issue #461.

Option

Description

Log Level

Enter the log level. Select from:

  • Panic

  • Debug

  • Error

  • Verbose

Log File Path

Path where you want to store the log files.

systemsettings

Option

Description

Cluster Password

Enter the password for the cluster.

Syslog

Add the syslog server IP address/FQDN for capturing the infrastructure logs of all the nodes in the cluster.

load-balancer-and-ingress-service(aka AKO)

Load-balancer-and-ingress-service add-on also known as AKO(AVI Kubernetes Operator) add-on.

Note:

  1. To install load-balancer-and-ingress-service(AKO) add-on for a Workload cluster, you must add AKOO(AVI Kubernetes Operator - Operator) on the Management cluster. For information about adding AKOO, see Add AVI Kubernetes Operator - Operator.

  2. Service engine group can not be shared by more than one TCA clusters, even if load-balancer-and-ingress-service(AKO) add-on is deleted from the original cluster or the original cluster is deleted already. To use a service engine group which was used by other cluster, delete the service engine group from Avi Controller UI and recreate it.

  3. To customize additional load-balancer-and-ingress-service(AKO) configurable fields and manage AKO objects(aviinfrasetting, gatewayclass, gateway) via the Custom Resources(CRs) tab, see Advanced configuration for load-balancer-and-ingress-service add-on.

Option

Description

Cloud Name

Enter the cloud name configured in the AVI Controller.

Default Service Engine Group

Enter the service engine group name configured in the AVI Controller.

Default VIP Network

Enter the VIP network name in the AVI Controller.

Default VIP Network CIDR

Enter the VIP network CIDR in the AVI Controller.

Ingress Configuration for AKO Deployment

Service Type

Enter the ingress method for the service. Choose from the following options:

  • Node Port

  • Cluster IP

  • Node Port Local - Available only for Antrea CNI.

Network Name

Enter the cluster node network name. To add a network, click Add Network.

CIDRs

You can enter multiple comma-separated CIDR values or use the <CR> tag to enter multiple CIDR values.

Promethues

Prometheus provides Kubernetes-native deployment and management of Prometheus and related monitoring components.

Note:

  1. To customize additional prometheus configurable fields via the Custom Resources(CRs) tab, see Advanced configuration for prometheus add-on.

  2. Some parameters(e.g. PVC parameters, service type, port) are immutable after prometheus add-on provisioned. See Configurable parameters.

Option

Description

Use Reference Configs

Click the toggle button to use the reference configurations.

Storage Class Name

The name of the Storage Class. Default Storage Class will be used if not set.

Access Mode

Choose from:

  • Read Write Once

  • Read Only Many

  • Read Write Many

Storage

Enter the size of the Persistent Volume Claim (PVC). The default value is 150 GB.

fluent-Bit

Note:

  1. Do not set cpu-manager-policy is to static for node pools as this may lead to crashing of fluent-bit deamonset pods.

  2. To customize additional fluent-bit configurable fields(inputs, outputs, filters, parsers) via the Custom Resources(CRs) tab, see Advanced configuration for fluent-bit addon.

  3. To update the provisioned fluent-bit configuration, manually restart all fluent-bit pods to make the new configuration take effect.

Option

Description

Use Reference Configs

Click the toggle button to use the reference configurations.

service

Service configuration for fluent-bit. Default value is:

[Service]
  Flush         5
  Log_Level     info
  Daemon        off
  Parsers_File  parsers.conf
  HTTP_Server   On
  HTTP_Listen   0.0.0.0
  HTTP_Port     2020

Outputs

You must enter the syslog server IP address.

whereabouts

This add-on has no configuration.

cert-manager

This add-on has no configuration.

Note:

In certain scenarios, the cainjector pod or webhook pod of cert-manager add-on can be in CrashLoopBackOff status while the cert-manager add-on status on UI will be Unhealthy. In such case, restart the CrashLoopBackOff pod with command kubectl delete pod -n cert-manager <crash-pod-name> to recover.

velero

Velero is used to back up and restore a workload cluster.

Note:

After changing the "Backup Storage" configuration (such as, Storage URL and Storage Bucketname), existing ResticRepositories CR should be deleted manually in order to continue using Restic to back up Persistent Volumes data.

kubectl delete ResticRepositories <resticrepository-name> -n velero

Option

Description

Credential

Access ID

Enter an ID to access backup storage.

Access Key

Enter password to access backup storage.

Backup Storage

Storage URL

Enter URL of the S3-compatible object storage service.

Region

Enter location of the bucket created in the S3-Compatible object storage server.

Note:

For example, enter minio if you are using the MinIO service.

Storage Bucket Name

Enter name of the storage bucket where the backup should be restored.

Note:

It is recommended to use a dedicated bucket for each TKG workload cluster.

CA certificate

Paste the CA certificate in PEM format.

Note:

  • This field appears only if the storage URL is in HTTPS format.

  • Also append https-proxy certificate if velero is behind https-proxy.

TKG standard extension

This addon is used to manage the TKG standard extensions, such as tkg-contour and tkg-harbor.

Note:

  • You must install cert-manager before installing any of the TKG standard extensions.

  • The following TKG standard extensions which are supported by the VMware Telco Cloud Automation addons cannot be installed through TKG standard extension: cert-manager, multus-cni, whereabouts, fluent-bit, promethesus.

  • For TKG standard extension configurations and other information, see Installing and Managing Packages with the Tanzu CLI.

Option

Description

Addon Name

Enter the addon name to be installed through TKG standard extension.

Note:

The addon name should be prefixed with tkg.