This topic describes how to update and troubleshoot the default configuration of core add-ons in Tanzu Kubernetes Grid.

Default Core Add-On Configuration

Tanzu Kubernetes Grid automatically manages the lifecycle of its core add-ons, which includes the CNI, Metrics Server, Pinniped, vSphere CPI, and vSphere CSI add-ons. For more information, see Core Add-ons in Deploying Management Clusters.

To review the default configuration of these add-ons, you can:

  • Download the following templates from projects.registry.vmware.com/tkg/tanzu_core/addons:

    • antrea-templates
    • calico-templates
    • metrics-server-templates
    • pinniped-templates
    • vsphere-cpi-templates
    • vsphere-csi-templates
  • Examine the Kubernetes secret for your target add-on by running the kubectl get secret CLUSTER-NAME-ADD-ON-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster.

For example, to review the default configuration of the Antrea add-on:

  • Review the Antrea templates:

    1. Locate the version tag for antrea-templates in the Tanzu Kubernetes release (TKr) that you used to deploy your cluster. You can retrieve the TKr by running the kubectl get tkr command against the management cluster:

      1. Run kubectl get clusters CLUSTER-NAME -n CLUSTER-NAMESPACE --show-labels.

      2. In the output, locate the value of tanzuKubernetesRelease. For example, tanzuKubernetesRelease=v1.20.5---vmware.1-tkg.1.

      3. Run kubectl get tkr TKR-VERSION, where TKR-VERSION is the value that you retrieved above. For example:

        kubectl get tkr v1.20.5---vmware.1-tkg.1 -o yaml
        
      4. In the output, locate the version tag under tanzu_core/addons/antrea-templates.

        Alternatively, you can review the TKr in ~/tanzu/tkg/bom/YOUR-TKR-BOM-FILE.

    2. Download the templates. For example:

      imgpkg pull -i projects.registry.vmware.com/tkg/tanzu_core/addons/antrea-templates:v1.3.1 -o antrea-templates
      
    3. Navigate to antrea-templates and review the templates.

  • Retrieve and review the Antrea add-on secret. To retrieve the secret, run the following command against the management cluster:

    kubectl get secret CLUSTER-NAME-antrea-addon -n CLUSTER-NAMESPACE
    

    Where:

    • CLUSTER-NAME is the name of your target cluster. If you want to review the Antrea add-on secret for a workload cluster, CLUSTER-NAME is the name of your workload cluster.
    • CLUSTER-NAMESPACE is the namespace of your target cluster.

Updating and Troubleshooting Core Add-on Configuration

You can update and troubleshoot the default configuration of a core add-on by modifying the following resources:

Type Resources Description
Configuration updates Add-on secret

To update the default configuration of a core add-on, you can:

Troubleshooting App custom resource (CR) and add-on secret

Same as above. Additionally, if you need to apply temporary changes to your add-on configuration, you can:

  • Pause secret reconciliation.
  • Pause app CR reconciliation.

This disables lifecycle management for the add-on. Use with caution. For more information, see Pause Core Add-on Lifecycle Management below.

For more information about add-on secrets and app CRs, see Key Components and Objects below.

Updating Core Add-on Configuration

You can update the default configuration of a core add-on by modifying the values.yaml section of the add-on secret or by adding an overlay to the add-on secret. These changes are persistent.

Update the values.yaml section

In the values.yaml section, you can update the following configuration settings:

Add-on Setting Description
Antrea antrea.config.defaultMTU By default, this parameter is set to null.
Pinniped dex.config.oidc.CLIENT_ID* (v1.3.0) or pinniped.upstream_oidc_client_id (v1.3.1+) The client ID of your OIDC provider.
Pinniped dex.config.oidc.CLIENT_SECRET (v1.3.0) or pinniped.upstream_oidc_client_secret (v1.3.1+) The client secret of your OIDC provider.
Pinniped dex.config.oidc.issuer (v1.3.0) or pinniped.upstream_oidc_issuer_url (v1.3.1+) The URL of your OIDC provider.
Pinniped dex.config.oidc.scopes (v1.3.0) or pinniped.upstream_oidc_additional_scopes (v1.3.1+) A list of additional scopes to request in the token response.
Pinniped dex.config.oidc.claimMapping (v1.3.0) or pinniped.upstream_oidc_claims (v1.3.1+) OIDC claim mapping.
Pinniped dex.config.ldap.host The IP or DNS address of your LDAP server. If you want to change the default port 636 to a different port, specify "host:port".
Pinniped dex.config.ldap.bindDN and dex.config.ldap.bindPW The DN and password for an application service account.
Pinniped dex.config.ldap.userSearch Search attributes for users.
Pinniped dex.config.ldap.groupSearch Search attributes for groups.
vSphere CSI vsphereCSI.provisionTimeout By default, this parameter is set to 300s.
vSphere CSI vsphereCSI.attachTimeout By default, this parameter is set to 300s.

* If you want to update a Pinniped setting that starts with dex., you must restart dex in the management cluster after you update the add-on secret.

To modify the values.yaml section of an add-on secret:

  1. Retrieve the add-on secret by running the kubectl get secret CLUSTER-NAME-ADD-ON-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. For example:

    kubectl get secret example-mgmt-cluster-antrea-addon -n tkg-system -o jsonpath={.data.values\\.yaml} | base64 -d > values.yaml
    
  2. Update the values.yaml section. You can update any of the values listed in the table above.

  3. Apply your update by running the kubectl apply command. Alternatively, you can use the kubectl edit command to update the add-on secret.

  4. After updating the secret, check the status of the add-on by running the kubectl get app command. For example:

    $ kubectl get app antrea -n tkg-system
    NAME           DESCRIPTION             SINCE-DEPLOY    AGE
    antrea         Reconcile succeeded     3m23s           7h50m
    

    If the returned status is Reconcile failed, run the following command to get details on the failure:

    kubectl get app antrea -n tkg-system -o yaml
    

The example below updates the default MTU for the Antrea add-on.

...
stringData:
  values.yaml: |
    #@data/values
    #@overlay/match-child-defaults missing_ok=True
    ---
    infraProvider: vsphere
    antrea:
      config:
        defaultMTU: 8900

Add an Overlay

If you want to update a configuration setting that is not supported by the default add-on templates, you can add an overlay to the add-on secret. The example below instructs Pinniped to use LoadBalancer instead of the default NodePort on vSphere:

...
stringData:
 overlays.yaml: |
   #@ load("@ytt:overlay", "overlay")
   #@overlay/match by=overlay.subset({"kind": "Service", "metadata": {"name": "pinniped-supervisor", "namespace": "pinniped-supervisor"}})
   ---
   #@overlay/replace
   spec:
     type: LoadBalancer
     selector:
       app: pinniped-supervisor
     ports:
       - name: https
         protocol: TCP
         port: 443
         targetPort: 8443
 values.yaml: |
   #@data/values
   #@overlay/match-child-defaults missing_ok=True
   ---
   infrastructure_provider: vsphere
   tkg_cluster_role: management

To add an overlay to an add-on secret:

  1. Retrieve the add-on secret by running the kubectl get secret CLUSTER-NAME-ADD-ON-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. For example:

    kubectl get secret example-mgmt-cluster-pinniped-addon -n tkg-system -o jsonpath={.data.values\\.yaml} | base64 -d > values.yaml
    
  2. Add your overlay.yaml section under stringData.

  3. Apply the update by running the kubectl apply command. Alternatively, you can use the kubectl edit command to update the add-on secret.

  4. After updating the secret, check the status of the add-on by running the kubectl get app command. For example:

    $ kubectl get app pinniped -n tkg-system
    NAME           DESCRIPTION             SINCE-DEPLOY    AGE
    pinniped       Reconcile succeeded     3m23s           7h50m
    

    If the returned status is Reconcile failed, run the following command to get details on the failure:

    kubectl get app pinniped -n tkg-system -o yaml
    

Troubleshooting Core Add-on Configuration

Before troubleshooting the core add-ons, review the following sections:

Key Components and Objects

Tanzu Kubernetes Grid uses the following components and objects for core add-on management.

Components in the management cluster:

  • kapp-controller, a local package manager: When you deploy a management cluster, the Tanzu CLI installs kapp-controller in the cluster. kapp-controller deploys tanzu-addons-manager and the core add-ons. It also deploys and manages kapp-controller in each Tanzu Kubernetes (workload) cluster that you deploy from that management cluster.
  • tanzu-addons-manager: Manages the lifecycle of the core add-ons in the management cluster and workload clusters that you deploy from your management cluster.
  • tkr-controller: Creates Tanzu Kubernetes releases (TKr) and BoM ConfigMaps in the management cluster.

Component in workload clusters:

kapp-controller deploys the core add-ons in the workload cluster in which it runs.

Objects:

  • Secret: The Tanzu CLI creates a secret for each core add-on, per cluster. These secrets define the configuration of the core add-ons. All add-on secrets are created in the management cluster. tanzu-addons-manager reads the secrets and uses the configuration information they contain to create app CRs.
  • App CR: For each add-on, tanzu-addons-manager creates an app CR in the target cluster. Then, kapp-controller reconciles the CR and deploys the add-on.
  • BoM ConfigMap: Provides metadata information about the core add-ons, such as image location, to tanzu-addons-manager.

You can use the following commands to monitor the status of these components and objects:

Command Description
kubectl get app ADD-ON -n tkg-system -o yaml Check the app CR in your target cluster. For example, kubectl get app antrea -n tkg-system -o yaml.
kubectl get cluster CLUSTER-NAME -n CLUSTER-NAMESPACE -o jsonpath={.metadata.labels.tanzuKubernetesRelease} In the management cluster, check if the TKr label of your target cluster points to the correct TKr.
kubectl get tanzukubernetesrelease TKR-NAME Check if the TKr is present in the management cluster.
kubectl get configmaps -n tkr-system -l 'tanzuKubernetesRelease=TKR-NAME' Check if the BoM ConfigMap corresponding to your TKr is present in the management cluster.
kubectl get app CLUSTER-NAME-kapp-controller -n CLUSTER-NAMESPACE For workload clusters, check if the kapp-controller app CR is present in the management cluster.
kubectl logs deployment/tanzu-addons-controller-manager -n tkg-system Check tanzu-addons-manager logs in the management cluster.
kubectl get configmap -n tkg-system | grep ADD-ON-ctrl Check if your updates to the add-on secret have been applied. The sync period is 5 minutes.

Pause Core Add-on Lifecycle Management

IMPORTANT: The commands in this section disable add-on lifecycle management. Whenever possible, use the procedures described in Updating Add-on Configuration above instead.

If you need to temporary pause add-on lifecycle management for a core add-on, you can use the commands below:

  • To pause secret reconciliation, run the following command against the management cluster:

    kubectl patch secret/CLUSTER-NAME-ADD-ON-NAME-addon -n CLUSTER-NAMESPACE -p '{"metadata":{"annotations":{"tkg.tanzu.vmware.com/addon-paused": ""}}}' --type=merge
    

    After you run this command, tanzu-addons-manager stops reconciling the secret.

  • To pause app CR reconciliation, run the following command against your target cluster:

    kubectl patch app/ADD-ON-NAME -n tkg-system -p '{"spec":{"paused":true}}' --type=merge
    

    After you run this command, kapp-controller stops reconciling the app CR.

If you want to temporary modify a core add-on app, pause secret reconciliation first and then pause app CR reconciliation. After you unpause add-on lifecycle management, tanzu-addons-manager and kapp-controller resume secret and app CR reconciliation:

  • To unpause secret reconciliation, remove tkg.tanzu.vmware.com/addon-paused from the secret annotations.

  • To unpause app CR reconciliation, update the app CR with {"spec":{"paused":false}} or remove the variable.

check-circle-line exclamation-circle-line close-line
Scroll to top icon