This topic explains how to retrieve the configuration files for a core package. It also lists the Antrea, Pinniped, vSphere CPI, and vSphere CSI package configuration settings that you can update after cluster creation. For troubleshooting information, see Updating and Troubleshooting Core Package Configuration.

Viewing the Configuration of a Core Package

Tanzu Kubernetes Grid automatically installs its core packages during cluster creation. Internally, these packages are called add-ons and their lifecycle is managed by tanzu-addons-manager, which is also installed as a core package. For more information, see Install and Configure Packages and Core Packages.

To view the configuration of a core package, or core add-on, you can:

  • Retrieve the Kubernetes secret for the add-on by running the kubectl get secret CLUSTER-NAME-PACKAGE-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. Where:

    • CLUSTER-NAME is the name of your target cluster. If you want to retrieve the secret for a core add-on that is installed in a workload cluster, CLUSTER-NAME is the name of your workload cluster.
    • PACKAGE-NAME is the name of the core package.
    • CLUSTER-NAMESPACE is the namespace of your target cluster.
  • Download the package configuration files from projects.registry.vmware.com/tkg/packages/core.

For example, to view the configuration of Antrea:

  • Retrieve the Antrea secret by running the following command against the management cluster:

    kubectl get secret CLUSTER-NAME-antrea-addon -n CLUSTER-NAMESPACE
    
  • Download the Antrea package configuration files:

    1. Locate the version tag for antrea in the Tanzu Kubernetes release (TKr) that you used to create the cluster. You can retrieve the TKr by running the kubectl get tkr command against the management cluster:

      1. Run kubectl get clusters CLUSTER-NAME -n CLUSTER-NAMESPACE --show-labels.

      2. In the output, locate the value of tanzuKubernetesRelease. For example, tanzuKubernetesRelease=v1.21.2---vmware.1-tkg.1.

      3. Run kubectl get tkr TKR-VERSION, where TKR-VERSION is the value that you retrieved above. For example:

        kubectl get tkr v1.21.2---vmware.1-tkg.1 -o yaml
        
      4. In the output, locate the version tag under packages/core/antrea.

        Alternatively, you can locate the version tag by reviewing the TKr in ~/.config/tanzu/tkg/bom/YOUR-TKR-BOM-FILE.

    2. Download the package configuration files. For example:

      imgpkg pull -b projects.registry.vmware.com/tkg/packages/core/antrea:v0.13.3_vmware.1-tkg.1 -o antrea
      
    3. Navigate to antrea and review the files.

Updating and Troubleshooting Core Package Configuration

You can update and troubleshoot the configuration of a core package by modifying the following resources:

Type Resources Description
Configuration updates Add-on Secret

To update the default configuration of a core package, you can:

Troubleshooting PackageInstall custom resource (CR) and add-on Secret

Same as above. Additionally, if you need to apply temporary changes to your package configuration, you can:

  • Pause Secret reconciliation.
  • Pause PackageInstall CR reconciliation.

This disables lifecycle management for the package. Use with caution. For more information, see Pause Lifecycle Management for a Core Package below.

For more information about add-on secrets and PackageInstall CRs, see Key Components and Objects below.

Updating Core Package Configuration

You can update the default configuration of a core package by modifying the values.yaml section of the add-on secret or by adding an overlay to the add-on secret. These changes are persistent.

Update the values.yaml Section

In the values.yaml section, you can update the following configuration settings:

Package Setting Description
Antrea antrea.config.defaultMTU By default, this parameter is set to null.
Pinniped pinniped.upstream_oidc_client_id The client ID of your OIDC provider.
Pinniped pinniped.upstream_oidc_client_secret The client secret of your OIDC provider.
Pinniped pinniped.upstream_oidc_issuer_url The URL of your OIDC provider.
Pinniped pinniped.upstream_oidc_tls_ca_data The base64-encoded CA bundle data used to verify TLS connections to your OIDC provider.
Pinniped pinniped.upstream_oidc_additional_scopes A list of additional scopes to request in the token response.
Pinniped pinniped.upstream_oidc_claims OIDC claim mapping.
Pinniped dex.config.ldap.host* The IP or DNS address of your LDAP server. If you want to change the default port 636 to a different port, specify "host:port".
Pinniped dex.config.ldap.bindDN* and dex.config.ldap.BIND_PW_ENV_VAR* The DN and password for an application service account.
Pinniped dex.config.ldap.userSearch* Search attributes for users. For schema, see the Dex documentation.
Pinniped dex.config.ldap.groupSearch* Search attributes for groups. For schema, see the Dex documentation.
vSphere CSI vsphereCSI.provisionTimeout By default, this parameter is set to 300s.
vSphere CSI vsphereCSI.attachTimeout By default, this parameter is set to 300s.

* If you want to update a Pinniped setting that starts with dex., see Updating Dex Settings After Management Cluster Deployment.

To modify the values.yaml section of an add-on secret:

  1. Retrieve the add-on secret by running the kubectl get secret CLUSTER-NAME-PACKAGE-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. For example:

    kubectl get secret example-mgmt-cluster-antrea-addon -n tkg-system -o jsonpath={.data.values\\.yaml} | base64 -d > values.yaml
    
  2. Update the values.yaml section. You can update any of the values listed in the table above.

  3. Apply your update by running the kubectl apply command. Alternatively, you can use the kubectl edit command to update the add-on secret.

  4. After updating the secret, check the status of the package by running the kubectl get packageinstall command. For example:

    $ kubectl get packageinstall antrea -n tkg-system
    NAMESPACE    NAME                   PACKAGE NAME                      PACKAGE VERSION                 DESCRIPTION                  AGE
    tkg-system   antrea                 antrea.tanzu.vmware.com           0.13.3+vmware.1-tkg.1           Reconcile succeeded          7d14h
    

    If the returned status is Reconcile failed, run the following command to get details on the failure:

    kubectl get packageinstall antrea -n tkg-system -o yaml
    
  5. To see more details, you can also inspect the add-on app resources by running the kubectl get app command. For example:

    $ kubectl get app antrea -n tkg-system
    NAME           DESCRIPTION             SINCE-DEPLOY    AGE
    antrea         Reconcile succeeded     3m23s           7h50m
    

    If the returned status is Reconcile failed, run the following command to get details on the failure:

    kubectl get app antrea -n tkg-system -o yaml
    

The example below updates the default MTU for Antrea.

...
stringData:
  values.yaml: |
    #@data/values
    #@overlay/match-child-defaults missing_ok=True
    ---
    infraProvider: vsphere
    antrea:
      config:
        defaultMTU: 8900

Add an Overlay

In some cases, you can add an overlay to the add-on secret. This enables you to customize the default configuration that is defined in the package configuration files. The example below instructs Pinniped to use LoadBalancer instead of the default NodePort on vSphere:

...
stringData:
 overlays.yaml: |
   #@ load("@ytt:overlay", "overlay")
   #@overlay/match by=overlay.subset({"kind": "Service", "metadata": {"name": "pinniped-supervisor", "namespace": "pinniped-supervisor"}})
   ---
   #@overlay/replace
   spec:
     type: LoadBalancer
     selector:
       app: pinniped-supervisor
     ports:
       - name: https
         protocol: TCP
         port: 443
         targetPort: 8443
 values.yaml: |
   #@data/values
   #@overlay/match-child-defaults missing_ok=True
   ---
   infrastructure_provider: vsphere
   tkg_cluster_role: management

To add an overlay to an add-on secret:

  1. Retrieve the add-on secret by running the kubectl get secret CLUSTER-NAME-PACKAGE-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. For example:

    kubectl get secret example-mgmt-cluster-pinniped-addon -n tkg-system -o jsonpath={.data.values\\.yaml} | base64 -d > values.yaml
    
  2. Add your overlays.yaml section under stringData.

  3. Apply the update by running the kubectl apply command. Alternatively, you can use the kubectl edit command to update the add-on secret.

  4. After updating the secret, check the status of the package by running the kubectl get packageinstall and kubectl get app commands as described in Update the values.yaml Section.

Troubleshooting Core Package Configuration

Before troubleshooting the core packages that are running in your target cluster, review the following sections:

Key Components and Objects

Tanzu Kubernetes Grid uses the following components and objects for core package management.

Components in the management cluster:

  • kapp-controller, a local package manager: When you deploy a management cluster, the Tanzu CLI installs kapp-controller in the cluster. kapp-controller installs tanzu-addons-manager and other core packages. It also installs and manages kapp-controller in each workload cluster that you deploy from that management cluster.
  • tanzu-addons-manager: Manages the lifecycle of the core packages, or core add-ons, in the management cluster and workload clusters that you deploy from your management cluster.
  • tkr-controller: Creates TKrs and BoM ConfigMaps in the management cluster.

Component in workload clusters:

kapp-controller installs core packages in the workload cluster in which it runs.

Objects:

  • Secret: The Tanzu CLI creates a Secret for each core add-on, per cluster. These secrets define the configuration of the core add-ons. tanzu-addons-manager reads the secrets and uses the configuration information that they contain to configure the add-ons. All add-on secrets are created in the management cluster.
  • PackageRepository CR: The tanzu-addons-manager creates a PackageRepository CR that references all core add-on Package CRs (see below).
  • Package CR: For each core add-on in the PackageRepository, kapp-controller creates a Package CR in the target cluster.
  • PackageInstall CR: For each core add-on Package, tanzu-addons-manager creates a PackageInstall CR in the target cluster to inform kapp-controller about which core packages it needs to install.
  • App CR: For each PackageInstall, kapp-controller creates an App CR in the target cluster. Then, kapp-controller reconciles the CR and installs the package.
  • BoM ConfigMap: Provides metadata information about the core add-ons, such as image location, to tanzu-addons-manager.

You can use the following commands to monitor the status of these components and objects:

Command Description
kubectl get packageinstall CORE-PACKAGE-NAME -n tkg-system -o yaml Check the PackageInstall CR in your target cluster. For example, kubectl get packageinstall antrea -n tkg-system -o yaml.
kubectl get app CORE-ADD-ON-NAME -n tkg-system -o yaml Check the App CR in your target cluster. For example, kubectl get app antrea -n tkg-system -o yaml.
kubectl get cluster CLUSTER-NAME -n CLUSTER-NAMESPACE -o jsonpath={.metadata.labels.tanzuKubernetesRelease} In the management cluster, check if the TKr label of your target cluster points to the correct TKr.
kubectl get tanzukubernetesrelease TKR-NAME Check if the TKr is present in the management cluster.
kubectl get configmaps -n tkr-system -l 'tanzuKubernetesRelease=TKR-NAME' Check if the BoM ConfigMap corresponding to your TKr is present in the management cluster.
kubectl get app CLUSTER-NAME-kapp-controller -n CLUSTER-NAMESPACE For workload clusters, check if the kapp-controller App CR is present in the management cluster.
kubectl logs deployment/tanzu-addons-controller-manager -n tkg-system Check tanzu-addons-manager logs in the management cluster.
kubectl get configmap -n tkg-system | grep CORE-ADD-ON-NAME-ctrl Check if your updates to the add-on secret have been applied. The sync period is 5 minutes.

Pause Lifecycle Management for a Core Package

IMPORTANT: The commands in this section disable core package lifecycle management. Whenever possible, use the procedures described in Updating Core Package Configuration above instead.

If you need to temporary pause lifecycle management for a core package, you can use the commands below:

  • To pause secret reconciliation, run the following command against the management cluster:

    kubectl patch secret/CLUSTER-NAME-CORE-ADD-ON-NAME-addon -n CLUSTER-NAMESPACE -p '{"metadata":{"annotations":{"tkg.tanzu.vmware.com/addon-paused": ""}}}' --type=merge
    

    After you run this command, tanzu-addons-manager stops reconciling the secret.

  • To pause PackageInstall CR reconciliation, run the following command against your target cluster:

    kubectl patch packageinstall/CORE-PACKAGE-NAME -n tkg-system -p '{"spec":{"paused":true}}' --type=merge
    

    After you run this command, kapp-controller stops reconciling the PackageInstall and corresponding App CR.

If you want to temporary modify the resources of a core add-on, pause secret reconciliation first and then pause PackageInstall CR reconciliation. After you unpause lifecycle management, tanzu-addons-manager and kapp-controller resume secret and PackageInstall CR reconciliation:

  • To unpause secret reconciliation, remove tkg.tanzu.vmware.com/addon-paused from the secret annotations.

  • To unpause PackageInstall CR reconciliation, update the PackageInstall CR with {"spec":{"paused":false}} or remove the variable.

check-circle-line exclamation-circle-line close-line
Scroll to top icon