View and Update Auto-Managed Package Configuration

This topic explains how to view the configuration of an auto-managed package installed from the tanzu-core repository. It also lists the Antrea, Pinniped, Calico, vSphere CPI, and vSphere CSI configuration settings that you can update after cluster creation. For troubleshooting information, see Updating and Troubleshooting Package Configuration below.

Viewing Package Configuration

Auto-managed packages, which are located in the tanzu-core repository, contain components that support basic cluster functionality, such as the Antrea or Calico container network interface and the Pinniped authentication component. In some cases, internal Tanzu Kubernetes Grid and Kubernetes resources refer to these components as addons.

To view the configuration of an auto-managed package and the add-on component that it contains, you can:

  • Retrieve the Kubernetes secret for the installed add-on component by running the kubectl get secret CLUSTER-NAME-PACKAGE-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. Where:

    • CLUSTER-NAME is the name of your target cluster. If you want to retrieve the secret for the add-on component that is installed in a workload cluster, CLUSTER-NAME is the name of your workload cluster.
    • PACKAGE-NAME is the name of the package.
    • CLUSTER-NAMESPACE is the namespace of your target cluster.

  • Download the package configuration files from projects.registry.vmware.com/tkg/packages/core.

For example, to view the configuration of Antrea:

  • Retrieve the Antrea secret by running the following command against the management cluster:

    kubectl get secret CLUSTER-NAME-antrea-addon -n CLUSTER-NAMESPACE

  • Download the Antrea package configuration files:

    1. Locate the version tag for antrea in the Tanzu Kubernetes release (TKr) that you used to create the cluster. You can retrieve the TKr by running the kubectl get tkr command against the management cluster:

      1. Run kubectl get clusters CLUSTER-NAME -n CLUSTER-NAMESPACE --show-labels.

      2. In the output, locate the value of tanzuKubernetesRelease. For example, tanzuKubernetesRelease=v1.23.10---vmware.1-tkg.1.

      3. Run kubectl get tkr TKR-VERSION, where TKR-VERSION is the value that you retrieved above. For example:

        kubectl get tkr v1.23.10---vmware.1-tkg.1 -o yaml

      4. In the output, locate the version tag under packages/core/antrea.

        Alternatively, you can locate the version tag by reviewing the TKr in ~/.config/tanzu/tkg/bom/YOUR-TKR-BOM-FILE.

    2. Download the package configuration files. For example:

      imgpkg pull -b projects.registry.vmware.com/tkg/packages/core/antrea:v1.2.3_vmware.4-tkg.1-advanced -o antrea

    3. Navigate to antrea and review the files.

To know more about the Antrea container networking features, see VMware Container Networking with Antrea 1.4.0 Release Notes. To know more about integrating an Antrea Container Cluster with VMware NSX, see Integration of Antrea Container Clusters.

Updating and Troubleshooting Package Configuration

You can update and troubleshoot the configuration of an auto-managed package by reviewing and modifying the resources below. Because auto-managed packages are managed by Tanzu Kubernetes Grid, you typically do not need to update their configuration.

Type Resources Description
Configuration updates Add-on component Secret

To update the default configuration of the add-on component installed by an auto-managed package, you can:

  • Modify the values.yaml section of the add-on component secret. For more information, see Update the values.yaml Section below.
  • Add an overlay to the add-on component secret. For more information, see Add an Overlay below.
Troubleshooting PackageInstall custom resource (CR) and add-on component Secret

Same as above. Additionally, if you need to apply temporary changes to your package configuration, you can:

  • Pause Secret reconciliation.
  • Pause PackageInstall CR reconciliation.

This deactivates lifecycle management for the package. Use with caution. For more information, see Pause Lifecycle Management for an Auto-Managed Package below.

For more information about add-on component secrets and PackageInstall CRs, see Key Terms below.

Updating Package Configuration

You can update the default configuration of the add-on component installed from an auto-managed package by modifying the values.yaml section of the add-on component secret or by adding an overlay to the secret. These changes are persistent.

Update the values.yaml Section

In the values.yaml section, you can update the following configuration settings:

Package Setting Description
Antrea antrea.config.defaultMTU Defaults to null.
Antrea antrea.config.tlsCipherSuites Include FIPS-enabled Cipher Suites, by default. To switch to other Cipher Suites, update the values under the tlsCipherSuites field.
Calico calico.config.vethMTU Defaults to “0”, which makes Calico auto-detect its maximum transmission size (MTU) setting. Set this parameter to specify a maximum packet size, in bytes, as a string.
Calico calico.config.skipCNIBinaries Defaults to true, which restricts Calico from overwriting the settings of existing CNI plugins during cluster creation. When you upgrade a cluster, to avoid overwriting, set calico.config.skipCNIBinaries=true.
Pinniped pinniped.supervisor_svc_external_dns An FQDN associated with a Pinniped Supervisor, used as a callback URL in the OIDC IDP client. Depending on Pinniped’s service type, you may also need to include the port number:
  • For LoadBalancer, on vSphere with NSX ALB, AWS, or Azure, do not include the port number: https://hr.tkg.example.com.
  • For NodePort, as on vSphere without NSX ALB, include the port number: https://hr.tkg.example.com:31234.
Pinniped pinniped.upstream_oidc_client_id The client ID of your OIDC provider.
Pinniped pinniped.upstream_oidc_client_secret The client secret of your OIDC provider.
Pinniped pinniped.upstream_oidc_issuer_url The URL of your OIDC provider.
Pinniped pinniped.upstream_oidc_tls_ca_data The base64-encoded CA bundle data used to verify TLS connections to your OIDC provider.
Pinniped pinniped.upstream_oidc_additional_scopes A list of additional scopes to request in the token response.
Pinniped pinniped.upstream_oidc_claims OIDC claim mapping.
Pinniped dex.config.ldap.host* The IP or DNS address of your LDAP server. If you want to change the default port 636 to a different port, specify “host:port”.
Pinniped dex.config.ldap.bindDN* and dex.config.ldap.BIND_PW_ENV_VAR* The DN and password for an application service account.
Pinniped dex.config.ldap.userSearch* Search attributes for users. For schema, see the Dex documentation.
Pinniped dex.config.ldap.groupSearch* Search attributes for groups. For schema, see the Dex documentation.
vSphere CSI vsphereCSI.provisionTimeout Defaults to 300s.
vSphere CSI vsphereCSI.attachTimeout Defaults to 300s.

* If you want to update a Pinniped setting that starts with dex., see Update Dex Settings After Management Cluster Deployment.

To modify the values.yaml section of an add-on component secret:

  1. Retrieve the secret by running the kubectl get secret CLUSTER-NAME-PACKAGE-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. For example:

    kubectl get secret example-workload-cluster-antrea-addon -n example-workload-cluster-namespace -o jsonpath="{.data.values\.yaml}" | base64 -d > values.yaml

  2. Update the values.yaml section. You can update any of the values listed in the table above.

  3. Apply your update by base64 encoding the edited values.yaml file and replacing it in the cluster secret. This command differs depending on the OS of your environment. For example:

    Linux:

    kubectl patch secret/example-workload-cluster-antrea-addon -n example-workload-cluster-namespace -p "{\"data\":{\"values.yaml\":\"$(base64 -w 0 < values.yaml)\"}}" --type=merge

    macOS:

    kubectl patch secret/example-workload-cluster-antrea-addon -n example-workload-cluster-namespace -p "{\"data\":{\"values.yaml\":\"$(base64 < values.yaml)\"}}" --type=merge

  4. After updating the secret, check the status of the package by running the kubectl get packageinstall command. For example:

    $ kubectl get packageinstall antrea -n tkg-system
NAMESPACE    NAME                   PACKAGE NAME                      PACKAGE VERSION                 DESCRIPTION                  AGE
tkg-system   antrea                 antrea.tanzu.vmware.com           0.13.3+vmware.1-tkg.1           Reconcile succeeded          7d14h

    If the returned status is Reconcile failed, run the following command to get details on the failure:

    kubectl get packageinstall antrea -n tkg-system -o yaml

  5. Run the kubectl get app command. For example:

    $ kubectl get app antrea -n tkg-system
NAME           DESCRIPTION             SINCE-DEPLOY    AGE
antrea         Reconcile succeeded     3m23s           7h50m

    If the returned status is Reconcile failed, run the following command to get details on the failure:

    kubectl get app antrea -n tkg-system -o yaml

The example below updates the default Maximum Transmission Unit (MTU) for Antrea.

stringData:
  values.yaml: |
    #@data/values
    #@overlay/match-child-defaults missing_ok=True
    ---
    infraProvider: vsphere
    antrea:
      config:
        defaultMTU: 8900
Note

Ensure that you configure the same MTU settings for all the nodes in a cluster. The firewall settings must allow for packets of the configured MTU size. To resolve any issues caused by different MTU settings on the nodes in a cluster, see Cluster Worker Nodes in NotReady Status Due to Mismatched MTUs.

Add an Overlay

In some cases, you can add an overlay to the add-on component secret. This enables you to customize the default configuration that is defined in the package configuration files. The example below instructs Pinniped to use the LoadBalancer service type instead NodePort, which is the default on vSphere when NSX Advanced Load Balancer (ALB) does not serve as the control plane endpoint:

...
stringData:
 overlays.yaml: |
   #@ load("@ytt:overlay", "overlay")
   #@overlay/match by=overlay.subset({"kind": "Service", "metadata": {"name": "pinniped-supervisor", "namespace": "pinniped-supervisor"}})
   ---
   #@overlay/replace
   spec:
     type: LoadBalancer
     selector:
       app: pinniped-supervisor
     ports:
       - name: https
         protocol: TCP
         port: 443
         targetPort: 8443
 values.yaml: |
   #@data/values
   #@overlay/match-child-defaults missing_ok=True
   ---
   infrastructure_provider: vsphere
   tkg_cluster_role: management

To add an overlay:

  1. Retrieve the secret by running the kubectl get secret CLUSTER-NAME-PACKAGE-NAME-addon -n CLUSTER-NAMESPACE command against the management cluster. For example:

    kubectl get secret example-workload-cluster-pinniped-addon -n example-workload-cluster-namespace -o jsonpath="{.data.values\.yaml}" | base64 -d > values.yaml

  2. Add your overlays.yaml section under stringData.

  3. Apply your update by base64 encoding the edited values.yaml file and replacing it in the cluster secret. This command differs depending on the OS of your environment. For example:

    Linux:

    kubectl patch secret/example-workload-cluster-pinniped-addon -n example-workload-cluster-namespace -p "{\"data\":{\"values.yaml\":\"$(base64 -w 0 < values.yaml)\"}}" --type=merge

    macOS:

    kubectl patch secret/example-workload-cluster-pinniped-addon -n example-workload-cluster-namespace -p "{\"data\":{\"values.yaml\":\"$(base64 < values.yaml)\"}}" --type=merge

  4. After updating the secret, check the status of the package by running the kubectl get packageinstall and kubectl get app commands as described in Update the values.yaml Section.

Troubleshooting Package Configuration

Before troubleshooting auto-managed packages, review the following sections:

Key Terms

Tanzu Kubernetes Grid uses the following resources to manage the lifecycle of auto-managed packages.

Components installed in the management cluster:

  • kapp-controller, a local package manager: When you deploy a management cluster, the Tanzu CLI installs kapp-controller in the cluster. kapp-controller installs tanzu-addons-manager and other auto-managed packages. It also installs and manages kapp-controller in each workload cluster that you deploy from that management cluster.
  • tanzu-addons-manager: Manages the add-on components that are installed, as auto-managed packages, in the management cluster and workload clusters that you deploy from your management cluster.
  • tkr-controller: Creates TKrs and BoM ConfigMaps in the management cluster.

Component installed in workload clusters:

kapp-controller installs auto-managed packages in the workload cluster in which it runs.

Objects:

  • Secret: The Tanzu CLI creates a Secret for each add-on component, per cluster. These secrets define the configuration of the add-on components. tanzu-addons-manager reads the secrets and uses the configuration information that they contain to configure the add-on components. All secrets are created in the management cluster.
  • PackageRepository CR: The tanzu-addons-manager creates a PackageRepository CR that references all add-on component Package CRs (see below).
  • Package CR: For each add-on component in the PackageRepository, kapp-controller creates a Package CR in the target cluster.
  • PackageInstall CR: For each add-on component Package, tanzu-addons-manager creates a PackageInstall CR in the target cluster to inform kapp-controller about which auto-managed packages it needs to install.
  • App CR: For each PackageInstall, kapp-controller creates an App CR in the target cluster. Then, kapp-controller reconciles the CR and installs the package.
  • BoM ConfigMap: Provides metadata information about the add-on components, such as image location, to tanzu-addons-manager.

You can use the following commands to monitor the status of these resources:

Command Description
kubectl get packageinstall PACKAGE-NAME -n tkg-system -o yaml Check the PackageInstall CR in your target cluster. For example, kubectl get packageinstall antrea -n tkg-system -o yaml.
kubectl get app PACKAGE-NAME -n tkg-system -o yaml Check the App CR in your target cluster. For example, kubectl get app antrea -n tkg-system -o yaml.
kubectl get cluster CLUSTER-NAME -n CLUSTER-NAMESPACE -o jsonpath={.metadata.labels.tanzuKubernetesRelease} In the management cluster, check if the TKr label of your target cluster points to the correct TKr.
kubectl get tanzukubernetesrelease TKR-NAME Check if the TKr is present in the management cluster.
kubectl get configmaps -n tkr-system -l ‘tanzuKubernetesRelease=TKR-NAME’ Check if the BoM ConfigMap corresponding to your TKr is present in the management cluster.
kubectl get app CLUSTER-NAME-kapp-controller -n CLUSTER-NAMESPACE For workload clusters, check if the kapp-controller App CR is present in the management cluster.
kubectl logs deployment/tanzu-addons-controller-manager -n tkg-system Check tanzu-addons-manager logs in the management cluster.
kubectl get configmap -n tkg-system | grep ADD-ON-NAME-ctrl Check if your updates to the add-on secret have been applied. The sync period is 5 minutes.

Pause Lifecycle Management for an Auto-Managed Package

Important

The commands in this section deactivate package lifecycle management. Whenever possible, use the procedures described in Updating Package Configuration above instead.

If you need to temporary pause lifecycle management for an auto-managed package, you can use the commands below:

  • To pause secret reconciliation, run the following command against the management cluster:

    kubectl patch secret/CLUSTER-NAME-ADD-ON-NAME-addon -n CLUSTER-NAMESPACE -p '{"metadata":{"annotations":{"tkg.tanzu.vmware.com/addon-paused": ""}}}' --type=merge

    After you run this command, tanzu-addons-manager stops reconciling the secret.

  • To pause PackageInstall CR reconciliation, run the following command against your target cluster:

    kubectl patch packageinstall/PACKAGE-NAME -n tkg-system -p '{"spec":{"paused":true}}' --type=merge

    After you run this command, kapp-controller stops reconciling the PackageInstall and corresponding App CR.

If you want to temporary modify the resources of an add-on component, pause secret reconciliation first and then pause PackageInstall CR reconciliation. After you unpause lifecycle management, tanzu-addons-manager and kapp-controller resume secret and PackageInstall CR reconciliation:

  • To unpause secret reconciliation, remove tkg.tanzu.vmware.com/addon-paused from the secret annotations.

  • To unpause PackageInstall CR reconciliation, update the PackageInstall CR with {"spec":{"paused":false}} or remove the variable.

check-circle-line exclamation-circle-line close-line
Scroll to top icon