Configure and Manage TLS Certificates for Pinniped and Dex

This topic explains how to configure custom TLS certificates for Pinniped and Dex in Tanzu Kubernetes Grid. It also explains how to update the Dex certificates in Tanzu Kubernetes Grid.

Configuring Custom TLS Certificates

By default, Tanzu Kubernetes Grid uses a self-signed Issuer to generate the TLS certificates that secure HTTPS traffic to Pinniped and Dex. You can optionally update the default configuration after deploying the management cluster, as follows:

1. Set a custom ClusterIssuer resource or your own TLS secret. See Set a ClusterIssuer Resource or a TLS Secret below.
2. Update your Pinniped configuration by redeploying the Pinniped add-on secret for the management cluster. See Update Your Pinniped Configuration below.

Note

Tanzu Kubernetes Grid deploys Dex only for LDAP identity providers. If you configure an OIDC identity provider when you create the management cluster or configure it as a post-creation step, Dex is not deployed.

Set a ClusterIssuer Resource or a TLS Secret

If you want to use a custom ClusterIssuer resource to generate the TLS certificates:

1. Verify that cert-manager is running in your management cluster. This component runs by default in all management clusters.
2. Obtain the name of an existing ClusterIssuer resource in the management cluster. For more information, see Issuer Configuration in the cert-manager documentation.
3. Specify the ClusterIssuer name in the custom_cluster_issuer field of the values.yaml section in the Pinniped add-on secret for the management cluster and then apply your changes. For instructions, see Update Your Pinniped Configuration below. After you complete the steps in this section, both the Pinniped the Dex certificate chains will be signed by your ClusterIssuer.

If you want to specify your own TLS secret directly:

1. Retrieve the IP address or DNS hostname of the Pinniped service, pinniped-supervisor, and if you are using an LDAP identity provider, of the Dex service, dexsvc:

   • The pinniped-supervisor service:

     • If the type of the service is set to LoadBalancer (vSphere with a load balancer, for example, NSX Advanced Load Balancer, Amazon Web Services (AWS), or Azure), retrieve the external address of the service by running:

       kubectl get service pinniped-supervisor -n pinniped-supervisor
       

     • If the type of the service is set to NodePort (vSphere without a load balancer), the IP address of the service is the same as the vSphere control plane endpoint. To retrieve the IP address, you can run the following command:

       kubectl get configmap cluster-info -n kube-public -o yaml
       

   • (LDAP only) The dexsvc service:

     • If the type of the service is set to LoadBalancer (vSphere with a load balancer, for example, NSX Advanced Load Balancer, Amazon Web Services (AWS), or Azure), retrieve the external address of the service by running:

       kubectl get service dexsvc -n tanzu-system-auth
       

     • If the type of the service is set to NodePort (vSphere without a load balancer), the IP address of the service is the same as the vSphere control plane endpoint. To retrieve the IP address, you can run the following command:

       kubectl get configmap cluster-info -n kube-public -o yaml
       

2. Create a kubernetes.io/tls secret in the pinniped-supervisor namespace if you are using an OIDC identity provider. If you are using an LDAP identity provider, create two kubernetes.io/tls secrets with the same name, one, for the Pinniped service, in the pinniped-supervisor namespace and one, for the Dex service, in the tanzu-system-auth namespace. To create a TLS secret, run:

   kubectl create secret generic SECRET-NAME -n SECRET-NAMESPACE --type kubernetes.io/tls --from-file tls.crt=FILENAME-1.crt --from-file tls.key=FILENAME-2.pem --from-file ca.crt=FILENAME-3.pem
   

   Replace the placeholder text as follows:

   • SECRET-NAME is the name you choose for the secret. For example, my-secret.
   • SECRET-NAMESPACE is the namespace in which to create the secret. This must be pinniped-supervisor for Pinniped and tanzu-system-auth for Dex.
   • FILENAME-* is the name of your tls.crt, tls.key, or ca.crt. The TLS certificate that you specify in the TLS secret for Pinniped must include the IP or DNS hostname of the Pinniped service from the step above. Similarly, the TLS certificate for Dex must include the IP address or DNS hostname of the Dex service that you retrieved above. The ca.crt field is required and contains the CA bundle that is used to verify the TLS certificate.

   For example, the resulting secret for Pinniped looks similar to the following:

   apiVersion: v1
   kind: Secret
   metadata:
    name: my-secret
    namespace: pinniped-supervisor
   type: kubernetes.io/tls
   data:
    tls.crt: |       MIIC2DCCAcCgAwIBAgIBATANBgkqh ...
    tls.key: |       MIIEpgIBAAKCAQEA7yn3bRHQ5FHMQ ...
    ca.crt:  |       MIIEpgIBAAKCAQEA7yn3bRHQ5FHMQ ...
   

   If the secret has been generated correctly, decoding tls.crt with openssl x509 -in tls.crt -text shows the IP address or DNS hostname in the Subject Alternative Name field.

3. Specify the secret name in the custom_tls_secret field of the values.yaml section in the Pinniped add-on secret for the management cluster and then apply your changes. For instructions, see Update Your Pinniped Configuration below.

4. If you want to configure a DNS hostname for the Pinniped service, specify the FQDN associated with a Pinniped Supervisor in the pinniped.supervisor_svc_external_dns field of the values.yaml section in the Pinniped add-on secret for the management cluster. Note that the value of pinniped.supervisor_svc_external_dns must start with https://. See Auto-Managed Package values.yaml Settings in detail. Then apply your changes. For instructions, see Update Your Pinniped Configuration below. Note that you must separately configure DNS for this hostname, for example by creating an A record in your DNS provider to resolve to the IP address of the Pinniped Supervisor Service. This hostname must be listed as a Subject Alternative Name in the TLS certificate that you configure for the Pinniped Supervisor.

Update Your Pinniped Configuration

To apply your changes, update the Pinniped configuration by following the steps below:

1. Save the Pinniped add-on secret for the management cluster to a file and decode the Base64-encoded string in the values.yaml section of the secret.

   kubectl get secret CLUSTER-NAME-pinniped-package -n tkg-system -o jsonpath="{.data.values\.yaml}" | base64 -d > FILENAME.yaml
   

   Replace the placeholder text as follows:

   • CLUSTER-NAME is the name of your management cluster.
   • FILENAME is the file name that you want to use for the secret. For example, values.yaml.

2. In the decoded text, do one of the following:

   • If you prepared a ClusterIssuer resource above, specify the name of the resource in the custom_cluster_issuer field. For example:

      ---
      infrastructure_provider: vsphere
      tkg_cluster_role: management
      custom_cluster_issuer: "my-cluster-issuer-name"
      pinniped:
         cert_duration: 2160h
         cert_renew_before: 360h
         supervisor_svc_endpoint: https://10.168.217.220:31234
         supervisor_ca_bundle_data: LS0tLS1CRUdJTiBDRVJUSUZJQ0F……
      ...
     

   • If you prepared your own TLS secret above, specify the name of the secret in the custom_tls_secret field. For example:

      ---
      infrastructure_provider: vsphere
      tkg_cluster_role: management
      custom_tls_secret: "my-tls-secret-name"
      pinniped:
         cert_duration: 2160h
         cert_renew_before: 360h
         supervisor_svc_endpoint: https://10.168.217.220:31234
         supervisor_ca_bundle_data: LS0tLS1CRUdJTiBDRVJUSUZJQ0F……
      ...
     

     If you configure the custom_tls_secret field, the custom_cluster_issuer is ignored.

     If you want to configure a DNS hostname for the Pinniped service, specify the FQDN that should be used for the Pinniped Supervisor in the pinniped.supervisor_svc_external_dns field. For example,

      ...
      pinniped:
        cert_duration: 2160h
        cert_renew_before: 360h
        supervisor_svc_endpoint: https://0.0.0.0:31234
        supervisor_ca_bundle_data: ca_bundle_data_of_supervisor_svc
        supervisor_svc_external_ip: 0.0.0.0
        supervisor_svc_external_dns: https://pinniped.example.com
      ...
     

3. Encode the values.yaml section again and update the Pinniped add-on secret. This command differs depending on the OS of your environment. For example:

   Linux:

   kubectl patch secret CLUSTER-NAME-pinniped-package -n tkg-system -p "{\"data\":{\"values.yaml\":\"$(base64 -w 0 < values.yaml)\"}}" --type=merge
   

   macOS:

   kubectl patch secret CLUSTER-NAME-pinniped-package -n tkg-system -p "{\"data\":{\"values.yaml\":\"$(base64 < values.yaml)\"}}" --type=merge
   

4. Confirm that the changes have been applied successfully:

   1. Get the status of the pinniped app:

      kubectl get app CLUSTER-NAME-pinniped -n tkg-system
      

      If the returned status is Reconcile failed, run the following command to get details on the failure:

      kubectl get app CLUSTER-NAME-pinniped -n tkg-system -o yaml
      

   2. Generate a kubeconfig file for the management cluster by running the tanzu mc kubeconfig get --export-file ./KUBECONFIG-MC-CLUSTER-NAME command. Then run a command such as kubectl get pods --kubeconfig ./KUBECONFIG-MC-CLUSTER-NAME using the kubeconfig. Additionally, if your management cluster manages any workload clusters, run tanzu cluster kubeconfig get <WORKLOAD-CLUSTER-NAME> --export-file ./KUBECONFIG-WORKLOAD-CLUSTER-NAME and then kubectl get pods --kubeconfig ./KUBECONFIG-WORKLOAD-CLUSTER-NAME against each of the existing clusters.

Update Dex Serving Certificate and Dex CA certificate

With clusters that use LDAP identity management, you might want to update the Dex certificates if:

• The Dex CA certificate has been updated or expired.
• The private key associated with the Dex CA is compromised.

Prerequisites

Before performing this procedure, ensure that you have:

• Configured the Pinniped addon with LDAP IDP. For more information, see Identity Providers - LDAP.
• Obtained the address of the Dex service by using the kubectl get service dexsvc -n tanzu-system-auth command.

Update the Dex Serving Certificate

1. Change the kubectl context to the management cluster, if you are currently in the workload cluster context. For more information, See Retrieve Workload Cluster kubeconfig.

2. Download the current Dex serving certificate:

   openssl s_client -connect ADDRESS:PORT -showcerts </dev/null | openssl x509 -noout -text > /tmp/OLD-FILE.txt
   

   Where:

   • ADDRESS is the Dex service address in Tanzu Kubernetes Grid.
   • OLD-FILE is the name in which you want to save the serving certificate text file, for example, before.txt.

3. Delete the current Dex serving certificate secret so that cert-manager re-creates it:

   kubectl delete secret -n tanzu-system-auth  dex-cert-tls
   

   The following is a sample output:

   secret "dex-cert-tls" deleted
   

4. Verify that a new Dex serving certificate is created:

   kubectl get secret -n tanzu-system-auth
   

   The following is a sample output:

   $kubectl get secret -n tanzu-system-auth
   NAME                  TYPE                                  DATA   AGE
   default-token-cg8f2   kubernetes.io/service-account-token   3      6m5s
   dex-ca-key-pair       kubernetes.io/tls                     3      5m50s
   dex-cert-tls          kubernetes.io/tls                     3      2s
   dex-token-p96gl       kubernetes.io/service-account-token   3      6m3s
   

5. Restart the Dex pods:

   kubectl delete pod -n tanzu-system-auth -l app=dex
   

   The following is a sample output:

   $ kubectl delete pod -n tanzu-system-auth -l app=dex
   pod DEX-POD deleted
   

6. Download the new Dex serving certificate:

   openssl s_client -connect ADDRESS:PORT -showcerts </dev/null  | openssl x509 -noout -text > /tmp/NEW-FILE.txt
   

   Where:

   • ADDRESS is the Dex service address in Tanzu Kubernetes Grid.
   • NEW-FILE is the name in which you want to save the new serving certificate text file, for example, after.txt.

7. Compare the old and the new certificates to ensure that the new certificate has been created:

   diff /tmp/OLD-FILE /tmp/NEW-FILE
   

   Where:

   • OLD-FILE is the name of the old Dex serving certificate.
   • NEW-FILE is the name of the new Dex serving certificate.

Update the Dex CA Certificate

1. Download the current Dex CA certificate:

   kubectl get secret -n tanzu-system-auth dex-cert-tls -o jsonpath={.data.ca\\.crt} | base64 -d | openssl x509 -noout -text > /tmp/OLD-FILE.txt
   

   Where OLD-FILE is the name in which you want to save the serving certificate text file, for example, ca-before.txt.

2. Download the current Dex serving certificate:

   openssl s_client -connect ADDRESS:PORT -showcerts </dev/null | openssl x509 -noout -text > /tmp/OLD-FILE.txt
   

   Where:

   • ADDRESS is the Dex service address in Tanzu Kubernetes Grid.

   • OLD-FILE is the name in which you want to save the serving certificate text file, for example, cert-before.txt.

3. Download the current Dex CA data:

   kubectl get oidcidentityprovider upstream-oidc-identity-provider -n pinniped-supervisor -o jsonpath={.spec.tls.certificateAuthorityData} | base64 -d  | openssl x509 -noout -text > /tmp/OLD-FILE.txt
   

   Where OLD-FILE is the name in which you want to save the serving certificate text file, for example, ca-data-before.txt.

4. Delete the Dex CA certificate secret so that cert-manager re-creates it:

   kubectl delete secret -n tanzu-system-auth dex-ca-key-pair
   

   The following is a sample output:

   secret "dex-ca-key-pair" deleted
   

5. Verify that a new Dex CA certificate is created:

   kubectl get secret -n tanzu-system-auth
   

   The following is a sample output:

   $ kubectl get secret -n tanzu-system-auth
   NAME                  TYPE                                  DATA   AGE
   default-token-cg8f2   kubernetes.io/service-account-token   3      25m
   dex-ca-key-pair       kubernetes.io/tls                     3      18s
   dex-cert-tls          kubernetes.io/tls                     3      17s
   dex-token-p96gl       kubernetes.io/service-account-token   3      25m
   
   

6. Delete the current Dex serving certificate secret so that cert-manager re-creates it:

   kubectl delete secret -n tanzu-system-auth  dex-cert-tls
   

   The following is a sample output:

   secret "dex-cert-tls" deleted
   

7. Verify that a new Dex serving certificate is created:

   kubectl get secret -n tanzu-system-auth
   

   The following is a sample output:

   kubectl get secret -n tanzu-system-auth
   NAME                  TYPE                                  DATA   AGE
   default-token-cg8f2   kubernetes.io/service-account-token   3      6m5s
   dex-ca-key-pair       kubernetes.io/tls                     3      5m50s
   dex-cert-tls          kubernetes.io/tls                     3      2s
   dex-token-p96gl       kubernetes.io/service-account-token   3      6m3s
   

8. Delete the Pinniped post-deploy job:

   kubectl delete job -n pinniped-supervisor pinniped-post-deploy-job
   

   The following is a sample output:

   job.batch "pinniped-post-deploy-job" deleted
   

9. Update the Pinniped add-on to sync your changes quickly, and wait a few minutes for the add-on to reconcile the changes:

   kubectl patch app pinniped -n tkg-system -p '{"spec":{"syncPeriod":"30s"}}' --type merge
   

   kubectl wait app pinniped -n tkg-system --for condition=ReconcileSucceeded --timeout 5m
   

10. Download the new Dex CA certificate:

    kubectl get secret -n tanzu-system-auth dex-cert-tls -o jsonpath={.data.ca\\.crt} | base64 -d | openssl x509 -noout -text > /tmp/NEW-FILE.txt
    

    Where NEW-FILE is the name in which you want to save the new Dex CA certificate text file, for example, ca-after.txt.

11. Download the new Dex serving certificate:

    openssl s_client -connect ADDRESS:PORT -showcerts </dev/null  | openssl x509 -noout -text > /tmp/NEW-FILE.txt
    

    Where:

    • ADDRESS is the Dex service address in Tanzu Kubernetes Grid.

    • NEW-FILE is the name in which you want to save the new Dex serving certificate text file, for example, after.txt.

    • Download the new Dex CA data:

    kubectl get oidcidentityprovider upstream-oidc-identity-provider -n pinniped-supervisor -o jsonpath={.spec.tls.certificateAuthorityData} | base64 -d  | openssl x509 -noout -text > /tmp/NEW-FILE.txt
    

    Where NEW-FILE is the name in which you want to save the new Dex CA data text file, for example, ca-data-after.txt.

12. Compare the old and the new CA certificates to ensure that the new CA certificate has been created:

    diff /tmp/OLD-FILE /tmp/NEW-FILE
    

    Where:

    • OLD-FILE is the name of the old Dex CA certificate.
    • NEW-FILE is the name of the old Dex CA certificate.

13. Compare the old and the new serving certificates to ensure that the new serving certificate has been created:

    diff /tmp/OLD-FILE /tmp/NEW-FILE
    

    Where:

    • OLD-FILE is the name of the old Dex serving certificate.
    • NEW-FILE is the name of the old Dex serving certificate.

14. Compare the old and the new CA data to ensure that the new CA data has been created:

    diff /tmp/OLD-FILE /tmp/NEW-FILE
    

    Where:

    • OLD-FILE is the name of the old Dex CA data file.
    • NEW-FILE is the name of the old Dex CA data file.