This topic explains how to configure custom TLS certificates for Pinniped and Dex in Tanzu Kubernetes Grid. It also explains how to update the Dex certificates in Tanzu Kubernetes Grid.
By default, Tanzu Kubernetes Grid uses a self-signed Issuer to generate the TLS certificates that secure HTTPS traffic to Pinniped and Dex. You can optionally update the default configuration after deploying the management cluster, as follows:
ClusterIssuer resource or your own TLS secret. See Set a ClusterIssuer Resource or a TLS Secret below.NoteTanzu Kubernetes Grid deploys Dex only for LDAP identity providers. If you configure an OIDC identity provider when you create the management cluster or configure it as a post-creation step, Dex is not deployed.
ClusterIssuer Resource or a TLS SecretIf you want to use a custom ClusterIssuer resource to generate the TLS certificates:
cert-manager is running in your management cluster. This component runs by default in all management clusters.ClusterIssuer resource in the management cluster. For more information, see Issuer Configuration in the cert-manager documentation.ClusterIssuer name in the custom_cluster_issuer field of the values.yaml section in the Pinniped add-on secret for the management cluster and then apply your changes. For instructions, see Update Your Pinniped Configuration below. After you complete the steps in this section, both the Pinniped the Dex certificate chains will be signed by your ClusterIssuer.If you want to specify your own TLS secret directly:
Retrieve the IP address or DNS hostname of the Pinniped service, pinniped-supervisor, and if you are using an LDAP identity provider, of the Dex service, dexsvc:
The pinniped-supervisor service:
If the type of the service is set to LoadBalancer (vSphere with a load balancer, for example, NSX Advanced Load Balancer, Amazon Web Services (AWS), or Azure), retrieve the external address of the service by running:
kubectl get service pinniped-supervisor -n pinniped-supervisor
If the type of the service is set to NodePort (vSphere without a load balancer), the IP address of the service is the same as the vSphere control plane endpoint. To retrieve the IP address, you can run the following command:
kubectl get configmap cluster-info -n kube-public -o yaml
(LDAP only) The dexsvc service:
If the type of the service is set to LoadBalancer (vSphere with a load balancer, for example, NSX Advanced Load Balancer, Amazon Web Services (AWS), or Azure), retrieve the external address of the service by running:
kubectl get service dexsvc -n tanzu-system-auth
If the type of the service is set to NodePort (vSphere without a load balancer), the IP address of the service is the same as the vSphere control plane endpoint. To retrieve the IP address, you can run the following command:
kubectl get configmap cluster-info -n kube-public -o yaml
Create a kubernetes.io/tls secret in the pinniped-supervisor namespace if you are using an OIDC identity provider. If you are using an LDAP identity provider, create two kubernetes.io/tls secrets with the same name, one, for the Pinniped service, in the pinniped-supervisor namespace and one, for the Dex service, in the tanzu-system-auth namespace. To create a TLS secret, run:
kubectl create secret generic SECRET-NAME -n SECRET-NAMESPACE --type kubernetes.io/tls --from-file tls.crt=FILENAME-1.crt --from-file tls.key=FILENAME-2.pem --from-file ca.crt=FILENAME-3.pem
Replace the placeholder text as follows:
SECRET-NAME is the name you choose for the secret. For example, my-secret.SECRET-NAMESPACE is the namespace in which to create the secret. This must be pinniped-supervisor for Pinniped and tanzu-system-auth for Dex.FILENAME-* is the name of your tls.crt, tls.key, or ca.crt. The TLS certificate that you specify in the TLS secret for Pinniped must include the IP or DNS hostname of the Pinniped service from the step above. Similarly, the TLS certificate for Dex must include the IP address or DNS hostname of the Dex service that you retrieved above. The ca.crt contains the CA bundle that is used to verify the TLS certificate.For example, the resulting secret for Pinniped looks similar to the following:
apiVersion: v1
kind: Secret
metadata:
name: my-secret
namespace: pinniped-supervisor
type: kubernetes.io/tls
data:
tls.crt: | MIIC2DCCAcCgAwIBAgIBATANBgkqh ...
tls.key: | MIIEpgIBAAKCAQEA7yn3bRHQ5FHMQ ...
ca.crt: | MIIEpgIBAAKCAQEA7yn3bRHQ5FHMQ ...
If the secret has been generated correctly, decoding tls.crt with openssl x509 -in tls.crt -text shows the IP address or DNS hostname in the Subject Alternative Name field.
Specify the secret name in the custom_tls_secret field of the values.yaml section in the Pinniped add-on secret for the management cluster and then apply your changes. For instructions, see Update Your Pinniped Configuration below.
To apply your changes, update the Pinniped configuration by following the steps below:
Save the Pinniped add-on secret for the management cluster to a file:
kubectl get secret CLUSTER-NAME-pinniped-package -n tkg-system -o yaml > FILENAME.yaml
Replace the placeholder text as follows:
CLUSTER-NAME is the name of your management cluster.FILENAME is the file name that you want to use for the secret. For example, pinniped-addon-secret.yaml.Decode the Base64-encoded string in the values.yaml section of the secret using the decoding tool of your choice. For example:
pbpaste | base64 -d > pinniped-configuration.yaml
In the decoded text, do one of the following:
If you prepared a ClusterIssuer resource above, specify the name of the resource in the custom_cluster_issuer field. For example:
---
infrastructure_provider: vsphere
tkg_cluster_role: management
custom_cluster_issuer: "my-cluster-issuer-name"
pinniped:
cert_duration: 2160h
cert_renew_before: 360h
supervisor_svc_endpoint: https://10.168.217.220:31234
supervisor_ca_bundle_data: LS0tLS1CRUdJTiBDRVJUSUZJQ0F……
...
If you prepared your own TLS secret above, specify the name of the secret in the custom_tls_secret field. For example:
---
infrastructure_provider: vsphere
tkg_cluster_role: management
custom_tls_secret: "my-tls-secret-name"
pinniped:
cert_duration: 2160h
cert_renew_before: 360h
supervisor_svc_endpoint: https://10.168.217.220:31234
supervisor_ca_bundle_data: LS0tLS1CRUdJTiBDRVJUSUZJQ0F……
...
If you configure the custom_tls_secret field, the custom_cluster_issuer is ignored.
Encode the values.yaml section again using the Base64 scheme. For example:
cat pinniped-configuration.yaml | base64 > encoded.txt
Copy the encoded string and paste it into the Pinniped add-on secret file that you retrieved from the cluster in step 1. For example, pinniped-addon-secret.yaml.
Redeploy the Pinniped add-on secret by running the kubectl apply -f command. For example:
kubectl apply -f pinniped-addon-secret.yaml
Confirm that the changes have been applied successfully:
Get the status of the pinniped app:
kubectl get app CLUSTER-NAME-pinniped -n tkg-system
If the returned status is Reconcile failed, run the following command to get details on the failure:
kubectl get app pinniped -n tkg-system -o yaml
Generate a kubeconfig file for the management cluster by running the tanzu mc kubeconfig get --export-file ./KUBECONFIG-MC-CLUSTER-NAME command. Then run a command such as kubectl get pods --kubeconfig ./KUBECONFIG-MC-CLUSTER-NAME using the kubeconfig. Additionally, if your management cluster manages any workload clusters, run tanzu cluster kubeconfig get <WORKLOAD-CLUSTER-NAME> --export-file ./KUBECONFIG-WORKLOAD-CLUSTER-NAME and then kubectl get pods --kubeconfig ./KUBECONFIG-WORKLOAD-CLUSTER-NAME against each of the existing clusters.
With clusters that use LDAP identity management, you might want to update the Dex certificates if:
Before performing this procedure, ensure that you have:
kubectl get service dexsvc -n tanzu-system-auth command.Change the kubectl context to the management cluster, if you are currently in the workload cluster context. For more information, See Retrieve Workload Cluster kubeconfig.
Download the current Dex serving certificate:
openssl s_client -connect ADDRESS -showcerts </dev/null | openssl x509 -noout -text > /tmp/OLD-FILE.txt
Where:
ADDRESS is the Dex service address in Tanzu Kubernetes Grid.OLD-FILE is the name in which you want to save the serving certificate text file, for example, before.txt.Delete the current Dex serving certificate secret so that cert-manager re-creates it:
kubectl delete secret -n tanzu-system-auth dex-cert-tls
The following is a sample output:
secret "dex-cert-tls" deleted
Verify that a new Dex serving certificate is created:
kubectl get secret -n tanzu-system-auth
The following is a sample output:
$kubectl get secret -n tanzu-system-auth
NAME TYPE DATA AGE
default-token-cg8f2 kubernetes.io/service-account-token 3 6m5s
dex-ca-key-pair kubernetes.io/tls 3 5m50s
dex-cert-tls kubernetes.io/tls 3 2s
dex-token-p96gl kubernetes.io/service-account-token 3 6m3s
Restart the Dex pods:
kubectl delete pod -n tanzu-system-auth -l app=dex
The following is a sample output:
$ kubectl delete pod -n tanzu-system-auth -l app=dex
pod DEX-POD deleted
Download the new Dex serving certificate:
openssl s_client -connect ADDRESS -showcerts </dev/null | openssl x509 -noout -text > /tmp/NEW-FILE.txt
Where:
ADDRESS is the Dex service address in Tanzu Kubernetes Grid.NEW-FILE is the name in which you want to save the new serving certificate text file, for example, after.txt.Compare the old and the new certificates to ensure that the new certificate has been created:
diff /tmp/OLD-FILE /tmp/NEW-FILE
Where:
OLD-FILE is the name of the old Dex serving certificate.NEW-FILE is the name of the new Dex serving certificate.Download the current Dex CA certificate:
kubectl get secret -n tanzu-system-auth dex-cert-tls -o jsonpath={.data.ca\\.crt} | base64 -d | openssl x509 -noout -text > /tmp/OLD-FILE.txt
Where OLD-FILE is the name in which you want to save the serving certificate text file, for example, ca-before.txt.
Download the current Dex serving certificate:
openssl s_client -connect ADDRESS -showcerts </dev/null | openssl x509 -noout -text > /tmp/OLD-FILE.txt
Where:
ADDRESS is the Dex service address in Tanzu Kubernetes Grid.
OLD-FILE is the name in which you want to save the serving certificate text file, for example, cert-before.txt.
Download the current Dex CA data:
kubectl get oidcidentityprovider upstream-oidc-identity-provider -n pinniped-supervisor -o jsonpath={.spec.tls.certificateAuthorityData} | base64 -d | openssl x509 -noout -text > /tmp/OLD-FILE.txt
Where OLD-FILE is the name in which you want to save the serving certificate text file, for example, ca-data-before.txt.
Delete the Dex CA certificate secret so that cert-manager re-creates it:
kubectl delete secret -n tanzu-system-auth dex-ca-key-pair
The following is a sample output:
secret "dex-ca-key-pair" deleted
Verify that a new Dex CA certificate is created:
kubectl get secret -n tanzu-system-auth
The following is a sample output:
$ kubectl get secret -n tanzu-system-auth
NAME TYPE DATA AGE
default-token-cg8f2 kubernetes.io/service-account-token 3 25m
dex-ca-key-pair kubernetes.io/tls 3 18s
dex-cert-tls kubernetes.io/tls 3 17s
dex-token-p96gl kubernetes.io/service-account-token 3 25m
Delete the current Dex serving certificate secret so that cert-manager re-creates it:
kubectl delete secret -n tanzu-system-auth dex-cert-tls
The following is a sample output:
secret "dex-cert-tls" deleted
Verify that a new Dex serving certificate is created:
kubectl get secret -n tanzu-system-auth
The following is a sample output:
kubectl get secret -n tanzu-system-auth
NAME TYPE DATA AGE
default-token-cg8f2 kubernetes.io/service-account-token 3 6m5s
dex-ca-key-pair kubernetes.io/tls 3 5m50s
dex-cert-tls kubernetes.io/tls 3 2s
dex-token-p96gl kubernetes.io/service-account-token 3 6m3s
Delete the Pinniped post-deploy job:
kubectl delete job -n pinniped-supervisor pinniped-post-deploy-job
The following is a sample output:
job.batch "pinniped-post-deploy-job" deleted
Update the Pinniped add-on to sync your changes quickly, and wait a few minutes for the add-on to reconcile the changes:
kubectl patch app pinniped -n tkg-system -p '{"spec":{"syncPeriod":"30s"}}' --type merge
kubectl wait app pinniped -n tkg-system --for condition=ReconcileSucceeded --timeout 5m
Download the new Dex CA certificate:
kubectl get secret -n tanzu-system-auth dex-cert-tls -o jsonpath={.data.ca\\.crt} | base64 -d | openssl x509 -noout -text > /tmp/NEW-FILE.txt
Where NEW-FILE is the name in which you want to save the new Dex CA certificate text file, for example, ca-after.txt.
Download the new Dex serving certificate:
openssl s_client -connect ADDRESS -showcerts </dev/null | openssl x509 -noout -text > /tmp/NEW-FILE.txt
Where:
ADDRESS is the Dex service address in Tanzu Kubernetes Grid.NEW-FILE is the name in which you want to save the new Dex serving certificate text file, for example, after.txt.
Download the new Dex CA data:
kubectl get oidcidentityprovider upstream-oidc-identity-provider -n pinniped-supervisor -o jsonpath={.spec.tls.certificateAuthorityData} | base64 -d | openssl x509 -noout -text > /tmp/NEW-FILE.txt
Where NEW-FILE is the name in which you want to save the new Dex CA data text file, for example, ca-data-after.txt.
Compare the old and the new CA certificates to ensure that the new CA certificate has been created:
diff /tmp/OLD-FILE /tmp/NEW-FILE
Where:
OLD-FILE is the name of the old Dex CA certificate.NEW-FILE is the name of the old Dex CA certificate.Compare the old and the new serving certificates to ensure that the new serving certificate has been created:
diff /tmp/OLD-FILE /tmp/NEW-FILE
Where:
OLD-FILE is the name of the old Dex serving certificate.NEW-FILE is the name of the old Dex serving certificate.Compare the old and the new CA data to ensure that the new CA data has been created:
diff /tmp/OLD-FILE /tmp/NEW-FILE
Where:
OLD-FILE is the name of the old Dex CA data file.NEW-FILE is the name of the old Dex CA data file.