Certificate rotation in Tanzu Mission Control Self-Managed

When a certificate expires or if the private key of a certificate is compromised, use the following procedures to rotate your certificates.

Rotating certificates for internal (east-west) traffic with `cert-manager`

Use the following steps to manually rotate certificates that are provided by TMC Self-Managed.

Identify the certificates that need to be rotated using the following command. The status field in the output indicates the validity of the certificate.
```
kubectl -n tmc-local get certificates -o wide
```

Identify, from the certificate resource, the Kubernetes TLS secret name.

kubectl -n tmc-local get certificate <CERTIFICATE_RESOURCE_NAME> -oyaml | yq .spec.secretName

Delete the certificate and the Kubernetes TLS secret name.

kubectl -n tmc-local delete certificate <CERTIFICATE_RESOURCE_NAME>
kubectl -n tmc-local delete secret <KUBERNETES_TLS_SECRET_NAME_FROM_2>

Restart the pods that use the certificate.

For a mapping of the certificate to a deployment/pod, see Mapping certificate name to TMC SM pod/deployment/stateful-set.
```
kubectl -n tmc-local rollout restart deployment <DEPLOYMENT_NAME> 
```
To restart the stateful-set (for example, kafka-controller, postgresql)
```
kubectl -n tmc-local rollout restart statefulset <STATEFULSET_NAME>
```
Note
minio-provisioning is an init job, and therefore it does not need to be restarted if its execution state is completed.

Verify the pod state:

kubectl -n tmc-local get pods | grep minio-provisioning

The restarted pods remain in container creating phase until the following are reconciled:

kapp-controller reconciles the certificate resource from the TMC Self-Managed package.
cert-manager reconciles the certificate resource and creates the TLS secret.

Rotating custom certificates for internal traffic

If you are using a custom certificate (one that is not provided by TMC Self-Managed), use the following steps to manually rotate the certificate.

Identify the certificate that needs to be rotated.
The status field in the output indicates the validity of the certificate.

kubectl get certificate -n tmc-local -ojsonpath="{range .items[*]}{.metadata.name} not before: {.status.notBefore} not after: {.status.notAfter}{'\n'}{end}"

Delete the Kubernetes TLS secret name.

kubectl -n tmc-local delete secret <KUBERNETES_TLS_SECRET_NAME>

Create a new Kubernetes TLS secret.

kubectl create secret tls <SECRET_NAME> \
--key <private-key-filename> \
--cert <certificate-filename>

Alternatively, if you already have an existing secret file:

kubectl create -f <my-secret.yaml>

Verify that the secret is created.
```
kubectl -n tmc-local get secrets
```
Restart the pods that use the secret.

For a mapping of the certificate to a deployment/pod, see Mapping certificate name to TMC SM pod/deployment/stateful-set.
```
kubectl -n tmc-local rollout restart deployment <DEPLOYMENT_NAME>
```
To restart the stateful-set (for example, kafka-controller, postgresql)
```
kubectl -n tmc-local rollout restart statefulset <STATEFULSET_NAME>
```
Note
minio-provisioning is an init job, and therefore it does not need to be restarted if its execution state is completed.

Verify the pod state:

kubectl -n tmc-local get pods | grep minio-provisioning

Mapping certificate name to TMC SM pod/deployment/stateful-set

Deployment Name	Secret Name
account-manager-server	account-manager-server-tls
agent-gateway-server	agent-gateway-instack-tls
api-gateway-server	api-gateway-instack-tls
audit-service-consumer	audit-consumer-tls
audit-service-server	audit-server-tls
auth-manager-server	server-tls
authentication-server	authentication-tls
cluster-agent-service-server	cluster-agent-service-server-tls
cluster-config-server	cluster-config-server-tls
cluster-object-service-server	cluster-object-service-server-tls
cluster-reaper-server	cluster-reaper-server-tls
cluster-secret-server	cluster-secret-server-tls
cluster-service-server	cluster-service-server-tls
cluster-sync-egest	cluster-sync-egest-tls
cluster-sync-ingest	cluster-sync-ingest-tls
dataprotection-server	dataprotection-server-tls
events-service-consumer	events-service-consumer-tls
events-service-server	events-service-server-tls
fanout-service-server	fanout-service-server-tls
feature-flag-service-server	feature-flag-service-tls
helm-deployment-server	helm-deployment-server-tls
inspection-server	inspection-server-tls
intent-server	intent-tls
kafka-exporter	kafka-exporter-cert
kafka-topic-controller	dev-ca-key
landing-service-server	landing-service-tls
minio-provisioning	minio-api-user-creds
onboarding-service-server	onboarding-service-server-tls
package-deployment-server	package-deployment-server-tls
policy-engine-server	policy-engine-server-tls
policy-insights-server	policy-insights-server-tls
policy-sync-service-server	policy-sync-service-server-tls
policy-view-service-server	policy-view-service-server-tls
provisioner-service-server	provisioner-service-server-tls
resource-manager-server	resource-manager-server-tls
telemetry-event-service-consumer	api-consumer-tls
tenancy-service-server	tenancy-service-tls
ui-server	ui-server-tls
wcm-server	wcm-server-tls

Stateful-set Name	Secret Name
kafka-controller	kafka-broker-cert
postgres-postgresql	postgres-tls

Rotating certificates for external (north-south) traffic

This procedure explains how to rotate the certificate provided by TMC Self-Managed through cert-manager that authenticates external traffic. Because this certificate is used for communication between TMC Self-Managed and the clusters it manages, you must re-establish that connection after rotating the certificate.

When to rotate the certificate?

When the certificate is due to expire soon, or the private key used by cert-manager to sign the certificates is compromised, rotate your certificate for external traffic.

Rotate the certificate

To rotate your certificate for external traffic:
1. Create a Kubernetes TLS secret using the new certificate.

kubectl create secret tls <secret-name> --key /home/kubo/certs/ca.key --cert /home/kubo/certs/ca.crt -n cert-manager

Verify the secret.
```
kubectl get secrets -n cert-manager
```

Create a cluster-issuer resource with the newly created secret.
Apply the following manifest with updated secret-name.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: <cluster-issuer-name>
  namespace: cert-manager
spec:
  ca:
    secretName: <secret-name>

In the /home/kubo/values.yaml file, replace the older cluster-issuer name with the newly created one. Also append the new certificate under trustedCAs section.
Delete the older cluster-issuer.

Delete the OIDC client.

kubectl -n tmc-local delete oidcclient client.oauth.pinniped.dev-auth-manager-pinniped-oidc-client; kubectl -n tmc-local delete secret client.oauth.pinniped.dev-auth-manager-pinniped-oidc-client-client-secret-generated

Re-install the tanzu-mission-control package with the appropriate version. This creates the secrets and certificates using the new cluster-issuer.

tanzu package installed list | grep tanzu-mission-control

tanzu package install tanzu-mission-control -p "tmc.tanzu.vmware.com" --version <version> --values-file /home/kubo/values.yaml --namespace tmc-local

Verify that the newly rotated certificate is copied over to /etc/ssl/certs.

Re-establish cluster connection after certificate rotation

Following a successful certificate rotation, the clusters and management clusters managed by TMC will be in a disconnected state. This is expected because your attached clusters and registered management clusters are still using the older certificates.

Use the following steps to re-connect your management clusters and re-attach your workload clusters. These commands push the new certificates down to the workload cluster or management cluster, allowing TMC Self-Managed to establish a successful TLS connection.

Restart the cluster-agent-service.

kubectl -n tmc-local rollout restart deploy cluster-agent-service-server

Re-register a management cluster.

tmc managementcluster reregister <management-cluster-name> --kubeconfig /home/kubo/.kube/config

Re-attach a cluster.

tmc cluster reattach -n <cluster-name> -k /home/kubo/.kube/config

After successful execution of the above steps, TMC Self-Managed re-connects with your workload clusters and management clusters.