Certificate rotation in Tanzu Mission Control Self-Managed (East-west traffic)

You might need to rotate certificates if the certificates expire or if the private key of the certificate is compromised.

Certificate rotation with cert-manager

We recommend using the following steps to manually rotate certificates that are provided with TMC Self-Managed.

Identify the certificate that needs to be rotated.
```
kubectl -n tmc-local get certificates -o wide
```
The status field in the output indicates the validity of the certificate.

Identify, from the certificate resource, the Kubernetes TLS secret name.

kubectl -n tmc-local get certificate <CERTIFICATE_RESOURCE_NAME> -oyaml | yq .spec.secretName

Delete the certificate and the Kubernetes TLS secret name.

kubectl -n tmc-local delete certificate <CERTIFICATE_RESOURCE_NAME>
kubectl -n tmc-local delete secret <KUBERNETES_TLS_SECRET_NAME_FROM_2>

Restart the pods that use the certificate.

For a mapping of the certificate to a deployment/pod, see Mapping certificate name to TMC SM pod/deployment/stateful-set.
```
kubectl -n tmc-local rollout restart deployment <DEPLOYMENT_NAME> 
```
To restart the stateful-set(eg. kafka-controller,postgresql)
```
kubectl -n tmc-local rollout restart statefulset <STATEFULSET_NAME>
```
Note
minio-provisioning is an init job hence it does not need to be restarted if its execution state is completed. Run the below command to verify the pod state:
```
kubectl -n tmc-local get pods | grep minio-provisioning
```

The restarted pods will be in container creating phase till the following are reconciled:

Kapp-controller reconciles the certificate resource from the TMC Self-Managed package.
cert-manager reconciles the certificate resource and creates the TLS secret.

Certificate rotation for custom certificates

A custom certificate is a certificate that is not provided by TMC Self-Managed. If you are using a custom certificate, use the following steps to manually perform the certificate-rotation.

Identify the certificate that needs to be rotated.

kubectl get certificate -n tmc-local -ojsonpath="{range .items[*]}{.metadata.name} not before: {.status.notBefore} not after: {.status.notAfter}{'\n'}{end}"

The status field in the output indicates the validity of the certificate.

Delete the Kubernetes TLS secret name.

kubectl -n tmc-local delete secret <KUBERNETES_TLS_SECRET_NAME>

Create a new Kubernetes TLS secret.

kubectl create secret tls <SECRET_NAME> \
--key < private key filename> \
--cert < certificate filename>

Alternatively, if you already have an existing secret file:

kubectl create -f my-secret.yaml

Verify that the secret is created.
```
kubectl -n tmc-local get secrets
```
Restart the pods that use the secret.

For a mapping of the certificate to a deployment/pod, see Mapping certificate name to TMC SM pod/deployment/stateful-set.
```
kubectl -n tmc-local rollout restart deployment <DEPLOYMENT_NAME>
```
To restart the stateful-set(eg. kafka-controller,postgresql)
```
kubectl -n tmc-local rollout restart statefulset <STATEFULSET_NAME>
```
Note
minio-provisioning is an init job hence it does not need to be restarted if its execution is completed. Run the below command to verify the pod state:
```
kubectl -n tmc-local get pods | grep minio-provisioning
```

Mapping certificate name to TMC SM pod/deployment/stateful-set

Deployment Name	Secret Name
account-manager-server	account-manager-server-tls
agent-gateway-server	agent-gateway-instack-tls
api-gateway-server	api-gateway-instack-tls
audit-service-consumer	audit-consumer-tls
audit-service-server	audit-server-tls
auth-manager-server	server-tls
authentication-server	authentication-tls
cluster-agent-service-server	cluster-agent-service-server-tls
cluster-config-server	cluster-config-server-tls
cluster-object-service-server	cluster-object-service-server-tls
cluster-reaper-server	cluster-reaper-server-tls
cluster-secret-server	cluster-secret-server-tls
cluster-service-server	cluster-service-server-tls
cluster-sync-egest	cluster-sync-egest-tls
cluster-sync-ingest	cluster-sync-ingest-tls
dataprotection-server	dataprotection-server-tls
events-service-consumer	events-service-consumer-tls
events-service-server	events-service-server-tls
fanout-service-server	fanout-service-server-tls
feature-flag-service-server	feature-flag-service-tls
helm-deployment-server	helm-deployment-server-tls
inspection-server	inspection-server-tls
intent-server	intent-tls
kafka-exporter	kafka-exporter-cert
kafka-topic-controller	dev-ca-key
landing-service-server	landing-service-tls
minio-provisioning	minio-api-user-creds
onboarding-service-server	onboarding-service-server-tls
package-deployment-server	package-deployment-server-tls
policy-engine-server	policy-engine-server-tls
policy-insights-server	policy-insights-server-tls
policy-sync-service-server	policy-sync-service-server-tls
policy-view-service-server	policy-view-service-server-tls
provisioner-service-server	provisioner-service-server-tls
resource-manager-server	resource-manager-server-tls
telemetry-event-service-consumer	api-consumer-tls
tenancy-service-server	tenancy-service-tls
ui-server	ui-server-tls
wcm-server	wcm-server-tls

Stateful-set Name	Secret Name
kafka-controller	kafka-broker-cert
postgres-postgresql	postgres-tls

Certificate rotation in Tanzu Mission Control Self-Managed (North-South traffic)

The document summarizes the steps the customers need follow to rotate the cert-manager certificate and followup tasks to ensure the TMC managed clusters are attached to the management plane.

Note
: Applicable if Cert-manager is used to manage the certificates & secrets used to authenticate North-South traffic to the cluster.

When to rotate the certificate?

When the private key used by cert-manager to sign the certificates is compromised or the certificate is expiring soon.

Rotate certificate

Follow the below guidelines: 1. Create a kubernetes TLS secret using the new certificate.

kubectl create secret tls <secret-name> --key /home/kubo/certs/ca.key --cert /home/kubo/certs/ca.crt -n cert-manager

Verify:

 kubectl get secrets -n cert-manager

Create a cluster-issuer resource with the newly created secret from the above step. Apply the following manifest with updated secret-name.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: <cluster-issuer-name>
  namespace: cert-manager
spec:
  ca:
    secretName: <secret-name>

Replace in /home/kubo/values.yaml file older cluster-issuer name with a newly created one. Also append the new certificate under trustedCAs section.
Delete the older cluster-issuer.

Delete the oidc client.

kubectl -n tmc-local delete oidcclient client.oauth.pinniped.dev-auth-manager-pinniped-oidc-client; kubectl -n tmc-local delete secret client.oauth.pinniped.dev-auth-manager-pinniped-oidc-client-client-secret-generated

Re-Install the TMC package with the available version. This will create the secrets and certificates using the new cluster-issuer.

tanzu package installed list | grep tanzu-mission-control

tanzu package install tanzu-mission-control -p "tmc.tanzu.vmware.com" --version <version> --values-file /home/kubo/values.yaml --namespace tmc-local

Finally, ensure the newly rotated certificate is copied over to /etc/ssl/certs.

Post Certificate rotation

Following a successful certificate rotation the clusters and management clusters under TMC management will be in a disconnected state. This is expected. Follow the steps needed to reattach the management-cluster and workload cluster.

Restart the cluster-agent-service.

kubectl -n tmc-local rollout restart deploy cluster-agent-service-server

Re-register management cluster.

tmc managementcluster reregister <management-cluster-name> --kubeconfig /home/kubo/.kube/config

Reattach cluster.

tmc cluster reattach -n <cluster-name> -k /home/kubo/.kube/config

After successful execution of the above steps the TMC would have established the connection with both the workload-cluster amd management-cluster. The above commands push the new certificates down to the cluster & management-cluster that were still using the older certificates hence allowing the TMC to establish a successful TLS connection.