Troubleshoot a Cluster with Different Roots of Trust in the Global Namespace

The clusters in a global namespace can use different roots of trust after one of the clusters is assigned a different Vault certificate authority (CA) integration. The use of different roots of trust in the global namespace breaks cross-cluster connectivity and causes CA-related problems in the cluster.

Problem

The use of different roots of trust for the clusters in a global namespace causes certificate authority (CA) errors, such as ca request failed, with the pods in a cluster's istio-system namespace. Also, requests to or from the pods in the cluster's namespaces included for Istio sidecar njection will fail with errors caused by failed mTLS connections.

Cause

Changing the Vault CA integration for a cluster in the global namespace can cause that cluster and the other clusters in the global namespace to use different roots of trust. In this case, the certificates issued by the old CA will still be present in the workloads because they won't have expired, and this will cause the noted problems.

Solution

1. To remedy the errors with the pods in the istio-system namespace, run the following commands.

   kubectl delete secret istiod-tls -n istio-system
   kubectl delete pod -l app=cert-manager-istio-csrkubectl rollout restart deploy -n istio-system

2. To remedy the problems with the pods in the namespaces that are included for Istio sidecar injection, do one of the following:

   • Drain the namespaces before changing the CA integration for the cluster. Once the CA is changed, re-deploy all the pods.

   • After you perform step 1, restart all the pods.