If your organization uses Vault as an internal certificate authority (CA) and you want to use certificates signed by Vault CA for secure mTLS communication between services within your service mesh, create an integration with Vault CA in Tanzu Service Mesh.

When your organization uses Vault as an internal CA, it can have one or multiple Vault servers that store a trusted root certificate and one or more intermediate certificates. Vault uses these CA certificates to generate and sign mTLS session certificates for authentication and encryption of services within the service mesh.

Every time a service needs an mTLS session certificate for secure communication with other services in the service mesh, the service sends a request to a Vault server from its Vault-connected cluster. Vault generates a certificate, signs it, and sends it back to the cluster. The service then uses the certificate to make mTLS authenticated and encrypted connections to other services.

If you want to use Vault CA for your service mesh, you must create a Vault integration in Tanzu Service Mesh. The integration defines a configuration that points to a Vault server. The Vault CA configuration contains such details as the URL of the Vault server, the endpoint on Vault to use for signing certificates, and the reference to the Vault server's bundle that contains the root certificate and the intermediate certificates.

After you create a Vault CA integration, you need to apply the CA label from the integration to a cluster when you onboard it. During onboarding, the Vault CA configuration is pushed to the services in the cluster, and the root and intermediate certificates are uploaded into the cluster's trust store.

Once the Vault CA configuration is applied to a cluster, the cluster is considered to be connected to the Vault server, and the services can send certificate generation and signing requests to the server. Because the cluster has the Vault server's root and intermediate certificates in its trust store, it trusts mTLS session certificates received from Vault CA.

You can view the CA health status for each connected cluster in the Tanzu Service Mesh UI.

Note:

You can also create a Vault CA integration by using the Tanzu Service Mesh API. For more information, see Create a Vault CA Integration in the Tanzu Service Mesh API Programming Guide.

Prerequisites

  • Access the Tanzu Service Mesh Console.

  • Verify that you are familiar with Vault concepts, such as PKI secrets engine, mount path, namespace, role, and policy. For information about these Vault concepts, see the HashiCorp Vault documentation.

  • Verify that you are familiar with the Kubernetes concept service account. For more information about service accounts, see the Kubernetes documentation.

  • Configure Vault to trust connections from your Kubernetes clusters. See step 1 in the following procedure.

Procedure

  1. To configure Vault to trust connections from your clusters and accept certificate generation and signing requests from them, perform the following steps for each cluster on Vault.
    1. In your terminal, run the following commands to set environment variables for the Vault server, the Vault namespace, and the Vault authentication token to access the Vault API. 
      export VAULT_ADDR="vault_server_URL:8200"; export VAULT_NAMESPACE="namespace"export VAULT_TOKEN=TOKEN
    2. Set up a PKI secrets engine. For example, run these commands to set up a PKI secrets engine. 
      vault secrets enable pki
 
# Create the root certificate
vault write -field=certificate pki/root/generate/internal common_name="svc" ttl=8760h > CA_cert.crt
 
vault write pki/config/urls issuing_certificates="$VAULT_ADDR/v1/pki/ca" \
  crl_distribution_points="$VAULT_ADDR/v1/pki/crl"
 
vault secrets enable -path=pki_int pki
 
vault write -format=json pki_int/intermediate/generate/internal common_name="svc Intermediate Authority" | jq -r '.data.csr' > pki_intermediate.csr
 
# This is the intermediate certificate
vault write -format=json pki/root/sign-intermediate \
  csr=@pki_intermediate.csr \
  format=pem_bundle ttl=4380h | jq -r '.data.certificate' > intermediate.cert.pem
 
vault write pki_int/intermediate/set-signed [email protected]
 
vault policy write policy_name - <<EOF
path "policy_name*" { capabilities = ["read", "list"] }
path "policy_name/sign/example-dot-com" { capabilities = ["create", "update"] }
path "policy_name/issue/example-dot-com" { capabilities = ["create"] }
EOF
 
vault write policy_name/roles/example-dot-com \
allowed_domains="svc" allow_subdomains=true \
max_ttl="720h" require_cn=false \
allowed_uri_sans="spiffe://domain.name/*"

      The last command is used to configure a signing role on Vault. Provide the following values:

      • allowed_domains. Specifies the domains that this role is allowed to issue certificates for. Set allowed domains to svc.

      • allow_subdomains. Specifies if clients can request certificates with common names (CNs) that are subdomains of the CNs allowed by the other role options. Set allow_subdomains to true.

      • require_cn. Set to false. This makes the common_name field optional during generation of a certificate.

      • allowed_uri_sans. Defines allowed Subject Alternative Name (SAN) URIs. Set allowed_uri_sans to the trust domain configured for the project that you use in Tanzu Service Mesh. For example, if you use the default project in Tanzu Service Mesh, set allowed_uri_sans to cluster.local.

    3. Set the SA_CA_CRT environment variable to the cluster’s client certificate. 
      export SA_CA_CRT=$(kubectl get secret -n default service-account-name -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
    4. Set the K8S_HOST environment variable to the URL of the Kubernetes API server. 
      export K8S_HOST=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")
    5. Set the CLUSTER_NAME environment variable to the name of the cluster and enable the Kubernetes authentication method. 
      export CLUSTER_NAME=cluster_name
vault auth enable --path=$CLUSTER_NAME kubernetes

      Make sure that the cluster name is unique.

    6. Use the /config endpoint to configure Vault to communicate with the Kubernetes cluster. 
      vault write auth/$CLUSTER_NAME/config \
  kubernetes_host="$K8S_HOST" \
  kubernetes_ca_cert="$SA_CA_CRT"

      This command imports the cluster’s client certificate to Vault so that connections from the cluster can be trusted by Vault.

    7. Create a named role. 
      vault write auth/$CLUSTER_NAME/role/role_name \
  bound_service_account_names=service_account_name \
  bound_service_account_namespaces=istio-system \
  policies=policy_name \
  ttl=1h

      bound_service_account_names is the name of the Kubernetes service account on the cluster that has access to the role. The role will be used to associate the service account with the set of Vault policies specified in policies within the role configuration.

      bound_service_account_namespaces is the namespace that is allowed to access the role. Set bound_service_account_namespaces to istio-system.

      Tanzu Service Mesh will create the specified service account in the istio-system namespace on the cluster during onboarding. When sending requests to Vault, a service will authenticate to Vault as the specified service account from the istio-system namespace.

      policies is the policy that gives read access to the paths of the PKI secrets engine paths. This references the name of the policy created in step b.

      Here is an example of a role.

      vault write auth/$CLUSTER_NAME/role/pki_int \
  bound_service_account_names=vault-issuer \
  bound_service_account_namespaces=istio-system \
  policies=pki_int \
  ttl=1h
  2. In the navigation pane on the left, click Admin > Integration.
  3. On the Integrations page, locate the Vault card.
  4. Select one of the following options:

    • If you are creating the first Vault integration account, click Configure at the bottom of the card.

    • If one or more Vault integration accounts exist and you are creating another account, click Add Account at the bottom of the card.

  5. In the New Vault Integration dialog box, provide the following information.

    • Name. Enter a name for the account.

    • Description. (Optional) Give a brief description of the account.

    • Label. The label associated with this Vault CA integration account. A Vault CA label is created automatically with the syntax Certificate Authority: Name of the Vault account. You can use this generated label or provide your own label. Click the automatically generated label, click New Value, type your label, click Add new below, and click Save.

    • URL. The URL of the Vault server accessible from the Kubernetes cluster, for example: https://vault-cluster-public-vault.z1.hashicorp.cloud:8200.

    • Service Account Name. The name of the Kubernetes service account that services will use to authenticate to the Vault server when sending requests from the cluster. An example of the service account name is vault-issuer. This value must be the service account name that you provided in step 1 g.

    • Role. The name of the Vault role, for example, pki_int. This value must be the name of the role that you created in step 1 g.

    • PKI Path. The Vault path that will be used for signing certificates. This path must use the sign endpoint and must match the path that you specified in the vault secrets enable command when you were creating a PKI secrets engine (see step 1 b). An example of PKI Path is pki-int/sign/example-dot-com.

    • Mount Path. (Optional) The Vault mount path to use when authenticating with Vault. If you specified the cluster name as the value of path when enabling the Kubernetes authentication method in step 1 e, use the format /v1/auth/my_cluster_name  to provide the mount path. If you leave the Mount Path box blank, the default mount path of /v1/auth/kubernetes will be used.

    • Namespace. (Optional) The namespace to use in Vault. Provide this value only if you use Vault Enterprise.

    • Certificate Authority Chain. (Optional) The CA bundle file that contains the root certificate and intermediate certificates that Vault uses to sign certificates. Your clusters will validate certificates issued by Vault CA against the root and intermediate certificates in the CA bundle. Provide the CA bundle file only if your organization uses a self-signed root certificate issued by an internal CA and the Vault server uses an HTTPS URL. If you use a root certificate issued by a publicly trusted CA, you don't need to provide a CA bundle.

    Select the name associated with the previously uploaded CA bundle. If you have not uploaded the CA bundle to Tanzu Service Mesh yet, perform the following steps.

    Note:

    The CA bundle file can contain a maximum of three certificates, including only one root certificate and a maximum of two intermediate certificates.

    1. Click New Certificate Chain.

    2. In the New Certificate Authority Chain dialog box, provide a name and optionally a description for the CA bundle.

    3. Next to Certificate Chain, click Select .PEM File and browse to the CA bundle file that you want to upload.

    The root and intermediate certificates in the CA bundle will be imported into the cluster’s trust store during onboarding.

  6. Click Save.
  7. In the Trust Domain window, enter the trust domain name.

    The trust domain corresponds to the trust root of a system and is part of a workload identity. A trust domain is required to onboard new clusters and create global namespaces.

    Important:

    The trust domain must exactly match the allowed_uri_sans value that you provided when you were creating a PKI secrets engine in step 1 b.

Results

The new account is added to the Vault integration card on the Integrations page. To view the details of the new Vault integration account and of the other existing integration accounts on the integration accounts details page, click View Details on the Vault integration card.

You can view the CA health status for the clusters connected to the Vault server specified in the integration account. A CA health status of CA: Connected means that the services in the cluster can successfully have certificates generated and signed by Vault CA. A CA health status of CA: Unknown means that the cluster is disconnected from the Vault server, and the services in the cluster cannot send certificate generation and signing requests to the server.

To view the CA health status of the connected clusters:

  • On the Global Namespaces page, click the name of the global namespace that you created. The global namespace details page displays the summary information about the global namespace, including its overall health state. The overall status of Healthy means that the configuration of the global namespace is synced and applied to the clusters that make up the global namespace. Verify that the CA: Connected status is displayed for each cluster within the global namespace by clicking the Healthy drop-down menu at the top of the page.

    To access the Global Namespaces page, in the navigation pane on the left, select Inventory > Global Namespaces.

  • On the Clusters tab, click the name of the cluster from the displayed table. The cluster details page displays the summary information about the cluster, including its overall health state. Verify that the CA: Connected status is displayed for the selected cluster by clicking the Healthy drop-down menu at the top of the page.

    To access the Clusters tab, in the navigation pane on the left, select Inventory > Clusters & Nodes and select the Clusters tab.

What to do next

  • To apply the Vault CA configuration from the integration account to a cluster so that the services deployed in the cluster can send certificate generation and signing requests to the Vault server, select the CA label from the integration account when you onboard the cluster (see step 7).

  • To edit a Vault integration account, click Edit in the Vault card. If you have more than one Vault account, in the lower-left corner of the card, click <number> Accounts, click the name of the account, and then click Edit. You can edit the following details:

    • Description. The description of the integration account in Tanzu Service Mesh.

    • Service Account Name. The name of the Kubernetes service account that services will use to authenticate to the Vault server when sending requests from the cluster.

Important:

Before changing the service account name in the integration account, change the service account name in the role configuration in Vault. See step 1 g.

  • To delete a Vault integration, click Delete. To confirm the deletion, click Confirm. If any clusters use the Vault CA configuration from the account, a warning is displayed. Select I understand the consequences of this action and click Confirm.

Warning:

If you delete a Vault CA integration, and the associated CA issued certificates for the service pods on a cluster, the pods will continue to have the certificates until they expire. When the certificates expire, mTLS connectivity between the services will fail.

Also, if a pod gets deleted, it won’t be automatically recreated because the CA was removed, and no new certificates can be created.

In this case, you will need to connect another CA to the cluster and restart all the pods so that they can start using the new CA’s certificates.

Note:

You can also edit or delete an integration account from the integration account details page.

  1. On the Vault card, click View Details.

  2. On the integration details page, in the section for the account, click Edit Configuration or click More Actions and then click Delete Account.