Certificate Verification Error While Connecting to the NSX Manager

Precheck fails while activating NSX Network Detection and Response.

Problem

The logs of the nsx-ndr-precheck job contain the following error:

2022-10-04 19:43:44,954 - nsx_api_client.nsx_api_client - ERROR - communication error: HTTPSConnectionPool(host='external-nsx-manager', port=443): Max retries exceeded with url: /api/v1/licenses (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

In these logs, the specific issue manifests with the error string "CERTIFICATE_VERIFY_FAILED" in the connection to the host "external-nsx-manager".

To know more information about how to access these logs, refer step 4 b of NSX Network Detection and Response Activation Precheck Failed.

Cause

The problem occurs because of issues with the certificate setup for the NSX Manager cluster.

Solution

1. To resolve the issue, make sure that you have correctly configured the certificates for each node in the NSX Manager cluster and for the cluster itself. Specifically verify:
   • Each certificate has a common name that matches the fully qualified domain name it is used for.
   • No two certificates use the same common name.
2. To verify that the certificate setup is correct, here are the two suggested approaches.
   1. Using a web browser:

      With a web browser, visit each of the NSX Manager nodes individually using their fully qualified domain names, and also visit the cluster domain name. At each of these domains, use the browser tools and view the certificate information and in particular Common Name of the certificate. Confirm that the common name exactly matches the visited domain name in the URL bar of the browser.

   2. Using the Command Line Tools:
      In a shell, for one of the NSX Manager nodes, run the following commands:

      • Extract the trusted CA certificate from the Kubernetes platform.

      napp-k get secret/nsx-ndr-precheck-ccpchk-nsx-manager-truststore   -o
                        jsonpath='{.data.ca\.crt}'| base64 -d > /tmp/ca.crt

      The command extracts the trusted certificate authorities from the Kubernetes platform into the /tmp/ca.crt file.

      • View the subject of the trusted certificates.

      openssl storeutl -noout -certs --text /tmp/ca.crt |grep "Subject:" 
             Subject: C=US, ST=CA, L=Palo Alto,
                          O=VMware Inc., OU=NSX,   CN=k8s-platform-ca
             Subject: C=US, ST=CA, L=Palo Alto,
                          O=VMware Inc., OU=NSX,
                      CN=vmnsxt-mgmt-p01.example.com
             Subject: C=US, ST=CA, L=Palo Alto,
                          O=VMware Inc., OU=NSX,
                      CN=vmnsxt-mgmt-p02.example.com
             Subject: C=US, ST=CA, L=Palo Alto,
                          O=VMware Inc., OU=NSX,
                      CN=vmnsxt-mgmt-p03.example.com
             Subject: C=US, ST=CA, L=Palo Alto,
                          O=VMware Inc., OU=NSX,
                      CN=vmnsxt-mgmt-p01.example.com

      The command parses those certificates and prints out the subject of each certificate. The command also includes the common name of the certificate which is after "CN=" in the sample example.

      Verify that the common names shown are all different and match the expected fully qualified domain names for the nodes of the NSX Manager cluster and for the cluster itself.

      In the earlier example, the common name vmnsxt-mgmt-p01.example.com occurs twice. The sample setup is configured incorrectly. The fully qualified domain name for the NSX Manager cluster is vmnsxt-mgmt.example.com, but the certificate uses an incorrect common name vmnsxt-mgmt-p01.example.com.