External Certificate Authority Design and Deployment on an On-Premises Orchestrator

Note: This guide is intended as a supplement to the External Certificate Authority documentation provided in the Operator Guide, which describes how to configure External CA in the Orchestrator.

The Use of Certificates in VMware SD-WAN

Device certificates are issued to SD-WAN Edges and Gateways and bind a public encryption key to an entity. They are used to establish trust between devices by verifying the authenticity of the public key presented by each device.

In VMware SD-WAN, certificates are used for mutual authentication during the establishment of VCMP overlay tunnels between SD-WAN endpoints. For example, Edge-to-Gateway, Edge-to-Hub, and Edge-to-Edge tunnels. This protects both data plane and management plane traffic traversing the SD-WAN overlay.

To protect management plane traffic, certificates are also inspected by the Orchestrator to authenticate Edge and Gateway client API calls. In VMware’s hosted deployment model, the VMware Edge Cloud Orchestrator has an integrated Certificate Authority that is used to generate and renew device certificates. However, customers may require that device certificates be generated and maintained using their own CA. An on-prem Orchestrator supports integration with an external CA to address these requirements. The Orchestrator will then act as a pass-through device between the SD-WAN devices and the External CA. Note that integration with an external CA is only supported for Orchestrators deployed on-premises, and not when using VMware’s hosted Orchestrators. The private key must be under VMware control to meet security and compliance requirements.

Expected Functions of an External CA

There are three functions that an External CA is expected to provide:

Provide a rooted chain of trust in the form of one or more PEM-encoded certificates.
This chain of certificates is distributed by the Orchestrator to Edges and Gateways to allow for mutual, certificate based authentication during the establishment of SD-WAN overlay tunnels.
The chain of trust is installed on the Orchestrator API Gateway (NGINX) allowing the Gateway to authenticate API calls from Edges and Gateways. All certificates in the chain must have:
1. Basic Constraint: CA: TRUE
2. Key Usage: CRITICAL
3. Include CRL signing
4. Certificate Signing
The chain must be continuous from a leaf certificate to a self-signed root certificate. The chain need not be rooted by a well-known public issuer, the root can be a private self-signed certificate.
Provide signed certificate revocation lists (CRLs) to the Orchestrator.
CRLs generated by the customer’s external CA are distributed by the Orchestrator to Edges and Gateways. Edges and Gateways do not communicate directly with the external CA. Also, the Orchestrator may be configured to automatically fetch CRLs from a CRL distribution point, or CRLs may be manually updated to the Orchestrator.

Figure 1. Configure Orchestrator polling when creating an External CA
Generate certificates on request for SD-WAN Gateways and Edges in response to CSRs generated by those devices.
Edges and Gateways generate RSA key pairs and CSRs when certificate enrollment or renewal is requested. The Orchestrator receives the CSR via the device heartbeat. From the Orchestrator the CSR can be delivered by several mechanisms to the customer CA, receiving a signed certificate in return.

Certificate Types and Requirements

There are two types of certificates, one for Edges, and the second for Gateways. Only the Common Name (CN) is reserved for VMware SD-WAN use.

The Edge Certificate:

It is composed of the literal string gateway concatenated with no spaces to the Gateway Logical-ID.
The Gateway Logical-ID is a globally unique identifier the Orchestrator generates to identify the Gateway.
Example: Subject: CN = gateway6f9589ca-7946-4b85-a2c2-be6e03f8a4c5

Only one CN can be defined. Other fields in the Distinguished Name (DN) are free for use by the external CA. For help in obtaining the Edge and Gateway Logical-IDs, see the section below “Locating Logical-IDs".

Gateway and Edge Certificate Requirements

2048-bit or 4096-bit RSA keys
X509v3 Extended Key Usage: Web Server Authentication, TLS Web Client Authentication, Key Usage = TLS Web Server & TLS Web Client
Certificates must be valid as of the time they are distributed to the Edge or Gateway. The validFrom time must be the current time or earlier.
Certificate lifetime is delegated to the external CA according to established customer policy.
- The Orchestrator automatically renews certificates when a configurable fraction of their lifetime has passed.
  - The default value is 33%, meaning that when 1/3 of the certificate’s lifetime has passed, the Orchestrator will request that the Edge or Gateway generate a new CSR to trigger the generation of a new certificate.
  - The lifetime fraction for renewal is configurable on the Orchestrator.
  - Renewal can also be windowed so that certificate generation is only done within a defined maintenance window.
Figure 2. Configure Certificate Lifetime for all Edges

Figure 3. Configure Certificate Lifetime for all Gateways

Example: Certificate and CSR Subject Names (DNs)

The Edge and Gateway, once activated, build a fixed CSR subject name, with just the CN attribute defined:

Edge:
- Subject: CN = VC11130031
Gateway:
- Subject: CN = 00:50:56:81:7e:2d

Integration with an external CA requires flexibility in both the generation of CSR subject names and in the handling of certificate subject names by both the Management Plane and the Data Plane.

Note: In generating certificates, the external CA can appropriate the O and OU fields for their own purposes.

Locating Logical IDs for use in Certificate Creation

The logical IDs used in certificate generation from CSRs may be obtained from the following locations:

Edge Logical ID
In the Orchestrator, the Edge Logical-ID may be found in the “Logical ID” column by going to “Monitor > Edges”. If it is not displayed, click on “Columns” at the bottom of the screen and select it.

Figure 4. Locating the Edge Logical ID
Gateway Logical ID
The Gateway Logical ID may be found on the Orchestrator UI by logging in as an Operator level user and navigating to Gateway Management > Gateways. Select the desired Gateway, then click the drop-down next to the Gateway name to obtain details about the Gateway, including its logical ID.

Figure 5. Locating the Gateway Logical ID

Orchestrator to External CA Communication

Two integration mechanisms are supported on the Orchestrator:

Synchronous API-based integration with an EJBCA CA.
Manual Mode, where transmission of CSRs and certificates are performed by users or by middleware using Orchestrator API calls.

1. Expose a rooted chain of trust in the form of one or more PEM-encoded certificates.

The PEM-encoded chain can be manually configured on the Orchestrator at the time of external CA definition on the Orchestrator, in the Orchestrator UI. The chain can be pulled from the External CA, or from customer middleware by API at the time of External CA definition on the Orchestrator.

2. Provision of signed certificate revocation lists.

CRLs may be manually pushed to the Orchestrator via API, manually uploaded to the Orchestrator from the Orchestrator UI, or pulled by the Orchestrator from a CRL distribution point on a configured schedule. The CRL distribution point is defined by an HTTP URL visible to the Orchestrator, and is configured when the CA is defined.

3. Certificate Enrollment

Certificate enrollment can be manual:

The CSR is downloaded by a customer administrator from the Orchestrator UI and delivered to the external CA for certificate generation.
- While logged in as an Operator-level user, in the Orchestrator UI, navigate to Certificate Authorities > Certificates Summary. Here, you can click on DOWNLOAD EDGE CSRs or DOWNLOAD GATEWAY CSRs.
Figure 6. Manual CSR Download for Edge or Gateway
- These CSRs may then be used to manually create a certificate in the external CA using the criteria described above.
The certificate generated by the eCA from the CSR is uploaded to the Orchestrator. It is then automatically distributed by the Orchestrator to the Edge or Gateway.
This is done in the UI by navigating to Certificate Authorities > Certificates Summary and selecting IMPORT CERTIFICATES.

Figure 7. Import Enrolledge Certificates
The certificates currently assigned by the Orchestrator to Edges and Gateways may be viewed and downloaded by navigating to Certificate Authorities > Certificates and selecting View Certificate next to the desired Edge or Gateway.
Key details of the issued certificate are displayed on that screen.

Figure 8. Certificate Detail for an Edge
Clicking Copy will copy the raw text of the certificate to the clipboard, if desired.

Certificate enrollment can be automated by customer middleware:

Middleware pulls CSR(s) by API.
Middleware then pushes certificates back by API.

Certificate enrollment can be automated by VMware custom integration with common 3rd party Certificate Authorities:

The Orchestrator pushes the CSR to an external CA using 3rd party API integration.
The Orchestrator receives a certificate on the return of API call.
PrimeKey EJBCA Enterprise edition is currently the only ‘out of the box’ integration supported. Microsoft CA Server integration is currently under consideration.

Certificate enrollment can also be automated by a custom integration developed in conjunction with VMware Professional Services.

External CA Modes

Manual Mode

The Orchestrator presents a UI allowing the user to poll/request CSR, install root CA's certificate and device certificate, and CRLs. This process is very similar to the Middleware mode using Southbound API, but instead of APIs, users interface with Orchestrator through UI.

Automated Mode

The Orchestrator integrates directly with an external CA (EJBCA, MS CA, etc) through REST APIs for certificate request, renewal, and revocation.

Middleware (Asynchronous) Mode

The Orchestrator integrates with an automation platform (for example, Cisco NSO), and the automation platform will interface with external CA and Orchestrator for certificate request, renewal, and revocation. There are Northbound and SouthBound APIs for integration with middleware and the function varies depending on whether northbound or southbound APIs are used.

Southbound API - This is where the Orchestrator receives REST APIs from the Middleware system for the certificate operations (CSR, issue, renewal, and so forth). The Middleware system in this case will be an automation engine where the device provisioning and configuration is done, and this includes certificate operations. Citi-group is the customer that uses this process.

Figure 11. Middleware Certificate Process with External CA (Southbound API based)
NorthBound API - This is where the Orchestrator initiates all certificate operations and uses a Middleware system as a proxy and protocol translation to the External CA server. An example of why a customer would use this method is the External CA does not support REST API and requires the Middleware system for protocol translation from the REST API (from the Orchestrator) to something that an external CA can support.

Troubleshooting

This section offers guidance on how to resolve common issues experienced when deploying and operating an external CA.

Edges deployed in High Availability (HA)

When two Edges are deployed in HA, there will be two certificates shown under a single Edge in Certificate Authority > Certificates > Latest Edge Certificates. In manual mode, a separate certificate must be generated by the external CA using the CSRs for the Active and Standby Edges.

Edges starting from the 5.1.0 version can enable HA as they are compatible with External CA. Edges running version 5.0.x or earlier software are not supported with external CA.

Connection errors due to certificate mismatches

If an Edge or Gateway fails to connect, it may be the result of a certificate mismatch. The first step is to check if the Edge or Gateway has Authentication Mode set to Certificate Acquire. A certificate mismatch can be indicated by logging into the CLI of the Edge or Gateway and viewing the output of the mgd.log file using the sudo tail –f /var/log/mgd.log command.

The certificates that have been pushed to the Edge and Gateway can also be decoded using the openssl utility in Linux and compared to the certificate in the Orchestrator. These certificates, as well as PKI keys derived from them, are stored on the Edge and Gateway in /etc/vc_private and /etc/vc_public.

Utility: debug.py --pki

The debug.py –pki utility may be used from the Edge or Gateway command line interface to view details about PKI settings, the trusted CA list, and the certificate:

Utility: debug.py --ike_sa

The debug.py --ike_sa utility can be used to show the details of current security associations. In the example below, the three existing security associations correspond to the Orchestrator and Primary and Secondary Gateways, respectively:

Utility: debug.py --ike_spd

Details of security protocols and encryption methods used per peer can be found using the debug.py --ike_spd utility:

Decoding Certificates using OpenSSL

The details and contents of certificates may be viewed by using the Linux openssl tool. It will show you the details of the certificate like Version, Serial Number, Signature Algorithm, Issuer, Validity, Subject, Public Key Algorithm and many more.

Some examples:

The full certificate may be seen by using the following command, where cert.pem is the file name of the certificate to be parsed out:
- openssl x509 -text -noout -in cert.pem

Specific fields within the certificate may be verified by piping the command to grep. For example:

openssl x509 -text -noout -in cert.pem | grep -a1 Basic
openssl x509 -text -noout -in ICA_from_Acumen.pem | grep -a1 "Subject Key"
openssl x509 -text -noout -in ICA_from_Acumen.pem | grep -a1 "Authority Key"
openssl x509 -text -noout -in ICA_from_Acumen.pem | grep -a1 X509v3

Certificate Authority Renewal

In the event of a CA compromise or the normal renewal of a CA, the Orchestrator includes the ability to renew the External CA. This is done by using the + Add CA function.

When the new CA is added the user has the option to retain the existing CA for continuity purposes for a period of time. When configuring a new CA, Edges and Gateways will be provisioned certificates from the newer CA and, depending upon the renewal interval, all Edges and Gateways will have certificates from the new CA. At this point the original CA can be decommissioned from the CA Summary section of the Orchestrator UI.