Configuring and Managing Multisite Deployments in Your VMware Cloud Director

To manage and monitor multiple, geographically distributed VMware Cloud Director installations or server groups and their organizations as single entities, you can use the VMware Cloud Director multisite feature.

Effective Implementation of a Multisite

When you associate two VMware Cloud Director sites, you enable the administration of the sites as a single entity. You also enable organizations at those sites to form associations with each other. See Create a Site Association in Your VMware Cloud Director. When an organization is a member of an association, organization users can use the VMware Cloud Director Tenant Portal to access organization assets at any member site, although each member organization and its assets are local to the site it occupies.

Note:

The sites must be with the same VMware Cloud Director API version, or one major version apart. For example, you can associate a VMware Cloud Director 10.1 (API version 34.0) site with a VMware Cloud Director site version 10.0, 10.1, 10.2 or 10.2.2, respectively API versions 33.0, 34.0, 35.0, or 35.2.

After you associate two sites, you can use the VMware Cloud Director API or the VMware Cloud Director Tenant Portal to associate organizations that occupy those sites. See the VMware Cloud Director API Programming Guide or the Configure and Manage Multisite Deployments topic in the VMware Cloud Director Sub-Provider and Tenant Guide.

A site or organization can form an unlimited number of associations with a peer, but each association includes exactly two members. Each site or organization must have its own private key. Association members establish a trust relationship by exchanging public keys, which are used to verify signed requests from one member to another.

Each site in an association is defined by the scope of a VMware Cloud Director server group (a group of servers that share a VMware Cloud Director database). Each organization in an association occupies a single site. The organization administrator controls access by organization users and groups to assets at each member site.

Site Objects and Site Associations

The installation or upgrade process creates a Site object that represents the local VMware Cloud Director server group. A system administrator whose authority extends to more than one VMware Cloud Director server group can configure those server groups as an association of VMware Cloud Director sites.

Site Names

Each Site object is created with a name attribute that is a UUID.

GET https://Site-B.example.com/api/site
...
<Site name="b5920690-fe13-4c31-8e23-9e86005e7a7b" ...>
   ...
   <RestEndpoint>https://Site-A.example.com/api/org/Org-A</RestEndpoint>
   <RestEndpointCertificate>-----BEGIN CERTIFICATE-----
      MIIDDTCCAfWgAwIBAgI...Ix0eSE= -----END CERTIFICATE-----
   </RestEndpointCertificate>
   ...
</Site>

While there is no requirement that the site name matches the hostname in the API endpoint, a system administrator can update the site name as an administrative convenience for VMware Cloud Director API users, with a request like this one:

PUT https://Site-B.example.com/api/site
content-type: application/vnd.vmware.vcloud.site+xml
...
<Site name="Site-B" ...>
   ...
   <RestEndpoint>https://Site-A.example.com/api/org/Org-A</RestEndpoint>
   <RestEndpointCertificate>-----BEGIN CERTIFICATE-----
      MIIDDTCCAfWgAwIBAgI...Ix0eSE= -----END CERTIFICATE-----
   </RestEndpointCertificate>
   ...
</Site>

The Site element in the request body must retain the formatting in which it was returned by the GET .../site request. Additional characters, particularly carriage-returns, line feeds, or spaces, before or after the certificates can cause the system to return a bad request exception.

Associations of Organizations

After site association is complete, organization administrators at any member site can begin associating their organizations.

Note: You cannot associate a System organization with a tenant organization. The System organization at any site can be associated only with the System organization at another site.

Authorization Headers and Request Fanout

The Session response to a successful login request includes an X-VMWARE-VCLOUD-ACCESS-TOKEN header whose value is an encoded key that you can use, and the value of the X-VMWARE-VCLOUD-TOKEN-TYPE header, to construct a JWT Authorization header to include in subsequent requests in place of the deprecated x-vcloud-authorization header, which does not authenticate you to association members. See Create a VMware Cloud Director API Session for more information about logging in to the VMware Cloud Director API.

You can make requests that fan out to multiple association members by specifying the multisite=value pair in the Accept header. If you want the request to fan out, the value can be global or a colon-separated list of location IDs. For information about obtaining the location IDs, see Authorized Locations. When you set the value to local, the request does not fan out but includes multisite metadata included on fan-out.

For example, when you make a request such as a query that retrieves lists of resources from an association of organizations, you can specify the multisite=global pair in the Accept header. By specifying the multisite=global pair, you fan out the request to all association members and return an aggregated list.

Accept: application/*;version=30.0;multisite=global

You can specify a colon-separated list of location IDs, for example, multisite=ID-a:ID-b:ID-x. Unless you include this value in the Accept header, the request returns only those resources owned by the organization that is the target of the request. Unless you are making a request to the same organization that you authenticated to, you must also include a X-VMWARE-VCLOUD-AUTH-CONTEXT header that specifies the name of the organization that will fulfill your request.

When an authentication request includes the multisite= value pair, the response includes Link elements if the request failed at any association member. The category of the failure is represented by the rel value of the link.

rel="fanout:failed": Member status was ACTIVE but authentication at the member failed for some other reason.
rel="fanout:skipped": Authentication at the member was skipped because the association status was ASYMMETRIC or UNREACHABLE.

The failed or skipped request URL is in the href the value of the Link.

Authorized Locations

When you authenticate to a site that is a member of an association, the Session response includes an AuthorizedLocations element that provides VMware Cloud Director API and VMware Cloud Director Tenant Portal endpoints for other association members. In this example:

Site-A.example.com and Site-B.example.com are associated.
The user logs in to Site-A as a system administrator.

Request:

POST https://Site-A.example.com/api/sessions
Authorization: Basic ...
Accept: application/*;version=30.0
...

Response:

200 OK
...
X-VMWARE-VCLOUD-ACCESS-TOKEN: eyJhbGciOiJSUzI1NiJ9....Rn4Xw
X-VMWARE-VCLOUD-TOKEN-TYPE: Bearer
Content-Type: application/vnd.vmware.vcloud.session+xml;version=30.0;multisite=global
...
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Session ...
   ...  
   <AuthorizedLocations>
      <Location>
         <LocationId>a93c9db9-7471-3192-8d09-a8f7eeda85f9@9a41...
         </LocationId>
         <SiteName>Site-A</SiteName>
         <OrgName>System</OrgName>
         <RestApiEndpoint>https://site-a.example.com
         </RestApiEndpoint>
         <UIEndpoint>https://site-a.example.com
         </UIEndpoint>
         <AuthContext>System</AuthContext>
      </Location>
      <Location>
         <LocationId>a93c9db9-7471-3192-8d09-a8f7eeda85f9@4f56...
         </LocationId>
         <SiteName>Site-B</SiteName>
         <OrgName>System</OrgName>
         <RestApiEndpoint>https://site-b.example.com
         </RestApiEndpoint>
         <UIEndpoint>https://site-b.example.com
         </UIEndpoint>
         <AuthContext>System</AuthContext>
      </Location>
   </AuthorizedLocations>
</Session>

User and Group Identities

Associations of sites and organizations must agree to use the same identity provider (IDP). User and group identities for all organizations in the association must be managed through this IDP.

Associations are free to choose the IDP that works best for them. See Managing Identity Providers in VMware Cloud Director.

Starting with VMware Cloud Director 10.4.1, service accounts can manage and monitor multiple, geographically distributed VMware Cloud Director installations or server groups and their organizations as single entities by using the multisite feature. If a service account is making a request to a different organization from the one that it is authenticated to, verify that the service account exists on the associated organization and that it has the same name and software ID. See Managing Service Accounts in VMware Cloud Director.

Site Access Control for Organization Users and Groups

Organization administrators can configure their IDP to generate user or group access tokens that are valid at all member sites, or valid at only a subset of member sites. While user and group identities must be the same in all member organizations, user and group rights are constrained by the roles those users and groups are assigned in each member organization. Assignment of a role to a user or group is local to a member organization, as are any custom roles you create.

Load Balancer Requirements

Effective implementation of a multisite deployment requires you to configure a load balancer that distributes requests arriving at an institutional endpoint such as https://vcloud.example.com to the endpoints for each member of the site association (for example, https://us.vcloud.example.com and https://uk.vcloud.example.com). If a site has more than one cell, you must also configure a load balancer that distributes incoming requests across all its cells, so that a request to https://us.vcloud.example.com can be handled by https://cell1.us.vcloud.example.com, https://cell2.us.vcloud.example.com, and so on.

Note: You must use the global load balancer, in this case https://vcloud.example.com, only for UI access. If you develop your own scripts or programs that use the REST API, those calls must target a particular site.

Starting with VMware Cloud Director 10.3, all client requests that arrive at the load-balancing endpoint for a multisite deployment are redirected. When a request arrives at the load-balancing endpoint, even if the site where the request arrives is the correct one, a redirect is issued and reflected in the user-visible URL to specify that the request was directed to the correct location.

For example, you can have a deployment consisting of two sites - https://site1.vcloud.example.com and https://site2.vcloud.example.com - behind a global load-balancing endpoint https://us.vcloud.example.com. Starting with VMware Cloud Director 10.3, when a request arrives at the load-balancing endpoint for an organization that is located at site 1 - https://us.vcloud.example.com/org1, if the request lands at site 1, then site issues a redirect to itself by forwarding the request to https://site1.vcloud.example.com/org1. VMware Cloud Director 10.2.x and earlier versions do not issue redirects when a load balancer receives a request for an organization that is located at the same place and the request is serviced through the public endpoint's URL https://us.vcloud.example.com/org1.

Network Connectivity Requirements

If you want to use the multisite feature, each cell at each site must be able to make REST API requests to the REST API endpoints of all sites. If you use the examples from the Load Balancer Requirements section, cell1.us.vcloud.example.com and cell2.us.vcloud.example.com must be able to reach the REST API endpoint for uk.example.com. The reverse is true for all cells under uk.example.com. This means that a cell must also be able to make REST API calls to it's own REST API endpoint, so cell1.us.vcloud.example.com must be able to make a REST API call to https://us.vcloud.example.com.

Making REST API requests to the REST API endpoints of all sites is necessary for of REST API fanout. For example, if the UI or an API client makes a multisite request to get a page of organizations from all sites and cell1.us.vcloud.example.com handle the request. The cell cell1 must make a REST API call to get a page of organizations from each site using the REST API endpoint configured for that site. When all sites return their page of organizations, cell1 collates the results and returns a single page of results containing the data from all other sites.

Sites and Certificates

When a site is associated with other sites, if you update its certificate, you might have to let the other sites know of the change. If you do not let the other sites know about the certificate change, the multisite fanout might be impacted.

If you are replacing a certificate on a site with a valid, well-signed certificate, then you do not need to inform the other sites. Because the certificate is valid and well-signed, the cells at the other sites can continue connecting to it in a secure manner without interruption.

If you are replacing a certificate on a site with a self-signed certificate, or if there is some other problem with the certificate that prevents automatic trust, then other sites need to know. For example, if the certificate expires, you must let the other sites know. At each of the other sites, you must upload the certificate into the Trusted Certificates in the Service Provider Admin Portal. See Import Trusted Certificates Using Your VMware Cloud Director Service Provider Admin Portal. When you import the certificate, the site where the certificate is uploaded can trust the site getting the new certificate.

Note: You can import these certificates to the Trusted Certificates of the other sites before you install them at the remote site. This ensures no interruptions in communication because both the old certificate and the new certificate are in the Trusted Certificates pool. You do not have to reassociate the sites.

Association Member Status

After you create an association of sites or organizations, the local system periodically retrieves the status of each remote association member and updates that status in the local site's VMware Cloud Director database.The member status is visible in the Status element of an SiteAssociationMember or OrgAssociationMember. The Status element can have one of three values:

ACTIVE: The association has been established by both parties, and communication with the remote party was successful.
ASYMMETRIC: The association has been established at the local site, but the remote site has not yet reciprocated.
UNREACHABLE: An association has been created by both parties, but the remote site is not currently reachable on the network.

In the Service Provider Admin Portal and Tenant Portal the statuses appear as Connected, Partially Connected, and Unreachable.

The member status "heartbeat" process runs with the identity of the multisite system user, a local VMware Cloud Director user account created in the System organization during VMware Cloud Director installation. Although this account is a member of the System organization, it does not have system administrator rights. It has only a single right, Multisite: System Operations, which gives it permission to make a VMware Cloud Director API request that retrieves the status of the remote member of a site association.