See the documentation on installing the latest release of the Services Toolkit to get started and see Topology for information about supported topologies.
This topic introduces a number of concepts. These are summarized as follows:
Projection Plane defines an “upstream” and “downstream” relationship between a pair of Kubernetes clusters, namely between a Service Cluster (upstream) and a Workload Cluster (downstream).
The UpstreamClusterLink resource is created on a Service Cluster. Its main purpose is to manage a Service Account that components running in a Workload Cluster use.
apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: UpstreamClusterLink
metadata:
name: workload-3c
namespace: services-toolkit
spec:
downstream:
# Name of the Workload Cluster. This will be used for debugging.
name: workload-3c
status:
# Created Service Account that will be used by the Workload Cluster
serviceAccount:
name: managed-service-account
observedGeneration: 1
conditions:
- lastTransitionTime: "2021-02-02T18:41:22Z"
status: "True"
type: Ready
- lastTransitionTime: "2021-02-02T18:41:22Z"
status: "True"
type: ServiceAccountReady
The DownstreamClusterLink resource is created on a Workload Cluster. Its primary purpose is to manage an API aggregation server that is eventually used to project specific APIs. This resource:
apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: DownstreamClusterLink
metadata:
name: services-2b
namespace: services-toolkit
spec:
proxy:
TLS:
# TLS cert to be used for the API proxy
secretName: omnia-isla
upstream:
kubeconfig:
# Secret containing the kubeconfig to connect to the Service Cluster
secretName: pumpkin-seeds
name: services-2b
status:
proxy:
# base64-encoded CA for the API proxy
caBundle: facade0ff1cebadc0ffee...
# Reference to the kubernetes Service providing access to the API proxy
serviceReference:
name: services-2b-proxy
namespace: services-toolkit
port: 443
conditions:
- lastTransitionTime: "2021-02-02T18:41:22Z"
status: "True"
type: Ready
- lastTransitionTime: "2021-02-02T18:41:22Z"
status: "True"
type: ServiceAccountReady
- lastTransitionTime: "2021-02-02T18:41:22Z"
status: "True"
type: ProxyDeploymentReady
- lastTransitionTime: "2021-02-02T18:41:22Z"
status: "True"
type: ProxyServiceReady
- lastTransitionTime: "2021-02-02T18:41:22Z"
status: "True"
type: ProxyConfigMapReady
- lastTransitionTime: "2021-02-02T18:41:22Z"
status: "True"
type: ProxyServiceAccountReady
The service account used by the proxy Deployment must have the following RBAC set up for it:
ClusteRoleBinding to the system:auth-delegator ClusterRole to delegate authentication decisions to the Kubernetes core API server.RoleBinding to the extension-apiserver-authentication-reader role in the kube-system namespace. This allows your extension API-server to access the extension-apiserver-authentication configmap.ClusterRoleBinding to a ClusterRole that provides get, list, and watch for namespaces. If such a ClusterRole doesn’t exist, you must create one.API Projection makes custom Kubernetes APIs installed on a Service Cluster (upstream) available in a Workload Cluster (downstream).
The purpose of the APIExportRoleBinding is to provide downstream users with necessary permissions on the Upstream Cluster. It does so by binding a user-specified ClusterRole to the service account referred to in the provided UpstreamClusterLink resource.
apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: APIExportRoleBinding
spec:
upstreamClusterLinkRef:
name: fish-sauce
namespace: project-alpha
clusterRoleRef:
name: cluster-1-a
The ClusterAPIGroupImport resource is a cluster-scoped resource created on the Workload Cluster. It expresses the intent to import an API group using the specified DownstreamClusterLink. Only one ClusterAPIGroupImport can exist per API Group.
After created, if a corresponding APIExportRole exists in the Service Cluster, a new custom Kubernetes API is available in the Workload Cluster and can be discovered by running the kubectl command api-resources.
apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ClusterAPIGroupImport
metadata:
name: rabbitmq.com
spec:
# This is the reference to the DownstreamClusterLink resources
downstreamClusterLinkRef:
name: services-2b
namespace: services-toolkit
# The api group that is to be projected
group: rabbitmq.com
# Version of the api to be projected. Optional, if not specified register all discovered versions
version: v1beta1
status:
conditions:
- type: Ready
lastTransitionTime: "2020-12-01T13:03:32Z"
status: "True"
- type: APIServicesReady
lastTransitionTime: "2020-12-01T13:03:28Z"
status: "True"
The APIResourceImport resource is a namespace-scoped resource created on the downstream cluster. Its presence indicates to the proxy whether a projected Group and Resource is available in a namespace. The proxy uses this information to decide whether to forward a particular request upstream. This is for convenience rather than policy enforcement, which the RBAC achieves upstream.
Resources are specified at the namespace scope rather than the cluster scope to allow different resources to be made available in different namespaces.
apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: APIResourceImport
metadata:
name: rabbitmq.com-import
namespace: team-1 # namespace scoped resource as it sets up ns RBAC
spec:
clusterApiImportRef:
name: rabbitmq.com
resources: [“rabbitmqclusters”]
status:
conditions:
- type: Ready
message: "Successfully reconciled"
lastTransitionTime: "2020-12-01T13:03:30Z"
status: "True"
- type: ResourcesAvailable
message: "Resources Ready"
lastTransitionTime: "2020-12-01T13:03:32Z"
status: "True"
The resource replication components are responsible for synchronizing core Kubernetes resources across multiple clusters. As of version v0.5.0, the resource replication only handles the Secret resources.
SecretExport is a namespaced resource indicating that the named secret is involved in replication. Services Toolkit places these resources on the services cluster. This resource sets up permissions for the local service account, which the Workload Clusters use to pull the secret across.
apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: SecretExport
metadata:
name: small-postgres-23.status.binding.name
namespace: project-1
labels:
# The following labels will be applied automatically
# to help with filtering and searching of SecretExport resources
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-group: sql.tanzu.vmware.com
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-version: v1
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-kind: Postgres
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-name: small-postgres-23
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-uid: cafe0123d09e
replication.apiresources.multicluster.x-tanzu.vmware.com/monitor-binding-uid: 0ff1ceca5cade
spec:
secret:
# The name of the secret in the current namespace to be replicated
name: pg-binding
serviceAccount:
# The name of the service account in the current namespace that will be used for replication
name: upstream-replication-sa
SecretImport is responsible for replicating the secret from the Service Cluster. Services Toolkit places the SecretImport in a user namespace of the Workload Cluster for each secret. Currently, the namespace on the Service Cluster has to match the namespace on the Workload Cluster.
apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: SecretImport
metadata:
namespace: project-1
name: small-postgres-23.status.binding.name
labels:
# The following labels will be applied automatically
# to help with filtering and searching of SecretImport resources
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-group: sql.tanzu.vmware.com
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-version: v1
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-kind: Postgres
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-name: small-postgres-23
replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-uid: cafe0123d09e
replication.apiresources.multicluster.x-tanzu.vmware.com/monitor-binding-uid: 0b5e55ed90dde55
spec:
secret:
# The name of the secret in the current namespace to be replicated
name: dumbo
remoteKubeconfig:
# The name of a secret in the current namespace holding a kubeconfig for the Service Cluster
name: energy-source
The two resources mentioned earlier handle a single Secret object replication. To set up replication of the specified secrets for every service instance of a given type, cluster-scoped resources ClusterResourceImportMonitor and ClusterResourceExportMonitor are used. Additionally, ResourceImportMonitorBinding and ResourceExportMonitorBinding are used to enable automatic replication in a namespace, and specify the connection details for replication for this namespace.
ClusterResourceImportMonitor is responsible for setting up watching on service instances. As a result, SecretImport resources can be produced when needed. ClusterResourceImportMonitor resources are defined on the Workload Cluster.
apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ClusterResourceImportMonitor
metadata:
name: postgres
labels:
# The following labels are required and must match the values in spec.resource
replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-group: sql.tanzu.vmware.com
replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-version: v1
replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-kind: Postgres
spec:
# The type of the resource owning the secrets to be replicated
resource:
group: sql.tanzu.vmware.com
version: v1
kind: Postgres
# The list of secrets to be replicated expressed as JSON path on the resource
secretPaths:
- .status.binding.name
By default, defining an ClusterResourceImportMonitor resource configures the resource type and secrets to be replicated, but does not enable replication. ResourceImportMonitorBinding enables the replication of secrets for service instances within a namespace. It references a secret containing the kubeconfig of the Service Cluster to pull the secrets from.
apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ResourceImportMonitorBinding
spec:
monitorRef:
# Name of the related cluster-scoped ClusterResourceImportMonitor
name: postgres
remoteKubeconfig:
# The name of a secret in the current namespace holding a kubeconfig for the Service Cluster
name: energy-source
ClusterResourceExportMonitor is responsible for setting up watching on service instances, so that as a result, SecretExport resources can be produced when needed. ClusterResourceExportMonitor resources are defined on the services cluster.
apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ClusterResourceExportMonitor
metadata:
name: postgres
labels:
# The following labels are required and must match the values in spec.resource
replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-group: sql.tanzu.vmware.com
replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-version: v1
replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-kind: Postgres
spec:
# The type of the resource owning the secrets to be replicated
resource:
group: sql.tanzu.vmware.com
version: v1
kind: Postgres
# The list of secrets to be replicated expressed as JSON path on the resource
secretPaths:
- .status.binding.name
By default, defining an ClusterResourceExportMonitor resource configures the resource type and secrets to be replicated, but does not enable replication. ResourceExportMonitorBinding enables the replication of secrets for service instances within a namespace. It provides the service account in the current namespace of the Service Cluster to pull the secrets from.
apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ResourceExportMonitorBinding
metadata:
name: cluster1-postgres
namespace: project-1
spec:
monitorRef:
# Name of the related cluster-scoped ClusterResourceImportMonitor
name: postgres
serviceAccount:
# Name of the service account in the current namespace used by the Workload Cluster to pull secrets.
name: upstream-replication-sa
