Service API Projection and Service Resource Replication for VMware Tanzu

Install

See the documentation on installing the latest release of the Services Toolkit to get started and refer to Topology for information on supported topologies.

Terminology

  • Service Resources - Things like databases, message queues, caches, DNS records, firewall rules, virtual networks, etc.
  • Service Resource Lifecycle API - Any Kubernetes API that can be used to manage the lifecycle (CRUD) of a Service Resource.
  • Service Cluster - A Kubernetes cluster that has Service Resource Lifecycle APIs installed and a corresponding controller managing their lifecycle.
  • Workload Cluster - A Kubernetes cluster that has developer-created applications running on it.

Concepts

This document introduces a number of concepts. These are briefly summarised below:

  • Projection Plane - Defines an “upstream” and “downstream” relationship between a pair of Kubernetes clusters, namely between a Service Cluster (upstream) and a Workload Cluster (downstream).
  • API Projection - Makes custom Kubernetes APIs installed on a Service Cluster (upstream) available in a Workload Cluster (downstream).
  • Resource Replication - Synchronizes core Kubernetes resources across Kubernetes clusters.

Resources

Projection Plane

UpstreamClusterLink and DownstreamClusterLink

The UpstreamClusterLink resource is created on a Service Cluster. Its main purpose is to manage a Service Account that will be used by components running in a Workload Cluster.

apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: UpstreamClusterLink
metadata:
  name: workload-3c
  namespace: services-toolkit
spec:
  downstream:
    # Name of the Workload Cluster. This will be used for debugging.
    name: workload-3c
status:
  # Created Service Account that will be used by the Workload Cluster
  serviceAccount:
    name: managed-service-account
  observedGeneration: 1
  conditions:
  - lastTransitionTime: "2021-02-02T18:41:22Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2021-02-02T18:41:22Z"
    status: "True"
    type: ServiceAccountReady

The DownstreamClusterLink resource is created on a Workload Cluster. Its primary purpose is to manage an API aggregation server that will eventually be used to project specific APIs. This resource does the following:

  • Contains information about the corresponding Service Cluster - url, name, ca cert and service account token.
  • Deploys the API-aggregation server that is configured to proxy to the Service Cluster using the provided service account token.
apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: DownstreamClusterLink
metadata:
  name: services-2b
  namespace: services-toolkit
spec:
  proxy:
    TLS:
      # TLS cert to be used for the API proxy
      secretName: omnia-isla
  upstream:
    kubeconfig:
      # Secret containing the kubeconfig to connect to the Service Cluster
      secretName: pumpkin-seeds
    name: services-2b
status:
  proxy:
    # base64-encoded CA for the API proxy
    caBundle: facade0ff1cebadc0ffee...
    # Reference to the kubernetes Service providing access to the API proxy
    serviceReference:
      name: services-2b-proxy
      namespace: services-toolkit
      port: 443
  conditions:
  - lastTransitionTime: "2021-02-02T18:41:22Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2021-02-02T18:41:22Z"
    status: "True"
    type: ServiceAccountReady
  - lastTransitionTime: "2021-02-02T18:41:22Z"
    status: "True"
    type: ProxyDeploymentReady
  - lastTransitionTime: "2021-02-02T18:41:22Z"
    status: "True"
    type: ProxyServiceReady
  - lastTransitionTime: "2021-02-02T18:41:22Z"
    status: "True"
    type: ProxyConfigMapReady
  - lastTransitionTime: "2021-02-02T18:41:22Z"
    status: "True"
    type: ProxyServiceAccountReady

Note: the service account used by the proxy Deployment must have the following RBAC set up for it: * A ClusteRoleBinding to the system:auth-delegator ClusterRole to delegate auth decisions to the Kubernetes core API server. * A RoleBinding to the extension-apiserver-authentication-reader role in the kube-system namespace. This allows your extension api-server to access the extension-apiserver-authentication configmap. * A ClusterRoleBinding to a ClusterRole that provides “get”, “list” and “watch” for namespaces, if such a ClusterRole doesn’t exist you will need to create one.

API Projection

APIExportRoleBinding

The purpose of the APIExportRoleBinding is to provide downstream users with necessary permissions on the Upstream Cluster. It does so by binding a user-specified ClusterRole to the service account referred to in the provided UpstreamClusterLink resource.

apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: APIExportRoleBinding
spec:
  upstreamClusterLinkRef:
    name: fish-sauce
    namespace: project-alpha
  clusterRoleRef:
    name: cluster-1-a

ClusterAPIGroupImport

The ClusterAPIGroupImport resource is a cluster-scoped resource created on the Workload Cluster. It expresses the intent to import an API group using the specified DownstreamClusterLink. Only one ClusterAPIGroupImport can exist per API Group.

Once created, if a corresponding APIExportRole exists in the Service Cluster, a new custom Kubernetes API will be available in the Workload Cluster and can be discovered via kubectl api-resources.

apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ClusterAPIGroupImport
metadata:
  name: rabbitmq.com
spec:
  # This is the reference to the DownstreamClusterLink resources
  downstreamClusterLinkRef:
    name: services-2b
    namespace: services-toolkit
  # The api group that is to be projected
  group: rabbitmq.com
  # Version of the api to be projected. Optional, if not specified register all discovered versions
  version: v1beta1
status:
  conditions:
  - type: Ready
    lastTransitionTime: "2020-12-01T13:03:32Z"
    status: "True"
  - type: APIServicesReady
    lastTransitionTime: "2020-12-01T13:03:28Z"
    status: "True"

APIResourceImport

The APIResourceImport resource is a namespace-scoped resource created on the downstream cluster. Its presence indicates to the proxy whether a projected Group and Resource is available in a given namespace. This information allows the proxy to decide if a particular request should be forwarded to upstream. It is worth noting this is for convenience rather than policy enforcement, which is achieved by the RBAC in upstream.

Resources are specified at the namespace scope rather than the cluster scope to allow different resources to be made available in different namespaces.

apiVersion: projection.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: APIResourceImport
metadata:
  name: rabbitmq.com-import
  namespace: team-1 # namespace scoped resource as it sets up ns RBAC
 spec:
   clusterApiImportRef:
      name: rabbitmq.com
   resources: [“rabbitmqclusters”]
status:
  conditions:
  - type: Ready
    message: "Successfully reconciled"
    lastTransitionTime: "2020-12-01T13:03:30Z"
    status: "True"
  - type: ResourcesAvailable
    message: "Resources Ready"
    lastTransitionTime: "2020-12-01T13:03:32Z"
    status: "True"

Resource Replication

The resource replication components are responsible for synchronizing core kubernetes resources across multiple clusters. As of version v0.5.0, the resource replication only handles the Secret resources.

SecretExport

SecretExport is a namespaced resource indicating that the named Secret is involved in the replication process. Services toolkit will place these resources on the services cluster. This resource is used to set up permissions for the local service account, which will be used by the Workload Clusters when pulling the secret across.

apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: SecretExport
metadata:
  name: small-postgres-23.status.binding.name
  namespace: project-1
  labels:
    # The following labels will be applied automatically
    # to help with filtering and searching of SecretExport resources
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-group: sql.tanzu.vmware.com
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-version: v1
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-kind: Postgres
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-name: small-postgres-23
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-uid: cafe0123d09e
    replication.apiresources.multicluster.x-tanzu.vmware.com/monitor-binding-uid: 0ff1ceca5cade
spec:
  secret:
    # The name of the secret in the current namespace to be replicated
    name: pg-binding
  serviceAccount:
    # The name of the service account in the current namespace that will be used for replication
    name: upstream-replication-sa

SecretImport

SecretImport is responsible for replicating the secret from the Service Cluster. Services Toolkit places the SecretImport in a user namespace of the Workload Cluster for each secret. Currently, the namespace on the Service Cluster has to match the namespace on the Workload Cluster.

apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: SecretImport
metadata:
  namespace: project-1
  name: small-postgres-23.status.binding.name
  labels:
    # The following labels will be applied automatically
    # to help with filtering and searching of SecretImport resources
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-group: sql.tanzu.vmware.com
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-version: v1
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-kind: Postgres
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-name: small-postgres-23
    replication.apiresources.multicluster.x-tanzu.vmware.com/secret-owner-uid: cafe0123d09e
    replication.apiresources.multicluster.x-tanzu.vmware.com/monitor-binding-uid: 0b5e55ed90dde55
spec:
  secret:
    # The name of the secret in the current namespace to be replicated
    name: dumbo
  remoteKubeconfig:
    # The name of a secret in the current namespace holding a kubeconfig for the Service Cluster
    name: energy-source

The two resources above handle a single Secret object replication. In order to automatically set up replication of the specified secrets for every service instance of a given type, cluster-scoped resources ClusterResourceImportMonitor and ClusterResourceExportMonitor are used. Additionally, ResourceImportMonitorBinding and ResourceExportMonitorBinding are used to enable automatic replication in a given namespace, and specify the connection details for replication for this namespace.

ClusterResourceImportMonitor

ClusterResourceImportMonitor is responsible for setting up watching on service instances, so that as a result, SecretImport resources could be produced when needed. ClusterResourceImportMonitor resources are defined on the Workload Cluster.

apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ClusterResourceImportMonitor
metadata:
  name: postgres
  labels:
    # The following labels are required and must match the values in spec.resource
    replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-group: sql.tanzu.vmware.com
    replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-version: v1
    replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-kind: Postgres
spec:
  # The type of the resource owning the secrets to be replicated
  resource:
    group: sql.tanzu.vmware.com
    version: v1
    kind: Postgres
  # The list of secrets to be replicated expressed as JSON path on the resource  
  secretPaths:
  - .status.binding.name

ResourceImportMonitorBinding

By default, defining an ClusterResourceImportMonitor resource configures the resource type and secrets to be replicated, but does not enable replication. ResourceImportMonitorBinding is used to enable the replication of secrets for service instances within a given namespace. It references a secret containing the kubeconfig of the Service Cluster to pull the secrets from.

apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ResourceImportMonitorBinding
spec:
  monitorRef:
    # Name of the related cluster-scoped ClusterResourceImportMonitor
    name: postgres
  remoteKubeconfig:
    # The name of a secret in the current namespace holding a kubeconfig for the Service Cluster
    name: energy-source

ClusterResourceExportMonitor

ClusterResourceExportMonitor is responsible for setting up watching on service instances, so that as a result, SecretExport resources could be produced when needed. ClusterResourceExportMonitor resources are defined on the services cluster.

apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ClusterResourceExportMonitor
metadata:
  name: postgres
  labels:
    # The following labels are required and must match the values in spec.resource
    replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-group: sql.tanzu.vmware.com
    replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-version: v1
    replication.apiresources.multicluster.x-tanzu.vmware.com/monitored-resource-kind: Postgres
spec:
  # The type of the resource owning the secrets to be replicated
  resource:
    group: sql.tanzu.vmware.com
    version: v1
    kind: Postgres
  # The list of secrets to be replicated expressed as JSON path on the resource
  secretPaths:
  - .status.binding.name

ResourceExportMonitorBinding

By default, defining an ClusterResourceExportMonitor resource configures the resource type and secrets to be replicated, but does not enable replication. ResourceExportMonitorBinding is used to enable the replication of secrets for service instances within a given namespace. It provides the service account in the current namespace of the Service Cluster to pull the secrets from.

apiVersion: replication.apiresources.multicluster.x-tanzu.vmware.com/v1alpha1
kind: ResourceExportMonitorBinding
metadata:
  name: cluster1-postgres
  namespace: project-1
spec:
  monitorRef:
    # Name of the related cluster-scoped ClusterResourceImportMonitor
    name: postgres
  serviceAccount:
    # Name of the service account in the current namespace used by the Workload Cluster to pull secrets.
    name: upstream-replication-sa
check-circle-line exclamation-circle-line close-line
Scroll to top icon