This guide provides a number of walkthroughs covering some of the most common use cases for the Services Toolkit.

  • Note It is recommended to be familiar with the User Roles that the toolkit considers.


See the documentation on installing the latest release of the Services Toolkit to get started.


Let's imagine the following scenario - A Service Operator wants to provide a RabbitMQ service to development teams, and they would like to make use of the RabbitMQ Cluster Kubernetes Operator to do this.

They would like to install the RabbitMQ Operator on a dedicated Kubernetes cluster (we will call this a Service Cluster). On the other hand, they would like to make the Service Resource Lifecycle API provided by the RabbitMQ Cluster Operator ( available in a cluster where Application Operators are running their workloads (we will call this an Application Workload Cluster).

This is where Services Toolkit and, specifically, Service API Projection comes in to play.

The Service API Projection component allows the Application Operators to perform any lifecycle operation (CRUD) on RabbitmqCluster resources from their Application Workload cluster, but with reconciliation of such resources occurring in the dedicated Service Cluster.

Install the RabbitMQ Operator on the Service Cluster

Let's now play the role of the Service Operator and install the Rabbitmq Cluster Operator on the Service Cluster. We will install a version of the operator that adheres to the binding spec. This will eventually allow any provisioned RabbitmqCluster resources to be automatically bound to Application Workloads.

Note In the future we hope to provide tooling that will allow any service resource, not just those that adhere to the k8s-service-bindings spec, to be bound to Application Workloads.

kapp --kubeconfig-context SERVICE-CLUSTER -y deploy --app rmq-operator \

You can verify that the Operator has been installed with the following:

kubectl --context=SERVICE-CLUSTER get crds

Link the two clusters

As the Service Operator we must now "link" the Application Workload Cluster and the Service Cluster. In practice, this means setting up a proxy in the Application Workload Cluster that is able to talk to the Service Cluster API.

We will use the kubectl-scp plugin to help us here. Please see the documentation for installing the plugin here.

With the plugin installed, execute the following:

kubectl scp link \
  --workload-kubeconfig-context=WORKLOAD-CLUSTER \

Note: On TKGm on AWS and certain other Kubernetes distributions you will need to provide the Service Cluster API Server address as an argument. e.g.

kubectl scp link \
  --workload-kubeconfig-context=kind-workload \

Federate the API Group

Next we will federate the API Group into the Application Workload Cluster. The act of federation can be split into two halves - projection and replication. Projection applies to custom API Groups, whereas replication applies to core Kubernetes resources (such as Secrets).

Before federating we will need to create a pair of target namespaces where instances of RabbitmqCluster will be created.

For now, the namespace name needs to be identical in the Application Workload and Service Cluster. See Topology for further information.

kubectl --context WORKLOAD-CLUSTER create namespace my-project-1
kubectl --context SERVICE-CLUSTER create namespace my-project-1

As with the previous cluster link step, we can use the kubectl-scp plugin to perform the federation step.

kubectl scp federate \
  --workload-kubeconfig-context=WORKLOAD-CLUSTER \
  --service-kubeconfig-context=SERVICE-CLUSTER \
  --namespace=my-project-1 \ \
  --api-version=v1beta1 \

Note: On TKGm on AWS and certain other Kubernetes distributions you will need to provide the Service Cluster API Server address as an argument. e.g.

kubectl scp federate \
  --workload-kubeconfig-context=kind-workload \
  --service-kubeconfig-context=kind-service \
  --namespace=my-project-1 \ \
  --api-version=v1beta1 \
  --api-resource=rabbitmqclusters \

ClusterServiceResource as a discovery mechanism

Now that the Service Resource Lifecycle API for RabbitmqCluster resources has been federated into the Application Workload Cluster, as the Service Operator, we want to now signify to Application Operators that they can use it as well as what its capabilities are. This is where the Service Offering APIs can help.

As a Service Operator, we can use the make-discoverable command on the kubectl-scp plugin, as follows:

kubectl scp make-discoverable \
  --workload-kubeconfig-context=WORKLOAD-CLUSTER \ \

Now, switching hats to the Application Operator role, I can use kubectl get clusterserviceresources to quickly list all the Service Resources that are available in my Application Workload cluster. I can also use the provided metadata, such as .spec.shortDescription ( lots more types of metadata coming soon!), to gather further contextual information about the Service Resources. This can help me to decide if RabbitmqCluster is something I am interested in provisioning.

  • Note ClusterServiceResource CRs do not contain any metadata that is specific to a given version of the API. Metadata that is specific to an API version is held in GVKDescriptor CRs. See Service Offering for further info.

For the sake of this tutorial, let's assume that RabbitmqCluster is exactly the Service Resource I am looking for.

Demonstrating usage with ClusterExampleUsage resources

Now that I have discovered a Service Resource I am interested in using, it would be nice to see some examples of how to use the API, so that I can get a better feel for what sorts of levers and options are available to me. ClusterExampleUsage CRs can be used to provide such examples, such as the following:

cat <<EOF | kubectl --context WORKLOAD-CLUSTER apply -f -
kind: ClusterExampleUsage
  name: rabbitmq-cluster-example-usage-1
    version: v1beta1
    kind: RabbitmqCluster
  description: |
    "Hello World" example for the RabbitmqCluster resource
  yaml: |
    kind: RabbitmqCluster
      name: example-rabbitmq-cluster-1

Service Operators can create as many or as few ClusterExampleUsage CRs as they feel necessary. Application Operators can then use kubectl get clusterexampleusages to quickly list all the examples. Alternatively they can choose to filter the list of resources using the following labels:


For example, kubectl get clusterexampleusages -l -l These labels are applied automatically to all ClusterExampleUsage CRs by the controller.

Tying it all together

Now that we have successfully federated the API and made it discoverable from the Application Workload cluster, let's put our Application Operator hat on and actually provision a RabbitmqCluster instance. We will use the ClusterExampleUsage from above to do so:

kubectl get clusterexampleusage rabbitmq-cluster-example-usage-1 -o jsonpath='{.spec.yaml}' | kubectl --context WORKLOAD-CLUSTER -n my-project-1 apply -f-

And we can use the following to see that the instance has been created:

kubectl --context WORKLOAD-CLUSTER get rabbitmqclusters -n my-project-1
check-circle-line exclamation-circle-line close-line
Scroll to top icon