Troubleshooting

Note

Starting with the Services Toolkit v0.10 release, you can find the Services Toolkit documentation in the Tanzu Application Platform v1.5 and later documentation.

This topic provides information for debugging and resolving issues with Services Toolkit.

Logs

Services Toolkit contains a single controller deployment. All logs related to Services Toolkit are on this deployment. The logs are in JSON format. You can obtain the logs by running:

kubectl logs -l app.kubernetes.io/name=services-toolkit-controller-manager -n services-toolkit

This is a sample of a log entry:

{
  "level": "info",
  "ts": 1639042469.5265353,
  "logger": "controller.resourceclaim",
  "msg": "Starting workers",
  "reconciler group": "services.apps.tanzu.vmware.com",
  "reconciler kind": "ResourceClaim",
  "worker count": 1
}

Inspecting Resources

Services Toolkit is comprised of a number of API endpoints, the majority of which are Custom Resource Definitions (CRDs). These resources configure and drive the behavior of Services Toolkit and contain the state of the system. A complete list of the APIs with how to obtain them for inspection follows.

Service Resource Claims

  • resourceclaims.services.apps.tanzu.vmware.com
  • resourceclaimpolicies.services.apps.tanzu.vmware.com
  • clusterinstanceclasses.services.apps.tanzu.vmware.com
  • instancequeries.claimable.services.apps.tanzu.vmware.com

Service Offerings

  • clusterresources.services.apps.tanzu.vmware.com
  • clusterexampleusages.services.apps.tanzu.vmware.com

Service API Projection - Experimental

  • downstreamclusterlinks.projection.apiresources.multicluster.x-tanzu.vmware.com
  • upstreamclusterlinks.projection.apiresources.multicluster.x-tanzu.vmware.com
  • apiexportrolebindings.projection.apiresources.multicluster.x-tanzu.vmware.com
  • apiresourceimports.projection.apiresources.multicluster.x-tanzu.vmware.com
  • clusterapigroupimports.projection.apiresources.multicluster.x-tanzu.vmware.com

Service Resource Replication - Experimental

  • clusterresourceexportmonitors.replication.apiresources.multicluster.x-tanzu.vmware.com
  • clusterresourceimportmonitors.replication.apiresources.multicluster.x-tanzu.vmware.com
  • resourceexportmonitorbindings.replication.apiresources.multicluster.x-tanzu.vmware.com
  • resourceimportmonitorbindings.replication.apiresources.multicluster.x-tanzu.vmware.com
  • secretexports.replication.apiresources.multicluster.x-tanzu.vmware.com
  • secretimports.replication.apiresources.multicluster.x-tanzu.vmware.com

For example, to find all ResourceClaims in all namespaces you can run:

kubectl get resourceclaim -A

Common issues

ResourceClaim errors with UnableToTrackReferencedResource

  • Symptom: A ResourceClaim is not functioning and running kubectl get ResourceClaim <name> shows error UnableToTrackReferencedResource.

  • Cause: This is due to the controller being unable to access (list or watch) the resource.

  • Resolution: Verify that the name of the Resource referenced in the ResourceClaim’s .spec.ref exists on the cluster and ensure RBAC exists to allow claiming is enabled. See ResourceClaims Permissions documentation.

ResourceClaim errors with ResourceNonBindable

  • Symptom: A ResourceClaim is not working and running kubectl get ResourceClaim <name> shows error ResourceNonBindable.

  • Cause: This is due to the Resource not adhering to the k8s-binding-spec Provisionable duck-type.

  • Resolution: Ensure that the Resource CRD implements the Provisionable type and was created successfully.

Unable to discover services using tanzu service plug-in

Warning: tanzu service types list and tanzu service instances list are deprecated commands. It is advised to make use of tanzu service classes list and tanzu service claimable list instead.

  • Symptom: tanzu service types list returns no service types found or tanzu service instances list return no service instances found.

  • Cause: There are no ClusterResource resources on the cluster, or​ there are no resources matching one of the referenced Service Types in a ClusterResource resource. It could be both.

  • Resolution: Create one or more ClusterResource resources, referencing the APIs to make discoverable.​ Create one or more resources matching the API Group/Kind referenced in a created ClusterResource resource​. For more information, see Service offering for VMware Tanzu.

Permission error running tanzu service instances list

Warning: tanzu service instances list is a deprecated command. Use tanzu service classes list and tanzu service claimable list instead.

  • Symptom: Running tanzu service instances list returns a 401 permission error.

  • Cause: The user running the command does not have sufficient RBAC permissions to get or list resources matching the API Group/Kinds defined in created ClusterResource resources.

  • Resolution: Ensure sufficient RBAC for all users for all resources that are referenced in ClusterResource resources that are installed on the cluster​. This is a manual step required to be taken by a Service Operator upon installing a new service. For example, RabbitMQ and making it discoverable.

Updates to a claimed secret do not propagate to the Workload

  • Symptom: An update to a claimed secret is not reflected in the workload using it. This can be because of password rotation or something similar.

  • Cause: While the secret was updated in the pod, because the secret is mounted there, the application still needs to re-read it. Many don’t do that.

  • Resolution: Restart the pods so that when the application starts up, it re-reads the updated mounted secret. If the pods scale to 0 because a Knative service manages them, they also re-read the secret on a scale-up.

check-circle-line exclamation-circle-line close-line
Scroll to top icon