This topic provides information for debugging and resolving issues with Services Toolkit.
Services Toolkit contains a single controller deployment. All logs related to Services Toolkit are on this deployment. The logs are in JSON format. You can obtain the logs by running:
kubectl logs -l app.kubernetes.io/name=services-toolkit-controller-manager -n services-toolkit
This is a sample of a log entry:
{
"level": "info",
"ts": 1639042469.5265353,
"logger": "controller.resourceclaim",
"msg": "Starting workers",
"reconciler group": "services.apps.tanzu.vmware.com",
"reconciler kind": "ResourceClaim",
"worker count": 1
}
Services Toolkit is comprised of a number of API endpoints, the majority of which are Custom Resource Definitions (CRDs). These resources configure and drive the behavior of Services Toolkit and contain the state of the system. A complete list of the APIs with how to obtain them for inspection follows.
For example, to find all ResourceClaims in all namespaces you can run:
kubectl get resourceclaim -A
UnableToTrackReferencedResourceSymptom: A ResourceClaim is not functioning and running kubectl get ResourceClaim <name> shows error UnableToTrackReferencedResource.
Cause: This is due to the controller being unable to access (list or watch) the resource.
Resolution: Verify that the name of the Resource referenced in the ResourceClaim’s .spec.ref exists on the cluster and ensure RBAC exists to allow claiming is enabled. See ResourceClaims Permissions documentation.
ResourceNonBindableSymptom: A ResourceClaim is not working and running kubectl get ResourceClaim <name> shows error ResourceNonBindable.
Cause: This is due to the Resource not adhering to the k8s-binding-spec Provisionable duck-type.
Resolution: Ensure that the Resource CRD implements the Provisionable type and was created successfully.
tanzu service plug-inWarning:
tanzu service types listandtanzu service instances listare deprecated commands. It is advised to make use oftanzu service classes listandtanzu service claimable listinstead.
Symptom: tanzu service types list returns no service types found or tanzu service instances list return no service instances found.
Cause: There are no ClusterResource resources on the cluster, or there are no resources matching one of the referenced Service Types in a ClusterResource resource. It could be both.
Resolution: Create one or more ClusterResource resources, referencing the APIs to make discoverable. Create one or more resources matching the API Group/Kind referenced in a created ClusterResource resource. For more information, see Service offering for VMware Tanzu.
tanzu service instances listWarning:
tanzu service instances listis a deprecated command. Usetanzu service classes listandtanzu service claimable listinstead.
Symptom: Running tanzu service instances list returns a 401 permission error.
Cause: The user running the command does not have sufficient RBAC permissions to get or list resources matching the API Group/Kinds defined in created ClusterResource resources.
Resolution: Ensure sufficient RBAC for all users for all resources that are referenced in ClusterResource resources that are installed on the cluster. This is a manual step required to be taken by a Service Operator upon installing a new service. For example, RabbitMQ and making it discoverable.
Symptom: An update to a claimed secret is not reflected in the workload using it. This can be because of password rotation or something similar.
Cause: While the secret was updated in the pod, because the secret is mounted there, the application still needs to re-read it. Many don’t do that.
Resolution: Restart the pods so that when the application starts up, it re-reads the updated mounted secret. If the pods scale to 0 because a Knative service manages them, they also re-read the secret on a scale-up.
