This topic for Tanzu Operations Manager operators gives you information about troubleshooting on-demand services.
You need the GUID of your service instance to run some BOSH commands. To retrieve the GUID, run the command:
service SERVICE-INSTANCE-NAME --guid
If you do not know the name of the service instance, run cf services
to see a listing of all service instances in the space. The service instances are listed in the name column.
This section provides information about how to troubleshoot specific errors or error messages.
Failed Installation |
|
---|---|
Symptom | On-Demand Services SDK fails to install. |
Cause | Reasons for a failed installation include:
|
Solution | To troubleshoot:
|
Cannot Create or Delete Service Instances |
|
---|---|
Symptom | If developers report errors such as: Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: create |
Cause | Reasons include:
|
Solution | To troubleshoot:
|
Broker Request Timeouts |
|
---|---|
Symptom | If developers report errors such as: Server error, status code: 504, error code: 10001, message: The request to the service broker timed out: https://BROKER-URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b |
Cause | Cloud Foundry might not be connected to the service broker, or there might be a large number of queued tasks. |
Solution | To troubleshoot:
|
Instance Does Not Exist |
|
---|---|
Symptom | If developers report errors such as: Server error, status code: 502, error code: 10001, message: Service broker error: instance does not exist |
Cause | The instance might have been deleted. |
Solution | To troubleshoot:
|
Cannot Bind to or Unbind from Service Instances |
|
---|---|
Symptom | If developers report errors such as: Server error, status code: 502, error code: 10001, message: Service broker error: There was a problem completing your request. Please contact your operations team providing the following information: service: example-service, service-instance-guid: 8d69de6c-88c6-4283-b8bc-1c46103714e2, broker-request-id: 15f4f87e-200a-4b1a-b76c-1c4b6597c2e1, operation: bind |
Cause | This might be due to authentication or network errors. |
Solution | To find out the issue with the binding:
|
Cannot Connect to a Service Instance |
|
---|---|
Symptom | Developers report that their app cannot use service instances that they created and bound. |
Cause | The error might originate from the service or be network related. |
Solution | To solve this issue, ask the user to send application logs that show the connection error. If the error originates from the service, then follow On-Demand Services SDK-specific instructions. If the issue appears to be network-related, then:
|
Cannot Update a Service Instance |
|
---|---|
Symptom | If developers report errors such as the following when trying to run cf-update-service : FAILED Server error, status code: 502, error code: 10001, message: Service broker error: Service cannot be updated at this time, please try again later or contact your operator for more information. |
Cause | Their service instance might not be running the latest service offering. |
Solution | Operators must run the If Until you resolve this issue, app developers cannot set parameters or change plans. For more information, see Upgrade All Service Instances and (Optional) Configure Maintenance Information. |
Upgrade All Service Instances Errand Fails |
|
---|---|
Symptom | The upgrade-all-service-instances errand fails. |
Cause | There might be a problem with a particular instance. |
Solution | To troubleshoot:
|
Missing Logs and Metrics |
|
---|---|
Symptom | No logs are being emitted by the on-demand broker. |
Cause | Syslog might not be configured correctly, or you might have network access issues. |
Solution | To troubleshoot:
|
This section provides information about troubleshooting on-demand broker components.
On-Demand service brokers add tasks to the BOSH request queue, which can back up and cause delay under heavy loads. An app developer who requests a new On-Demand Services SDK instance sees create in progress
in the Cloud Foundry Command Line Interface (cf CLI) until BOSH processes the queued request.
Tanzu Operations Manager deploys two BOSH workers to process its queue.
The VM or Disk type that you configured in the plan page of the tile in Tanzu Operations Manager might not be large enough for the On-Demand Services SDK service instance to start. See tile-specific guidance on resource requirements.
If you rotated any UAA user credentials then you might see authentication issues in the service broker logs.
To resolve this, redeploy the On-Demand Services SDK tile in Tanzu Operations Manager. This provides the broker with the latest configuration.
Caution You must ensure that any changes to UAA credentials are reflected in the Tanzu Operations Manager credentials
tab of the VMware Tanzu Application Service for VMs tile.
Common issues with networking include:
Issue | Solution |
---|---|
Latency when connecting to the On-Demand Services SDK service instance to create or delete a binding. | Try again or improve network performance. |
Firewall rules are blocking connections from the On-Demand Services SDK service broker to the service instance. | Open the On-Demand Services SDK tile in Tanzu Operations Manager and verify that the two networks configured in the Networks pane allow access to each other. |
Firewall rules are blocking connections from the service network to the BOSH Director network. | Ensure that service instances can access the Director so that the BOSH agents can report in. |
Apps cannot access the service network. | Configure Cloud Foundry application security groups to allow runtime access to the service network. |
Problems accessing BOSH’s UAA or the BOSH director. | Follow network troubleshooting and verify that the BOSH Director is online. |
To validate connectivity:
View the BOSH deployment name for your service broker by running:
bosh deployments
SSH into the On-Demand Services SDK service broker by running:
bosh -d DEPLOYMENT-NAME ssh
If no BOSH task-id
appears in the error message, look in the broker log using the broker-request-id
from the task.
Use the cf ssh
command to access to the app container, then connect to the On-Demand Services SDK service instance using the binding included in the VCAP_SERVICES
environment variable.
If developers report errors such as:
Message: Service broker error: The quota for this service plan has been exceeded. Please contact your Operator for help.
If developers report errors such as:
Message: Service broker error: The quota for this service has been exceeded. Please contact your Operator for help.
To find out if there is an issue with the On-Demand Services SDK deployment:
Inspect the VMs by running:
bosh -d service-instance_GUID vms --vitals
For additional information, run:
bosh -d service-instance_GUID instances --ps --vitals
If the VM is failing, follow the service-specific information. Any unadvised corrective actions (such as running BOSH restart
on a VM) can cause issues in the service instance.
This section provides general techniques for troubleshooting, which might include the following:
Failed operations (create, update, bind, unbind, delete) cause an error message. You can retrieve the error message later by running the cf CLI command cf service INSTANCE-NAME
.
$ cf service myservice Service instance: myservice Service: super-db Bound apps: Tags: Plan: dedicated-vm Description: Dedicated Instance Documentation url: Dashboard: Last Operation Status: create failed Message: Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: create Started: 2017-03-13T10:16:55Z Updated: 2017-03-13T10:17:58Z
Use the information in the Message
field to debug further. Provide this information to Support when filing a ticket.
The task-id
field maps to the BOSH task ID. For more information about a failed BOSH task, use the bosh task TASK-ID
.
The broker-request-guid
maps to the portion of the On-Demand Service Broker log containing the failed step. Access the broker log through your syslog aggregator, or access BOSH logs for the broker by typing bosh logs broker 0
. If you have more than one broker instance, repeat this process for each instance.
Before following these procedures, log in to the cf CLI and the BOSH CLI.
You can access logs using Tanzu Operations Manager by clicking on the Logs tab in the tile and downloading the broker logs.
To access logs using the BOSH CLI:
To identify the on-demand broker (ODB) deployment run:
bosh deployments
To view VMs in the deployment run:
bosh -d DEPLOYMENT-NAME instances
To SSH onto the VM run:
bosh -d DEPLOYMENT-NAME ssh
To Download the broker logs run:
bosh -d DEPLOYMENT-NAME logs
The archive generated by BOSH includes the following logs:
Log Name | Description |
---|---|
broker.stdout.log | Requests to the on-demand broker and the actions the broker performs while orchestrating the request (e.g. generating a manifest and calling BOSH). Start here when troubleshooting. |
bpm.log | Control script logs for starting and stopping the on-demand broker. |
post-start.stderr.log | Errors that occur during post-start verification. |
post-start.stdout.log | Post-start verification. |
drain.stderr.log | Errors that occur while running the drain script. |
To target an individual service instance deployment, retrieve the GUID of your service instance with the following cf CLI command:
cf service MY-SERVICE --guid
To view VMs in the deployment, run:
bosh -d service-instance_GUID instances
To SSH into a VM, run:
bosh -d service-instance_GUID ssh
To download the instance logs, run:
bosh -d service-instance_GUID logs
From the BOSH CLI, you can run service broker errands that manage the service brokers and perform mass operations on the service instances that the brokers created. These service broker errands include:
register-broker
registers a broker with the Cloud Controller and lists it in the Marketplace.
deregister-broker
deregisters a broker with the Cloud Controller and removes it from the Marketplace.
upgrade-all-service-instances
upgrades existing instances of a service to its latest installed version.
delete-all-service-instances
deletes all instances of service.
orphan-deployments
detects "orphan" instances that are running on BOSH but not registered with the Cloud Controller.
To run an errand:
bosh -d DEPLOYMENT-NAME run-errand ERRAND-NAME
For example:
bosh -d my-deployment run-errand deregister-broker
The register-broker
errand:
Run this errand whenever the broker is re-deployed with new catalog metadata to update the Marketplace.
Plans with deactivated service access are only visible to admin Cloud Foundry users. Non-admin Cloud Foundry users, including Org Managers and Space Managers, cannot see these plans.
This errand deregisters a broker from Cloud Foundry.
The errand:
Use the Delete All Service Instances errand to delete any existing service instances.
To run the errand:
bosh -d DEPLOYMENT-NAME run-errand deregister-broker
The upgrade-all-service-instances
errand:
When you make changes to the plan configuration, the errand upgrades all the On-Demand Services SDK service instances to the latest version of the plan.
If any instance fails to upgrade, the errand fails immediately. This prevents systemic problems from spreading to the rest of your service instances.
This errand uses the Cloud Controller API to delete all instances of your broker service offering in every Cloud Foundry org and space. It deletes only instances the Cloud Controller knows about. It does not delete orphan BOSH deployments.
Important Orphan BOSH deployments do not correspond to a known service instance. While rare, orphan deployments can occur. Use the orphan-deployments
errand to identify them.
The delete-all-service-instances
:
CautionUse extreme caution when running this errand. Use it only when you want to destroy all of the on-demand service instances in an environment.
To run the errand:
bosh -d service-instance_GUID delete-deployment
A service instance is defined as "orphaned" when the BOSH deployment for the instance is still running, but the service is no longer registered in Cloud Foundry.
The orphan-deployments
errand collates a list of service deployments that have no matching service instances in Cloud Foundry and return the list to the operator. It is then up to the operator to remove the orphaned BOSH deployments.
To run the errand:
bosh -d DEPLOYMENT-NAME run-errand orphan-deployments
If orphan deployments exist---The errand script does the following:
[stdout]
header[stderr]
headerFor example:
[stdout] [{"deployment\_name":"service-instance\_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}] [stderr] Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete. Errand 'orphan-deployments' completed with error (exit code 10)
These details are also available through the BOSH /tasks/
API endpoint for use in scripting:
$ curl 'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?type=result' | jq .
{
"exit_code": 10,
"stdout": "[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]\n",
"stderr": "Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete.\n",
"logs": {
"blobstore_id": "d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d"
}
}
If no orphan deployments exist---The errand script:
None
[stdout] [] [stderr] None Errand 'orphan-deployments' completed successfully (exit code 0)
If the errand encounters an error during running---The errand script does the following:
To clean up orphaned instances, run the following command on each instance:
Caution Running this command might leave IaaS resources in an unusable state.
bosh delete-deployment service-instance_SERVICE-INSTANCE-GUID
To retrieve the admin credentials for a service instance from BOSH CredHub:
cf service SERVICE-INSTANCE-NAME --guid
For example: $ cf service my-service-instance --guid 12345678-90ab-cdef-1234-567890abcdefIf you do not know the name of the service instance, you can list service instances in the space with
cf services
. Find the values for BOSH_CLIENT
and BOSH_CLIENT_SECRET
:
BOSH_CLIENT
and BOSH_CLIENT_SECRET
.credhub api https://BOSH-DIRECTOR-IP:8844 \
--ca-cert=/var/tempest/workspaces/default/root_ca_certificate
Where BOSH-DIRECTOR-IP
is the IP address of the BOSH Director VM. $ credhub api https://10.0.0.5:8844 \
--ca-cert=/var/tempest/workspaces/default/root_ca_certificate
credhub login \
--client-name=BOSH-CLIENT \
--client-secret=BOSH-CLIENT-SECRET
For example:
$ credhub login \ --client-name=credhub \ --client-secret=abcdefghijklm123456789
Use the CredHub CLI to retrieve the credentials :
credhub get -n /p-bosh/service-instance_GUID/admin_password
In the output, the password appears under value
. Record the password.$ credhub get \ -n /p-bosh/service-instance_70d30bb6-7f30-441a-a87c-05a5e4afff26/admin_password
id: d6e5bd10-3b60-4a1a-9e01-c76da688b847 name: /p-bosh/service-instance_70d30bb6-7f30-441a-a87c-05a5e4afff26/admin_password type: password value: UMF2DXsqNPPlCNWMdVMcNv7RC3Wi10 version_created_at: 2018-04-02T23:16:09Z
To identify which apps are using a specific service instance using the name of the BOSH deployment:
service-instance_
leaving you with the GUID.cf curl /v2/service_instances/GUID/service_bindings
resources
, with each item referencing a service binding, which contains the APP-URL
. To find the name, org, and space for the app, run:
cf curl APP-URL
and record the app name under entity.name
.cf curl SPACE-URL
to obtain the space, using the entity.space_url
from the curl. Record the space name under entity.name
. cf curl ORGANIZATION-URL
to obtain the org, using the entity.organization_url
from the curl. Record the organization name under entity.name
. Important When running cf curl
ensure that you query all pages, because the responses are limited to a certain number of bindings per page. The default is 50. To find the next page, curl the value under next_url
.
To view usage statistics for any service, run:
Run:
bosh -d DEPLOYMENT-NAME vms --vitals
To view process-level information, run:
bosh -d DEPLOYMENT-NAME instances --ps
Quota saturation and total number of service instances are available through ODB metrics emitted to Loggregator. These are the metric names:
Metric Name | Description |
---|---|
on-demand-broker/SERVICE-NAME-MARKETPLACE/quota_remaining |
global quota remaining for all instances across all plans |
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/quota_remaining |
quota remaining for a particular plan |
on-demand-broker/SERVICE-NAME-MARKETPLACE/total_instances |
total instances created across all plans |
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/total_instances |
total instances created for a given plan |
Importants Quota metrics are not emitted if no quota was set.
To reinstall a tile in the same environment where it was previously uninstalled:
cf login
cf m
bosh log-in
bosh deployments
bosh delete-deployment BROKER-DEPLOYMENT-NAME
Find the answer to your question and navigate product discussions and solutions by searching the VMware Tanzu Knowledge Base.
You can file a ticket with Support. Include the error message from cf service YOUR-SERVICE-INSTANCE
.
To expedite troubleshooting, provide your service broker logs and your service instance logs. If your cf service YOUR-SERVICE-INSTANCE
output includes a task-id
, provide the BOSH task output.