Broker and Service Management

This topic tells you how to manage the on-demand broker and service plans created using On-Demand Services SDK.

Broker Management Errands

This section describes how to manage your broker with BOSH errands. You can run these errands using the BOSH CLI.

Register Broker

The register-broker errand registers the broker with Cloud Foundry and enables access to plans in the service catalog. Run this errand whenever the broker is re-deployed with new catalog metadata to update the Cloud Foundry catalog.

Add the Errand to the Manifest

To add the register-broker errand to the manifest, do the following:

Add the following instance groups to your manifest:

- name: register-broker
  lifecycle: errand
  instances: 1
  jobs:
    - name: register-broker
      release: ODB-RELEASE-NAME
      properties:
        broker_name: BROKER-NAME
        broker_uri: BROKER-URI # optional, only required when a route has been registered
  vm_type: VM-TYPE
  stemcell: STEMCELL
  networks: [NETWORK]
  azs: [AZ]

# Optional. Add the cf_service_access property to the broker job:
- name: broker
  ...
  jobs:
    - name: broker
      ...
      properties:
        ...
        service_catalog:
          ...
          plans:
            - name: PLAN-NAME
              ...
              cf_service_access: enable | disable | manual | org-restricted # optional, defaults to enable
              service_access_org: ORG-NAME # required if "cf_service_access" is set to "org-restricted"

If the broker_uri property is set, you must register a route for your broker with Cloud Foundry. See the Route Registration section for more details.
(Optional) Use the cf_service_access property to enable access to individual plans. Set the property on each plan in the broker job. The values accepted are the following:
- enable: the errand activates access for that plan. This is the default value.
- disable: the errand deactivates access for that plan.
- manual: the errand does not modify service access for this plan.
- org-restricted: the errand enables access to the org set as the service_access_org in the plan definition.
Only Cloud Foundry admin users can see plans with deactivated service access. Org Managers and Space Managers cannot see these plans.

Note When the broker URI of this errand is not configured, the errand defaults to a BOSH-provided IP address or BOSH-provided BOSH DNS address. See Impact on links in the BOSH documentation.

When the broker is using TLS, the broker certificate must contain this BOSH provided address in its Subject Alternative Names section, otherwise the certificate cannot be verified by Cloud Foundry. For details about how to insert a BOSH DNS address into a config server generated certificate, see BOSH DNS Addresses in Config Server Generated Certs in the BOSH documentation.

Run the Errand

To run the register-broker errand, do the following:

Run the command:

bosh -d DEPLOYMENT-NAME run-errand register-broker

Where DEPLOYMENT-NAME is the name of your deployment.

For example:

$ bosh -d cf run-errand register-broker

Delete all Service Instances

Caution Use extreme caution when running this errand. You should only use it when you want to totally destroy all of the on-demand service instances from Cloud Foundry.

The delete-all-service-instances errand deletes service instances of your broker’s service offering in every org and space of Cloud Foundry. Because the errand uses the Cloud Controller API, it only deletes instances the Cloud Controller knows about.

The errand does not delete orphan BOSH deployments, which do not correspond to a known service instance. Orphan BOSH deployments should never happen, but in practice they do. Use the orphan-deployments errand to identify them. For more information, see Orphan Deployments below.

The errand does the following:

Unbinds all apps from the service instances.
Deletes all service instances sequentially. Each service instance deletion includes:
1. Running any pre-delete errands
2. Deleting the BOSH deployment of the service instance
3. Removing any ODB-managed secrets from CredHub
4. Checking for instance deletion failure, which results in the errand failing immediately
Determines whether any instances have been created while the errand was running. If new instances are detected, the errand returns an error. In this case, VMware recommends running the errand again.

Add the Errand to the Manifest

To add the delete-all-service-instances errand to the manifest, do the following:

Add the following instance group to your manifest:
```
- name: delete-all-service-instances
  lifecycle: errand
  instances: 1
  jobs:
    - name: delete-all-service-instances
      release: ODB-RELEASE-NAME
      properties:
        polling_interval_seconds: INTERVAL-IN-SECONDS # defaults to 60
        polling_initial_offset_seconds: OFFSET-IN-SECONDS # defaults to 5

  vm_type: VM-TYPE
  stemcell: STEMCELL
  networks: [{name: NETWORK}]
  azs: [AZ]
```
Where:
- INTERVAL-IN-SECONDS: The interval in seconds before a service instance is deleted.
- OFFSET-IN-SECONDS: The offset in seconds before polling Cloud Foundry to check if the instance has been deleted.
  Notes
  - The polling_interval_seconds default is set to 60 seconds because the Cloud Controller itself polls the on-demand broker every 60 seconds. Setting your polling interval to anything lower than 60 seconds does not speed up the errand.
  - The polling_initial_offset_seconds default is set to 5 seconds. In systems with more load, consider increasing the polling offset.

Run the Errand

To run the delete-all-service-instances errand, do the following:

Run the command:

bosh -d DEPLOYMENT-NAME run-errand \
delete-all-service-instances

Where DEPLOYMENT-NAME is the name of your deployment.

For example:

$ bosh -d cf run-errand \
delete-all-service-instances

Deregister Broker

The deregister-broker errand deregisters a broker from Cloud Foundry. It requires that there are no existing service instances of your broker’s service offering. If you run this errand with service instances remaining, it fails. Run the delete-all-service-instances-and-deregister-broker errand to remove remaining service instances and deregistering the broker. For more information, see Delete All Service Instances and Deregister Broker below.

Add the Errand to the Manifest

To add the deregister-broker errand to the manifest, do the following:

Add the following instance group to your manifest:

- name: deregister-broker
  lifecycle: errand
  instances: 1
  jobs:
    - name: deregister-broker
      release: ODB-RELEASE-NAME
      properties:
        broker_name: BROKER-NAME
  vm_type: VM-TYPE
  stemcell: STEMCELL
  networks: [{name: SERVICE-NETWORK}]
  azs: [AZ]

Run the Errand

To run the deregister-broker errand, do the following:

Run the command:

bosh -d DEPLOYMENT-NAME run-errand deregister-broker

Where DEPLOYMENT-NAME is the name of your deployment.

For example:

$ bosh -d cf run-errand deregister-broker

Delete All Service Instances and Deregister Broker

Caution Use extreme caution when running this errand. You should only use it when you want to destroy all of the on-demand service instances and deregister the broker from Cloud Foundry.

The delete-all-service-instances-and-deregister-broker errand performs a similar operation to the errands delete-all-service-instances and deregister-broker. For more information on both, see Delete All Service Instances and Deregister Broker below.

This errand does the following:

Deactivates service access to the service offering for all orgs and spaces. The errand deactivates service access to ensure that new instances cannot be provisioned during the lifetime of the errand.
Unbinds all apps from the service instances.
Deletes all service instances sequentially. Each service instance deletion includes:
1. Running any pre-delete errands
2. Deleting the BOSH deployment of the service instance
3. Removing any ODB-managed secrets from CredHub
4. Checking for instance deletion failure, which results in the errand failing immediately
Determines whether any instances have been created while the errand was running. If new instances are detected, the errand returns an error. In this case, VMware recommends running the errand again.
Deregisters the broker from Cloud Foundry.

Add the Errand to the Manifest

To add the delete-all-service-instances-and-deregister-broker errand to the manifest, do the following:

Add the following instance group to your manifest:
```
- name: delete-all-service-instances-and-deregister-broker
  lifecycle: errand
  instances: 1
  jobs:
    - name: delete-all-service-instances-and-deregister-broker
      release: ODB-RELEASE-NAME
      properties:
        broker_name: BROKER-NAME
        polling_interval_seconds: INTERVAL-IN-SECONDS # defaults to 60
        polling_initial_offset_seconds: OFFSET-IN-SECONDS # defaults to 5

  vm_type: VM-TYPE
  stemcell: STEMCELL
  networks: [{name: NETWORK}]
  azs: [AZ]
```
Where:
- INTERVAL-IN-SECONDS: The interval in seconds before a service instance is deleted.
- OFFSET-IN-SECONDS: The offset in seconds before polling Cloud Foundry to check if the instance has been deleted.
  Notes
  - The polling_interval_seconds default is set to 60 seconds because the Cloud Controller itself polls the on-demand broker every 60 seconds. Setting your polling interval to anything lower than 60 seconds does not speed up the errand.
  - The polling_initial_offset_seconds default is set to 5 seconds. In systems with more load, consider increasing the polling offset.

Run the Errand

To run the delete-all-service-instances-and-deregister-broker errand, do the following:

Run the command:

bosh -d DEPLOYMENT-NAME run-errand \
delete-all-service-instances-and-deregister-broker

Where DEPLOYMENT-NAME is the name of your deployment.

For example:

$ bosh -d cf run-errand \
delete-all-service-instances-and-deregister-broker

Orphan deployments

Note The deployment for a service instance is orphaned when the BOSH deployment is still running but the service is no longer registered in Cloud Foundry.

The orphan-deployments errand lists service deployments that have no matching service instances in Cloud Foundry and returns the list to the operator.

Add the Errand to the Manifest

To add the orphan-deployments errand to the manifest, do the following:

Add the following instance group to your manifest:

- name: orphan-deployments
  lifecycle: errand
  instances: 1
  jobs:
  - name: orphan-deployments
    release: ODB-RELEASE-NAME
    properties:
      broker_uri: BROKER-URI # optional
      tls: # optional
        ca_cert: BROKER-CA-CERT
        disable_ssl_cert_verification: TRUE|FALSE # defaults to false
  vm_type: VM-TYPE
  stemcell: STEMCELL
  networks: [{name: NETWORK}]
  azs: [AZ]

The orphan-deployments errand can be configured to use a Service Instances API. This might be required if your broker is deployed without Cloud Foundry. For more information, see Service Instances API.

Run the Errand

To run the orphan-deployments errand, do the following:

Run the command:

bosh -d DEPLOYMENT-NAME run-errand orphan-deployments

Where DEPLOYMENT-NAME is the name of your deployment.

For example:

$ bosh -d cf run-errand orphan-deployments

See if orphan deployments are present. If orphan deployments are present, the errand outputs a list that resembles the following:

Exit Code  10
Stdout     [{"deployment_name":"service-instance_bebbcf14-14ef-4eae-8fbd-656d15f2b4b5"}]

Stderr     [orphan-deployments] 2019/04/03 14:56:02.489064 Orphan BOSH
deployments detected with no corresponding service instance in the
platform. Before deleting any deployment it is recommended to verify the
service instance no longer exists in the platform and any data is safe to
delete

Delete an Orphan deployment

Caution Deleting the BOSH deployment destroys the service instance. Any data present is lost.

To delete an orphan deployment, do the following:

Run the command:

bosh -d DEPLOYMENT-NAME delete-deployment

Where DEPLOYMENT-NAME is the name of the orphaned deployment returned in the output of the errand.

For example:

bosh -d service-instance_aoeu39fgn-8125-05h2-9023-9vbxf7676f3 \
delete-deployment

Upgrade all Service Instances

The upgrade-all-service-instances errand upgrades all existing service instances to the latest service offering. If you have made changes to the plan definition or uploaded a new tile into Tanzu Operations Manager, VMware recommends that you upgrade all on-demand service instances to the latest software or plan definition.

The upgrade-all-service-instances errand does the following:

Collects all of the service instances the on-demand broker has registered
For each instance the errand does the following, serially:
- Issues an upgrade command to the on-demand broker
- Regenerates the service instance manifest based on its latest configuration from the tile
- Deploys the new manifest for the service instance
- Waits for this operation to complete, and then proceeds to the next instance
Adds to a retry list any instances that have ongoing BOSH tasks at the time of upgrade
Retries any instances in the retry list until all are upgraded

If any instance fails to upgrade, the errand fails immediately. This prevents systemic problems from spreading to the rest of your service instances.

To add the upgrade-all-service-instances errand to the manifest, follow the procedure in Upgrade All Service Instances.

Recreate all Service Instances

Note ODB only supports the recreate-all-service-instances errand in the following BOSH versions:

266.10.0–266.10.x
267.10.0–267.10.x
268.2.2–268.2.x
268.4.0 and later

Where x represents the latest version in that release line.

The recreate-all-service-instances errand recreates all service instance VMs managed by a broker. You might want use this errand when doing, for example, the following:

Rotating the Tanzu Operations Manager root certificate authority (CA). For more information about rotating CAs, see Rotate CAs and Leaf Certificates.
A full restore of the platform during disaster recovery or migration.

Add the Errand to the Manifest

To add the recreate-all-service-instances errand to the manifest, do the following:

Add the following instance group to your manifest:

- name: recreate-all-service-instances
  lifecycle: errand
  instances: 1
  jobs:
    - name: recreate-all-service-instances
      release: ODB-RELEASE-NAME
      properties:
        polling_interval_seconds: POLLING-INTERVAL-IN-SECONDS # defaults to 60
        attempt_interval_seconds: ATTEMPT-INTERVAL-IN-SECONDS # defaults to 60
        attempt_limit: NUMBER-OF-ATTEMPTS # defaults to 5
        broker_uri: BROKER-URI # optional
        tls: # optional
          ca_cert: BROKER-CA-CERT
          disable_ssl_cert_verification: TRUE|FALSE # defaults to false
  vm_type: VM-TYPE
  stemcell: STEMCELL
  networks: [{name: NETWORK}]
  azs: [AZ]

You can configure the behavior of this errand using following properties:

Property	Description
`polling_interval_seconds`	This controls the wait between checking the status of successfully submitted BOSH re-create tasks.
`attempt_interval_seconds`	When there are BOSH tasks in progress on the service instance to re-create, the instance is put in a retry queue. This property controls the pause between retries.
`attempt_limit`	The number of times to check whether each instance is available to be recreated. After an instance reaches the limit, the errand terminates.

Run the Errand

To run the recreate-all-service-instances errand, do the following:

Run the command:

bosh -d DEPLOYMENT-NAME run-errand recreate-all-service-instances

Where DEPLOYMENT-NAME is the name of your deployment.

For example:

$ bosh -d cf run-errand recreate-all-service-instances

Service Management

This section describes how to update, deactivate, and remove service plans. For how to upgrade the broker and service plans, see Upgrading.

Update the Broker

To modify the broker, do the following:

Make any necessary changes to the core broker configuration in the broker manifest. For more information about the core broker configuration, see Configure Your Broker.
Deploy the broker.

Update the Service Offering

To modify the service offering, do the following:

Change properties in the service_catalog of the broker manifest. For example, update the service metadata.
Change properties in the service_deployment of the broker manifest. For example, update the jobs used from a service release.
Deploy the broker.
Run the register-broker errand to update the Marketplace. For how to run the errand, see Register Broker.
Run the upgrade-all-service-instances errand to apply updated plans to existing service instances. For how to run the errand, see Upgrade All Service Instances.

Caution When Cloud Foundry registers the broker, do not change service_id or plan_id for any plan.

Deactivate Service Plans

To deactivate access to a service plan, do the following:

Run the following command:
```
cf disable-service-access SERVICE-NAME-FROM-CATALOG -p PLAN-NAME
```
Where:
- SERVICE-NAME-FROM-CATALOG is the name of the service from the service catalog.
- PLAN-NAME is the name of the plan you want to deactivate.
For example:
```
cf disable-service-access my-service -p small
```

Note When a plan has the property cf_service_access: disable in the service_catalog the register-broker errand deactivates service access to that plan. For more information on the errand, see Register Broker.

Remove Service Plans

You can remove service plans if there are no instances using the plan.

To remove a plan, do the following:

Remove the service plan from the broker manifest.
Run the register-broker errand to update the Marketplace. For more information about this errand, see Register Broker.

Caution If any service instances remain on a plan that has been removed from the catalog, the On-Demand Service Broker fails to start.