Upgrading the On-Demand Broker

This topic for Tanzu Operations Manager operators and BOSH operators gives you information about upgrading the on-demand broker provided by On-Demand Services SDK, upgrading service offerings, and upgrading service instances.

For product versions and upgrade paths, see Upgrade Planner.

Upgrade the Broker

To upgrade the broker, do the following:

Download a new version of the on-demand service broker BOSH release from BOSH.io Releases.

Upload the release to the BOSH Director by running:

bosh -e BOSH-DIRECTOR-NAME upload-release RELEASE-FILE-NAME.tgz

Make any necessary changes to the core broker configuration in the broker manifest. For more information about the core broker configuration, see Configure Your Broker.

Deploy the broker by running:

bosh -e BOSH-DIRECTOR-NAME -d DEPLOYMENT-NAME deploy DEPLOYMENT-MANIFEST.yml

Upgrade the Service Offering

The service offering is made of the following:

Service catalog
Service adapter BOSH release
Service BOSH releases
Service stemcells

To upgrade a service offering, do the following:

Make any changes to the service catalog in the broker manifest. For more information about the service catalog, see the Starter Snippet.
Upload any new service BOSH releases to the BOSH Director by running:
```
bosh -e BOSH-DIRECTOR-NAME upload-release RELEASE-FILE-NAME.tgz
```
Make any changes to service releases in the broker manifest.
Upload any new service stemcells to the BOSH Director.
```
bosh -e BOSH-DIRECTOR-NAME upload-stemcell STEMCELL-LOCATION
```
Where STEMCELL-LOCATION is the path or URL of the stemcell.
Make any changes to the service stemcells in the service_deployment broker manifest.

Deploy the broker by running:

bosh -e BOSH-DIRECTOR-NAME -d DEPLOYMENT-NAME deploy DEPLOYMENT-MANIFEST.yml

New service instances are created using the latest service offering. To upgrade all existing instances, you can run the upgrade-all-service-instances errand. See Upgrade All Service Instances below.

Caution Until a service instance has been upgraded, cf update-service operations are blocked and an error is shown. For more information, see Cannot Update a Service Instance.

Upgrade All Service Instances

To upgrade all existing service instances after the service offering has been updated or upgraded, do the following:

Add the following instance group to your broker manifest:

- name: upgrade-all-service-instances
  lifecycle: errand
  instances: 1
  jobs:
    - name: upgrade-all-service-instances
      release: ODB-RELEASE-NAME
      properties:
        canaries: NUMBER-OF-CANARIES # defaults to 0
        canary_selection_params:
          cf_org: ORG # specifying service instances to upgrade as canaries
          cf_space: SPACE # specifying service instances to upgrade as canaries
        max_in_flight: NUMBER-OF-PARALLEL-UPGRADES # defaults to 1
        polling_interval_seconds: POLLING-INTERVAL-IN-SECONDS # defaults to 60
        attempt_interval_seconds: ATTEMPT-INTERVAL-IN-SECONDS # defaults to 60
        attempt_limit: NUMBER-OF-ATTEMPTS # defaults to 5
        broker_uri: BROKER-URI # optional
        tls: # optional
          ca_cert: BROKER-CA-CERT
          disable_ssl_cert_verification: TRUE|FALSE # defaults to false
  vm_type: VM-TYPE
  stemcell: STEMCELL
  networks: [{name: NETWORK}]
  azs: [AZ]

The errand properties allow fine-tuning of the behavior of the upgrade job:

Property	Description
`max_in_flight`	Sets the limit for the number of upgrades occurring concurrently. The number of simultaneous upgrades is limited by the number of available BOSH workers. See workers in the Cloud Foundry BOSH documentation. Set the `max_in_flight` value to lower than this limit to avoid over-saturating BOSH.
`canaries`	Sets the number of canary instances to upgrade first. If all canary instances upgrade, the remaining instances are upgraded. If a canary instance fails to upgrade or the `attempt_limit` is reached, the upgrade fails. No further instances are upgraded, and the errand exits with an error; however, all in-flight upgrades can complete. Canary instances are upgraded in parallel, respecting the `max_in_flight` value. Note: Canary instances are selected in a non-deterministic way using all available instances. If a selected instance is busy or was deleted, another instance is selected. If all instances are busy, the errand retries, respecting the `attempt_limit` and `attempt_interval_seconds`.
(Optional) `canary_selection_params`	Use this to specify an org and a space that you want canaries to be sourced from during an upgrade. If an org is specified, then a space must also be provided and vice versa. If `canaries` is specified, the broker upgrades that number of instances present in the org and space. If fewer instances are present than specified, the broker upgrades as many instances as possible in that org and space. Note: If `canary_selection_params` are specified and no instances exist in that org or space, no canaries are chosen. If other instances exist, the broker fails, alerting you to chose different selection criteria. If `canary_selection_params` is specified but empty, it is treated as if none was provided.
`polling_interval_seconds`	This controls the wait between checking the status of the successfully submitted BOSH upgrade job. If service instances have in-progress BOSH operations, upgrade requests are rejected and the errand queues those instances for a retry: `attempt_interval_seconds` determines the time to wait between retrying upgrades. `attempt_limit` sets the number of times these instances are retried for upgrade.

Deploy the broker manifest by running:

bosh -e BOSH-DIRECTOR-NAME -d DEPLOYMENT-NAME deploy DEPLOYMENT-MANIFEST.yml

Run the errand with the following command:

bosh -d DEPLOYMENT-NAME run-errand upgrade-all-service-instances

Note The upgrade-all-service-instances errand triggers service instance lifecycle errands configured for the broker. For more information, see Service Instance Lifecycle Errands.

Note When the broker URI of this errand is not configured, the errand defaults to a BOSH-provided IP address or BOSH-provided BOSH DNS address. See Impact on links in the BOSH documentation.

When the broker is using TLS, the broker certificate must contain this BOSH provided address in its Subject Alternative Names section, otherwise the certificate cannot be verified by Cloud Foundry. For details about how to insert a BOSH DNS address into a config server generated certificate, see BOSH DNS Addresses in Config Server Generated Certs in the BOSH documentation.

Service Instances API

The service instances API is used with an on-demand service broker without a connection to Cloud Foundry. It consists of an HTTP endpoint that the orphan-deployments and upgrade-all-service-instances errands can use to retrieve the list of instances known to the platform.

You can configure the API endpoint in the broker job.

Caution The Service Instances API is an advanced feature. Configuring a Service Instances API is not required, unless you are operating in very specific circumstances, such as running an on-demand service broker with no connection to Cloud Foundry.

You are required to provide an endpoint which satisfies the API requirements detailed below and also guarantees that all service instances which need to be upgraded are part of that response. If you configure an endpoint which does not satisfy these criteria, some service instances may become unusable.

Note The Service Instances API definition is at v0.1.0 and should be considered subject to change.

General Request and Response

The API endpoint must handle the request below with the appropriate response.

Example Request:

curl -u BASIC-AUTH-USERNAME:BASIC-AUTH-PASSWORD \
-X GET \
http://SERVICE-INSTANCES-API-URL

Example Response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {"service_instance_id": GUID-1, "plan_id": "dedicated"},
  {"service_instance_id": GUID-2, "plan_id": "dedicated"},
  {"service_instance_id": GUID-3, "plan_id": "dedicated"},
  [...]
]

The Services Instances API returns other status codes such as 3xx, 4xx, or 5xx for non-successful scenarios.

The plan IDs listed in the response must correspond to the plan IDs in each plan’s definition in the ODB deployment manifest, not any other ID that may be assigned by the service controller.

Filtered Request and Response

The API endpoint must be able to provide a filtered response based on query parameters passed in the request.

The canary_selection_params property defines how to filter the canary instances from of a set of instances. If the on-demand service broker is configured to use canary_selection_params, The Service Instances API must respond with a filtered list of service instances when the canary_selection_params are passed as query parameters.

For example, if the canary_selection_params are configured as following:

canary_selection_params:
  cf_org: staging-org
  cf_space: staging-space

Then, the Service Instances API returns a filtered list of instances when cf_org and cf_space are passed as query parameters in the request.

Example Request:

curl -u BASIC-AUTH-USERNAME:BASIC-AUTH-PASSWORD \
-X GET \
http://SERVICE-INSTANCES-API-URL?cf_org=staging-org&cf_space=staging-space

Example Response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {"service_instance_id": GUID-1, "plan_id": "dedicated"}
]

The instances in the filtered list are used as canary instances and upgraded before the rest. The number of canaries taken from this list can be configured by specifying the canaries property in the on-demand service broker manifest.

Configure the Broker to Use the Service Instances API

To configure the broker to use a Service Instance API provider for errands, update the broker’s configuration in its deployment manifest to include the service_instances_api section:

- name: upgrade-all-service-instances
  ...
  jobs:
    - name: broker
      ...
      properties:
        ...
        # Add the following section:
        service_instances_api:
          # required:
          url: SERVICE-INSTANCES-API-URL
          # optional:
          root_ca_cert: ROOT-CA-CERT
          # optional - defaults to false:
          disable_ssl_cert_verification: TRUE|FALSE
          authentication:
            # required - currently the only supported authentication type:
            basic:
              username: USERNAME
              password: PASSWORD
 ...

Where SERVICE-INSTANCES-API-URL is the URL of the Service Instances API provider.