VMware recommends that you perform Concourse backups with BOSH Backup and Restore (BBR). The process involves the following prerequisites:
bbr
, the BOSH Backup and Restore (BBR) CLI toolfly
, Concourse's CLI toolThis guide details the process of setting this system up. Once set up, you can follow the steps in the the Performing a Backup section of this guide.
Then, once you've created a valid BBR backup of Concourse, you can use the Performing a restore with BBR guide.
Use the table of contents on the right to navigate this guide.
You can use the bosh releases
command to see what you have installed so far. A backup-and-restore-sdk
release is required for each Concourse deployment you want to backup from or restore to.
Any Concourse deployment you wish to backup from or restore to must have the backup-and-restore-sdk
release. If you are starting from scratch on a new workstation, the instructions in this section cover cloning the concourse-bosh-deployment
repository to a local directory, so that you can craft a new deployment with the appropriate release and configuration. If you already have a deployment directory with manifest and ops files on your workstation, navigate to that directory before jumping ahead to the Set up the BBR SDK on your BOSH Environment below.
Clone the concourse-bosh-deployment repository by running the following command:
git clone https://github.com/concourse/concourse-bosh-deployment.git
Move to the concourse-bosh-deployment
directory:
cd concourse-bosh-deployment
All the paths used in this topic are relative to this directory.
Checkout the release that corresponds to your working version of Concourse. For example:
git checkout v5.5.11
git checkout v4.2.4
git checkout v3.13.0
For a list of all Concourse releases, see concourse-bosh-deployment in GitHub.
Checking out a release rather than a branch means that git produces the following output:
Note: checking out 'RELEASE'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at HASH... COMMIT-MESSAGE
If you already have your CA certificate and have already logged in to and aliased your BOSH environment, you can skip this section.
Get a CA certificate for your BOSH Director.
If you created your BOSH Director manually, retrieve the credentials that were created during setup.
If you are working with an Operations Manager-deployed BOSH Director, log in to Operations Manager and access the following endpoint in your Tanzu Operations Manager domain:
https://OPS-MANAGER-DOMAIN/download_root_ca_cert
Where OPS-MANAGER-DOMAIN
is the Operations Manager domain.
This will create a file in your downloads folder called root_ca_certificate
.
Copy and rename the root_ca_certificate
file into the working directory. For example, on OSX:
mv ~/Downloads/root_ca_certificate ./ca-cert.crt
This certificate file can be used whenever a CLI command asks for a --ca-cert
flag and value.
Ensure you are logged in to your BOSH environment with the appropriate BOSH Director credentials. For example, if you saved your CA certificate as ca-cert.crt
, run the following command:
bosh -e BOSH-ENVIRONMENT-IP login --ca-cert=ca-cert.crt
Where BOSH-ENVIRONMENT-IP
is your BOSH environment IP address.
If you set up a BOSH Director directly, the director credentials were returned as a file after the setup process finished.
If you have an Operations Manager-deployed BOSH Director, you can find the director credentials in the Operations Manager credentials tab or at the following endpoint in your Operations Manager domain:
http://OPS-MANAGER-DOMAIN/api/v0/deployed/director/credentials/director_credentials
Where OPS-MANAGER-DOMAIN
is your Operations Manager domain.
Give your environment an alias by running the following command:
bosh -e BOSH-ENVIRONMENT-IP alias-env ALIAS --ca-cert=ca-cert.crt
Where:
BOSH-ENVIRONMENT-IP
is your BOSH environment IP addressALIAS
is the alias you're creating BOSH environmentUsing an alias for your BOSH environment substantially reduces the keystrokes needed for commands in future. You can use this alias whenever you target this environment by using the -e
flag in a BOSH command.
This section details the task of uploading the BOSH backup-and-restore-sdk
release to a Concourse environment.
Upload the backup-and-restore-sdk
release by running the following upload-release
command:
bosh \
-e BOSH-ENVIRONMENT-ALIAS \
upload-release \
--sha1 364838c384f2edec80866b4abf2397c4c5d15c62 \
https://bosh.io/d/github.com/cloudfoundry-incubator/backup-and-restore-sdk-release?v=1.15.1
Where BOSH-ENVIRONMENT-ALIAS
is your BOSH environment alias.
If you run the bosh releases
command again, you should see backup-and-restore-sdk
in the list.
An alternative approach is to download the latest release from the BOSH Backup and Restore Release Page and pass in the path to the file in the following format:
bosh -e BOSH-ENVIRONMENT-ALIAS upload-release PATH-TO-FILE
If you're backing up version 3.13.0, you need to create and edit a file called backup-atc.yml
in ./cluster/operations
. Versions 4.2.4 and onwards of the concourse-bosh-deployment
repo come with this file already.
For example:
vim ./cluster/operations/backup-atc.yml
Edit this file so that it has the following contents:
---
- type: replace
path: /releases/-
value:
name: backup-and-restore-sdk
url: https://bosh.io/d/github.com/cloudfoundry-incubator/backup-and-restore-sdk-release?v=((bbr_sdk_version))
sha1: ((bbr_sdk_sha1))
version: ((bbr_sdk_version))
- type: replace
path: /instance_groups/name=web/jobs/-
value:
name: bbr-atcdb
release: concourse
properties:
postgresql:
database: atc
role:
name: ((postgres_role))
password: ((postgres_password))
postgresql_database: atc
- type: replace
path: /instance_groups/name=web/jobs/-
value:
name: database-backup-restorer
release: backup-and-restore-sdk
Save and close the backup-atc.yml
file.
Open your versions.yml
file:
vim versions.yml
Update versions.yml
by adding lines for the backup-and-restore-sdk's version and SHA1. The two properties you need to define are bbr_sdk_version
and bbr_sdk_sha1
.
To deploy backup-and-restore-sdk 1.15.1, copy and paste the highlighted lines into your versions.yml
file:
# this file is partially maintained by CI; the concourse and garden-runc
# versions and sha1s are automatically bumped, while the rest are preserved
# as-is.
#
# this should make getting started easy while being easy enough to maintain
# manually. feel free to PR sane defaults along with newly supported
# infrastructures and such!
---
concourse_version: '5.5.11'
concourse_sha1: '876a8aef1c790c6e8578a3b7c11c2265072f0565'
garden_runc_version: '1.13.1'
garden_runc_sha1: '54cbb89cae1be0708aa056185671665d7f4b2a4f'
postgres_version: '36'
postgres_sha1: '3dd10b417b21cfa3257f1cc891e9e46f02fefe16'
virtualbox_cpi_version: '0.0.12'
virtualbox_cpi_sha1: 537c77b087eeda859b1eb89553efa6f808214d50
virtualbox_stemcell_version: '3468.5'
virtualbox_stemcell_sha1: 25301a41c15caad859b66efc0fd75bacdfc57dab
bbr_sdk_version: '1.15.1'
bbr_sdk_sha1: '364838c384f2edec80866b4abf2397c4c5d15c62'
# this file is partially maintained by CI; the concourse and garden-runc
# versions and sha1s are automatically bumped, while the rest are preserved
# as-is.
#
# this should make getting started easy while being easy enough to maintain
# manually. feel free to PR sane defaults along with newly supported
# infrastructures and such!
---
concourse_version: '4.2.4'
concourse_sha1: '56122002ce49af09b03b8c3a49bdb7578ee2868a'
garden_runc_version: '1.13.1'
garden_runc_sha1: '54cbb89cae1be0708aa056185671665d7f4b2a4f'
postgres_version: '36'
postgres_sha1: '3dd10b417b21cfa3257f1cc891e9e46f02fefe16'
virtualbox_cpi_version: '0.0.12'
virtualbox_cpi_sha1: 537c77b087eeda859b1eb89553efa6f808214d50
virtualbox_stemcell_version: '3468.5'
virtualbox_stemcell_sha1: 25301a41c15caad859b66efc0fd75bacdfc57dab
bbr_sdk_version: '1.15.1'
bbr_sdk_sha1: '364838c384f2edec80866b4abf2397c4c5d15c62'
# this file is partially maintained by CI; the concourse and garden-runc
# versions and sha1s are automatically bumped, while the rest are preserved
# as-is.
#
# this should make getting started easy while being easy enough to maintain
# manually. feel free to PR sane defaults along with newly supported
# infrastructures and such!
---
concourse_version: '3.13.0'
concourse_sha1: 'aaa4e4d42adb2293abc79422351ca71ed548f95c'
garden_runc_version: '1.13.1'
garden_runc_sha1: '54cbb89cae1be0708aa056185671665d7f4b2a4f'
postgres_version: '36'
postgres_sha1: '3dd10b417b21cfa3257f1cc891e9e46f02fefe16'
virtualbox_cpi_version: '0.0.12'
virtualbox_cpi_sha1: 537c77b087eeda859b1eb89553efa6f808214d50
virtualbox_stemcell_version: '3468.5'
virtualbox_stemcell_sha1: 25301a41c15caad859b66efc0fd75bacdfc57dab
bbr_sdk_version: '1.15.1'
bbr_sdk_sha1: '364838c384f2edec80866b4abf2397c4c5d15c62'
If you want to use a different version of backup-and-restore-sdk-release
, see backup-and-restore-sdk Release to find the version and SHA1 for your release.
Deploying Concourse with BOSH requires you to set some variables to fill in details in the manifest that are written as placeholders. These are in the format ((placeholder))
. The currently deployed Concourse manifest contains the values that you will use to fill out each of the variables.
The preferred approach is to create a file to define a key/value pair for each required variable, and check it into version control so that it can be safely and easily maintained over time. You might already have a solution for this; if not, follow the steps below.
The following steps describe setting variables that are necessary for deployment in a yaml file. If you prefer to set them on the command line when you run bosh deploy
, you can pass them in with the --var KEY=VALUE
flag and syntax. For more information about bosh deploy
, see Deploy in the BOSH documentation.
Run the following command to fetch the manifest and put it in a deployed-manifest.yml
file:
bosh \
-e BOSH-ENVIRONMENT-ALIAS \
-d NAME-OF-CONCOURSE-DEPLOYMENT \
manifest > deployed-manifest.yml
Sending the response to a file is a security precaution that ensures that any included keys are not echoed to your terminal session history. Ensure that you destroy this file after you are done with this backup process.
If you do not already have a variable file from a previous deployment, create a file to store your Concourse and BOSH-related environment variables. For example, use vim to create a file called variables.yml
by running:
vim variables.yml
This file will hold all of the variables that BOSH needs to interpolate into references in the manifest file.
If you have existing variables file, open that file instead.
The benefit of creating a file for these variables is that you can check this file into your version control, track changes over time, and more easily document and share this information with others in your organization. This also saves you from looking up configuration details every time you need to change your Concourse deployment.
Ensure you have the correct key/value pairs in your variables file:
deployed-manifest.yml
file that created in step 1 of this section.---
local_user:
username: CHOOSE-A-USERNAME
password: CHOOSE-A-PASSWORD
deployment_name: DEPLOYMENT-NAME
db_persistent_disk_type: PERSISTANT-DISK-TYPE-FOR-DATABASE
db_vm_type: VM-TYPE-FOR-DATABASE-INSTANCE
external_url: EXTERNAL-URL
network_name: NETWORK-NAME
postgres_password: PASSWORD
web_ip: WEB-IP
web_network_name: WEB-NETWORK-NAME
web_vm_type: VM-TYPE-FOR-WEB-INSTANCE
worker_vm_type: VM-TYPE-FOR-WORKER-INSTANCE
---
local_user:
username: CHOOSE-A-USERNAME
password: CHOOSE-A-PASSWORD
deployment_name: DEPLOYMENT-NAME
db_persistent_disk_type: PERSISTANT-DISK-TYPE-FOR-DATABASE
db_vm_type: VM-TYPE-FOR-DATABASE-INSTANCE
external_url: EXTERNAL-URL
network_name: NETWORK-NAME
postgres_password: PASSWORD
web_ip: WEB-IP
web_network_name: WEB-NETWORK-NAME
web_vm_type: VM-TYPE-FOR-WEB-INSTANCE
worker_vm_type: VM-TYPE-FOR-WORKER-INSTANCE
---
deployment_name: DEPLOYMENT-NAME
db_persistent_disk_type: PERSISTANT-DISK-TYPE-FOR-DATABASE
db_vm_type: VM-TYPE-FOR-DATABASE-INSTANCE
external_url: EXTERNAL-URL
network_name: NETWORK-NAME
postgres_password: PASSWORD
web_ip: WEB-IP
web_network_name: WEB-NETWORK-NAME
web_vm_type: VM-TYPE-FOR-WEB-INSTANCE
worker_vm_type: VM-TYPE-FOR-WORKER-INSTANCE
Where:
DEPLOYMENT-NAME
is your Concourse deployment name. You can find this by running:
bosh -e BOSH-ENVIRONMENT-ALIAS deployments
NETWORK-NAME
is the name subproperty of the network property in your deployed manifest.
PASSWORD
is the password sub-property of the postgres property in your deployed manifest.
bosh \
-e BOSH-ENVIRONMENT-ALIAS \
-d TARGET-CONCOURSE-DEPLOYMENT \
manifest | grep “(("
This command searches through your deployed manifest for any placeholders that are being interpolated after deployment. If this returns any values, this could indicate that you have something like CredHub involved. Before you proceed, stop to investigate your current deployment, or get in touch with someone who has the necessary context.
Save and close the file.
The Concourse manifest has some defaults set for availability zones (AZs). You need to modify these defaults to suit a given environment. To ensure these are set correctly, follow these steps:
Retrieve the values of the azs
properties in your currently deployed manifest.
To find these, reference the deployed-manifest.yml
file created in the last section, or run the bosh manifest
command to re-create it:
bosh \
-e BOSH-ENVIRONMENT-ALIAS \
-d CONCOURSE-DEPLOYMENT-NAME \
manifest > deployed-manifest.yml
Open the concourse-bosh-deployment
repository's cluster Concourse manifest file:
vim ./cluster/concourse.yml
Search through this manifest file to find any azs
properties and edit each one of these so that they match the availability zones (azs) in your deployed-manifest.yml
file. There should be three in total, one in each instance group.
For example, if your availability zone is default
:
...
instance_groups:
- name: web
...
azs: [default]
...
- name: db
...
azs: [default]
...
- name: worker
...
azs: [default]
...
...
Save and close the ./cluster/concourse.yml
file.
Deploy the new setup to your BOSH environment. The bosh deploy
command uses a manifest that acts as a recipe for the deployment that you are creating on your cloud environment. When you run bosh deploy
, you need to incorporate the following:
./cluster/concourse.yml
versions.yml
variables.yml
./cluster/operations/backup-atc.yml
filecluster-creds.yml
Operations files make substitutions in your manifest file to customize your deployment. The example below demonstrates a simple setup with basic authentication.
To change the authentication type, change the -o ./cluster/operations/basic-auth.yml
line.
Edit and run this snippet, substituting your environment alias and deployment name, along with any additional ops files you want to add.
For example:
bosh -e BOSH-ENVIRONMENT-ALIAS deploy -d CONCOURSE-DEPLOYMENT-NAME ./cluster/concourse.yml \
-l versions.yml \
-l variables.yml \
-o ./cluster/operations/backup-atc.yml \
-o ./cluster/operations/basic-auth.yml \
-o ./cluster/operations/privileged-http.yml \
-o ./cluster/operations/static-web.yml \
--vars-store cluster-creds.yml
When prompted to review the deployment, you should see the addition of the backup-and-restore-sdk and its configuration. If you continue, BOSH prepares the deployment and starts compiling packages for you.
If you have CPI errors, try running the deployment script again. In rare cases, the operation can time out while packages are being built, causing a CPI timeout. The deployment should be successful if you re-run it a second time, as by that point the packages are already built.
To install the BBR CLI on your computer, do the following:
Check if BBR is already installed by running the following command. If this returns a version number instead of an error, you can skip to the next section.
bbr -v
Download the latest release's asset file that corresponds to your operating system from the bosh-backup-and-restore GitHub repository. For example, Apple OSX users should download the darwin-amd64 file.
There is no Windows release for the BBR CLI. If you are on Windows, you can SSH into a jumpbox where you can use a Linux release of BBR. For instructions, see Installing Bosh Backup and Restore (BBR) in the Cloud Foundry documentation. You can then continue to the Performing a Backup section below.
Move the file into your local /bin and rename it to bbr
by running the following command:
mv ~/Downloads/BBR-BINARY-FILE-NAME /usr/local/bin/bbr
Make the file executable by running the following command:
chmod +x /usr/local/bin/bbr
To run a backup with the bbr
command, do the following:
Retrieve the UAA BBR Client Credentials, username and password, from the following endpoint on your environment:
https://YOUR-OPS-MANAGER-DOMAIN/api/v0/deployed/director/credentials/uaa_bbr_client_credentials
Run bbr
with the following command. Make sure to use the environment IP address instead of using the environment alias you used with the bosh
commands above.
bbr deployment \
--target ENVIRONMENT-IP-ADDRESS \
--deployment TARGET-CONCOURSE-DEPLOYMENT-NAME \
--username UAA-BBR-USERNAME \
--password UAA-BBR-PASSWORD \
--ca-cert CA-CERTIFICATE-FILE \
backup
A successful backup results in the creation of a directory named after your deployment name and the date and time of the backup.
Move the backup directory to a secure location. Note that this backup might include sensitive information, such as passwords in builds, depending on how you have configured your pipelines.
If you need to restore a backup, see Performing a Restore.
When you are ready to continue with the upgrade process, proceed to the guide for Upgrading Concourse with BOSH or Upgrading Concourse with Helm.