Upgrading Concourse involves following these steps:
This step-by-step tutorial includes details for checking out sample manifest and supporting files for the desired deployment version from the
concourse-bosh-deployment repository. These can be used as a template for customizing your release.
The guide below is written to provide necessary details for someone starting this process from scratch on a new workstation. You can skip certain steps if you have deployed from
concourse-bosh-deployment before and already have a working directory, credentials, and/or cloud configuration variables.
Note If you have not backed up your current Concourse deployment, follow the steps in Backup Concourse before proceeding.
concourse-bosh-deploymentDirectory on Your Local Machine
Success Skip step 1 if you already have `concourse-bosh-deployment` cloned to a local directory.
concourse-bosh-deployment repository to your workstation by running the following command:
git clone https://github.com/concourse/concourse-bosh-deployment.git
Navigate to the
concourse-bosh-deployment directory by running the following command:
All the paths used in this guide are relative to this directory.
Check your Git Status VMware recommends checking your Git status whenever you enter the `concourse-bosh-deployment` directory.Do so by running the following command:
If you encounter unstaged files that can be overwritten by checking out the new release branch,
git stash your changes, check out the release, and then
git stash pop and resolve any merge conflicts.
Check out the
concourse-bosh-deployment Git tag that corresponds to your desired upgrade version. For example, with the latest
git checkout v7.4.2
Tip Run `git tag` to see a list of the different releases.
Checking out a release, rather than a branch, produces the following output:
Note: checking out RELEASE. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, use `-b` with the checkout command again. For example: git checkout -b <new-branch-name> HEAD is now at HASH COMMIT-MESSAGE
RELEASE is the release version number.
HASH is the hash key of the most recent commit.
COMMIT-MESSAGE is the most recent commit message.
Note If you already have your CA certificate and have already logged in to and aliased your BOSH environment, you can skip this section.
Get a CA certificate for your BOSH Director.
If you created your BOSH Director manually, retrieve the credentials that were created during setup.
If you are working with an Ops Manager-deployed BOSH Director, log in to Ops Manager and access the following endpoint in your Ops Manager domain:
OPS-MANAGER-DOMAIN is the Ops Manager domain.
This will create a file in your downloads folder called
Copy and rename the
root_ca_certificate file into the working directory. For example, on OSX:
mv ~/Downloads/root_ca_certificate ./ca-cert.crt
Tip This certificate file can be used whenever a CLI command asks for a
--ca-cert flag and value.
Ensure you are logged in to your BOSH environment with the appropriate BOSH Director credentials. For example, if you saved your CA certificate as
ca-cert.crt, run the following command:
bosh -e BOSH-ENVIRONMENT-IP login --ca-cert=ca-cert.crt
BOSH-ENVIRONMENT-IP is your BOSH environment IP address.
Finding director credentials If you set up a BOSH Director directly, the director credentials were returned as a file after the setup process finished.
If you have an Ops Manager-deployed BOSH Director, you can find the director credentials in the Ops Manager credentials tab or at the following endpoint in your Ops Manager domain:
OPS-MANAGER-DOMAIN is your Ops Manager domain.
Give your environment an alias by running the following command:
bosh -e BOSH-ENVIRONMENT-IP alias-env ALIAS --ca-cert=ca-cert.crt
BOSH-ENVIRONMENT-IPis your BOSH environment IP address
ALIASis the alias you're creating BOSH environment
Using an alias for your BOSH environment substantially reduces the keystrokes needed for commands in future.
Tip You can use this alias whenever you target this environment by using the
-e flag in a BOSH command.
Deploying with BOSH requires setting some variables to fill in details in your manifest. These are written as placeholders in the following format:
You can do this in a number of ways, but VMware recommends creating a file to define a key-value pair for each required variable, and checking it into version control so that it can be maintained and tracked over time. Follow the steps below to do so.
Tip The following steps describe setting variables that are necessary for deployment in a YAML file. If you prefer to set them on the command line when you run `bosh deploy`, you can pass them in at that time with the `--var KEY=VALUE` flag and syntax. For more information, see [Deploy](https://bosh.io/docs/cli-v2/#deploy) in the BOSH documentation.
To find the values needed to fill out each of the variables in the next step, check the currently deployed Concourse manifest. Use the following command to tell BOSH to fetch the manifest and put it in a
bosh \ -e BOSH-ENVIRONMENT-ALIAS \ -d CONCOURSE-DEPLOYMENT-NAME \ manifest > deployed-manifest.yml
BOSH-ENVIRONMENT-ALIASis your BOSH environment alias
CONCOURSE-DEPLOYMENT-NAMEis your Concourse deployment name.
This command produces a file that contains a value for each of the variables that you define in the next step.
Note Sending the response to a file is a security precaution that ensures that any included keys are not echoed to your terminal session history. Destroy this file after you are done with this upgrade process.
Create a file to store your Concourse- and BOSH-related environment variables, if you do not already have one from a previous deployment. For example, the following command uses Vim to create a file called
This file holds all the variables that BOSH needs to interpolate into references in the manifest file. The manifest file then serves as a recipe for the new deployment.
Why create a file for these variables? The benefit to doing things this way is that you can check this file into your version control, track changes over time, and more easily document and share this information with others in your org. It also saves you the trouble of looking up configuration details every time you need to change your Concourse deployment.
Add the key-value pairs shown in the snippet below into the file.
--- local_user: username: USERNAME password: PASSWORD deployment_name: DEPLOYMENT-NAME db_persistent_disk_type: PERSISTENT-DISK-TYPE db_vm_type: VM-TYPE external_url: EXTERNAL-URL network_name: NETWORK-NAME postgres_password: POSTGRESQL-PASSWORD web_ip: WEB-IP web_network_name: WEB-NETWORK-NAME web_vm_type: WEB-VM-TYPE worker_vm_type: WORKER-VM-TYPE azs: AVAILABILITY-ZONES
DEPLOYMENT-NAME is your deployment name
PERSISTENT-DISK-TYPE is the persistent disk type for the database
VM-TYPE is the VM type for the database instance
EXTERNAL-URL is the external URL
NETWORK-NAME is the name subproperty of the network property in your deployed manifest
POSTGRESQL-PASSWORD is the password subproperty of the PostgreSQL property in your deployed manifest
WEB-IP is your web IP
WEB-NETWORK-NAME is your web network name
WEB-VM-TYPE is the VM type for the web instance
WORKER-VM-TYPE is the VM type for the worker instance
AVAILABILITY-ZONES is the
name of one of the
azs in your
Replace each value placeholder with the appropriate properties in the
deployed-manifest.yml file that you created earlier.
Optional Tip: If Concourse is deployed with a component such as CredHub, these variables might be automatically interpolated. You can check if this is the case by running the following command:
bosh \ -e BOSH-ENVIRONMENT-ALIAS \ -d TARGET-CONCOURSE-DEPLOYMENT \ manifest | grep “(("
If the search returns any values, CredHub or a similar component might be involved. In this case, investigate your current deployment, or get in touch with whoever has the necessary context, before proceeding further.
Save and close your variables file.
Your deployed manifest, located at
./cluster/concourse.yml, has some defaults set for availability zones (AZs). Often these must be customized to suit a given environment. To ensure these are set correctly, follow these steps:
Check the AZ names in your currently deployed manifest.
To find these, you can reference the
deployed-manifest.yml file we created in the last section, or run the
bosh manifest command to re-create it:
bosh \ -e BOSH-ENVIRONMENT-ALIAS \ -d CONCOURSE-DEPLOYMENT-NAME \ manifest > deployed-manifest.yml
Open the Concourse manifest file of the
concourse-bosh-deployment repository cluster by running the following command:
Change each of the three
az properties in the manifest file so that they match the AZs in your
deployed-manifest.yml file. For example, if your AZ is
default, your manifest file looks like this:
... instance_groups: - name: web ... azs: [default] ... - name: db ... azs: [default] ... - name: worker ... azs: [default] ... ...
Tip You can use a find-and-replace operation in a text editor to change all these values simultaneously.
Save and close the
??? tip "Tip for upgrading very large Concourses" If you have many pipelines with many containers, pause your pipelines before you start the upgrade. Then, after you finish the upgrade, wait for all of your workers to be ready and then turn on your pipelines one by one.
VMware recommends doing this to prevent your pipelines from overwhelming the first workers that come online, which could cause a crash.
To deploy your new Concourse version, you need the following information:
--vars-storeflag with the name of a file in which BOSH can store your cluster credentials, such as
About the `cluster-creds.yml` file BOSH creates this file for you if it does not already exist. If this is the case, you still need to specify a filename for BOSH to use. In the case that you already have a `cluster-creds.yml` file, specify it here so that BOSH does not re-create it.
If you specify a file that does not exist, and BOSH re-creates this file for you, it also re-creates things such as passwords and secrets that exist within the file.
When you are ready, navigate to your terminal and run the following
bosh deploy command:
bosh deploy \ -e BOSH-ENVIRONMENT-ALIAS \ -d CONCOURSE-DEPLOYMENT-NAME ./cluster/concourse.yml \ -l versions.yml \ -l variables.yml \ -o ./cluster/operations/backup-atc.yml \ -o ./cluster/operations/basic-auth.yml \ -o ./cluster/operations/privileged-http.yml \ -o ./cluster/operations/static-web.yml \ --vars-store cluster-creds.yml
Danger Different Concourse deployments require different environment variables and operations files. If you encounter an error, check the error message for clues about additional variables that need to be set. For more information, see the Concourse documentation.