Upgrading Concourse with Helm

These topics provide a high-level overview of upgrading a Concourse deployment using Helm, including command requirements and common errors.

Notes about breaking changes and other information can be found in the release notes.

This topic describes how to upgrade from v6.3.x to v6.7.x - if you are using a previous release of Concourse for VMware Tanzu, consult previous upgrade guides to first migrate to v6.3.x before you continue.

Introduction

Helm is the package manager for Kubernetes, a tool that streamlines installing and managing Kubernetes applications. It creates Kubernetes objects that can be submitted to Kubernetes clusters, and materialized into a Concourse deployment using Kubernetes constructs (Deployments, StatefulSets, PersistentVolumeClaims, etc).

A Helm Chart is a versioned package of pre-configured Kubernetes resources. Deploying Concourse via Helm Chart makes it easy to deploy, scale, maintain, or upgrade your deployment in the future. This guide aims to walk an operator through the step by step process of deploying with Helm.

Once you've deployed Concourse to a Kubernetes cluster, you can use the following process to keep it up to date as new releases arrive on the Broadcom Support portal.

Helm v2 vs Helm v3

Prior to Concourse v6.7.x, the recommended way to deploy Concourse via Helm was using Helm v2. As of v6.7.x, Helm v2 is now deprecated and the required way is to use Helm v3. You can read about v3 here.

The recommended upgrade path is to upgrade Helm v2 → Helm v3 first, and then Concourse v6.3.x → Concourse v6.7.x. For additional details on migrating Helm v2 apps to v3, see the Helm documentation.

For the rest of this article, whenever helm is used, we are referring to the Helm v3 CLI. This CLI will be the one packaged in the Broadcom Support portal release. You can check what version of the CLI you're using by running:
helm version
The output looks something like this:
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}

Using the Helm v2 CLI

This example requires the use of Helm v2 CLI to figure out the release name, the v6.3.1 Broadcom Support portal release contains a helm binary that is v2.

In the following example, the RELEASE-NAME is upgrade-test:

$ helm2 list
NAME          REVISION  UPDATED                   STATUS    CHART             APP VERSION NAMESPACE
upgrade-test  1         Tue Jan 26 16:28:12 2021  DEPLOYED  concourse-11.1.0  6.3.1       default

You should see output that looks similar to:

2021/01/27 16:01:10 Release "upgrade-test" will be converted from Helm v2 to Helm v3.
2021/01/27 16:01:10 [Helm 3] Release "upgrade-test" will be created.
2021/01/27 16:01:10 [Helm 3] ReleaseVersion "upgrade-test.v1" will be created.
2021/01/27 16:01:10 [Helm 3] ReleaseVersion "upgrade-test.v1" created.
2021/01/27 16:01:10 [Helm 3] Release "upgrade-test" created.
2021/01/27 16:01:10 Release "upgrade-test" was converted successfully from Helm v2 to Helm v3.
2021/01/27 16:01:10 Note: The v2 release information still remains and should be removed to avoid conflicts with the migrated v3 release.
2021/01/27 16:01:10 v2 release information should only be removed using `helm 2to3` cleanup and when all releases have been migrated over.

Backup

In the case of a Helm-deployed Concourse pointed to a Postgres database that has been deployed and managed with BOSH, you can use BBR to perform a backup.

Otherwise, use the pg_dump utility to backup your Postgres database. For more information on the tool, click here to view a guide on postgresql.org.

Next, download, tag, and push images to the internal registry.

Download Concourse Helm Chart and load images into Docker

If you have not already done so, visit the Broadcom Support portal and download the Concourse Helm Chart.
Unarchive the Helm Chart tarball to a local directory. For example, with version v7.9.1, the tarball will be called concourse-7.9.1.tgz.
```
mkdir concourse-helm
```
```
tar xvzf ./concourse-7.9.1.tgz -C ./concourse-helm
```
```
cd ./concourse-helm
```
Load the container images into a local Docker client by running the following docker load commands one at a time:
```
docker load -i ./images/concourse.tar
docker load -i ./images/postgres.tar
```
If you are using Helm v2 (i.e. Concourse versions < v6.7.x) you'll also need to load the helm image:
```
docker load -i ./images/helm.tar
```
These images are quite large, and there will be no output until Docker is done loading.

Success: After the loading finishes, you'll see:
```
Loaded image: IMAGE-NAME
```

Tag and push the loaded images to internal registry

This step assumes that the current docker client has already authenticated against the internal registry through a regular docker login.

In addition to logging in, if you're using a registry with self-signed certificates, you should also make sure your registry has been added to the Insecure Registries section of the Daemon tab in the Docker settings ui for your current workstation.

For more information about certificates and secure registry concerns, see this article: Test an insecure registry.

Begin by exporting a pair of variables to your shell to be reused throughout this process. In your terminal, run the following commands.
- If using Harbor (internal registry):
```
export INTERNAL_REGISTRY=INTERNAL-REGISTRY
export PROJECT=PROJECT-NAME
```
  Where:
  - INTERNAL-REGISTRY is the domain of your internal registry - if you are using Harbor, this must correspond to the URL (without scheme)
  - PROJECT-NAME is the name of the project in your registry. If the project does not exist already you will need to make it.
- If using Docker Hub:
```
export USERNAME=DOCKERHUB-USERNAME
```
  Where:
  - DOCKERHUB-USERNAME is your username on hub.docker.io
The .tar file you downloaded contains a directory called images. You need to extract the tag of each image so that you can appropriately tag the images with the internal registry and project name details from the last step.

To do this, run the following commands:
```
export CONCOURSE_IMAGE_TAG=$(cat ./images/concourse.tar.name | cut -d ':' -f 2)
export POSTGRES_IMAGE_TAG=$(cat ./images/postgres.tar.name | cut -d ':' -f 2)
```
If you are using Helm v2 (i.e. Concourse versions < v6.7.x) you'll also need to tag the helm image:
```
export HELM_IMAGE_TAG=$(cat ./images/helm.tar.name | cut -d ':' -f 2)
```

Tag the images so their names include the internal registry address.

If using Harbor (internal registry):

docker tag concourse/concourse:$CONCOURSE_IMAGE_TAG $INTERNAL_REGISTRY/$PROJECT/concourse:$CONCOURSE_IMAGE_TAG
docker tag dev.registry.pivotal.io/concourse/postgres:$POSTGRES_IMAGE_TAG $INTERNAL_REGISTRY/$PROJECT/postgres:$POSTGRES_IMAGE_TAG

If using Docker Hub:

docker tag concourse/concourse:$CONCOURSE_IMAGE_TAG $USERNAME/concourse:$CONCOURSE_IMAGE_TAG
docker tag dev.registry.pivotal.io/concourse/postgres:$POSTGRES_IMAGE_TAG $USERNAME/postgres:$POSTGRES_IMAGE_TAG

If you are using Helm v2 (i.e. Concourse versions < v6.7.x) you'll also need to tag the helm image.

If using Harbor (internal registry):

docker tag dev.registry.pivotal.io/concourse/helm:$HELM_IMAGE_TAG $INTERNAL_REGISTRY/$PROJECT/helm:$HELM_IMAGE_TAG

If using Docker Hub:

docker tag dev.registry.pivotal.io/concourse/helm:$HELM_IMAGE_TAG $USERNAME/helm:$HELM_IMAGE_TAG

Push the images to the internal registry by running the following commands in your terminal.
- If using Harbor (internal registry):
```
docker push $INTERNAL_REGISTRY/$PROJECT/concourse:$CONCOURSE_IMAGE_TAG
docker push $INTERNAL_REGISTRY/$PROJECT/postgres:$POSTGRES_IMAGE_TAG
```
- If using Docker Hub:
```
docker push $USERNAME/concourse:$CONCOURSE_IMAGE_TAG
docker push $USERNAME/postgres:$POSTGRES_IMAGE_TAG
```
If you are using Helm v2 (i.e. Concourse versions < v6.7.x) you'll also need to tag the helm image.
- If using Harbor (internal registry):
```
docker push $INTERNAL_REGISTRY/$PROJECT/helm:$HELM_IMAGE_TAG
```
- If using Docker Hub:
```
docker push $USERNAME/helm:$HELM_IMAGE_TAG
```
You must have the necessary credentials (and authorization) to push to the targeted project.

Upgrade Helm

Helm provides a plugin to migrate from Helm v2 to Helm v3 called 2to3. This plugin is also available as a binary in the release downloaded from the Broadcom Support portal.
- Using the provided binary
```
2to3 convert RELEASE-NAME
```
- Using the Helm plug-in system
```
helm 2to3 convert RELEASE-NAME
```
Where:
- RELEASE-NAME is the name of your existing Concourse release. Discover this by running the helm2 list command and looking for a chart that starts with concourse.
You should now be able to run helm list and see a release with the same name has been created.

Upgrade Concourse

First, get previous deployment values

Helm uses a set of configuration values to template the different Kubernetes objects that Concourse requires into a release. To ensure you maintain the exact same configuration between the current version of Concourse and the upgraded version you are about to deploy, it is important to know what configuration is already deployed.

Version Control: VMware recommends checking in these configuration values to a source control management (SCM) system like git so that you can track changes over time. If you have your configuration values in to an SCM system, check them out to your local machine so that they can be updated now.

If you're unsure of what has been deployed, and don't have access to the values files locally, you can fetch the deployment values that your existing release is using by running the helm get values command. This will allow you to recreate the exact configuration with your upgraded release.

Get your previous deployment-values.yml file from SCM, or download it using this command:
```
helm get values RELEASE-NAME > deployment-values.yml
```
Where:
- RELEASE-NAME is the name of your existing Concourse release. Discover this by running the helm list command and looking for a chart that starts with concourse.
Open the file in your favorite text editor. For example, with vim:
```
vim deployment-values.yml
```
Change the imageTag value to match the new version of Concourse. For example, using the v6.7.3 Helm Chart, this image tag will be 6.7.3-ubuntu.

The following is a sample snippet; make sure the rest of the file you edit reflects your existing Concourse deployment:
```
---
image: INTERNAL-REGISTRY/PROJECT/concourse
imageTag: CONCOURSE-IMAGE-TAG
imagePullSecrets: ["regcred"] #remove this line for public image registry
postgresql:
    image:
        registry: INTERNAL-REGISTRY
        repository: PROJECT/postgres
        tag: POSTGRES-IMAGE-TAG
        pullSecrets: ["regcred"] # remove for public image registry
```
Where:
- INTERNAL-REGISTRY/PROJECT is your registry address and project.
- CONCOURSE-IMAGE-TAG is the output of
```
cat ./images/concourse.tar.name | cut -d ':' -f 2
```
  For example, 6.7.3-ubuntu
- imagePullSecrets is the pull secret listed under the key spec.template.spec in the following file:
```
helm init --tiller-image $INTERNAL_REGISTRY/$PROJECT/helm:IMAGE-TAG --service-account tiller --dry-run --debug > tiller.yml
```
  Remove if using a public repository.
- POSTGRES-IMAGE-TAG is the output of
```
cat ./images/postgres.tar.name | cut -d ':' -f 2
```
- pullSecrets is the pull secret created in step 6a. Remove if using a public repository.

Upgrading

Now that the deployment values are updated, use the helm upgrade command to upgrade your existing release using the new values.

Upgrade your release:
```
helm upgrade RELEASE-NAME ./charts/ --values ./deployment-values.yml
```
Where:
- RELEASE-NAME is the name of your existing Concourse release. Discover this by running the helm list command and looking for a chart that starts with concourse.

Successful deployment

If Kubernetes has accepted the manifest correctly, then you will see the following response followed by more information about your cluster:

NAME: DEPLOYMENT-NAME
LAST DEPLOYED: DEPLOYMENT-DATE
NAMESPACE: default
STATUS: DEPLOYED
...

To determine whether your pods are up and running correctly, use the following command:

kubectl get pods

This will show you a list of deployed pods as well as their current status. It should look something like this:

$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
concourse-postgresql-0           1/1     Running   0          15m
concourse-web-57ccf57f46-zsd25   1/1     Running   0          9m52s
concourse-worker-0               1/1     Running   0          8m45s
concourse-worker-1               1/1     Running   0          9m45s

Clean up Helm v2

Once you're satisfed with the upgrade and have no more Helm v2 releases left on the Kubernetes cluster, you can cleanup all the Helm v2 components by running:

Using the provided binary
```
2to3 cleanup
```
Using the Helm plug-in system
```
helm 2to3 cleanup
```