Using a secrets store to store credentials

Secrets stores such as CredHub can be used to store secure properties that you don't want committed into a config file. Within your pipeline, the config file can then reference that secrets store value for runtime evaluation.

Platform Automation Toolkit contains two tasks to help with retrieving these credentials in the tasks that use them:

• If you're using Concourse version 5 or newer the prepare-tasks-with-secrets task can be used with any Concourse supported secrets store.
• The credhub-interpolate task can be used only with CredHub.

Using prepare-tasks-with-secrets

The prepare-tasks-with-secrets task takes a set of tasks and modifies them to include environment variables that reference the variables found in the config files provided. This allows use of the native Concourse secrets handling and provides support for any secret store Concourse supports.

The prepare-tasks-with-secrets task replaces the credhub-interpolate task on Concourse versions 5.x+ and provides the following benefits:

• Support for all native Concourse secrets stores including CredHub and Vault.
• CredHub credentials are no longer required by the task so they can be completely handled by Concourse.
• Secrets are no longer written to disk, which alleviates some security concerns.

The prepare-tasks-with-secrets task can be used in two different ways:

• Adding to a pipeline without an already implemented credhub-interpolate task
• Replacing an already implemented credhub-interpolate task

When using prepare-tasks-with-secrets, all secrets must exist in either a secrets store or a vars file found under VARS_PATHS. If a vars from a config file can't be found in CredHub, it must be available in a YAML file found under VARS_PATHS in prepare-tasks-with-secrets. This will prevent these credentials from being added as environment variables to the task; this would result in Concourse being unable to find them in the secrets store.

To understand how prepare-tasks-with-secrets modifies the Platform Automation Toolkit tasks, the following is an example of how a task will be changed:

1. Authenticate with your CredHub instance.

2. Generate a username and password.

   credhub generate --name="/concourse/:team_name/:pipeline_name/vcenter_login" --type=user --username=some-user
   

3. Create a director configuration file that references the properties using the om interpolation syntax.

   properties-configuration:
     iaas_configuration:
       vcenter_host: ((vcenter_host))
       vcenter_username: ((vcenter_login.username))
       vcenter_password: ((vcenter_login.password))
   

4. (Optional) Create vars files with additional variables not stored in the secrets store.

   This is recommended only for non-secret variables. It's more secure to store secrets in the secrets store. If using multiple foundations, there are some cases where a foundation-specific key might not be sensitive, but should be extracted to allow reuse of the config file between foundations. If using a single config file for multiple foundations, vars files may be used instead of storing the variables in a secrets store.

   For example:

   vcenter_host: vcenter.example.com
   

5. Configure your pipeline to use the prepare-tasks-with-secrets task.

   • The config input is required and is a directory that contains your configuration file from (3).
   • The tasks input is required and is the set of tasks that will be modified.
   • The vars input and VARS_PATHS param are only required if vars files are being used in subsequent tasks.
   • The output_mapping section is required and is where the modified tasks will be.

   The declaration in a pipeline might look similar to this example:

   - task: prepare-tasks-with-secrets
     file: platform-automation-tasks/tasks/prepare-tasks-with-secrets.yml
     image: platform-automation-image
     input_mapping:
       tasks: platform-automation-tasks
       config: deployments
       vars: deployments  # required only if using vars
     output_mapping:
       tasks: platform-automation-tasks
     params:
       CONFIG_PATHS: ((foundation))/config
       VARS_PATHS: ((foundation))/vars # required only if using vars
   

   Unlike with credhub-interpolate, there is no concept of SKIP_MISSING. As such, if there are credentials that will be filled in future jobs by vars files, those vars files must be provided in the vars input and the VARS_PATHS param.

   This task will replace all of the tasks provided in the tasks input with the modified tasks. The modified tasks include an extended params section with the secret references detected from the config files.

6. Use the modified tasks in future jobs.

What the prepare-tasks-with-secrets task is doing

Here's an example of what prepare-tasks-with-secrets is doing internally. Given an original task and the previously provided config/vars files:

# Original Platform Automation Toolkit Task
platform: linux

inputs:
- name: platform-automation-tasks
- name: config # contains the director configuration file
- name: env # contains the env file with target OpsMan Information
- name: vars # variable files to be made available
  optional: true
- name: secrets
  # secret files to be made available
  # separate from vars, so they can be store securely
  optional: true
- name: ops-files # operations files to custom configure the product
  optional: true

params:
  VARS_FILES:
  # - Optional
  # - Filepath to the Tanzu Operations Manager vars YAML file
  # - The path is relative to root of the task build,
  #   so `vars` and `secrets` can be used.

  OPS_FILES:
  # - Optional
  # - Filepath to the Tanzu Operations Manager operations YAML files
  # - The path is relative to root of the task build

  ENV_FILE: env.yml
  # - Required
  # - Filepath of the env config YAML
  # - The path is relative to root of the `env` input

  DIRECTOR_CONFIG_FILE: director.yml
  # - Required
  # - Filepath to the director configuration YAML file
  # - The path is relative to the root of the `config` input

run:
  path: platform-automation-tasks/tasks/configure-director.sh

The prepare-tasks-with-secrets task modifies the original task to embed the variables found in director.yml, in the params section. Any variables found in the vars.yml file will not be included in the modified task. The params added will have a prefix of OM_VAR, to avoid collisions. The task is a programmatically modified YAML file, so the output loses the comments and keys are sorted.

# prepare-job-with-secrets Generated Task
inputs:
- name: platform-automation-tasks
- name: config
- name: env
- name: vars
  optional: true
- name: secrets
  optional: true
- name: ops-files
  optional: true

params:
  DIRECTOR_CONFIG_FILE: director.yml
  ENV_FILE: env.yml
  OM_VAR_vcenter_password: ((vcenter_password))
  OM_VARS_ENV: OM_VAR
  OPS_FILES:
  VARS_FILES:

platform: linux

run:
  path: platform-automation-tasks/tasks/configure-director.sh

Replacing credhub-interpolate with prepare-tasks-with-secrets

If you already have implemented the credhub-interpolate task in your pipeline, this solution should be a drop-in replacement if you are not using vars files. You must be using Concourse 5.x or greater.

If you are using vars files, the vars input and the VARS_PATHS param will also need to be set on the prepare-tasks-with-secrets task.

For example, if the existing credhub-interpolate task looks like this:

---
platform: linux

inputs:
- name: platform-automation-tasks
- name: files
# contains YAML files with extension `.yml`.
# Each one of these files will have their values interpolated from credhub.
# For examples, run: `credhub interpolate --help`
# (minimum version >= 2.1.0 required)

outputs:
- name: interpolated-files
# Contains only YAML files found and interpolated by this task.
# Maintains the filestructure of the `files` input.

# all params are required to be filled out
params:

  CREDHUB_CLIENT:
  CREDHUB_SECRET:
  CREDHUB_SERVER:
  # - Required
  # - Credentials to talk to credhub server

  CREDHUB_CA_CERT:
  # - Optional
  # - This is only necessary if your Concourse worker
  #   is not already configured to trust the CA used for CredHub.
  # - If more than one CA cert is required (ie the UAA),
  #   the CA certs can be concatenated together and separated by a newline.
  #   For example,
  #   CREDHUB_CA_CERT: |
  #     -----BEGIN CERTIFICATE-----
  #     ...credhub cert...
  #     -----END CERTIFICATE-----
  #     -----BEGIN CERTIFICATE-----
  #     ...UAA cert...
  #     -----END CERTIFICATE-----

  PREFIX:
  # - Required
  # - Prefix flag used by credhub interpolate

  INTERPOLATION_PATHS: '.'
  # - Required
  # - Path the contains the files to read from
  # - This is a space separated list of directories
  #   the paths are all evaluated relative to files/

  SKIP_MISSING: true
  # Optional
  # Change to false to have strict interpolation
  # and fail if params are missing from vars

run:
  path: platform-automation-tasks/tasks/credhub-interpolate.sh

In this task definition, you defined the prefix and CredHub authorization credentials. The new prepare-tasks-with-secrets task uses Concourse's native integration with CredHub (and other credential managers). The task definition can be replaced with the following:

- task: prepare-tasks-with-secrets
  file: platform-automation-tasks/tasks/prepare-tasks-with-secrets.yml
  input_mapping:
    tasks: platform-automation-tasks
    config: configuration
    vars: configuration  # required only if using vars
  output_mapping:
    tasks: platform-automation-tasks
  params:
    CONFIG_PATHS: ((foundation))/config
    VARS_PATHS: ((foundation))/vars # required only if using vars

If using vars files in subsequent tasks, the vars input and the VARS_PATHS param must be used to prevent interpolation errors in those subsequent tasks.

For the preceding, also note:

• The output_mapping, which is required. This will replace all platform-automation-tasks with the modified tasks. The modification now includes an extended params, which now includes the secrets references detected from the config files.
• The INTERPOLATION_PATHS is now CONFIG_PATHS. The concept of reading the references from the config files is still here, but no interpolation actually happens.
• The PREFIX is no longer defined or provided. Since the tasks are using Concourse's native credential management, the lookup path is predetermined. For example, /concourse/:team_name/:cred_name or /concourse/:team_name/:pipeline_name/:cred_name.

Using credhub-interpolate

The credhub-interpolate task can only be used with CredHub.

If you are using Concourse 5.x+, it is recommended that you use the prepare-tasks-with-secrets task instead.

An example workflow: Storing an SSH key

1. Authenticate with your CredHub instance.

2. Generate an SSH key: credhub generate --name="/concourse/:team_name/:pipeline_name/opsman_ssh_key" --type=ssh

3. Create an Tanzu Operations Manager configuration file that references the name of the property.

   opsman-configuration:
     azure:
       ssh_public_key: ((opsman_ssh_key.public_key))
   

4. Configure your pipeline to use the credhub-interpolate task. It takes an input called files, which should contain your configuration file from the earlier step.

   The declaration in a pipeline might look like this:

   jobs:
   - name: example-job
     plan:
     - get: platform-automation-tasks
     - get: platform-automation-image
     - get: config
     - task: credhub-interpolate
       image: platform-automation-image
       file: platform-automation-tasks/tasks/credhub-interpolate.yml
       input_mapping:
         files: config
       params:
         # depending on CredHub configuration
         # ether CA cert or secret are required
         CREDHUB_CA_CERT: ((credhub_ca_cert))
         CREDHUB_SECRET: ((credhub_secret))
   
         # all required
         CREDHUB_CLIENT: ((credhub_client))
         CREDHUB_SERVER: ((credhub_server))
         PREFIX: /concourse/:team_name/:pipeline_name
         SKIP_MISSING: true
   

   This task will reach out to the deployed CredHub and fill in your entry references and return an output named interpolated-files that can then be read as an input to any tasks that follow.

   Our configuration will now look like this:

   opsman-configuration:
   azure:
     ssh_public_key: ssh-rsa AAAAB3Nz...
   

If using this, you need to ensure that the Concourse worker can talk to CredHub. Depending on how you deployed CredHub and/or the worker, this may not be possible. Using credhub-interpolate inverts control; now workers need to access CredHub. With prepare-tasks-with-secrets and other uses of the Concourse native integration, the ATC retrieves secrets from CredHub and passes them to the worker.

Defining multiline certificates and keys in config files

There are three ways to include certificates in the YAML files that are used by Platform Automation Toolkit tasks.

1. Direct inclusion in YAML file:

   # An incomplete base.yml response from om staged-config
   product-name: cf
   
   product-properties:
     .uaa.service_provider_key_credentials:
       value:
         cert_pem: |
           -----BEGIN CERTIFICATE-----
           ...<Some Cert>...
           -----END CERTIFICATE-----
         private_key_pem: |
           -----BEGIN RSA PRIVATE KEY-----
           ...<Some Private Key>...
           -----END RSA PRIVATE KEY-----
   
     .properties.networking_poe_ssl_certs:
       value:
         -
           certificate:
             cert_pem: |
               -----BEGIN CERTIFICATE-----
               ...<Some Cert>...
               -----END CERTIFICATE-----
             private_key_pem: |
               -----BEGIN RSA PRIVATE KEY-----
               ...<Some Private Key>...
               -----END RSA PRIVATE KEY-----
   

2. Secrets manager reference in YAML file:

   # An incomplete base.yml
   product-name: cf
   
   product-properties:
     .uaa.service_provider_key_credentials:
       value:
         cert_pem: ((uaa_service_provider_key_credentials.certificate))
         private_key_pem: ((uaa_service_provider_key_credentials.private_key))
   
     .properties.networking_poe_ssl_certs:
       value:
         -
           certificate:
             cert_pem: ((networking_poe_ssl_certs.certificate))
             private_key_pem: ((networking_poe_ssl_certs.private_key))
   

   This example assumes the use of CredHub.

   CredHub supports a --type=certificate credential type which allows you to store a certificate and private key pair under a single name. The cert and key can be stored temporarily in local files or can be passed directly on the command line.

   An example of the file storage method:

   credhub set --type=certificate \
     --name=uaa_service_provider_key_credentials \
     --certificate=./cert.pem \
     --private=./private.key
   

3. Using vars files:

   Vars files are a mix of the two previous methods. The cert/key is defined inline in the vars file.

   #vars.yml
   uaa_service_provider_key_credentials_cert_pem: |
     -----BEGIN CERTIFICATE-----
     ...<Some Cert>...
     -----END CERTIFICATE-----
   uaa_service_provider_key_credentials_private_key: |
     -----BEGIN RSA PRIVATE KEY-----
     ...<Some Private Key>...
     -----END RSA PRIVATE KEY-----
   networking_poe_ssl_certs_cert_pem: |
     -----BEGIN CERTIFICATE-----
     ...<Some Cert>...
     -----END CERTIFICATE-----
   networking_poe_ssl_certs_private_key: |
     -----BEGIN RSA PRIVATE KEY-----
     ...<Some Private Key>...
     -----END RSA PRIVATE KEY-----
   

   It is then referenced as a ((parameter)) in the base.yml.

   # An incomplete base.yml
   product-name: cf
   
   product-properties:
     .uaa.service_provider_key_credentials:
       value:
         cert_pem: ((uaa_service_provider_key_credentials_cert_pem))
         private_key_pem: ((uaa_service_provider_key_credentials_private_key))
   
     .properties.networking_poe_ssl_certs:
       value:
         -
           certificate:
             cert_pem: ((networking_poe_ssl_certs_cert_pem))
             private_key_pem: ((networking_poe_ssl_certs_private_key))
   

Storing values for multi-foundation deployment

Concourse-supported secrets store

If you have multiple foundations, store relevant keys to each foundation in a different pipeline path, and Concourse reads the values in appropriately. When sharing the same base.yml across foundations, it is recommended that you have a different pipeline for each foundation.

Vars files

Vars files can be used for your secrets handling. They are not recommended, but are sometimes required based on your foundation setup.

Consider the following example (which only uses vars files and does not use a secrets store):

# base.yml
# An incomplete YAML response from om staged-config
product-name: cf

product-properties:
  .cloud_controller.apps_domain:
    value: ((cloud_controller_apps_domain))
  .cloud_controller.encrypt_key:
    value:
      secret: ((cloud_controller_encrypt_key.secret))
  .properties.security_acknowledgement:
    value: X
  .properties.cloud_controller_default_stack:
    value: default

network-properties:
  network:
    name: DEPLOYMENT
  other_availability_zones:
  - name: AZ01
  singleton_availability_zone:
    name: AZ01

resource-config:
  diego_cell:
    instances: 5
    instance_type:
      id: automatic
  uaa:
    instances: 1
    instance_type:
      id: automatic

The first foundation has the following vars.yml, optional for the configure-product task.

# vars.yml
cloud_controller_encrypt_key.secret: super-secret-encryption-key
cloud_controller_apps_domain: cfapps.domain.com

The vars.yml can then be passed to configure-product with base.yml as the config file. The configure-product task will then substitute the ((cloud_controller_encrypt_key.secret)) and ((cloud_controller_apps_domain)) specified in vars.yml and configure the product as normal.

An example of how this might look in a pipeline. (Resources are not listed in this example):

jobs:
- name: configure-product
  plan:
  - aggregate:
    - get: platform-automation-image
      params:
        unpack: true
    - get: platform-automation-tasks
      params:
        unpack: true
    - get: configuration
    - get: variable
  - task: configure-product
    image: platform-automation-image
    file: platform-automation-tasks/tasks/configure-product.yml
    input_mapping:
      config: configuration
      env: configuration
      vars: variable
    params:
      CONFIG_FILE: base.yml
      VARS_FILES: vars.yml
      ENV_FILE: env.yml

If deploying more than one foundation, a unique vars.yml should be used for each foundation.

Using the tasks with vars files

• Using prepare-tasks-with-secrets and vars files: CredHub and vars files may be used together to interpolate variables into base.yml. This use case is described in Using prepare-tasks-with-secrets.

• Using credhub-interpolate and vars files: CredHub and vars files may be used together to interpolate variables into base.yml. Using the same example from earlier in this topic.

Using credhub-interpolate with vars files

Using credhub-interpolate and vars files together:

# base.yml
# An incomplete YAML response from om staged-config
product-name: cf

product-properties:
  .cloud_controller.apps_domain:
    value: ((cloud_controller_apps_domain))
  .cloud_controller.encrypt_key:
    value:
      secret: ((cloud_controller_encrypt_key.secret))
  .properties.security_acknowledgement:
    value: X
  .properties.cloud_controller_default_stack:
    value: default

network-properties:
  network:
    name: DEPLOYMENT
  other_availability_zones:
  - name: AZ01
  singleton_availability_zone:
    name: AZ01

resource-config:
  diego_cell:
    instances: 5
    instance_type:
      id: automatic
  uaa:
    instances: 1
    instance_type:
      id: automatic

There is one parametrized variable that is secret and you might not want to have stored in a plain text vars file, ((cloud_controller_encrypt_key.secret)), but ((cloud_controller_apps_domain)) is fine in a vars file.

To support a base.yml with credentials from multiple sources (that is, CredHub and vars files), you must use SKIP_MISSING: true in the credhub-interpolate task. This is enabled by default by the credhub-interpolate task.

The workflow is the same as CredHub, but when passing the interpolated base.yml as a config into the next task, you add in a vars file to fill in the missing variables.

An example of how this might look in a pipeline (resources not listed), assuming:

• The ((base.yml)) above
• ((cloud_controller_encrypt_key.secret)) is stored in CredHub
• ((cloud_controller_apps_domain)) is stored in director-vars.yml

jobs:
- name: example-credhub-interpolate
  plan:
  - get: platform-automation-tasks
  - get: platform-automation-image
  - get: config
  - task: credhub-interpolate
    image: platform-automation-image
    file: platform-automation-tasks/tasks/credhub-interpolate.yml
    input_mapping:
      files: config
    params:
      # depending on CredHub configuration
      # ether CredHub CA cert or CredHub secret are required
      CREDHUB_CA_CERT: ((credhub_ca_cert))
      CREDHUB_SECRET: ((credhub_secret))

      # all required
      CREDHUB_CLIENT: ((credhub_client))
      CREDHUB_SERVER: ((credhub_server))
      PREFIX: /concourse/:team_name/:pipeline_name
      SKIP_MISSING: true
- name: example-configure-director
  plan:
  - get:
  - task: configure-director
    image: platform-automation-image
    file: platform-automation-tasks/tasks/configure-director.yml
    params:
      VARS_FILES: vars/director-vars.yml
      ENV_FILE: env/env.yml
      DIRECTOR_CONFIG_FILE: config/director.yml

credhub-interpolate and multiple key lookups

When using the credhub-interpolate task with a CredHub in a single foundation or multi-foundation, it's best to avoid duplicating identical credentials (duplication makes credential rotation more difficult).

To have CredHub read in credentials from multiple paths (not relative to your PREFIX), you must provide the absolute path to any credentials not in your relative path.

For example, using an alternative base.yml:

# An incomplete YAML response from om staged-config
product-name: cf

product-properties:
  .cloud_controller.apps_domain:
    value: ((cloud_controller_apps_domain))
  .cloud_controller.encrypt_key:
    value:
      secret: ((/alternate_prefix/cloud_controller_encrypt_key.secret))
  .properties.security_acknowledgement:
    value: X
  .properties.cloud_controller_default_stack:
    value: default

Suppose there is an example job in which the prefix is defined as "foundation1." The parameterized values in the previous example are interpolated as follows:

• ((cloud_controller_apps_domain)) uses a relative path for CredHub. When running credhub-interpolate, the task prepends the PREFIX. This value is stored in CredHub as /foundation1/cloud_controller_apps_domain.

• ((/alternate_prefix/cloud_controller_encrypt_key.secret)) (note the leading slash) uses an absolute path for CredHub. When running credhub-interpolate, the task does not prepend the prefix. This value is stored in CredHub at its absolute path /alternate_prefix/cloud_controller_encrypt_key.secret.

Any value with a leading / slash will never use the PREFIX to look up values in CredHub, so you can have multiple key lookups in a single interpolate task.