Retrieving external dependencies

In this topic, you will find a reference pipeline that illustrates the tasks and provides an example of a basic pipeline design. You know your environment and constraints; VMware recommend that you look at the tasks that make up the pipeline, and see how they can be arranged for your specific automation needs. For a deeper dive into each task see the Task Reference.

These Concourse pipelines are examples on how to use the tasks. If you use a different CI/CD platform, you can use these Concourse files as examples of the inputs, outputs, and arguments used in each step in the workflow.

Prerequisites

Deployed Concourse

Note Platform Automation Toolkit is based on Concourse CI. We recommend that you have some familiarity with Concourse before getting started. If you are new to Concourse, see Installing Concourse with BOSH.
Persisted datastore that can be accessed by Concourse resource (for example, s3, gcs, minio)
A set of valid download-product-config files: Each product has a configuration YAML of what version to download from the Broadcom Support portal.
Broadcom Support portal access to Platform Automation Toolkit

Retrieval from the Broadcom Support portal (formerly Tanzu Network)

The pipeline downloads dependencies consumed by the tasks and places them into a trusted s3-like storage provider. This helps other Concourse deployments without internet access retrieve task dependencies.

The pipeline requires configuration for the download-product task. See the following for examples that you can use.

Important Note the unique regex format for blob names, for example: \[p-healthwatch,(.*)\]p-healthwatch-.*.pivotal. The Broadcom Support portal file names will not always contain the necessary metadata to accurately download files from a blobstore (for example, s3, gcs, azure), so the product slug and version are prepended when using download-product. For more information about how this works, and what to expect when using download-product, see the download-product task reference

Healthwatch

---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: p-healthwatch

file-glob: "healthwatch-2*.pivotal"
product-version-regex: ^2\.1\..*$
stemcell-iaas: google

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: healthwatch-stemcell

Operations Manager

---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: ops-manager

file-glob: "ops-manager-gcp*.yml"
product-version-regex: ^2\.10\.\d+$ #^2\.\d+\.\d+$|^2\.\d+\.\d+-rc.*$|^2\.\d+\.\d+-alpha.*$

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))

PKS

---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: pivotal-container-service

file-glob: "pivotal-container-service*.pivotal"
product-version-regex: ^1\.12\..*$
stemcell-iaas: google

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: pks-stemcell

Tanzu Application Service (TAS)

---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: elastic-runtime

file-glob: "srt*.pivotal"
product-version-regex: ^2\.11\..*$
stemcell-iaas: google

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: tas-stemcell

Full Pipeline and Reference Configurations

There is a git repository, docs-platform-automation-reference-pipeline-config, containing containing the full pipeline file, along with other pipeline and configuration examples.

This can be useful when you want to take a fully assembled pipeline as a starting point; the rest of this document covers the sections of the full pipeline in more detail.

Pipeline components

Resource types

This custom resource type uses the pivnet-resource to pull down and separate both pieces of the Platform Automation Toolkit product (tasks and image) so they can be stored separately in S3.

resource_types:
- name: pivnet
  type: docker-image
  source:
    repository: pivotalcf/pivnet-resource
    tag: latest-final

Product resources

These are the S3 resources where Platform Automation Toolkit download-product outputs will be stored. Each product/stemcell needs a separate resource defined. Platform Automation Toolkit will not create these resources for you.

resources:
- name: opsman-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[ops-manager,(.*)\]ops-manager-gcp.*.yml

- name: pks-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[pivotal-container-service,(.*)\]pivotal-container-service-.*.pivotal

- name: pks-stemcell
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: pks-stemcell/\[stemcells-ubuntu-xenial,(.*)\]light-bosh-stemcell-.*-google.*\.tgz

- name: tas-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[elastic-runtime,(.*)\]srt-.*.pivotal

- name: tas-stemcell
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: tas-stemcell/\[stemcells-ubuntu-xenial,(.*)\]light-bosh-stemcell-.*-google.*\.tgz

- name: healthwatch-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[p-healthwatch,(.*)\]healthwatch-\d.*.pivotal

- name: healthwatch-pas-exporter
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[p-healthwatch,(.*)\]healthwatch-pas-exporter.*.pivotal

- name: healthwatch-pks-exporter
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[p-healthwatch,(.*)\]healthwatch-pks-exporter.*.pivotal

- name: healthwatch-stemcell
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: healthwatch-stemcell/\[stemcells-ubuntu-xenial,(.*)\]light-bosh-stemcell-.*-google.*\.tgz

- name: telemetry-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[pivotal-telemetry-collector,(.*)\]telemetry-collector-.*.tgz

Platform Automation Toolkit resources

platform-automation-pivnet is downloaded directly from the Broadcom Support portal and will be used to download all other products from the Broadcom Support portal.

platform-automation-tasks and platform-automation-image are S3 resources that will be stored for internet-restricted, or faster, access. Platform Automation Toolkit will not create this resource for you.

- name: platform-automation-pivnet
  type: pivnet
  source:
    api_token: ((pivnet_token))
    product_slug: platform-automation
    product_version: 2\.(.*)
    sort_by: semver

- name: platform-automation-tasks
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: platform-automation-tasks-(.*).zip

- name: platform-automation-image
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: platform-automation-image-(.*).tgz

Configured resources

You need to add your download-product configuration files to your configurations repo. Platform Automation Toolkit will not create these resources for you. For more details, see Inputs and outputs.

- name: configuration
  type: git
  source:
    private_key: ((docs-ref-pipeline-repo-key.private_key))
    uri: ((docs-ref-pipeline-repo-uri))
    branch: develop
    submodules: all
    depth: 1

Trigger resources

- name: daily
  type: time
  source:
    interval: 24h

Secrets handling

This example shows how to load secrets stored in an external credential manager such as CredHub. Concourse supports several credential managers natively.

The configuration below uses the prepare-tasks-with-secrets task to load secrets from your external configuration files.

# This task is used in multiple jobs
# The YAML anchor "*prepare-tasks-with-secrets" is used in its place
prepare-tasks-with-secrets: &prepare-tasks-with-secrets
  image: platform-automation-image
  file: platform-automation-tasks/tasks/prepare-tasks-with-secrets.yml
  params:
    CONFIG_PATHS: config/download-product-pivnet
  input_mapping:
    tasks: platform-automation-tasks
    config: configuration
  output_mapping:
    tasks: platform-automation-tasks

Jobs

Each job corresponds to a box on the visual representation of your Concourse pipeline. These jobs consume the resources defined above.

jobs:
- name: fetch-opsman
  plan:
  - in_parallel:
    - get: daily
      trigger: true
    - get: platform-automation-image
      params:
        unpack: true
    - get: platform-automation-tasks
      params:
        unpack: true
    - get: configuration
  - task: prepare-tasks-with-secrets
    <<: *prepare-tasks-with-secrets
  - task: download-opsman-image
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product-pivnet/download-opsman.yml
    input_mapping:
      config: configuration
  - in_parallel:
    - put: opsman-product
      params:
        file: downloaded-product/*
- name: fetch-pks
  plan:
  - in_parallel:
    - get: daily
      trigger: true
    - get: platform-automation-image
      params:
        unpack: true
    - get: platform-automation-tasks
      params:
        unpack: true
    - get: configuration
  - task: prepare-tasks-with-secrets
    <<: *prepare-tasks-with-secrets
  - task: download-pks-product-and-stemcell
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product-pivnet/download-pks.yml
    input_mapping:
      config: configuration
    output_mapping: {downloaded-stemcell: pks-stemcell}
  - in_parallel:
      - put: pks-product
        params:
          file: downloaded-product/*.pivotal
      - put: pks-stemcell
        params:
          file: pks-stemcell/*.tgz

- name: fetch-tas
  plan:
    - in_parallel:
      - get: daily
        trigger: true
      - get: platform-automation-image
        params:
          unpack: true
      - get: platform-automation-tasks
        params:
          unpack: true
      - get: configuration
    - task: prepare-tasks-with-secrets
      <<: *prepare-tasks-with-secrets
    - task: download-tas-product-and-stemcell
      image: platform-automation-image
      file: platform-automation-tasks/tasks/download-product.yml
      params:
        CONFIG_FILE: download-product-pivnet/download-tas.yml
      input_mapping:
        config: configuration
      output_mapping: {downloaded-stemcell: tas-stemcell}
    - in_parallel:
        - put: tas-product
          params:
            file: downloaded-product/*.pivotal
        - put: tas-stemcell
          params:
            file: tas-stemcell/*.tgz

- name: fetch-healthwatch
  plan:
    - in_parallel:
      - get: daily
        trigger: true
      - get: platform-automation-image
        params:
          unpack: true
      - get: platform-automation-tasks
        params:
          unpack: true
      - get: configuration
    - task: prepare-tasks-with-secrets
      <<: *prepare-tasks-with-secrets
    - task: download-healthwatch-product-and-stemcell
      image: platform-automation-image
      file: platform-automation-tasks/tasks/download-product.yml
      params:
        CONFIG_FILE: download-product-pivnet/download-healthwatch.yml
      input_mapping:
        config: configuration
      output_mapping: {downloaded-stemcell: healthwatch-stemcell}
    - in_parallel:
        - put: healthwatch-product
          params:
            file: downloaded-product/*healthwatch-2*.pivotal
        - put: healthwatch-stemcell
          params:
            file: healthwatch-stemcell/*.tgz

- name: fetch-healthwatch-pas-exporter
  plan:
    - in_parallel:
      - get: daily
        trigger: true
      - get: platform-automation-image
        params:
          unpack: true
      - get: platform-automation-tasks
        params:
          unpack: true
      - get: configuration
    - task: prepare-tasks-with-secrets
      <<: *prepare-tasks-with-secrets
    - task: download-healthwatch-pas-exporter-and-stemcell
      image: platform-automation-image
      file: platform-automation-tasks/tasks/download-product.yml
      params:
        CONFIG_FILE: download-product-pivnet/download-healthwatch-pas-exporter.yml
      input_mapping:
        config: configuration
    - in_parallel:
        - put: healthwatch-pas-exporter
          params:
            file: downloaded-product/*healthwatch-pas-exporter*.pivotal

- name: fetch-healthwatch-pks-exporter
  plan:
    - in_parallel:
      - get: daily
        trigger: true
      - get: platform-automation-image
        params:
          unpack: true
      - get: platform-automation-tasks
        params:
          unpack: true
      - get: configuration
    - task: prepare-tasks-with-secrets
      <<: *prepare-tasks-with-secrets
    - task: download-healthwatch-pks-exporter-and-stemcell
      image: platform-automation-image
      file: platform-automation-tasks/tasks/download-product.yml
      params:
        CONFIG_FILE: download-product-pivnet/download-healthwatch-pks-exporter.yml
      input_mapping:
        config: configuration
    - in_parallel:
        - put: healthwatch-pks-exporter
          params:
            file: downloaded-product/*healthwatch-pks-exporter*.pivotal

- name: fetch-telemetry-collector
  plan:
    - in_parallel:
        - get: daily
          trigger: true
        - get: platform-automation-image
          params:
            unpack: true
        - get: platform-automation-tasks
          params:
            unpack: true
        - get: configuration
    - task: prepare-tasks-with-secrets
      <<: *prepare-tasks-with-secrets
    - task: download-telemetry-product
      image: platform-automation-image
      file: platform-automation-tasks/tasks/download-product.yml
      params:
        CONFIG_FILE: download-product-pivnet/download-telemetry.yml
      input_mapping:
        config: configuration
    - in_parallel:
        - put: telemetry-product
          params:
            file: downloaded-product/*

- name: fetch-platform-automation
  # We use the pivnet resource to bootstrap the pipeline,
  # and because this product is part of the pipeline, not the foundation
  plan:
  - get: platform-automation-pivnet
    trigger: true
  - in_parallel:
    - put: platform-automation-tasks
      params:
        file: platform-automation-pivnet/*tasks*.zip
    - put: platform-automation-image
      params:
        file: platform-automation-pivnet/*image*.tgz