In this topic, you will find a reference pipeline that illustrates the tasks and provides an example of a basic pipeline design. You know your environment and constraints; VMware recommend that you look at the tasks that make up the pipeline, and see how they can be arranged for your specific automation needs. For a deeper dive into each task see the Task Reference.
These Concourse pipelines are examples on how to use the tasks. If you use a different CI/CD platform, you can use these Concourse files as examples of the inputs, outputs, and arguments used in each step in the workflow.
Deployed Concourse
Platform Automation Toolkit is based on Concourse CI. We recommend that you have some familiarity with Concourse before getting started. If you are new to Concourse, see Installing Concourse with BOSH.
Persisted datastore that can be accessed by Concourse resource (for example, s3, gcs, minio)
A set of valid download-product-config files: Each product has a configuration YAML of what version to download from the Broadcom Support portal.
Broadcom Support portal access to Platform Automation Toolkit
The pipeline downloads dependencies consumed by the tasks and places them into a trusted s3-like storage provider. This helps other Concourse deployments without internet access retrieve task dependencies.
The pipeline requires configuration for the download-product task. See the following for examples that you can use.
Note the unique regex format for blob names, for example: \[p-healthwatch,(.*)\]p-healthwatch-.*.pivotal
. The Broadcom Support portal file names will not always contain the necessary metadata to accurately download files from a blobstore (for example, s3, gcs, azure), so the product slug and version are prepended when using download-product
. For more information about how this works, and what to expect when using download-product
, see the download-product
task reference
Healthwatch
---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: p-healthwatch
file-glob: "healthwatch-2*.pivotal"
product-version-regex: ^2\.1\..*$
stemcell-iaas: google
s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: healthwatch-stemcell
Operations Manager
---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: ops-manager
file-glob: "ops-manager-gcp*.yml"
product-version-regex: ^2\.10\.\d+$ #^2\.\d+\.\d+$|^2\.\d+\.\d+-rc.*$|^2\.\d+\.\d+-alpha.*$
s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
PKS
---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: pivotal-container-service
file-glob: "pivotal-container-service*.pivotal"
product-version-regex: ^1\.12\..*$
stemcell-iaas: google
s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: pks-stemcell
Tanzu Application Service (TAS)
---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: elastic-runtime
file-glob: "srt*.pivotal"
product-version-regex: ^2\.11\..*$
stemcell-iaas: google
s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: tas-stemcell
There is a git repository, docs-platform-automation-reference-pipeline-config, containing containing the full pipeline file, along with other pipeline and configuration examples.
This can be useful when you want to take a fully assembled pipeline as a starting point; the rest of this document covers the sections of the full pipeline in more detail.
This custom resource type uses the pivnet-resource to pull down and separate both pieces of the Platform Automation Toolkit product (tasks and image) so they can be stored separately in S3.
resource_types:
- name: pivnet
type: docker-image
source:
repository: pivotalcf/pivnet-resource
tag: latest-final
These are the S3 resources where Platform Automation Toolkit download-product
outputs will be stored. Each product/stemcell needs a separate resource defined. Platform Automation Toolkit will not create these resources for you.
resources:
- name: opsman-product
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: \[ops-manager,(.*)\]ops-manager-gcp.*.yml
- name: pks-product
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: \[pivotal-container-service,(.*)\]pivotal-container-service-.*.pivotal
- name: pks-stemcell
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: pks-stemcell/\[stemcells-ubuntu-xenial,(.*)\]light-bosh-stemcell-.*-google.*\.tgz
- name: tas-product
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: \[elastic-runtime,(.*)\]srt-.*.pivotal
- name: tas-stemcell
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: tas-stemcell/\[stemcells-ubuntu-xenial,(.*)\]light-bosh-stemcell-.*-google.*\.tgz
- name: healthwatch-product
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: \[p-healthwatch,(.*)\]healthwatch-\d.*.pivotal
- name: healthwatch-pas-exporter
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: \[p-healthwatch,(.*)\]healthwatch-pas-exporter.*.pivotal
- name: healthwatch-pks-exporter
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: \[p-healthwatch,(.*)\]healthwatch-pks-exporter.*.pivotal
- name: healthwatch-stemcell
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: healthwatch-stemcell/\[stemcells-ubuntu-xenial,(.*)\]light-bosh-stemcell-.*-google.*\.tgz
- name: telemetry-product
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: \[pivotal-telemetry-collector,(.*)\]telemetry-collector-.*.tgz
platform-automation-pivnet
is downloaded directly from the Broadcom Support portal and will be used to download all other products from the Broadcom Support portal.
platform-automation-tasks
and platform-automation-image
are S3 resources that will be stored for internet-restricted, or faster, access. Platform Automation Toolkit will not create this resource for you.
- name: platform-automation-pivnet
type: pivnet
source:
api_token: ((pivnet_token))
product_slug: platform-automation
product_version: 2\.(.*)
sort_by: semver
- name: platform-automation-tasks
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: platform-automation-tasks-(.*).zip
- name: platform-automation-image
type: s3
source:
access_key_id: ((s3_access_key_id))
bucket: ((s3_pivnet_products_bucket))
region_name: ((s3_region_name))
secret_access_key: ((s3_secret_access_key))
regexp: platform-automation-image-(.*).tgz
You need to add your download-product
configuration files to your configurations repo. Platform Automation Toolkit will not create these resources for you. For more details, see Inputs and outputs.
- name: configuration
type: git
source:
private_key: ((docs-ref-pipeline-repo-key.private_key))
uri: ((docs-ref-pipeline-repo-uri))
branch: develop
submodules: all
depth: 1
- name: daily
type: time
source:
interval: 24h
This example shows how to load secrets stored in an external credential manager such as CredHub. Concourse supports several credential managers natively.
The configuration below uses the prepare-tasks-with-secrets
task to load secrets from your external configuration files.
# This task is used in multiple jobs
# The YAML anchor "*prepare-tasks-with-secrets" is used in its place
prepare-tasks-with-secrets: &prepare-tasks-with-secrets
image: platform-automation-image
file: platform-automation-tasks/tasks/prepare-tasks-with-secrets.yml
params:
CONFIG_PATHS: config/download-product-pivnet
input_mapping:
tasks: platform-automation-tasks
config: configuration
output_mapping:
tasks: platform-automation-tasks
Each job corresponds to a box on the visual representation of your Concourse pipeline. These jobs consume the resources defined above.
jobs:
- name: fetch-opsman
plan:
- in_parallel:
- get: daily
trigger: true
- get: platform-automation-image
params:
unpack: true
- get: platform-automation-tasks
params:
unpack: true
- get: configuration
- task: prepare-tasks-with-secrets
<<: *prepare-tasks-with-secrets
- task: download-opsman-image
image: platform-automation-image
file: platform-automation-tasks/tasks/download-product.yml
params:
CONFIG_FILE: download-product-pivnet/download-opsman.yml
input_mapping:
config: configuration
- in_parallel:
- put: opsman-product
params:
file: downloaded-product/*
- name: fetch-pks
plan:
- in_parallel:
- get: daily
trigger: true
- get: platform-automation-image
params:
unpack: true
- get: platform-automation-tasks
params:
unpack: true
- get: configuration
- task: prepare-tasks-with-secrets
<<: *prepare-tasks-with-secrets
- task: download-pks-product-and-stemcell
image: platform-automation-image
file: platform-automation-tasks/tasks/download-product.yml
params:
CONFIG_FILE: download-product-pivnet/download-pks.yml
input_mapping:
config: configuration
output_mapping: {downloaded-stemcell: pks-stemcell}
- in_parallel:
- put: pks-product
params:
file: downloaded-product/*.pivotal
- put: pks-stemcell
params:
file: pks-stemcell/*.tgz
- name: fetch-tas
plan:
- in_parallel:
- get: daily
trigger: true
- get: platform-automation-image
params:
unpack: true
- get: platform-automation-tasks
params:
unpack: true
- get: configuration
- task: prepare-tasks-with-secrets
<<: *prepare-tasks-with-secrets
- task: download-tas-product-and-stemcell
image: platform-automation-image
file: platform-automation-tasks/tasks/download-product.yml
params:
CONFIG_FILE: download-product-pivnet/download-tas.yml
input_mapping:
config: configuration
output_mapping: {downloaded-stemcell: tas-stemcell}
- in_parallel:
- put: tas-product
params:
file: downloaded-product/*.pivotal
- put: tas-stemcell
params:
file: tas-stemcell/*.tgz
- name: fetch-healthwatch
plan:
- in_parallel:
- get: daily
trigger: true
- get: platform-automation-image
params:
unpack: true
- get: platform-automation-tasks
params:
unpack: true
- get: configuration
- task: prepare-tasks-with-secrets
<<: *prepare-tasks-with-secrets
- task: download-healthwatch-product-and-stemcell
image: platform-automation-image
file: platform-automation-tasks/tasks/download-product.yml
params:
CONFIG_FILE: download-product-pivnet/download-healthwatch.yml
input_mapping:
config: configuration
output_mapping: {downloaded-stemcell: healthwatch-stemcell}
- in_parallel:
- put: healthwatch-product
params:
file: downloaded-product/*healthwatch-2*.pivotal
- put: healthwatch-stemcell
params:
file: healthwatch-stemcell/*.tgz
- name: fetch-healthwatch-pas-exporter
plan:
- in_parallel:
- get: daily
trigger: true
- get: platform-automation-image
params:
unpack: true
- get: platform-automation-tasks
params:
unpack: true
- get: configuration
- task: prepare-tasks-with-secrets
<<: *prepare-tasks-with-secrets
- task: download-healthwatch-pas-exporter-and-stemcell
image: platform-automation-image
file: platform-automation-tasks/tasks/download-product.yml
params:
CONFIG_FILE: download-product-pivnet/download-healthwatch-pas-exporter.yml
input_mapping:
config: configuration
- in_parallel:
- put: healthwatch-pas-exporter
params:
file: downloaded-product/*healthwatch-pas-exporter*.pivotal
- name: fetch-healthwatch-pks-exporter
plan:
- in_parallel:
- get: daily
trigger: true
- get: platform-automation-image
params:
unpack: true
- get: platform-automation-tasks
params:
unpack: true
- get: configuration
- task: prepare-tasks-with-secrets
<<: *prepare-tasks-with-secrets
- task: download-healthwatch-pks-exporter-and-stemcell
image: platform-automation-image
file: platform-automation-tasks/tasks/download-product.yml
params:
CONFIG_FILE: download-product-pivnet/download-healthwatch-pks-exporter.yml
input_mapping:
config: configuration
- in_parallel:
- put: healthwatch-pks-exporter
params:
file: downloaded-product/*healthwatch-pks-exporter*.pivotal
- name: fetch-telemetry-collector
plan:
- in_parallel:
- get: daily
trigger: true
- get: platform-automation-image
params:
unpack: true
- get: platform-automation-tasks
params:
unpack: true
- get: configuration
- task: prepare-tasks-with-secrets
<<: *prepare-tasks-with-secrets
- task: download-telemetry-product
image: platform-automation-image
file: platform-automation-tasks/tasks/download-product.yml
params:
CONFIG_FILE: download-product-pivnet/download-telemetry.yml
input_mapping:
config: configuration
- in_parallel:
- put: telemetry-product
params:
file: downloaded-product/*
- name: fetch-platform-automation
# We use the pivnet resource to bootstrap the pipeline,
# and because this product is part of the pipeline, not the foundation
plan:
- get: platform-automation-pivnet
trigger: true
- in_parallel:
- put: platform-automation-tasks
params:
file: platform-automation-pivnet/*tasks*.zip
- put: platform-automation-image
params:
file: platform-automation-pivnet/*image*.tgz