This topic tells you how to upgrade a dependency version in a custom buildpack. These procedures enable Cloud Foundry (CF) operators to maintain custom buildpacks that contain dependencies outside of the dependencies in the CF system buildpacks.
The procedures in this topic refer to the tools used by the CF buildpacks team, but they do not require the following specific tools. You can use any continuous integration (CI) system and workflow management tool to update dependencies in custom buildpacks.
The CF buildpacks team uses the following tools to update dependencies:
When the New Releases job in the notifications pipeline detects a new version of a tracked dependency in a buildpack, it creates a Tracker story about building and including the new version of the dependency in the buildpack manifests. It also posts a message as the dependency-notifier
to the #buildpacks channel in the Cloud Foundry Slack channel.
For all dependencies, you must build the binary from source or acquire the binary as a tarball from a trusted source. For most dependencies, the CF buildpacks team builds the binaries from source.
The following steps assume you are using a Concourse deployment of the buildpacks-ci
pipelines and Pivotal Tracker.
To build the binary for a dependency:
Go to the public-buildpacks-ci-robots
directory and verify no uncommitted changes exist.
$ cd ~/workspace/public-buildpacks-ci-robots $ git status
Run the git pull
command in the directory to ensure it contains the most recent version of the contents.
$ git pull -r
Go to the binary-builds
directory.
$ cd binary-builds
Locate the YAML file for the buildpack that you want to build a binary. The directory contains YAML files for all the packages and dependencies tracked by the CF buildpacks team. Each YAML file correlates to the build queue for one dependency or package, and uses the naming format DEPENDENCY-NAME.yml
. For example, the YAML file tracking the build queue for Ruby is named ruby-builds.yml
and contains the following contents:
---
ruby: []
Different buildpacks use different signatures for verification. Determine which signature your buildpack requires by consulting the list in the buildpacks section of this topic and follow the instructions to locate the SHA256, MD5, or GPG signature for the binary:
For the SHA256 of a file, run shasum -a 256 FILE-NAME
.
md5 FILE-NAME
.For the GPG signature (for Nginx), see the Nginx Downloads page.
Add the version and verification for the new binary to the YAML file as attributes of an element under the dependency name. For example, to build the Ruby 2.3.0 binary verified with SHA256, add the following to the YAML file:
---
ruby:
- version: 2.3.0
sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507
Important Do not preface the version number with the name of the binary or language. For example, specify 2.3.0
for version
instead of ruby-2.3.0
.
You can enqueue builds for multiple versions at the same time. For example, to build both the Ruby 2.3.0 binary and the Ruby 2.3.1 binary, add the following to the YAML file:
---
ruby:
- version: 2.3.0
sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507
- version: 2.3.1
sha256: b87c738cb2032bf4920fef8e3864dc5cf8eae9d89d8d523ce0236945c5797dcd
Use the git add
command to stage your changes:
$ git add .
Use the git commit -m "YOUR-COMMIT-MESSAGE [#STORY-NUMBER]"
command to commit your changes using the Tracker story number. Replace YOUR-COMMIT-MESSAGE
with your commit message, and STORY-NUMBER
with the number of your Tracker story.
$ git commit -m "make that change [#1234567890]"
Run git push
to push your changes to the remote origin.
$ git push
Pushing your changes initiates the binary building process, which you can monitor at the binary-builder pipeline of your own buildpacks-ci
Concourse deployment. When the build completes, it adds a link to the Concourse build run to the Tracker story specified in the commit message for the new release.
Binary builds are run by the Cloud Foundry Binary Builder and the dependency-builds pipeline.
After you build the binary for a dependency that you want to access and download from a URL, follow the instructions to add the dependency version to the buildpack manifest
The following steps assume you are using a Concourse deployment of the buildpacks-ci
pipelines and Pivotal Tracker.
To add the dependency version to the buildpack manifest, use these steps:
Go to the directory of the buildpack for which you want to update dependencies and run git checkout develop
to check out the develop
branch.
$ cd ~/workspace/ruby-buildpack $ git checkout develop
Edit the manifest.yml
file for the buildpack to add or remove dependencies.
dependencies:
- name: ruby
version: 2.3.0
md5: 535342030a11abeb11497824bf642bf2
uri: https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/ruby/ruby-2.3.0-linux-x64.tgz
cf_stacks:
- cflinuxfs4
ruby-2.1.9
and ruby-2.1.8
.uri
and the md5
from the build-BINARY-NAME
job that ran in the Concourse dependency-builds pipeline and add them to the manifest.In the PHP buildpack, you might see a modules
line for each PHP dependency in the manifest. Do not include this modules
line in your new PHP dependency entry. The modules
line is added to the manifest by the ensure-manifest-has-modules
Concourse job in the php-buildpack
when you commit and push your changes. You can see this in the output logs of the build-out
task.
Replace any other mentions of the old version number in the buildpack repository with the new version number. The CF buildpack team uses Ag for text searching.
$ ag OLD-VERSION
Run the following command to package and upload the buildpack, set up the org and space for tests in the specified CF deployment, and run the CF buildpack tests:
$ BUNDLE_GEMFILE=cf.Gemfile buildpack-build
If the command fails, you might need to fix or change the tests, fixtures, or other parts of the buildpack.
Once the test suite completely passes, use git commands to stage, commit, and push your changes:
$ git add . $ git commit -m "YOUR-MESSAGE[#TRACKER-STORY-ID]" $ git push
Monitor the LANGUAGE-buildpack
pipeline in Concourse. Once the test suite builds, the specs-lts-develop
job and specs-edge-develop
job, pass for the buildpack, you can deliver the Tracker story for the new Dependency release. Copy and paste links for the successful test suite builds into the Tracker story.
The following list contains information about the buildpacks maintained by the CF buildpacks team:
Go:
GO-VERSION.linux-amd64.tar.gz
, provided by Google on the Go Downloads page.Godep:
.tar.gz
file from the Godep GitHub releases page.Verified with: The SHA256 of the source Example: Automated enqueuing of binary build for Godep 72.
The buildpacks-ci dependency-builds pipeline automates the process of detecting, uploading, and updating Godep in the manifest.
Node:
node-vVERSION.tar.gz
file listed on https://nodejs.org/dist/vVERSION/SHASUMS256.txt
For example, for Node version 4.4.6, the CF buildpacks team verifies with the SHA256 for node-v4.4.6.tar.gz
on its SHASUMS256 page.Python:
Gzipped source tarball
, listed on https://www.python.org/downloads/release/python-VERSION/
, where VERSION
has no periods. For example, for Python version 2.7.12
, use the MD5 for the Gzipped source tarball
on its downloads page.OpenJDK:
JRuby:
.tar.gz
file from the JRuby Downloads page.Ruby:
Bundler:
.gem
file from Rubygems.PHP:
.tar.gz
file from the PHP Downloads page.public-buildpacks-ci-robots
repository. For PHP5 versions, the CF buildpacks team enqueues builds in the binary-builds/php-builds.yml
file. For PHP7 versions, the CF buildpacks team enqueues builds in the binary-builds/php7-builds.yml
file.Nginx:
gpg-rsa-key-id
and gpg-signature
of the version. The gpg-rsa-key-id
is the same for each version/build, but the gpg-signature
is different. This information is located on the Nginx Downloads page.HTTPD:
.tar.bz2
file from the HTTPD Downloads page.Composer:
composer.phar
file from the Composer Downloads page.For Composer, there is no build process as the composer.phar
file is the binary. In the manual process, connect to the appropriate S3 bucket using the correct AWS credentials. Create a new directory with the name of the composer version, for example 1.0.2
, and put the appropriate composer.phar
file into that directory.
For Composer v1.0.2
, connect and create the php/binaries/trusty/composer/1.0.2
directory. Then place the composer.phar
file into that directory so the binary is available at php/binaries/trusty/composer/1.0.2/composer.phar
.
The buildpacks-ci dependency-builds pipeline automates the process of detecting, uploading, and updating Composer in the manifest.
Example: Automated enqueuing of binary build for Composer 1.1.2.
Nginx:
gpg-rsa-key-id
and gpg-signature
of the version. The gpg-rsa-key-id
is the same for each version/build, but the gpg-signature
is different. This information is located on the Nginx Downloads page.The Binary buildpack has no dependencies.