This topic explains how VMware’s development practices, automated build tools, and organizational structures work together to create and support stable releases of VMware Tanzu Operations Manager (Ops Manager).
Ops Manager teams building system components receive frequent feedback, which helps to secure code from exposure to vulnerability.
Every Ops Manager release follows a strict workflow and passes through numerous quality and compliance checks before distribution.
Teams build tests into the product consistently and run them automatically with any code change.
VMware releases, patches, and supports multiple versions of Ops Manager simultaneously. This section explains the versioning and support conventions VMware follows.
VMware numbers Ops Manager releases following a semantic versioning format, X.Y.Z, where X and Y indicate major and minor releases, and Z designates patch releases. Major and minor releases change Ops Manager functionality. Patch releases are backward-compatible security patches or bug fixes. For more information about semantic versioning, see the Semantic Versioning 2.0.0 website.
As of Pivotal Cloud Foundry (PCF) v1.8, VMware supports each major and minor platform release according to the following criteria:
VMware supports the release for at least nine months following its first publication date.
VMware supports the last three major or minor releases, even if this extends coverage beyond nine months.
Support includes maintenance updates and upgrades, bug and security fixes, and technical help. For more details about support standards, technical guidance, and publication phases, see Support Lifecycle Policy on the VMware Tanzu website. For a definition of VMware support in legal terms, see VMware Tanzu Software Support Services Terms & Conditions on the VMware Tanzu website.
Patch releases are more frequent and less predictable than major and minor releases. The PCF v1.6.x line provides a good example of their frequency. PCF v1.6.1 was released on October 26, 2015. Through August 2016, 36 additional patches of Elastic Runtime v1.6.x and 18 patches of Ops Manager v1.6.x provided security and bug fixes to customers.
VMware identifies security issues using standard nomenclature from Common Vulnerabilities and Exposures (CVE), Ubuntu Security Notices (USN), and other third-party sources. For more information about security fixes in core Ops Manager code or packaged dependencies, see Ops Manager v3.0 Release Notes and VMware Tanzu Application Service for VMs v3.0 Release Notes.
The Pivotal Application Security Team maintains a running list of security fixes in Ops Manager and Ops Manager dependencies. To see their most recent findings, see Pivotal Application Security Team on the VMware Tanzu website.
All Ops Manager releases pass through extensive test suites that include automated unit, integration, and acceptance tests on multiple IaaSes. Regardless of this extensive testing, VMware recommends that you test major and minor releases in a non-production environment before implementing them across your deployment. Upgrade your production environment as soon as possible after you validate the new release on your test environment.
This section describes VMware’s software development processes and explains compliance and regulatory standards to which VMware software adheres.
VMware’s development process relies on a strict workflow with continuous automated testing. VMware R&D does not separate engineering and testing teams. Rather, every member of each engineering team is responsible for ensuring the quality of their code. They write tests for all of the software components that they develop, often before writing the software itself.
With every software change, automated build pipelines trigger these tests for the new software component and for everything it touches. If a new code check-in does not pass its tests or causes a failure elsewhere, it pauses the build pipeline for the entire team, or sometimes all of VMware R&D. The transparency of this process encourages developers to work together to address code issues quickly.
VMware applies the following automated testing approaches, scenarios, and frameworks to Ops Manager components and to the release as a whole:
Unit tests: Development teams write unit tests to express and validate desired functional behavior of product components. Typical frameworks used are RSpec and Ginkgo. These tests run continuously throughout the development cycle.
OSS integration tests: The Release Integration team exercises a full deployment of open-source Cloud Foundry to validate all end-user features. They maintain the Cloud Foundry Acceptance Test (CATs) suite alongside the OSS cf-release. Cloud Foundry component teams also contribute acceptance test suites at the OSS Integration Test level. These tests exercise and validate their components’ functional, performance, and integration health.
Ops Manager integration tests: The Ops Manager Release Engineering (RelEng) team validates the quality and cross-product integration health of the commercial Ops Manager release. RelEng runs OSS Acceptance Tests against all supported releases. These tests run on full Ops Manager instances configured to represent diverse real-world customer scenarios on various IaaSes and using both internal and external load balancer, database, blobstore, and user store solutions.
In addition to its automated unit and integration testing, VMware deploys all upgrades slated for upcoming Ops Manager releases on at-scale test environments. Prior to each major or minor commercial release, VMware runs the entire Ops Manager suite of services on several internally-managed large integration environments that run customer-like data and workloads.
All Ops Manager product teams participate in go-to-market steps for each release, as is often required for shipping a legally compliant product. Examples include Open Source License File attribution and an Export Compliance classification.
VMware uses established processes to track, disclose, and remediate vulnerabilities in Ops Manager and related dependent components. This section explains how VMware identifies vulnerabilities and implements fixes for them.
VMware has an established process to track and patch vulnerabilities in software dependencies and Ops Manager software. Additionally, the Pivotal Application Security Team page describes a responsible disclosure process for reporting vulnerabilities identified in VMware software by third parties.
VMware uses multiple methods to identify security vulnerabilities in VMware software and dependencies internally, including:
VMware also monitors externally-reported vulnerabilities from many sources, including:
When VMware discovers a potential security vulnerability in Ops Manager, the security team opens an issue to assess it. If it confirms the vulnerability exists, VMware identifies and updates affected components with plans to backport the fix to stable releases. Fixes are implemented on a target timeline based on the severity level of the vulnerability. For more information, see Ops Manager Security Overview and Policy.
The following flowchart details the steps that VMware performs on a typical high-priority CVE, to publish a patch release fix on VMware Network. A detailed description of the steps follow.
The triage flowchart shows the following steps:
Vulnerability reported in a Cloud Foundry Dependency
OSS Cloud Foundry team addresses vulnerability
Story detailing the vulnerability and acceptance criteria for fiex priorized on OSS backlog
Unit Tests written before coding the fix
Fix written; Unit Tests pass
Code committed; OSS Integration Tests pass
Check: Does Product Owner verify that fix meets acceptance criteria?
If yes, fix is ready for integration.
VMware Tanzu team integrates fix
Story detailing the issue and acceptance criteria for fix prioritized on Commercial backlog
Fix patched in all release lines under General Support
Check: Have all tests passed each stage?
If yes, patched builds kick off integration pipeline.
Release Engineering Integration Pipelines validate quality
Tests pass by scenario:
Check: Have all pipelines passed on every supported IaaS?
If yes, ready for Publishing
Release is published for distribution
Post to VMware Network; with High CVE vulnerab ility release the fix with as fast a turnaround as possible
Customer receives announcements and downloads release for their IaaS
