Testing, release, and security lifecycle for Tanzu Operations Manager

This topic explains how VMware’s development practices, automated build tools, and organizational structures work together to create and support stable releases of VMware Tanzu Operations Manager.

• Tanzu Operations Manager teams building system components receive frequent feedback, which helps to secure code from exposure to vulnerability.

• Every Tanzu Operations Manager release follows a strict workflow and passes through numerous quality and compliance checks before distribution.

• Teams build tests into the product consistently and run them automatically with any code change.

Release mechanics

VMware releases, patches, and supports multiple versions of Tanzu Operations Manager simultaneously. This section explains the versioning and support conventions VMware follows.

Versioning

VMware numbers Tanzu Operations Manager releases following a semantic versioning format, X.Y.Z, where X and Y indicate major and minor releases, and Z designates patch releases. Major and minor releases change Tanzu Operations Manager functionality. Patch releases are backward-compatible security patches or bug fixes. For more information about semantic versioning, see the Semantic Versioning 2.0.0 website.

Support

VMware supports each major and minor platform release according to the following criteria:

• VMware supports the release for at least nine months following its first publication date.

• VMware supports the last three major or minor releases, even if this extends coverage beyond nine months.

Support includes maintenance updates and upgrades, bug and security fixes, and technical help. For more details about support standards, technical guidance, and publication phases, see Product Lifecycle Policy on the VMware Tanzu website. For a definition of VMware support in legal terms, see Broadcom Support Services Policies on the VMware Tanzu website.

Patch releases

Patch releases are more frequent and less predictable than major and minor releases.

VMware identifies security issues using standard nomenclature from Common Vulnerabilities and Exposures (CVE), Ubuntu Security Notices (USN), and other third-party sources. For more information about security fixes in core Tanzu Operations Manager code or packaged dependencies, see Tanzu Operations Manager v3.0 Release Notes and VMware Tanzu Application Service for VMs v3.0 Release Notes.

The Tanzu security team maintains a running list of security fixes in Tanzu Operations Manager and Tanzu Operations Manager dependencies. To see their most recent findings, see Product Security Center on the VMware Tanzu website.

Upgrading

All Tanzu Operations Manager releases pass through extensive test suites that include automated unit, integration, and acceptance tests on multiple IaaSes. Regardless of this extensive testing, VMware recommends that you test major and minor releases in a non-production environment before implementing them across your deployment. Upgrade your production environment as soon as possible after you validate the new release on your test environment.

Release testing, integration, and validation

This section describes VMware’s software development processes and explains compliance and regulatory standards to which VMware software adheres.

Test-Driven Development

VMware’s development process relies on a strict workflow with continuous automated testing. VMware R&D does not separate engineering and testing teams. Rather, every member of each engineering team is responsible for ensuring the quality of their code. They write tests for all of the software components that they develop, often before writing the software itself.

With every software change, automated build pipelines trigger these tests for the new software component and for everything it touches. If a new code check-in does not pass its tests or causes a failure elsewhere, it pauses the build pipeline for the entire team, or sometimes all of VMware R&D. The transparency of this process encourages developers to work together to address code issues quickly.

VMware applies the following automated testing approaches, scenarios, and frameworks to Tanzu Operations Manager components and to the release as a whole:

• Unit tests: Development teams write unit tests to express and validate desired functional behavior of product components. Typical frameworks used are RSpec and Ginkgo. These tests run continuously throughout the development cycle.

• OSS integration tests: The Release Integration team exercises a full deployment of open-source Cloud Foundry to validate all end-user features. They maintain the Cloud Foundry Acceptance Test (CATs) suite alongside the OSS cf-release. Cloud Foundry component teams also contribute acceptance test suites at the OSS Integration Test level. These tests exercise and validate their components’ functional, performance, and integration health.

• Tanzu Operations Manager integration tests: The Tanzu Operations Manager Release Engineering (RelEng) team validates the quality and cross-product integration health of the commercial Tanzu Operations Manager release. RelEng runs OSS Acceptance Tests against all supported releases. These tests run on full Tanzu Operations Manager instances configured to represent diverse real-world customer scenarios on various IaaSes and using both internal and external load balancer, database, blobstore, and user store solutions.

Additional pre-release gates: Internal, PWS, and Compliance

In addition to its automated unit and integration testing, VMware deploys all upgrades slated for upcoming Tanzu Operations Manager releases on at-scale test environments. Prior to each major or minor commercial release, VMware runs the entire Tanzu Operations Manager suite of services on several internally-managed large integration environments that run customer-like data and workloads.

All Tanzu Operations Manager product teams participate in go-to-market steps for each release, as is often required for shipping a legally compliant product. Examples include Open Source License File attribution and an Export Compliance classification.

Patch releases: security and bug fixes

VMware uses established processes to track, disclose, and remediate vulnerabilities in Tanzu Operations Manager and related dependent components. This section explains how VMware identifies vulnerabilities and implements fixes for them.

Identifying security vulnerabilities

VMware has an established process to track and patch vulnerabilities in software dependencies and Tanzu Operations Manager software. Additionally, the Product Security Center page describes a responsible disclosure process for reporting vulnerabilities identified in VMware software by third parties.

VMware uses multiple methods to identify security vulnerabilities in VMware software and dependencies internally, including:

• Security notifications from Canonical for their Ubuntu operating system, provided through VMware’s commercial relationship with Canonical.
• Software component scans that use third-party security software to update continuously from external security vulnerability sources.
• Dependency analysis software that identifies and catalogs software dependencies.
• Security vulnerability notifications from known software dependencies.

VMware also monitors externally-reported vulnerabilities from many sources, including:

• Third-party security analysis requested by VMware.
• Cloud Foundry Foundation security notifications from member companies.
• Customer, prospect and other third-party security reports.

When VMware discovers a potential security vulnerability in Tanzu Operations Manager, the security team opens an issue to assess it. If it confirms the vulnerability exists, VMware identifies and updates affected components with plans to backport the fix to stable releases. Fixes are implemented on a target timeline based on the severity level of the vulnerability. For more information, see Tanzu Operations Manager Security Overview and Policy.

Fix, test, and release lifecycle

The following flowchart details the steps that VMware performs on a typical high-priority CVE, to publish a patch release fix on the Broadcom Support portal. A detailed description of the steps follows.

The triage flowchart shows the following steps:

1. Vulnerability reported in a Cloud Foundry Dependency.

2. OSS Cloud Foundry team addresses vulnerability.

   1. Story detailing the vulnerability and acceptance criteria for fix prioritized on OSS backlog.

   2. Unit Tests written before coding the fix.

   3. Fix written; Unit Tests pass.

   4. Code committed; OSS Integration Tests pass.

   5. Check: Does Product Owner verify that fix meets acceptance criteria?

   6. If yes, fix is ready for integration.

3. VMware Tanzu team integrates fix.

   1. Story detailing the issue and acceptance criteria for fix prioritized on Commercial backlog.

   2. Fix patched in all release lines under General Support.

   3. Check: Have all tests passed each stage?

   4. If yes, patched builds kick off integration pipeline.

4. Release Engineering Integration Pipelines validate quality.

   1. Tests pass by scenario:

      • New installation scenario: integration tests pass.
      • Upgrade scenario: Integration Tests pass.
      • Additional customer scenarios: Integration Tests pass.

   2. Check: Have all pipelines passed on every supported IaaS?

   3. If yes, ready for Publishing.

5. Release is published for distribution.

   1. Post to VMware Network; with High CVE vulnerab ility release the fix with as fast a turnaround as possible.

   2. Customer receives announcements and downloads release for their IaaS.