VMware Telco Cloud Platform 4.0 | 16 APR 2024

Check for additions and updates to these release notes.

What's New

VMware Telco Cloud Platform Release 4.0 is a major milestone release, providing a unified Telco Cloud Platform to address the needs of both Cloud-Native Network Functions (CNFs) and Virtual Network Functions (VNFs).

VMware Telco Cloud Platform is now offered in two editions, providing flexibility to match functionality and cost to your cloud footprint and requirements. Telco Cloud Platform Essential Edition provides a simple and fast path to network modernization. Telco Cloud Platform Advanced Edition adds automation, orchestration, and assurance across multiple layers, domains, and vendors to deliver a scalable telco-grade network.

Telco Cloud Platform 4.0 introduces several new features to simplify the lifecycle management of the platform for CNFs, including CaaS automation upgrades with Workflow hub enhancements in Telco Cloud Automation and support for multiple Tanzu Kubernetes Grid versions. Similar to the previous release, this release includes extended support for a Kubernetes version, allowing users to remain on a Kubernetes version for an additional one year and plan for upgrade. This release also includes key security features such as preventing network function deployment into Kubernetes system namespace and providing granular security defaults at the user namespace level instead of system-wide.

Note:

To support VNFs and CNFs, Telco Cloud Platform 4.0 includes features from the latest versions of vSphere and NSX. NSX is available as part of Telco Cloud Platform Advanced Edition. This release also supports VMware Cloud Director 10.4.2, which is part of Telco Cloud Infrastructure - Cloud Director Edition 3.0.

General Updates

  • Product Name Change

    VMware Telco Cloud Platform - 5G Edition is now called VMware Telco Cloud Platform.

  • Telco Cloud Platform Essential and Advanced Editions:

    • Telco Cloud Platform Essentials:

      • Supports only vSphere networking configuration, but does not include advanced networking configuration options available with NSX.

      • Supports only CaaS cluster creation and lifecycle management, but does not support onboarding and instantiation of Cloud Native Network Functions (CNFs).

      • Supports VNF Onboarding and instantiation through Telco Cloud Automation

    • Telco Cloud Platform Advanced: Includes the following additional capabilities over Essentials:

      • NSX Networking

        • Enhanced data plane use cases

        • Overlay and T1 / T0 routing

        • eVPN Architecture

      • vSAN licensing (1 Tib core vs 100 Gib core in Essentials)

      • Telco Cloud Service Assurance

      • Additional Telco Cloud Automation functionalities

        • CNF Onboarding, design, and lifecycle management

        • Workflow hub

        • Network slicing

  • Integrating VMware Telco Cloud Infrastructure - Cloud Director Edition

    VMware Telco Cloud Infrastructure - Cloud Director Edition is now integrated with VMware Telco Cloud Platform, allowing users to leverage Telco Cloud Infrastructure features through Telco Cloud Platform and perform VNF lifecycle management through VMware Cloud Director.

Workload Management, Storage, and Reliability Enhancements

VMware ESXi 8.0 Update 2b includes the following key features and enhancements. This release also inherits features and enhancements from VMware ESXi 8.0 Update 2.

  • Reduced Downtime for vCenter Upgrade: Introduces the Reduced Downtime Upgrade method in vSphere on-prem. This method uses a migration-based approach for upgrading the vCenter to a newer version. During the upgrade, the old vCenter and its services remain online while its data and configuration are copied to the new vCenter. Hence, the downtime is reduced further to a few minutes.

  • Resilient vCenter Patching: Enables vSphere administrators to receive pre-check messages for vCenter patching, requiring acknowledgment that the backup is taken before patching vCenter. In addition to backup completion, vCenter automatically performs an OS-level Logical Volume Manager (LVM) snapshot before a patch/update task.

  • Non-disruptive Certificate Management: Introduces non-intrusive certificate management that enables vSphere administrators to renew and replace the vCenter SSL/TLS certificate without service restarts. Hence, the annual certificate can be renewed without impacting vCenter productivity.

  • Reliable Network Configuration Recovery: Enhances the Distributed Key-Value Store to include vSphere Distributed Switch configuration, including vSphere Distributed Switch instances used by NSX. This feature ensures proper and reliable network configuration recovery of a vCenter failure from a backup. When a vCenter is restored from a backup, the latest vSphere Distributed Switch information in the ESXi cluster is reconciled with the vCenter database.

  • vSphere Identity Federation with Azure AD: Enhances vSphere Identity Federation with Azure Active Directory support, while continuing to support other authentication services such as LDAPS, ADFS, and Okta. Federated identity helps both security and compliance efforts.

  • Enhanced vSAN Witness Support in vSphere Lifecycle Manager: Enhances the vSAN witness node support in vSphere Lifecycle Manager to account for shared vSAN witness nodes. Hence, the image definition of the vSAN witness node can be managed independent of the vSAN clusters it is part of.

Other important features such as improved placement for GPU workloads, VM compatibility with HW version 21, and QOS for GPU workloads provide workload performance improvements.

For more information, see the VMware ESXi 8.0U2b Release Notes and VMware ESXi 8.0 U2 Release Notes.

Carrier-Grade Resilient Networking and Security

Important:

VMware NSX is available as part of VMware Telco Cloud Platform Advanced Edition.

VMware NSX 4.1.2.1 contains various new functionalities for virtualized networking, security, and management. This release also inherits features and enhancements from VMware NSX 4.1.2:

  • Layer 3 Networking:

    • Supports GRE tunnels in default Tier-0 Gateways and Tier-0 VRF Gateways.

    • Supports Dynamic routing (BGP) and static routing over GRE tunnels.

  • Layer 2 Networking: Allows modification of overlay VLANs in the default uplink profile.

  • Debug Packet Drops on Edge platform: Provides a tool to define better where packet drops are appearing and define if the Edge platform is dropping packets. Granular filtering is available per data flow.

  • Security Enhancements in the following areas:

    • Intrusion Detection and Prevention (IDS/IPS)

    • Distributed IDS/IPS Packet Capture

    • Distributed Firewall

    • FQDN Filtering

    • Network Detection and Response

    • Distributed Malware Detection and Prevention

    • Platform Security: NSX Manager now supports the highest version of Transport Layer Security (TLS) v1.3

      Note: Some of these security features require an additional license.

  • Operational & Management features include:

    • Installation & in-place upgrades

    • User experience improvements

    • Operations and Monitoring: alarms for remote logging

    • Multi-tenancy support with Terraform for NSX projects within Tenants

    • Tenant-aware logging within a project

  • Scale Enhancements: Includes updates to maximum scale levels. For more information, see VMware Configuration Maximums.

For more information, see the VMware NSX 4.1.2.1 Release Notes and VMware NSX 4.1.2 Release Notes.

Carrier-Grade Kubernetes Infrastructure

VMware Tanzu Standard for Telco introduces various key features as part of VMware Tanzu Kubernetes Grid 2.5:

  • Supported Kubernetes versions:

    • 1.26

    • 1.27 (1 year extended support)

    • 1.28

    Note: Kubernetes 1.27 is supported for two years instead of the standard one year.

  • Supports Photon OS 5 with Kernel 6.1 and cgroup v2

  • Introduces Harbor v2.9.1 with OCI only

  • Supports Dual Stack IPv4/IPv6 on Kubernetes 1.28 classy clusters for Greenfield deployment.

  • Independent delivery of Tanzu Standard packages: Whereabouts, Multus CNI, and cert-manager packages are delivered through Tanzu Standard Packages

  • Tanzu Diagnostics commands to troubleshoot standalone management and workload clusters.

For more information about these features and enhancements, see the VMware Tanzu Kubernetes Grid 2.5 Release Notes.

Carrier-Grade VNF and CNF Automation and Orchestration

VMware Telco Cloud Automation 3.1 introduces various new features and enhancements:

  • Multi-TKG support: Supports multiple Tanzu Kubernetes Grid releases (2.1.1 and 2.5). This feature allows users to continue to use Kubernetes 1.24.x clusters from the previous Tanzu Kubernetes Grid release (2.1.1), while being able to create new workload clusters with the new Kubernetes version from the new Tanzu Kubernetes Grid release (2.5).

  • Cluster Rehoming: Enables selective upgrading of legacy Kubernetes workload clusters to the latest K8s version. This feature avoids the need to upgrade all legacy workload clusters soon after the management cluster upgrade, providing flexibility on maintenance windows. Keeping the source management cluster (Tanzu Kubernetes Grid 2.1.1) allows support for all LCM/CRUD Operations on Kubernetes 1.24 clusters until End-of-Life (EOL).

  • CaaS cluster upgrade improvements:

    • Supports workload cluster upgrade from the UI

    • Granular Updates for CNF Operations with Detailed Events

    • Provides better visibility during CaaS upgrade

  • Skip-level upgrade: Supports skip-level upgrade from Telco Cloud Automation 2.3 to 3.1 while skipping Telco Cloud Automation 3.0.

  • Network Function LCM improvements: Introduces information about the CNF operation progress, CNF LCM detailed helm and k8s events are collected and displayed in the CNF operation detail view on the TCA GUI. In addition, when deleting an empty Kubernetes workload cluster, the associated VIM is also deleted.

  • Workflow Hub improvements and new workflows: Introduces enhancements to Queuing, payload schema validation, Telco Cloud Service Assurance integration, workflow library upgrade, new BMA/ZTP workflows to manage host provisioning and upgrade from K8S Extended support to the next Extended support workflow.

  • GitOps support - phase II (Tech Preview): Possibility to manage CNF LCM with GitOps, as an alternative to ETSI including Dynamic Infrastructure Policy.

  • VNF Enhancements: TCA can now consume Datacenter Group backed Networks for VMware Cloud Director.

  • CaaS ENS RSS Spec readiness (Tech Preview): Enables NetQ Receive Side Scaling (RSS) vNICs requests to be offloaded to a physical NIC through a feature flag using an API call. This feature improves the packet performance of the receive-side data. The RSS configuration is disabled by default.

  • Certificate Observability: Provides certificates monitoring and alerts of all Systems connected to the TCA platform.

  • Enhancements to TCA Platform Security:

    • Prevents NFs into system NameSpace: Prevents deployment of CNFs to system namespaces. The list of reserved namespaces can be edited in Telco Cloud Automation.

    • Upgraded secure runtime (java), fixing pentest findings, and fixing CVEs

    • Pod Security Admission (PSA) and Kubernetes policy configuration:

      • Users can configure the PSA policy and the Kubernetes policy separately

      • Users can define defaults that apply to all user-created namespaces, rather than apply cluster-wide

      • PSA status can be shown in the GUI

  • Airgap server enhancements: Provides several Airgap server enhancements including Health observability, multi-TKG data, improved upgrades with synchronization and reduced overall data size, and also certificate renewal.

  • Harbor certificate renewal and automation: Allows users to specify and update the Harbor certificate under Partner systems in Telco Cloud Automation.

For more information about these features and enhancements, see the VMware Telco Cloud Automation 3.1 Release Notes.

Operations Management

VMware Telco Cloud Platform includes components such as Aria Operations, Aria Operations for Logs, and Telco Cloud Service Assurance that provide operations capabilities such as fault management, accounting, and performance for the entire Telco Cloud Platform.

VMware Telco Cloud Service Assurance 2.4 includes several new features and enhancements.

Important:

VMware Telco Cloud Service Assurance is available as part of VMware Telco Cloud Platform Advanced Edition.

  • Notification Console is enhanced to reorder the Notification Details tabs based on users' needs from the notification panel.

  • Alerting feature is now enabled with multiple threshold conditions supported by different severity levels of the alarm. Users can now create alarms with different severity conditions based on different threshold conditions or different time intervals for particular threshold levels.

  • Alarms creation is now enabled with the Clear condition option where a user can define a condition to clear an alarm in the same definition where the alarm triggers.

  • User Defined Fields (UDFs) in the alarms are now enhanced with a combination of multiple properties, tags, and static data, which enables users to provide all the required details in User Defined Fields. The UDF in creating Alarm Definition is enabled with a combination of static values and multiple tags, properties, and static data such as description. This feature helps users to provide all the details such as Property (Router/Host) located in Tag (tag.Location) with value Tag (tag.Details) in the User Defined Fields.

  • Alarm definition is now enabled with an EDIT option for filters.

  • vCenter Collector is now available out of the box to collect performance data from configured vCenter endpoints.

  • EDAA is enabled by default in all service definitions of domain managers (IP, SAM, ESM, NPM, and MPLS).

  • New device certifications are added in the IP domain manager.

For more information about these features, see the VMware Telco Cloud Service Assurance 2.4 Release Notes.

VMware Aria Operations 8.16 includes several features and enhancements. For more information, see the VMware Aria Operations 8.16 Release Notes.

VMware Aria Operations for Logs 8.16 includes several features and enhancements. For more information, see the VMware Aria Operations for Logs 8.16 Release Notes

Components

Telco Cloud Platform Essentials

To download these components, see the Telco Cloud Platform 4.0 Essentials Product Downloads page.

Optional Add-On Components

Note:

Additional license is required.

Telco Cloud Platform Advanced

To download these components, see the Telco Cloud Platform 4.0 Advanced Product Downloads page.

Optional Add-On Components

Note:

Additional license is required.

Validated Patches

Support for Backward Compatibility of CaaS Layer with IaaS Layer

VMware Telco Cloud Platform Release 4.0 supports backward compatibility of its CaaS layer components (Telco Cloud Automation and Tanzu Kubernetes Grid) with the IaaS Layer components (vSphere and NSX) in earlier versions of Telco Cloud Platform. With this feature, you can upgrade the CaaS layer components to their latest versions while using earlier versions of the IaaS layer components.

For more information, see Software Version Support and Interoperability in the Telco Cloud Automation Deployment Guide and Supported Features on Different VIM Types in the Telco Cloud Automation User Guide.

Resolved Issues

Note: For information about the entire list of fixes in each Telco Cloud Platform component, see the corresponding product release notes.

  • Unable to Perform Rollback on CNF if it was Instantiated Without the Node Customization Configured

    If a CNF was instantiated without the node customizations configured in Telco Cloud Automation, rollback cannot be performed on the CNF. Additional customizations can be performed only through the CNF upgrade.

    This issue is resolved.

  • CNF Instantiation Wizard Does Not Show Warning for CNFs Deployed in Node Pools When Selecting Node Pools for Customization

    When selecting the node pools for customization in Telco Cloud Automation, the CNF Instantiation wizard does not show any warning message that the CNFs are already deployed on the selected node pools.

    This issue is resolved.

  • Management Cluster Upgrade from v1.25 to v1.26 Fails Occasionally

    In Telco Cloud Automation, the Management Cluster upgrade from v1.25 to v1.26 fails occasionally with the following message:

    Cluster upgraded successfully but post configuration failed.

    This issue is resolved.

Known Issues

Note: For information about the entire list of known issues in each Telco Cloud Platform component, see the corresponding product release notes.

  • Bare Metal Edge Devices Experience Packet Drops Causing Network Performance Degradation

    Bare Metal Edge devices using NSX 4.1.2.1 experience packet drops, which affect the network traffic flow and result in degraded network performance.

    Workaround: Upgrade the NSX Edge version from 4.1.2.1 to 4.1.2.4.

  • VMware Aria Operations 8.16 Fails to Integrate with Tanzu Kubernetes Classy Standard Clusters 1.27 and 1.28 Running TLS 1.3

    VMware Aria Operations 8.16 fails to integrate with Tanzu Kubernetes Classy Standard Clusters (1.27 and 1.28) that are running Transport Layer Security (TLS) version 1.3. Hence, the Classy Standard Clusters 1.27 and 1.28 cannot be monitored.

    Note: TLS 1.3 is not supported in Aria Operations 8.16.

    Workaround: Follow one of these workarounds:

    • Upgrade to Aria Operations 8.18.

    • If you are using Aria Operations 8.16, change TLS 1.3 to TLS 1.2 on the Classy Standard clusters.

      1. Navigate to CaaS Infrastructure > Cluster Instances in the TCA UI.

      2. Select the cluster you want to modify, and add the following topology variable in the Edit Cluster Configuration > Configuration > Cluster Info section:

        • Key: Security

        • Value: minimumTLSProtocol: tls_1.2

  • Incorrect Template Used for Control Plane and Worker Node Deployments When Upgrading Management Cluster from Kubernetes version 1.24 to 1.25

    If vCenter contains multiple templates of the same version during the management cluster upgrade from Kubernetes version 1.24 to 1.25 (Tanzu Kubernetes Grid 2.2), an incorrect template is used for control plane and worker node deployments.

    In this issue, node cloning might fail in vCenter if the auto-selected node is not accessible by the selected vSphere cluster. vCenter reports a Clone VM Task Failure with the following message:

    Cannot connect to host, and cluster creation fails.

    Workaround: Delete redundant Kubernetes cluster templates from vCenter Server.

  • vCenter Server Upgrade to 8.0 U2 Stuck for an Extended Time in Airgapped Environment

    vCenter Server upgrade to 8.0 U2 is stuck for an extended time in an airgapped environment.

    Workaround: Do one of the following:

    • If the upgrade is in progress, wait until the upgrade completes.

    • Retry the upgrade through VAMI or the full patch ISO.

  • Techsupport Bundle Generation for CaaS Clusters Might Fail When Run in Parallel

    The techsupport bundle generation for CaaS clusters might fail if it is run in parallel.

    In this issue, the Support bundle service allows a user to trigger multiple support bundle requests simultaneously, while KBS allows only one CaaS cluster log collection request at a time.

    Workaround: Wait until the previous techsupport bundle generation completes and then retry the subsequent bundle generation.

    Note: The Support bundle service displays a tooltip that a subsequent request to collect CaaS cluster logs will fail if one is already running.

  • Migration to TCA 3.0 or 3.1 Not Supported if Compute Cluster Domains Exist in TCA 2.3.x Infrastructure Automation

    If compute cluster domains exist in Telco Cloud Automation 2.3.x Infrastructure Automation, migration to TCA 3.0 or 3.1 is not supported.

    Workaround:

    1. Revert the partially migrated appliances using the tcamigctl tool.

    2. In the TCA Manager Web UI (443), delete compute cluster domains under Infrastructure Automation > Domains > Compute Cluster.

    3. Retry the migration.

    Important:

    Compute Clusters functionality in the Infrastructure Automation is deprecated and migration is not supported for Compute Clusters. Deleting Compute Clusters is a prerequisite for migration.

  • Multitenancy Not supported for Certificate Observability Service

    Unless a non-default Tenant shares the Endpoint with the default Tenant or the default Tenant inherits the Endpoint as a part of parent-child relationship, the Endpoint is not shown in the view for a Default Tenant login.

    For the Default Tenant login, though the Endpoint owned by other Tenants (non-default) is not listed in the portal, the Endpoint may get listed in the Connected Endpoints listing.

    Workaround: NA

  • CNF Upgrade Retry Skips Nodecustomization if Previous Nodecustomization Failed During CNF Upgrade

    The CNF upgrade retry skips nodecustomization if the previous nodecustomization failed during the CNF upgrade.

    Workaround:

    1. Roll back the failed CNF upgrade.

    2. Perform a fresh upgrade instead of retry.

  • Airgap rsync Operation Might Fail Occasionally if it is Run Multiple Times

    The airgap rsync operation might fail occasionally if it is run multiple times.

    Workaround: Run the following commands on the airgap server as a root user:

    1. Remove the existing content from the following location:

      rm -f /etc/yum.repos.d/*

    2. Copy the content from the backup location:

      cp /usr/local/airgap/backup_repo/* /etc/yum.repos.d/

    3. Run the rsync operation using the copied content:

      agctl rsync

  • Management Cluster Upgrade Might Fail Due to Default Timeout in TCA API

    The management cluster upgrade might fail due to the default timeout (about 3.5 hours) in the TCA API. If the upgrade task is running in the backend, inconsistent cluster status appears in the TCA UI and backend.

    Workaround:

    1. Identify the backend status of the management cluster upgrade from the k8s-bootstrapper pod.

      1. Log in to k8s-bootstrapper pod from TCA-CP:

        # kubectl exec -it <k8s-bootstrapper-pod-name> -ntca-cp-cn bash

      2. Identify the cluster ID using the management cluster name:

        # curl http://localhost:8888/api/v1/managementclusters 

      3. Identify the cluster upgrade status using the management cluster ID:

        # curl http://localhost:8888/api/v1/managementcluster/<target-mc-id>/status 

    2. Retry the management cluster upgrade based on the cluster upgrade status:

      • If the cluster upgrade is complete and its status is running, retry the management cluster upgrade from the TCA UI.

      • If the cluster upgrade is in progress and its status is upgrading, wait until the upgrade is complete and retry.

  • capv User Account Gets Locked After Three Unsuccessful Login Attempts in 15 Minutes

    The capv user account gets locked after three unsuccessful login attempts in 15 minutes. The following message appears in the Journal log:

    Mar 27 07:15:55 cp-stardard-cluster-1-control-plane-zdfgm sshd[3767202]: pam_faillock(sshd:auth): Consecutive login failures for user capv account temporarily locked

    In this issue, the Photon operating system automatically locks the user account as per the Photon 5 STIG requirement (PHTN-50-000108).

    Workaround:

    1. Log in to TCA-CP as an admin and change to the root user.

    2. SSH in to workload cluster endpoint as a capv user.

    3. Release the locked account:

      # faillock --user capv --reset

  • Workload Cluster Upgrade Might Leave a Few Pods in Terminating State

    Sometimes, the workload cluster upgrade might leave a few pods in the terminating state.

    Workaround:

    1. SSH into the workload cluster control plane.

    2. Identify the node with the status "NotReady,SchedulingDisabled":

      kubectl get node

    3. Clear the node:

      kubectl delete node <node_name>

  • Cluster Creation Might Fail if Too Many Kubernetes Node Templates are on vCenter

    If too many Kubernetes node templates are on vCenter, the cluster creation might fail with the following error in Tanzu Kubernetes Grid log reports.

    Workaround:

    1. Delete all unused Kubernetes node templates from vCenter Server.

    2. Retry the cluster creation operation.

    Note: You can also increase the resources allocated to the tkr-source-controller. For instructions on increasing resources, see KB92524.

  • Node Pool Forced Deletion Might Stuck in Processing State if Cell Site Host Goes Down

    If the cell site host goes down, the forced deletion of a node pool might get stuck in the processing state.

    Workaround: Remove the host from the vCenter inventory.

    Note: After the host is removed, the node pool is deleted successfully without requiring any additional action from TCA.

  • Management Cluster Creation Fails When vCenter Server Password Ends with Colon

    If the vCenter Server password ends with the colon character, the management cluster creation fails in Telco Cloud Automation.

    Workaround: Change the vCenter Server password so it does not end with a colon.

End of General Support Guidance

VMware Product Lifecycle Matrix outlines the End of General Support (EoGS) dates for VMware products. Lifecycle planning is required to keep each component of the VMware Telco Cloud Platform solution in a supported state. Plan the component updates and upgrades according to the EoGS dates. To ensure that the component versions are supported, you may need to update the Telco Cloud Platform solution to its latest maintenance release.

VMware pre-approval is required to use a product past its EoGS date. To discuss the extended support for products, contact your VMware representative.

Release Notes Change Log

Date

Change

Oct 1, 2024

VMware vCenter Server 8.0 Update 3b is added to the Validated Patches section.

Sep 10, 2024

VMware Telco Cloud Service Assurance 2.3.1 is replaced with VMware Telco Service Assurance 2.4.

Note: You can now use Telco Service Assurance 2.4 with Telco Cloud Platform 4.0.

Aug 9, 2024

The following issue is added to the Known Issues section;

  • Bare Metal Edge Devices Experience Packet Drops Causing Network Performance Degradation

Aug 7, 2024

VMware Aria Operations 8.16 is replaced with VMware Aria Operations 8.18 in the Components section.

Aug 7, 2024

The following issue is added to the Known Issues section:

  • VMware Aria Operations 8.16 Fails to Integrate with Tanzu Kubernetes Classy Standard Clusters 1.27 and 1.28 Running TLS 1.3

Jul 10, 2024

VMware vCenter Server is replaced with VMware vCenter.

Support Resources

For additional support resources, see the VMware Telco Cloud Platform documentation page.

check-circle-line exclamation-circle-line close-line
Scroll to top icon