RSS
 

VMware Telco Cloud Platform 5G Edition 2.7 | 19 JAN 2023

Check for additions and updates to these release notes.

What's New

Telco Cloud Platform 5G Edition Release 2.7 provides various key features and enhancements across carrier-grade workload compute, networking, network function automation and orchestration, and Kubernetes infrastructure areas.

This release delivers high-availability features such as Affinity/Anti-Affinity capability for CaaS clusters and backup/restore of the CaaS clusters using Velero. This release also delivers Node Autoscaler for CaaS clusters and dual-stack IPv6 support for Telco Cloud Automation. Other key features of this release include new functionalities in the IaaS layer for virtualized networking, security, and migration from NSX Data Center for vSphere, and several infrastructure and security bug fixes.

Carrier-Grade VNF and CNF Automation and Orchestration

VMware Telco Cloud Automation 2.2 introduces various new features and enhancements. Some of the key features are as follows:

  • Affinity/Anti-Affinity for CaaS Clusters: Supports application high availability that allows Tanzu Kubernetes nodes to run on different hosts, thereby enabling applications to run even if a host fails.

  • Backup and Restore of CaaS Clusters: Supports backup and restore of CaaS clusters using Velero for increased resiliency.

  • IPv6 Dual Stack Support: Support for both IPv4 and IPv6 environments with some interfaces on IPv4 while others can be migrated to IPv6.

  • Improved CaaS Add-ons: Support for Whereabouts, Cert Manager, and Enhanced Multus.

In addition, Telco Cloud Automation delivers many enhancements related to infrastructure automation, CNF support, Airgap support, and dedicated NFS interface support for CaaS workload clusters and other areas.

For more information about these features and enhancements, see the VMware Telco Cloud Automation 2.2 Release Notes.

Carrier-Grade Kubernetes Infrastructure

VMware Tanzu Standard for Telco introduces various key features as part of VMware Tanzu Kubernetes Grid 1.6.1. This release also inherits enhancements from Tanzu Kubernetes Grid 1.6. Some of the key features are as follows:

  • Support for Kubernetes versions:

    • v1.23.10

    • v1.22.13

    • v1.21.14

  • Wherabouts Support for IP Address Management (IPAM) CNI plugin, with Multus CNI to dynamically assign IP addresses to pods across the cluster.

  • Supports the deployment of a workload cluster to an edge device.

  • Security Hardening for Photon OS 3 in compliance with Security Technical Implementation Guides (STIG) standards by default.

  • Fixes for security vulnerabilities:

    • Kernel upgrade: CVE-2022-0492, CVE-2022-0847

    • Golang upgrade: CVE-2022-23806, CVE-2022-23772, CVE-2022-23773

For more information about these features and enhancements, see the VMware Tanzu Kubernetes Grid 1.6.1 Release Notes.

Carrier-Grade Resilient Networking and Security

VMware NSX-T Data Center 3.2.2.1 includes the following key enhancements and bug fixes:

  • NSX Data Center for vSphere to NSX-T Data Center migration: Allows pausing or removing of hosts during in-place migration. The Migration Coordinator supports pauses between hosts or the removal of hosts during the host migration phase. In addition, a new migration mode is added to the Migration Coordinator for Lift-and-Shift - Configuration and Edge Migration. This mode allows migrating both configuration and edge, and establishes a performance-optimized distributed bridge between the NSX Data Center for vSphere (NSX-V) and NSX-T Data Center (NSX-T) environments. Hence, the connectivity between the source and destination environments is maintained during the lift-and-shift migration.

  • Install and upgrade process: Allows the registration of multiple NSX management clusters to a single vCenter Server. This feature allows the isolation of NSX lifecycle management per cluster and has all NSX workloads connect to the NSX on vSphere Distributed Switch (vDS).

  • Ease of operations:

    • Backup & restore: Allows reminders to be sent for configuring the backups in the NSX Manager UI.

    • Upgrade readiness: NSX Upgrade Evaluation Tool is now integrated with pre-upgrade checks as part of the NSX framework. This feature eliminates the need for getting compliance approvals or versioning with a separate appliance. When you run the pre-upgrade checks, NSX checks the readiness of your NSX deployment for a successful NSX Manager upgrade.

  • Bug fixes: In addition, this release includes the following key bug fixes that are critical to NSX deployments:

    • Baremetal Edge Management and support for Mellanox NICs

    • NVDS to VDS Migration

    • NSX-V to NSX-T Migration

    • EVPN Route-Server mode

For more information, see the VMware NSX-T Data Center 3.2.2.1 Release Notes.

Workload Management, Storage, and Reliability Enhancements

VMware vCenter Server 7.0 Update 3h includes new features and fixes for vCenter Server.

For more information, see the VMware vCenter Server 7.0 Update 3h Release Notes.

Support for Backward Compatibility of CaaS Layer with IaaS Layer

VMware Telco Cloud Platform 5G Edition Release 2.7 supports backward compatibility of its CaaS layer components (Telco Cloud Automation and Tanzu Kubernetes Grid) with the IaaS Layer components (vSphere and NSX-T Data Center) in earlier versions of Telco Cloud Platform 5G Edition. With this feature, you can upgrade the CaaS layer components to their latest versions while using earlier versions of the IaaS layer components.

For more information, see Software Version Support and Interoperability in the Telco Cloud Automation Deployment Guide and Supported Features on Different VIM Types in the Telco Cloud Automation User Guide.

Components

Mandatory Add-On Components

Note:

An additional license is required.

Optional Add-On Components

Note: Additional license is required.

Validated Patches

End of General Support Guidance

VMware Product Lifecycle Matrix outlines the End of General Support (EoGS) dates for VMware products. Lifecycle planning is required to keep each component of the VMware Telco Cloud Platform solution in a supported state. Plan the component updates and upgrades according to the EoGS dates. To ensure that the component versions are supported, you may need to update the Telco Cloud Platform solution to its latest maintenance release.

VMware pre-approval is required to use a product past its EoGS date. To discuss the extended support of products, contact your VMware representative.

Note: If you purchase NSX-T Data Center as part of the Telco Cloud Platform bundles, NSX-T Data Center is entitled to the support lifecycle specific to the Telco Cloud Platform bundles. The entitlement details for NSX-T 3.2.x are as follows:  

  • General Availability: 2021-12-16

  • End of General Support: 2026-12-16

The Technical Guidance phase is not available for this product lifecycle. To receive new severity 1 bug fixes and security updates, upgrade NSX-T Data Center to the latest maintenance release in the 3.2.x release series.

Resolved Issues

Note: For information about the entire list of fixes in each Telco Cloud Platform 5G component, see the corresponding product release notes.

  • Node Pool Customizations Fail when the Node Pool is in Maintenance Mode

    Whenever the node pool is in maintenance mode, users are now prompted appropriately on the node pool listing page and the CNF instantiation page of the Telco Cloud Automation UI.

  • Helm Service Fails to Start on the TCA-CP Appliance

    When the TCA-CP database has orphan CNF entries, the Helm service fails to start on the TCA-CP appliance.

    This issue is fixed in Telco Cloud Automation 2.2.

  • VM Hardware Version Upgrade Not Applied on the Newly Created Node Pools of v2 CaaS Workload Clusters

    The node customization option to upgrade the VM hardware version to the latest is not applied on the newly created node pools of v2 CaaS workload clusters.

    This issue is fixed in Telco Cloud Automation 2.2.

  • CaaS Cluster Operations Fail After Backing Up and Restoring from One VMware Telco Cloud Automation Appliance to Another

    When you perform backup and restore from one Telco Cloud Automation 2.1 appliance to another, TCA-CP Minikube’s kubeconfig is not updated and the cluster operator resources are not restored. This results in the failure of CaaS Cluster operations.

    This issue is fixed in Telco Cloud Automation 2.2.

  • Configuration of vmconfig-operator and Istio add-ons for Management Cluster Not Supported from Telco Cloud Automation UI

    The configuration of vmconfig-operator and Istio add-ons for management clusters is not supported in the Telco Cloud Automation UI.

    This functionality is now supported in Telco Cloud Automation 2.2.

  • Alert Manager Pod of Prometheus Add-on Fails to Boot

    When deploying Prometheus to a Tanzu Kubernetes cluster, the alertmanager pod of the Prometheus add-on fails to boot. It might get stuck in the CrashLoopBackOff state in the vCenter 70u2 and vCenter 70u3 deployments.

    This issue is fixed in the Tanzu Kubernetes Grid 1.6.1.

  • Modification of ako-operator Add-on Configuration Not Supported

    The modification of ako-operator add-on configurations such as Avi Controller credentials and certificates is not supported.

    This issue is fixed in Telco Cloud Automation 2.2. You can now modify the Avi Controller certificate and credentials.

  • Custom Port Harbor Add-On Gets Stuck When Adding Harbor to the V2 clusters

    When adding the custom port harbor to V2 clusters, the harbor add-on gets stuck.

    This issue is fixed in Telco Cloud Automation 2.2.

  • Design Network Function with VMXNET3 Adapter Fails to Onboard CSAR Package

    In CSAR designer, the Add Network Adapter with the device type vmxnet3 does not show the Resource Name. This causes the failed onboarding of the CSAR package.

    This issue is fixed from Telco Cloud Automation 2.1.1 onwards.

  • Edit Operation Fails for Workload Clusters Transformed from CaaS V1 to v2

    After transforming a workload cluster associated with TCA-CPs (excluding TCA-CPs used for management clusters), the day-2 operations such as creating a cluster or nodepool and editing a cluster or nodepool fail.

    This issue is fixed in Telco Cloud Automation 2.2.

    Note:

    If the transform operation was done before upgrading Telco Cloud Automation to 2.2, follow the instructions in KB89361.

Known Issues

Note: For information about the entire list of known issues in each Telco Cloud Platform 5G component, see the corresponding product release notes.

  • Upgrading a CaaS Workload Cluster Sometimes Might Lead to Upgrade Failure

    When you upgrade a CaaS workload cluster, sometimes the upgrade fails with the following error:

    "timeout: poll control plane ready for removing SCTPSupport=true"

    Workaround:

    • Caution:

      Before upgrading to Telco Cloud Automation 2.2, we recommend that you do the following to avoid the error:

      • Ensure that the CaaS cluster has multiple control plane nodes

      • Ensure that the CaaS cluster is upgraded to a supported Kubernetes version in Telco Cloud Automation 2.1

    • If the error still occurs, contact the VMware Technical Support team to debug and fix the error.

  • vSphere CSI Not Supported if the Cluster is Deployed Across Multiple vCenter Servers

    Workaround: N/A

  • CaaS Cluster Deployments are Not Supported on vCenter Servers Containing Multiple Data Centers

    Workaround: N/A

  • AKO Deployment Fails on the CaaS v2 Workload Cluster When the Cluster Name Length is Greater Than 29 Characters

    If the length of the CaaS v2 workload cluster name is greater than 29 characters, the installation of the load-balancer-and-ingress-service add-on for the cluster fails.

    Workaround: Ensure that the length of the workload cluster name is less than or equal to 29 characters.

  • TCA Appliance Manager UI and API Show GMT Time Zone Instead of UTC Time Zone

    Telco Cloud Automation 2.2 Appliance Manager UI and API show the default GMT time zone instead of UTC.

    Note:

    This change does not impact any functionality.

    Workaround: Not Applicable

  • No Proxy Field Marked Optional While Configuring Proxy in Partner Systems

    While configuring Proxy in Partner systems, the No Proxy field is marked optional. The following error occurs when the Proxy configuration details are not specified in this field:

    Unable to create k8s cluster with error Proxy. NoProxy: No proxy list can’t be empty. CIDR of cluster node network is mandatory at least

    Workaround: Ensure that at least the following details are specified in the No Proxy field:

    • Node network CIDR, vCenter FQDN/IP address, Harbor URL, NFS server FQDNs

    Important:

    The Proxy settings cannot be modified after the initial configuration. Hence, add all the node network CIDR ranges to be used by CaaS clusters that are created through proxy.

  • Harbor Credentials Not Cleared When Airgap Operations fail with a Wrong Harbor Password

    If an airgap operation such as Setup, Sync, and Deploy fails with a wrong harbor password, the Harbor credentials are not cleared and all other operations fail.

    Workaround: Clear the Harbor credential file:

    {root-dir/airgap/scripts/vars/harbor-credential.yaml}

  • Certificate Generation Skipped in the Airgap Deployment Phase

    In the airgap deployment phase, the certificate is not generated even when the user-inputs.yml file requests for certificate generation. The certificate is auto-generated only when the FQDN changes or when no certificates are in the root directory.

    Workaround: To generate new certificates in the airgap deployment phase, do the following tasks:

    1. Ensure that the value of auto_generate is set to True in the user-inputs.yml.

    2. Use a different FQDN in the setup phase or remove the {root-dir}/airgap/certs/ folder.

  • cainjector and webhook Pods of the Cert-Manager Add-on Stuck in CrashLoopBackOff Status

    The cainjector and webhook pods of the cert-manager add-on are stuck in the CrashLoopBackOff status. Hence, the provision status of the cert-manager add-on is unhealthy in the TCA UI.

    Workaround: Restart the CrashLoopBackOff pod using the following command:

    kubectl delete pod -n cert-manager <crash-pod-name>

  • Old Kubernetes Versions Displayed in the Telco Cloud Automation UI Even After Upgrading Telco Cloud Automation

    If the TCA Manager and TCA-CPs under it are not on the same and latest Telco Cloud Automation version during the cluster template creation or during the v1 cluster upgrade, old Kubernetes versions are displayed in the Telco Cloud Automation UI.

    Workaround: Upgrade the TCA-CPs registered on the TCA Manager to the same and latest Telco Cloud Automation version.

  • CaaS v2 Cluster Creation with Antrea is Stuck in Processing State When Using an NSX overlay Network Enabled with EDP

    When you use an NSX overlay network enabled with Enhanced Data Path (EDP) in TCA, Antrea is stuck in the processing state. It does not create the v2 cluster node pool or provision the nodeconfig-operator.

    Workaround: Do one of the following tasks:

    • (Workaround 1) Modify the value of ANTREA_DISABLE_UDP_TUNNEL_OFFLOAD from false to true on the cluster operator:

      Important:

      This is the preferred workaround. This workaround is required only on the management cluster's control plane node. However, this workaround is not persistent and it needs to be re-applied whenever the tca-kubecluster-operator pods are rebooted.

      1. Log in to the management cluster Control Plane node IP through SSH.

      2. Access the Cluster Operator pod by running the following command:

      3. Run the following command:

        • sed -i 's/ANTREA_DISABLE_UDP_TUNNEL_OFFLOAD: false/ANTREA_DISABLE_UDP_TUNNEL_OFFLOAD: true/g' /root/.config/tanzu/tkg/providers/config_default.yaml

    • (Workaround 2) Set the value of disableUdpTunnelOffload to true in the antrea add-on.

      Important:

      Follow this workaround only when workaround 1 is not applied. Apply this workaround after the workload cluster is created and is in an error state.

      1. Create a v2 workload cluster.

      2. Click Add-Ons.

      3. Choose the Antrea addon name antrea-tca-addon and click Edit corresponding to the antrea-tca-addon.

      4. Click Cancel on the pop-up page to configure No SNAT & Traffic Encap Mode.

      5. Click Edit corresponding to Antrea and then click Next.

      6. Click Custom Resources (CRs), append disableUdpTunnelOffload: true under spec/config/stringData/values.yamls/antrea/config, and then click Deploy.

    Sample:

    metadata:  resourceVersion: 1  name: antrea-tca-addon  clusterName: {cluster_name}spec:  name: antrea  clusterRef:    name: {cluster_name}    namespace: {cluster_name}  config:    stringData:      values.yaml: |        antrea:          config:            disableUdpTunnelOffload: true

  • Airgap Techsupport Logs Collection Fails Due to Insufficient Space in /tmp

    If the airgap server is running for a long period, the Airgap techsupport logs collection fails due to insufficient space in /tmp.

    Workaround:

    1. Open the ansible playbook {root-dir}/airgap/scripts/playbooks/airgap-support.yml.

    2. Under the Create logs dir and Package support-bundle section, change the default log location from /tmp to a different folder.

  • Cell Site Host Provisioning Marked Successful in Telco Cloud Automation Even Though Network Migration of Host to the Distributed Switch Fails

    While provisioning a cell site host under a domain with two or more Distributed Virtual Switches in Infrastructure Automation, if a switch with the management network is not mapped to use vmnic that has the VMKernel network (vmk0) interface, the network migration of the host to the switch fails. However, the host provisioning is marked as successful.

    Workaround: If a cell site group domain has multiple Distributed Virtual Switches, do the following:

    1. Modify the configuration of the cell site group domain to map the management network switch to the vmnic that has the vmk0 interface attached.

    2. Perform a full resynchronization on the host.

    After the configuration, the host is provisioned again and the network migration is successfully completed from the host to the relevant switch.

  • Fluent-bit Pod Stuck in CrashLoopBackoff State on Worker Nodes Where cpu-manager-policy is Set to Static

    The Fluent-bit pod is stuck in the CrashLoopBackoff state on worker nodes where the cpu-manager-policy is set to static.

    Workaround: Change the cpu-manager-policy to none.

  • New Active Directory Users Cannot Log in to Telco Cloud Automation

    New Active Directory users configured with the option 'Change Password on next logon' cannot log in to Telco Cloud Automation.

    Workaround: Before you log in to Telco Cloud Automation, set the user password in the Active Directory.

  • Deployment of IPv6-based Workload Clusters with Kubernetes Versions Older than 1.22.x Results in the Failure of Cluster Creation

    Due to existing limitations in Tanzu Kubernetes Grid, the deployment of IPv6-based workload clusters with Kubernetes versions older than 1.22.x results in the failure of cluster creation. If a workload cluster has more than one control plane node, this may further lead to IP address conflicts where VSPHERE_CONTROL_PLANE_ENDPOINT IP address is assigned to the node and host network pods.

    Workaround: Deploy IPv6-based workload clusters with Kubernetes version 1.22.x or later.

  • Avi Kubernetes Operator Uninstallation Does Not Delete Objects in AVI Controller

    When you uninstall the load-balance-and-ingress-service add-on from a Workload cluster, objects in Avi Controller are not deleted automatically.

    Workaround: Delete the objects manually from the Avi Controller UI.

  • vmconfig-operator Does Not Support vCenter Access Through Custom Port

    vmconfig-operator supports the cluster deployment within vCenter environments that run only on port 443. vCenter environments with custom ports are not supported.

    Workaround: Not Applicable

  • vSphere CSI Does Not Support Topologies Where Workload Cluster is Deployed Across Multiple vCenter Servers

    vSphere CSI does not support topologies where the workload cluster is deployed across multiple vCenter servers.

    Workaround: For workload clusters that span across multiple vCenter Servers, use the NFS_Client CSI driver.

Release Notes Change Log

Date

Changes

25 OCT 2023

VMware vCenter Server 7.0 Update 3o is added to the Validated Patches section.

1 JUN 2023

NSX-T Data Center 3.2.2 is replaced with NSX-T Data Center 3.2.2.1.

Support Resources

For additional support resources, see the VMware Telco Cloud Platform documentation page.

check-circle-line exclamation-circle-line close-line
Scroll to top icon