VMware Telco Cloud Service Assurance 2.0.1 Release Notes

VMware Telco Cloud Service Assurance 2.0.1 | 04 AUG 2022

Check for additions and updates to these release notes.

About VMware Telco Cloud Service Assurance

VMware Telco Cloud Service Assurance is a real-time automated service assurance solution designed to holistically monitor and manage complex 5G NFV virtual and physical infrastructure and services end to end, from mobile core to the RAN to the edge. From single pane of glass, VMware Telco Cloud Service Assurance provides cross‑domain, multi‑layer, automated assurance in a multi‑vendor and multi‑cloud environment. It provides operational intelligence to reduce complexity, perform rapid root cause analysis and see how problems impacts services and customers across all layers lowering costs, and improved customer experience.

For information about setting up and using VMware Telco Cloud Service Assurance, see the VMware Telco Cloud Service Assurance Documentation.

What's New

VMware Telco Cloud Service Assurance release 2.0.1 brings together various features and enhancements across platform, networking, and virtual infrastructure management areas. This release introduces new functionalities like, the deployment of VMware Telco Cloud Service Assurance with VMware Telco Cloud Automation (TCA), Backup and Restore, and new usecase support for Cisco-ACI, IP-SLA, and vIMS.

Note: VMware Telco Cloud Service Assurance v2.0.1 does not support Kubernetes 1.20.x deployments.

5G Core and vRAN Assurance
Platform Modernization
Enhanced Assurance Use Cases
User Interface Enhancements
Identity Management System Enhancements
Logs Enhancements
REST API Enhancements
Domain Manager Enhancements

5G Core and vRAN Assurance

Support for Network Slice connectivity maps.
Grafana has been updated to the latest version to take advantage of new widgets like HeatMap and so on.
Support has been added for the Business Impact Management (BIM): BIM calculates the impact of network infrastructure and application root-cause problem events on business elements, and creates business impact notifications. The business impact notifications enable business enterprises to immediately identify disruptions and failures, and to prioritize the resolution of the infrastructure and application problems in accordance with their business impact.

Platform Modernization

VMware Telco Cloud Service Assurance can now be easily deployed using VMware Telco Cloud Automation (TCA) Orchestrator as a Cloud Native Service (CSAR) package similar to onboarding and instantiating Container Network Function.
Backup and Restore feature in VMware Telco Service Assurance provides enhanced data protection.

Enhanced Assurance Use Cases

Following use cases are now supported in VMware Telco Cloud Service Assurance for data collection and assurance:

Cisco ACI Collector collects data from Cisco Application Centric Infrastructure (ACI) deployment. It is a software-defined networking solution that facilitates the application agility and data center automation.

vIMS collector collects data from a vIMS clear water deployment. vIMS provides a scalable deployment of IP Multimedia Subsystem (IMS) in virtualized and cloud environments for voice, video, and messaging services.

Cisco IP-SLA is specifically designed for most common IP-SLA probes running on Cisco IP-SLA enabled devices.

User Interface Enhancements

Enhanced user interface with ability to edit Alarms based on domain and sub-domain classification, clone notification views, filter RCAs, and so on.

Identity Management System Enhancements

Added REST API support to provision Active Directory or LDAP as Identity Stores to KeyCloak.

Logs Enhancements

Exporting North Bound Interface (NBI) service's logs to Kibana log viewer.

REST API Enhancements

Added REST API support for VMware Telco Cloud Service Assurance metrics.

Domain Manager Enhancements

Support for MPLS and NPM Domain Managers is added.

MPLS Domain Manager: MPLS enhances network performance by introducing virtual circuits called label switched paths (LSPs) to IP networks. Packets are switched rather than routed through the network. And because the fundamental principles of virtual circuits are based on traffic separation and segmentation, MPLS is ideal for building provider-provisioned Layer 2 (L2) and Layer 3 (L3) VPNs.

NPM Domain Manager: Network Protocol Manager relies on the topology, topology updates, and topology status updates received from IP Availability Manager to discover and monitor routing-enabled devices and BGP, EIGRP, IS-IS, or OSPF services.

Network Protocol Manager collectively represents each of the following Network Protocol Management Suite products:

Network Protocol Manager for BGP
Network Protocol Manager for EIGRP
Network Protocol Manager for IS-IS
Network Protocol Manager for OSPF

Fixed Issues

After editing the alarm definition, the legacy job active alarm is not getting cleared.
Prometheus restarts observed.
vROps Contextual Launch button's name in vRops notifications is Launch External Application instead of Launch vROps Application".

Known Issues

Get user federation authentication management APIs are having permission issue in the TKG and Azure deployments.

User federation Auth Management APIs are not applicable for VMware Telco Cloud Service Assurance 2.0.1 release.
Follow the steps to resolve the issue:
1. Login to the Keycloak administration console as administrator user at https://<URL>/auth.
2. Select master realm from the left pane.
3. Navigate to Realm settings > Login.
4. Update the Require SSL parameter to none.
5. Click Save to save the configuration change.
VMware Telco Cloud Service Assurance job Instantiation and Terminate status is not showing correct status in VMware Telco Cloud Automation user interface.

VMware Telco Cloud Service Assurance job Instantiation and Terminate status is shown as success, even though the VMware Telco Cloud Service Assurance deployment is in progress.
To check the VMware Telco Cloud Service Assurance job Instantiation & Terminate status, using the following kubectl command:
```
root [ ~/tcx-deployer/scripts ]# kubectl get tcxproduct
```
Getting an error message during migration of IP, SAM, and ESM.
Following error message appears, during migration of IP, SAM, and ESM, and .conflict files are created for sm_merge and version.pm:
```
----
Merge Process aborted/opt/InCharge11/IP/smarts/local/bin/system/sm_merge: 
line 1: $'\177ELF\002\001\001': command not found/opt/InCharge11/IP/smarts/local/bin/system/sm_merge: 
line 2: $'w\267P\343\316\301\024W\026': command not found
------
```
There is no functionality impact, and the errors can be ignored.
API to get the Kubernetes cluster status is not working in Azure deployment.

API endpoint is tcsa/api/deployment/v1/info

This bug has been observed only in the Azure deployment and having no functionality impact.
Incremental scale fails when VMware Telco Cloud Service Assurance scale is triggered without the Node or VM scale up.

Post deployment if the incremental scale is triggered without scaling up the VM or Node, the incremental scale fails with error: Insufficient CPU capacity.

Post that, increases the VM or Node capacity as per the footprint and re-trigger the incremental scale. Again, the incremental scale fails even though sufficient resource capacity is provided.
Ensure that the Node or VM scale is done as per the destination footprint:
1. Run the command: kubectl delete validatingwebhookconfiguration admin-operator-webhook.
2. Re-trigger the incremental scale operation.
Incremental scale passed and all the apps are scaled up as per the destination footprint specified.
The log_level messages are displaying 'unknown' in Service logs (Kibana logs).

When user navigates to Administration > Service Logs, and clicks on the application service logs, the filter log level displays 'unknown' fields for log_level messages for any service. For example: Apiservice, elasticsearch, and so on.. .
Unable to delete the cloned console of default Summary View.

Note: You are able to perform all required operations using Edit option.
Occasionally, ElasticSearch data service pods are crashing in longevity setup.

Pods like API, Catalog, ElasticSearch, Esdb Proxy, Metic and Event services are crashing, and unable to login to VMware Telco Cloud Service Assurance.
VMware Telco Cloud Service Assurance currently does not support connections to SAM server when Broker is configured in secure mode.

Currently there is no workaround. Broker must configured in non-authenticate mode.

Note: EDAA related operations including the Acknowledge, Ownership, Server Tools, Browse Details > Containment and Browse Details > Domain Manager are not supported when Broker is configured in secure mode.
When the number of hops of connectivity is increased, you may experience performance issues in the topology maps.

There might be performance issues in the rendering of Redundancy Group and SDN connectivity map types in the Map Explorer view. This issue is observed on deployments with a complex topology where the topology maps may stop working when the number of hops of connectivity is increased.
Broker failover is not supported in VMware Telco Cloud Service Assurance.

Primary Broker fails in the Domain manager failover environment.
Currently when a Broker (multi-broker) failover happens, then it requires a manual intervention where you need to log in to VMware Telco Cloud Service Assurance and change the Broker IP address to point to the new Broker IP.

Procedure:
1. Go to https://IPaddress of the Control Plane Node.
2. Navigate to Administration > Configuration > Smarts Integration
3. Delete the existing Smarts Integration Details.
4. Re-add the Smarts Integration Details by pointing it to secondary Broker.
Weekly indexes are not displayed while creating custom reports, only daily and hourly index are shown part of reports.
Procedure for workaround:
1. Select Configurations > Data Sources from the left side menu bar
2. Click Add Data Source.
3. Select Elasticsearch.
4. Enter relevant name based on the metric-type for which the weekly index needs to be created (for example: Week-Network-Interface) and the Elastic http url as http://elasticsearch:9200, refer any other VMware Telco Cloud Service Assurance data sources
5. Enter Index Name based on the metric type for which the weekly index needs to be create ([vsametrics-week-networkinterface-]YYYY.MM) and select Pattern "Monthly"
6. Enter the Time Field Name timestamp and Version 7+.
7. Keep the rest of the fields to default value.
8. Click Save & Test.
Notification count mismatch between SAM and VMware Telco Cloud Service Assurance UI due to non-filtering of notification with Owner field set to SYSTEM. By default in VMware Telco Cloud Service Assurance there are no filters set.
Manually apply the filter to remove notifications with Owner field not containing SYSTEM in VMware Telco Cloud Service Assurance Notification Console window by following below steps:
1. Go to Default Notification Console.
2. Click Customize View.
3. Go to Filters and provide Filter Set Name, for example Filterout SYSTEM Notifications.
4. Filter Section Add Attribute with below condition:
  Property = Owner
  
  Expression = regex
  
  Value = ~(SYSTEM+)
5. Click Update.
Verify the Default Notification Console has only those notifications whose owner not set to SYSTEM. The default notification count must match between SAM and VMware Telco Cloud Service Assurance UI.
The Containment, Browse detail, Notification Acknowledge/Unacknowledge does not work when the primary Tomcat server fails in a HA environment.

In a Failover deployment, when the primary Tomcat fails, the UI operations including the Notification Acknowledgement, Containment, Browse Detail, and Domain Managers fail.
When the primary Tomcat instance fails in a failover environment, then you can manually point the VMware Telco Cloud Service Assurance to a secondary Tomcat instance.

Procedure:
1. Go to https://IPaddress of the Control Plane Node.
2. Navigate to Administration > Configuration > Smarts Integration
3. Delete the existing Smarts Integration Details.
4. Re-add the Smarts Integration Details by editing the EDAA URL and pointing it to the secondary Tomcat Instance.
The SAM server is getting listed in the Domain Manager section instead of Presentation SAM section.

During Smart integration and configuration, INCHARGE SA (SAM server) is getting listed in the Domain Manager section. This problem occurs only when, the SAM server is started in Non-EDAA Mode.

To get listed under Presentation SAM section, start the SAM server in EDAA Mode.
While starting server, the Map error warning message appears for INCHARGE-SA and INCHARGE-OI in respective logs.

No functional impacts.
User needs to mandatorily discover ESX Servers for getting Virtual Machine Down event. Currently the Virtual Machine Down event is not generated if the corresponding ESX Servers are not discovered in IP Server. So, its recommended to discover Virtual Machines to get proper Root cause events.
On RHEL 7.8 version machine when SAM services are started, brcontrol shows IPv6 entry for servers due to which communication between servers is getting impacted.

On RHEL 7.8 version, if you start any domain manager as a service, the domain gets registered to a broker using both v4 and v6 IP address space. Due to this issue domain manager v6 entry will go to DEAD state in brcontrol output and the communication between the servers is failing sometimes due to this issue.

Note: Issue also detected on some machines with RHEL 7.2 and 7.6.

To avoid a domain running in v6 mode, allow only v4, by setting the below flag in runcmd_env.sh file:

SM_IP_VERSIONS=v4

Restart the domain manager, after updating runcmd_env.sh file.
Topology synchronization is taking more than 10 minutes for 25k devices, when latency between SAM and VMware Telco Cloud Service Assurance is more than 5 milliseconds.

When the latency increases topology synchronization time increases.

Ensure that the latency between VMware Telco Cloud Service Assurance (Topology Collector) and SAM Presentation server is less than 5 milliseconds.
Notification processing rate is slower, when the latency between SAM and VMware Telco Cloud Service Assurance is greater than 5 milliseconds.

When the latency between VMware Telco Cloud Service Assurance and SAM Presentation server increases, notification processing rate goes down.

Ensure that the latency between VMware Telco Cloud Service Assurance (Notification Collector) and SAM Presentation server is less than 5 milliseconds.
KPI feature will be supported in upcoming release of VMware Telco Cloud Service Assurance.
Post Incremental scale from one footprint to another footprint, VMware Telco Cloud Service Assurance user interface shows older base footprint instead of destination upgraded footprint.

Post Incremental scale from 50k to 100k, About option in the VMware Telco Cloud Service Assurance user interface shows Footprint : 50k instead of 100k.

There is no workaround and it has no functional impact, since the actual incremental scale happens properly only in the About UI footprint information is not updated.
In Service Logs, the dark theme is not working. When user selects the Service Logs in the user interface, the default theme is not changing.

Kibana GUI dark theme support is not available in the opendistro version of Kibana.