The release notes cover the following topics:
What's New in this Release
With Smart Assurance 10.1.2 release, the following features, enhancements, and changes are introduced:
[VM-ER-8]: MBIM server logging has been enhanced to capture the device details with Device name and duration of maintenance windows. Henceforth, the details of the scheduled maintenance device can be retrieved later for any debugging or auditing purposes.
[ER-1063]: The processing of the Dying Gasp trap for SMARTS has been enhanced, so that it can process the trap from Cisco Device running on the IOS release 12.2(60)EZ10 and later.
[VM-ER-3]: SAM has been enhanced to allow operators to add any device component into the maintenance windows if the device component is present in the SAM topology.
[VM-ER-54]: The VeloCloud VEdge instance in the ESM server has a new attribute “activationTime” that captures the activation time for the edge.
- [VM-ER-59] [VM-ER-102]: The VeloCloud VEdge instance in the ESM server has been enhanced with new additional attributes:
- Model: Represents the model number of the Edge. The “modelNumber” property that is returned by the getEdge REST API is used to populate this field.
- Type: Represents the device family of the Edge. The “deviceFamily” property that is returned by the getEdge REST API is used to populate this field.
- Location: It is of the form <country>,<city>,<street>. Represents the location of the Edge.
- haSerialNumber: Represents the HA serial number for this edge. The haSerialNumber property that is returned by the getEdge REST API is used to populate this field.
- [VM-ER-81]: The VeloCloud VEdge instance in the ESM server has the below two attributes populated with the proper value:
- id: Represents the Edge id as maintained by the VCO.
- TenantId: Represents the tenant id for the Edge.
- [VM-ER-63]: If the user changes the configuration at a Tenant/Edge level, then to get the new changes, the incremental discovery can be used. The incremental discovery can be triggered for an existing Tenant/Edge. The rediscovery can only be triggered using a remote API client or dmctl command. Refer to, VeloCloud SD-WAN Monitoring section in the Deployment Scenario guide for more information.
- [VM-ER-95]: In SAM, the Element Name attribute of the SAM notification is the same as the device name whenever there is a notification on AggregatePort. This behavior needs to be explicitly enabled by setting the “ElementNameMapToDevice” attribute of the ICS_NotificationFactory::ICS-NotificationFactory and reattaching the underlying domain or restarting the SAM server.
For security vulnerabilities addressed in Smarts, see Smart Security Update for Multiple Vulnerabilities.
Third-party support 10.1.2 changes
- Java is upgraded to OpenJDK Runtime Environment Zulu 11.0.7.
- ASAM java upgraded to 1.8.0_252.
- SAM console java upgraded to 1.8.0_252.
- Tomcat upgraded to 9.0.37 (for SAM 64 bit and 32 bit).
- Jackson-databind upgraded to 2.11 (for IP only).
For OSL (Open Source License) file (open_source_license_VMware_Smart_Assurance_GA.txt), navigate to <BASEDIR>/smarts/setup/osl.
The VMware Smart Assurance SAM, IP, ESM, MPLS, and NPM Managers Support Matrix available from the VMware Support website provides the latest platform and interoperability information. For detailed information about platform support and interoperability, refer support matrix for your release.
Note: In SMARTS 10.1.2 release, some of the document(s) do not require modification. The older version document(s) are released as it is.
- SMARTA-329 / SR-19008628706
Issue with InputPacketBroadcastRate calculation query.
Resolution:The InputPacketBroadcastRate formula has been modified, to calculate rate, on the basis of ifHCInBroadcastPkts packets.
- SMARTA-642 / SR-20093959701
ESM with VeloCloud feature enabled consumes 100% CPU, and the domain becomes unusable once this happens.
Resolution:The FD leak in the ESM monitoring subsystem has been fixed to resolve this issue.
- SMARTA-700 / SR-20110190003
ESM with VeloCloud feature enabled reports “java.lang.OutOfMemoryError: Java heap space”.
Resolution: ESM server is changed to consume the kafka message at a configurable rate. This is achieved by two configurable parameters specified in esm-param.conf. The parameter and its description are given below. When these parameters are not set the fix is not enabled and ESM server consumes all the outstanding messages in the kafka bus.
The below parameter is for specifying the maximum size of the kafka adapter internal queue size at which the consumer is stopped:
# to minimize the memory footprint of the ESM server.
# The below parameter is for specifying the minimum size of the kafka adapter internal queue size at which the consumer is restarted.
- SMARTA-556 / SR- 19080029711
VeloCloud discovery for the MSP user takes longer time.
Resolution: The redundant loops has been fixed, which brought down the discovery time to an acceptable range.
- SMARTA-716 / SR- 20093544701
Changing the VeloCloud Edge/Enterprise name in the VCO breaks the monitoring for that Edge.
Resolution: Instead of using the Edge name and Enterprise name in naming the VEdge instance the Edge id and Enterprise id
are used. The DisplayName property of the VEdge will still have useful reference to the enterprise name.
- SMARTA-684 / SR- 20103429002
Partitions are having devices from other customers, despite using IP tagging.
Resolution: During the partition creation, SMARTS invoke getNeighbors() method, and this method has been altered not to return
IPNetwork tagged neighbors. Hence, code has been modified to ensure partition does contain members from non-tagged
- SMARTA-648 / SR- 20095306401
Smarts IP slow startup and dull performance.
Resolution: As part of the fix, a new flag (DisableIPNetworkGuessPattern) has been introduced in tpmgr-param.conf. Customer can update this flag with the respective IPNetwork (in the current case its 10.0.0.0) for which SMARTS will not guess the Netmask for the same and will be excluded being part of IPNetwork.
The default value of the flag is : 10.107.119.0|FDCC:0:0:BD:0:0:0:0. The User can set any value to it for the first time via the tpmgr-param.conf.
Post which; if the flag value needs to be changed, the below steps can be followed:
Step 1: To reset the Flag, use the below command:
./dmctl -s <Incharge_domain_manager_Name> -b <broker>:<port> invoke ICF_TopologyManager::ICFTopologyManager insertParameter DisableIPNetworkGuessPattern " "
Step 2: Execute the below command to check if the value is set properly.
./dmctl -s <Incharge_domain_manager_Name> -b <broker>:<port> invoke ICF_TopologyManager::ICF-TopologyManager findParameter DisableIPNetworkGuessPattern
Step 3: A new value can be set again or override by executing the below command:
./dmctl -s <Incharge_domain_manager_Name> -b <broker>:<port> invoke ICF_TopologyManager::ICF-TopologyManager insertParameter DisableIPNetworkGuessPattern "10.0.0.0"
- SMARTA-549 / SR- 1907832321
The failover is not able to activate ESM domain due to errors in actions logs.
Resolution: The obsoleted properties of the ESM server are removed from the relevant failover scripts. The changes are available in both ESM and SAM.
- SMARTA-501 / SR- 19068962310
The IP Domain manager running for more than a day for a single device discovery.
- SMARTA-693 / SR-20103165702
The SM_Config utility exports the details to default /local instead of SM_WRITEABLE.
Resolution: When Initiating a remote action, the environment variable “SM_WRITEABLE” is set via SSH.
- SMARTA-780 / SR 20115216704
The filter pattern is not working in System Name or IP Pattern tab in Application Signatures.
Resolution: The code has been modified to enhance the filter pattern to neutralize the systems for the application creation code block.
To incorporate the LoadBalancer class available into NPM and to effectively discover and monitor LoadBalancers on NPM domain manager.
Resolution: Code has been modified to add the LoadBalancer class to the NPM domain manager, to import classes to create protocol endpoints and monitor them.
- SMARTA-630 / SR- 19089925812
Multiple Fiberlink Line Failure alerts have been reported for same port.
Resolution: Upstream/Downstream RCA correlation has been enhanced to deduce a single root cause for LOS ( Loss Of Signal) failures at multiple WDM (OTS-OMS-OCH) layers.
- SMARTA-766 / SR- 20119403804
Multiple Fiberlink notifications generated for IsSignalDegradeDetected event.
Resolution: Upstream/Downstream RCA correlation has been enhanced to deduce a single root cause for SD (Signal Degrade) and LOS failures at multiple WDM (OTS-OMS-OCH) layers .
The known issues are grouped as follows.
- Known Issues in SAM
- Known Issues in ESM
- Known Issues in IP
- Known Issues in MPLS
- Known Issues in NPM
- Known Issues in Smart Assurance UI
- Common Issues to all products
- Known Issues in Smart Assurance Management Pack
- Known Issues in ASAM
- Known Issues in ACM
Installing SAM console on the Linux platform is not creating applications "VMware Smarts Global Console" icon for quick launch. User needs to go to install directory and type sm_gui command only, the icon option is not available. This appears when previous version 9.5.1. or 9.6 console is already installed.
The map icons for VEdge and VGateway are missing for “SDWAN VCO Connectivity” in the 10.0 SAM console.
Workaround: User has to either use 10.1 SAM console or edit map icons for VEdge and VGateway classes.
Default DCF Password is not working for DCF SAM-OI Communication. Notification will not be pulled from vROPS to SAM-OI.
Workaround: Create a new user and password in DCF at
admin1:admin123at users file, and restart the Webservice-Gateway in DCF.
All Services are not installed post upgrade for SAM and SAM-Console.
Refer Services for the Service Assurance Manager section in VMware Smart Assurance Installation Guide to install services manually.
Incase of Console mode installation and upgrade, below irrelevant message is displayed in command prompt which does not have any functional impact:
cwd: C:\Users\Administrator\AppData\Local\Temp\2\I1559131575\Windows cmd: "C:\Users\Administrator\AppData\Local\Temp\2\I1559131575\Windows\resource\jre\bin\java.exe" --add-opens java.base/jdk.internal.loader=ALL-UNNAMED -Xms16777216 -Xmx50331648 -classpath "C:\Users\Administrator\AppData\Local\Temp\2\I1559131575\InstallerData\IAClasses.zip;C:\Users\Administrator\AppData\Local\Temp\2\I1559131575\InstallerData\Execute.zip;C:\Users\Administrator\AppData\Local\Temp\2\I1559131575\Windows\InstallerData\Execute.zip;C:\Users\Administrator\AppData\Local\Temp\2\I1559131575\InstallerData\Resource1.zip;C:\Users\Administrator\AppData\Local\Temp\2\I1559131575\Windows\InstallerData\Resource1.zip;C:\Users\Administrator\AppData\Local\Temp\2\I1559131575\InstallerData;C:\Users\Administrator\AppData\Local\Temp\2\I1559131575\Windows\InstallerData;" com.zerog.lax.LAX "C:/Users/Administrator/AppData/Local/Temp/2/I1559131575/Windows/setup-CONSOLE-10_0_0_0-win.lax" "C:/Users/Administrator/AppData/Local/Temp/2/lax827D.tmp" -i console
The following map error warning message appears for INCHARGE-SA and INCHARGE-OI in respective logs, while starting the server.
WARNING: register package 'map_error' version 468026d2 failed (0x276f4e00) [May 11, 2020 2:50:50 AM EDT +804ms] t@375936768 InCharge Framework FDXAM-*-DXA_TOPOCONC-TopologySync is running in CONCURENT mode with minimum delay=120 seconds. WARNING: register package 'Map_mm' version a1894c1c failed (0x276f4e00) [May 11, 2020 2:50:52 AM EDT +072ms] t@375936768 InCharge Framework
There is no functional impact due to this issue.
The following ASL error message appears for INCHARGE-SA and INCHARGE-SA-PRES server, while starting the servers.
[May 11, 2020 2:50:50 AM EDT +710ms] t@881858304 InCharge Framework ASL-W-ERROR_RULE_SOURCE-While executing rule set '/opt/INCHARGE10120/SAM/smarts/rules/import.asl' ASL-ERROR_ACTION-While executing action at: ASL-CALL_STACK_RULE- RuleName: INSERT, Line: 181 ASL-CALL_STACK_RULE- RuleName: OBJECT_PROPERTY, Line: 177 ASL-CALL_STACK_RULE- RuleName: OBJECT_PROPERTIES, Line: 171 ASL-CALL_STACK_RULE- RuleName: OBJECT_VALUE, Line: 138 ASL-ERROR_INSERT-While attempting to insert into property 'MapViews' of object 'Map_Containment::Map-Containment' MR-CALL_OBJ_PROP-Object: Map-Containment property: Map-Containment; in file "/work/redcurrent/DMT-10.1.2.0/3/smarts/repos/mr/obj.c" at line 2102 MR-KEY_DUPLICATION-violation of uniqueness requirement for a key [May 11, 2020 2:50:52 AM EDT +095ms] t@881858304 InCharge Framework
There is no functional impact due to this issue.
The Rabbit MQ service is not started in SAM, due to the following error:
[root@vmwbgc052 bin]# ./sm_rabbitmq --ignoreme ERROR: epmd error for host vmwbgc052: address (cannot connect to host/port) [root@vmwbgc052 bin]#
Workaround: If the RabitMQ process fails to start, you need to add the following mapping in the /etc/hosts file:
127.0.0.1 <FQDN of the host>
In some hosts, the pre-installed services such as smarts-tomcat and smarts-elasticsearch may fail to start.
Workaround: Restart the ic-serviced daemon process by running the following commands:
/etc/init.d/ic-serviced stop /etc/init.d/ic-serviced stop
Note: If the ic-business-dashboard service in SAM Console does not start, then restart the ic-serviced daemon process by running the earlier commands.
After SAM 10.1.2.0 upgrade, epmd service-related errors appear in the SAM upgrade log:
/opt/InCharge/SAM/smarts/toolbox/OTP/erts-9.2/bin/epmd (Text file busy) at com.zerog.util.expanders.ExpandToDiskZip.ag(Unknown Source) at com.zerog.util.expanders.ExpandToDiskPMZ.ab(Unknown Source)
SMARTS NCM adapter fails to connect to VSA domain managers when there are a higher number of domains registered with the broker. As broker.getDomainManagers() API returning all domain managers attached to the broker and due to the character variable size restriction at the NCM adapter code, there are failures in processing the domain manager list. The acceptable characters are limited to 256.
Workaround: Connect to the broker where there is less number of domain managers registered.
Clearwater SNMP collector does not support SNMP V3, Clearwater supports SNMP v2c and UDP protocol.
Whenever user performs discoverAll in ESM, it triggers the discovery for all the hosts in the ESM Server and as well all the underlying INCHARGE-AM-PM servers. The host discovery triggers an Orchestrator discovery. Also the INCHARGE-AM-PM discovery triggers another Orchestrator discovery. So if there is a Orchestrator discovery which is InProgress, then if there another request for the same orchestrator discovery, then this request will be blocked and the device will be moved to the pending list.
There is no functionality impact as one of the discovery is completed successfully.
Badge Reports are having negative value in vIMS performance reports in vROps.
Manage/Unmanage option is not available for NetworkConnection Class instances.
To Unmanage the NetworkConnection, user needs to Unmanage one of the connected interface of that NetworkConnection.
When all the EdgeNodes of NSX-T goes down which has Tier 0 Router due to ESX(Host)Down or VM going down then Compute VMs host Unresponsive will not be explained by the Host(ESX) / VM Down RCA alert.
Currently there is no workaround available however RCA alert will be shown only impact to Host unresponsive will be not available for NSX-T Compute VMs. In the impact list EdgeNode Down, LogicalRouter(T0) Down will be shown.
User needs to mandatorily discover ESX Servers for getting Virtual Machine Down event. Currently the Virtual Machine Down event is not generated if the corresponding ESX Servers are not discovered in IP Server. So, its recommended to discover Virtual Machines to get proper Root cause events.
Manage / Unmanage option is not available for network connection class instances, However the Instrumented by relationship is available with NetworkConnection_Fault class.
While deploying clearwater collector through DCC, user have an option to provide Community string and port through clientConnect.conf, but it is limited to only one. The user can update the community string in agents-groups.xml file of DCF collector, however string does not get encrypted.
When Kafka is not reachable, monitoring failure event appears in SAM after reconfiguration of ESM Server.
- If Kafka server goes down, a notification for VCD monitoring failure appears in SAM once ESM Server is reconfigured.
- When Kafka is brought back, the failure notification is cleared from SAM once ESM server is reconfigured.
Exception appears when selecting "Tiered application map" by right clicking on any map icon.
Mapping of ElementName attribute to the device for notification on AggregatePort is missing. ElementName appears as Aggregate Port instance name instead of Device name.
Workaround: Detach the INCHARGE-AM-PM from SA and underly it again. Once you re-attach INCHARGE-SA-PRES to the domain manager, you may lose all the active notifications.
The following error message appears in the ESM log while discovering VCD and its dependant components like NSX-T, vCenter, etc..
NV_MESSAGE-*-NV_GENERIC-MSG ERR : [Thread-8 DmtObjectFactory]:insert(TransportNodeInterface::10.107.146.14/10.107.146.69::ConnectedVia += Tunnel:): SVIF-E-EREMOTE-Remote error occurred. See exception chain for detail.; in file "/work/redcurrent/DMT-10.1.2.0/3/smarts/skclient/linux_rhAS50-x86-64/optimize/SmLocalInterfaceHandler.c" at line 3623' should be empty.
In the VeloCloud use case, the 'haSerialNumber' attribute is not getting updated for the incremental discovery of VEdge in VSA10.1.2.
Objects deleted in IP are not getting deleted in vROps even after object deletion time interval.
To delete the objects manually:
- Go to Administration > Configuration > Inventory Explorer.
- Select the objects to be deleted and click delete button (Bulk deletion is also supported).
A set of reports are created by default when user installs the Smart Assurance Adapter pack. These reports are retained in the dashboard even after uninstalling the management pack. Ideally, these reports must be deleted as part of the management pack uninstallation process.
User can manually delete the reports from the dashboard.
When a new credentials added without any username and password while configuring an instance of the adapter and edit it later with correct username and password, the password field wont be updated resulting the connection failure.
Edit the credentials again and update the password filed again and save it.
Special characters are present after migration from Windows to Linux.
Patch files are getting migrated, when performing migration on ESM 10.0.0.1 and in IP 10.0.0.2 to 10.1.0.0 respectively.
This is applicable only if migration is triggered on the below two products and patch versions:
- ESM - 10.0.0.1 (ESM 10.0 patch 1)
- IP - 10.0.0.2 (IP 10.0 Patch 2)
When performing migration only on the above two versions, you need to follow the below procedure:
Step 1: Collect the backup from the old installation using sm_migrate utility.
-Run 'sm_perl sm_migrate.pl --old=<dir> --archive=<tar/zip> --sitemod=<local_directories>' from the old installation.
Step 2: Perform manual migration on the customized files.
-Copy the archive file from ‘step1’ to new host.
-Run 'sm_perl sm_migrate.pl --archive=<tar/zip> --new=<dir>' from the new installation.
-Once the sm_migrate utility backs up all the customizations under smarts directory (Eg: /opt/InCharge/IP/smarts/.migrate.bkp.10.0.0.2) and prompts for merging the customizations with below options, the user has to select “N” to not merge.
a. Press 'n' to skip FileMergeUtilty.
b. Press any other key to start FileMergeUtilty...[y]n
Step 3: You need to manually copy customizations from backup directory (Eg: /opt/InCharge/IP/smarts/.migrate.bkp.10.0.0.2) to respective local directory and rename the files to remove ‘.local’ extension.
Null pointer exceptions are observed in IP server log file after Cisco ACI Discovery.
ASL error present in INCHARGE-MPLS-TOPOLOGY after starting server from fresh install.
Post migration from Windows to Linux, EIGRP server is not coming up.
While starting the EIGRP server use option --ignore-restore-errors.
EIGRP classes are not populated in NPM EIGRP, due to ASL error.
Null Value filtering is not possible form log view Filter.
When we scroll the scroll bar in notification window, the column name must not be hide, it must be fridge. Working fine in Chrome, but in IE, the column name is hiding.
In case number of notification is 0 or 1 in Smart Assurance UI, the edit field in filter pop up is not visible.
The Smart Assurance UI fails to display live notifications above 10,000.
Expanded notification view does not provide distinguish between parent and child(impacted) notifications.
During Smarts Metric collector installation, if user wants to collect only Metrics or Topology data separately, and selects option 2 or 3 as displayed below:
 AM/PM Topology & Metrics
 AM/PM Metrics
 AM/PM Topology
The installation fails with errors.
During the collector installation user needs to select the default Option 1 to collect both metric and topology data. Once the installation is complete:
- To collect only the Metric data, open <DCF Install Location>/Collecting/Collector-Manager/<Collector Name>/conf/collecting.xml
- Change the value enabled="false" in the following line <collector enabled="true" name="smarts-INCHARGE-AM-PM-Install-0-topo" next="Kafka File" type="Smarts-Collector" config="Smarts-Collector/topology/conf/smarts-INCHARGE-AM-PM-Install-0-topo.xml" />
- To collect only the Topology data, open <DCF Install Location>/Collecting/Collector-Manager/<Collector Name>/conf/collecting.xml
- Change the value enabled="false" in the following line <collector enabled="true" name="smarts-INCHARGE-AM-PM-Install-0-metrics" next="Kafka File" type="Smarts-Collector" config="Smarts-Collector/topology/conf/smarts-INCHARGE-AM-PM-Install-0-metrics.xml" />
- Restart the collector process.
During product upgrade, patch folder is not highlighted.
On RHEL 7.8 version, if you start any domain manager as a service, the domain gets registered to a broker using both v4 and v6 IP address space. Due to this issue domain manager v6 entry will go to DEAD state in brcontrol output and the communication between the servers is failing sometimes due to this issue.
Note: Issue also detected on some machines with RHEL 7.2 and 7.6
Workaround: To avoid a domain running in v6 mode, allow only v4, by setting the below flag in runcmd_env.sh file:
Restart the domain manager, after updating runcmd_env.sh file.
"-help" command installer displays following invalid options:
- -i [gui]
- -jvmxms <size>
- -jvmxmx <size>
- -add <feature_name_1> [<feature_name_2 ...]
- -remove <feature_name_1> [<feature_name_2 ...]
Smart Assurance products in silent mode are getting installed in root folder, when user disable or comment the user install directory (<Products>SUITE.installLocation=/opt/InCharge) in silent response file.
Exception appears in Collector logs when Orchestrator closes connection during discovery.
Few Log messages are tagged as ERROR in the collector log incorrectly.
The vCenter Management Pack does not have any API to detect floating IP. If a VNF (sprout or bono) is configured with floating IP and private IP, only the private IP is used to establish the relationship between VNFs (P-CSCF or I/S-CSCF) and VirtualMachines.
ASAM server is not running in service way, but the user able to run via the Server way. When the user installs any application with java 11 enabled on the VM, ic-serviced (sm_serviced) is also installed, which is compiled with java 11. And, when the user installs another application with java 8 installed (in this case ASAM), then the application crashes during the starting of service (ic-asam). This is because all the executables installed under ASAM bin directory are compiled with java 8 which in turn not compatible with ic-serviced, hence crashing only in the service way starting.
Workaround: User needs to stop ic-serviced and start it again from the ASAM bin directory and then again try to start the server in service way. Also, the ASAM must be installed on a fresh VM without any other application installed along with it (except SAM CONSOLE Linux which is also on java 8).
When ACM is upgraded to 10.1.2, ASL error appears while performing full discovery (DiscoverALL) and after restarting the INCHARGE-AM, INCHARGE-OI, and ACM servers. Following error message appears in ACM log:
ASL-W-ERROR_RULE_SOURCE-While executing rule set '/opt/InCharge1012ACM/ACM/smarts/rules/app-sig/standard-probe.asl' ASL-ERROR_ACTION-While executing action at: ASL-CALL_STACK_RULE- RuleName: CREATE_TOPOLOGY, Line: 224 ASL-CALL_STACK_RULE- RuleName: START, Line: 73 ASL-ERROR_INVOKE-While attempting to invoke operation 'makeSoftwareServiceOnHost' of object 'Application_SourceObjectFactory::Application-SourceObjectFactory-AS-IANA-smarts-broker_hostname=10.62.72.138-STANDARD' APPF-NULL_OBJECT-Null Object 'DXA_TopologySource::Application-SourceObjectFactory-AS-IANA-smarts-broker_hostname=10.62.72.138-STANDARD'