Troubleshooting Steps

Some of the common troubleshooting steps are given below, which user must follow if discovery or monitoring is not working.

Debug flags for VIO Feature

Logging and troubleshooting:
- Debug flags for OpenStack feature are present under $SM_HOME/conf/esm/ESM.import file.
- Modify the file using sm_edit utility.
- Update EnableOpenStackDiscoveryDebug = TRUE (Default value : FALSE) for enabling the discovery related logging.
- Update EnableOpenStackMonitoringDebug = TRUE (Default value : FALSE) for enabling the monitoring related logging.
- Change LogLevel = DEBUG (Default value : ERR) to increase logging levels if required.
- No restart is required.
Logs and log locations:
- ESM server logs can be found under $SM_HOME/local/log directory.

VIO errors and logging for troubleshooting

Scenario 1- Errors observed due to Controller/OpenStack ManagementServer Down:

[April 20, 2019 4:44:48 PM GMT+05:30 +436ms] t@3682531072 Discovery #14
NV_MESSAGE-*-NV_GENERIC-MSG ERR : [Thread-2365 OpenStackDiscoveryProbe]:Exception in OpenStack Discovery
 [April 20, 2019 4:44:48 PM GMT+05:30 +436ms] t@446584576 platform
MAIN_MSG-*-STDFD_ERR-stderr: java.lang.Exception: Could not retrieve credentials for OpenStack : 10.234.61.189
        at com.emc.asd.openstack.OpenStackDiscoveryProbe.invoke(OpenStackDiscoveryProbe.java:321)
        at com.smarts.java_probe.ProbeRunner.invoke(ProbeRunner.java:69)
 
[April 20, 2019 4:44:48 PM GMT+05:30 +763ms] t@3682531072 Discovery #14
TM-W-EPROBE-Probe driver failed when probing element 'SNMPAgent-10.234.61.189'. Discovery error is 'java.lang.Exception: Could not retrieve credentials for OpenStack : 10.234.61.189'

Scenario 2- Errors observed due to NSX Manager Down

[April 20, 2019 4:21:32 PM GMT+05:30 +943ms] t@446584576 platform
MAIN_MSG-*-STDFD_ERR-stderr: java.lang.Exception: Could not retrieve credentials for NSX-T : 10.234.24.70
        at com.emc.asd.nsx.NSXTDiscoveryProbe.invoke(NSXTDiscoveryProbe.java:166  )
        at com.smarts.java_probe.ProbeRunner.invoke(ProbeRunner.java:69)
 [April 20, 2019 4:21:33 PM GMT+05:30 +180ms] t@3686733568 Discovery #9
TM-W-EPROBE-Probe driver failed when probing element 'SNMPAgent-10.234.24.70'. Discovery error is 'java.lang.Exception: Could not retrieve credentials for NSX-T                      : 10.234.24.70’

Scenario 3- Errors observed due to VIO vCenter Down

ASL-W-ERROR_RULE_SOURCE-While executing rule set '/opt/INCHARGE10/ESM/smarts/rules/nv/openstack-post.asl'
 ASL-ERROR_ACTION-While executing action at:
 ASL-CALL_STACK_RULE-   RuleName: VMWARE_VIO_MAP_CLUSTER_MANAGER_TO_VM, Line: 675
 ASL-CALL_STACK_RULE-   RuleName: VMWARE_VIO_MAP_SERVICS_CLUSTER_MANAGER_TO_VM, Line: 662
 ASL-CALL_STACK_RULE-   RuleName: START, Line: 54
 ASL-ERROR_RESULT_ASSIGNMENT-While assigning to the variable 'mgmtVM' in rule 'VMWARE_VIO_MAP_CLUSTER_MANAGER_TO_VM'
 ASL-UNDEFINED_INTERMEDIATE-Function 'GET_MGMT_SRV_VM_OBJ' failed to return a value

MAIN_MSG-*-STDFD_ERR-stderr: java.net.UnknownHostException: compute01: Name or service not known
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
        at java.net.InetAddress.getAllByName(InetAddress.java:1192)
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)
        at java.net.InetAddress.getByName(InetAddress.java:1076)
        at com.emc.asd.common.Utils.getFQDNFromName(Utils.java:116)
        at com.emc.asd.factory.dmt.DmtObjectFactory.getHostNameByName(DmtObjectFactory.java:1280)
        at com.emc.asd.openstack.keystone.Tenant.discoverComponents(Tenant.java:205)
        at com.emc.asd.openstack.OpenStack.discoverOpenstack(OpenStack.java:110)
        at com.emc.asd.openstack.OpenStackDiscoveryProbe.invoke(OpenStackDiscoveryProbe.java:339)
        at com.smarts.java_probe.ProbeRunner.invoke(ProbeRunner.java:69)