This section describes about some common troubleshooting steps, which user can follow in case either discovery or monitoring feature for Cisco ACI SDN is not working as expected.

Trobleshooting Cisco ACI Discovery and Monitoring feature:

Domain Manager logs are located at $INSTALL_DIR/smarts/local/logs

Enable Discovery and Monitoring logs in Smart Assurance IP Availability and Performance Manager:

Run dmctl <domain name> -b <broker> put ICF_TopologyManager::ICF-TopologyManager::DebugEnabled TRUE

Run dmctl -s <domain name> -b <broker> invoke ICF_TopologyManager::ICF-TopologyManager insertParameter DebugReconfigure TRUE

Enable Debug Logs for DCF Collectors

DCF Collector (Discovery & Monitoring)

To enable logs for a particular Collector used for discover/monitoring in DCF, first get the corresponding collector ID.

To get Collector Id:

Run dmctl -s <domain name> -b <broker> invoke ICF_PersistentDataSet::ACICollectorIDs get.

The above command returns two entries for each ACI. ACI Controller is associated to two Collectors, one is for discovery and another one is for monitoring, output of the command indicates the same.

Steps to enable logs:

  1. To increase the log level, edit /opt/DCF/Collecting/Collector-Manager/<collector Id>/conf/logging.properties
  2. Change the default INFO level to one of – CONFIG, FINE, FINER & FINEST (lowest value).
  3. The new log level takes effect in the next discovery/monitoring cycle.

Enable file connector logs for a collector:

Edit /opt/DCF/Collecting/Collector-Manager/<collector Id>/conf/collecting.xml. Change enabled=true for the file collector.

<connector enabled="true" name="File" type="File-Connector" config="Collector-Manager/<collector Id>/conf/file-connector.xml"/>

Following table describes some common reasons for Cisco ACI fabric discovery failure.
Comment Description Troubleshooting
Kafka consumer has failed! The Kafka cluster is not configured correctly. Refer to Troubleshooting Methodologies -1.
Cannot find Matching kafka Broker Settings. The Kafka cluster is configured in Cisco ACI Access setting does not have a corresponding Kafka Bus Access setting. Refer to Troubleshooting Methodologies -2.
Cannot find Matching DCC Settings. The Data Collector is configured in Cisco ACI Access setting does not have a corresponding Data Collector Access setting. Refer to Troubleshooting Methodologies -2.
Timeout - Messages not received from DCF. The Data Collector does not push the Messages consistently. Refer to Troubleshooting Methodologies -3.
Not able to connect to Cisco ACI Controller. HTTP Connection with Cisco ACI Controller is not properly established. Refer to Troubleshooting Methodologies -4.
Controller Failed to create block for [apicIp/Name]. The Collection block in Data Collector cannot be created. Refer to Troubleshooting Methodologies -5.
Controller Failed to getConfig for Collector instance [apicIp/Name]. The configurations of a collection block in Data Collector cannot be fetched.

Refer to Troubleshooting Methodologies -5.
Controller Failed to set configurations for [apicIp/Name]. The configurations of a collection block in Data Collector cannot be set. Refer to Troubleshooting Methodologies -5.

Troubleshooting Methodologies -1

If the error message "Discovery Error [kafka consumer has failed!]" appears during discovery:

  • Check kafka credentials and port details.
  • Check if the kafka process is running on the server.
  • Check if kafka topic name is configured.

Troubleshooting Methodologies -2

If the error message "Cannot find Matching kafka Broker Settings" appears during discovery:

  • Check Device Access Setting for the Cisco ACI kafka Broker detail matches with the Device Access Setting for kafka Broker detail.

If the error message "Cannot find Matching DCC Settings" appears during discovery:

  • Check Device Access Setting for Cisco ACI Data Collector address detail matches with Device Access Setting for Data Collector Access Setting Hostname detail.

Troubleshooting Methodologies -3

In case discovery fails with the error message “Timeout - Messages not received from DCF”:

  • Check if the DCF collector corresponding to this discovery has started.
    • A quick way is to check the timestamp of the discovery failure message and then comparing it with last update in the logs of the corresponding collector.
  • In case the collector does not start, contact to Smart Assurance Support.

Troubleshooting Methodologies -4

In case discovery fails with the error message “Not able to connect to Cisco ACI Controller”:

  • Check if all the information provided in Cisco ACI Device Access settings is proper. Generally, incorrect Cisco ACI port or username/password is the reason.

Troubleshooting Methodologies -5

In case the discovery fails with error message “Controller Failed”:

  • Retry discovery as it mostly causes due to a huge number of requests sent to DCF, in case the issue persists contact Smart Assurance support.