Some of the common troubleshooting steps are described here, which user must follow if discovery or monitoring is not working as expected.

Troubleshooting Methodologies -1

DCF Collector (Discovery & Monitoring)
  • Getting Collector Id for a VCO:
    • Run “dmctl -s <server name> invoke ICF_PersistentDataSet::VCOTopologyCollectorInstanceIds get”
      • Returns two blocks for each VCO. Each block has a collector Id and a tag to indicate if it is discovery/monitoring collector .
  • Logs and log locations:
    • Collector logs are located at /opt/DCF/Collecting/Collector-Manager/<collector Id>/logs/
  • Logging and troubleshooting:
    • Edit /opt/DCF/Collecting/Collector-Manager/<collector Id>/conf/logging.properties to increase log level
    • Change the default INFO level to one of – CONFIG, FINE, FINER & FINEST (lowest value)
    • The new log level takes effect in the next discovery
  • The kafka messages can also be configured to be written to file:
    • Edit /opt/DCF/Collecting/Collector-Manager/<collector Id>/conf/collecting.xml. Change enabled=true for file collector
      • <connector enabled="true" name="File" type="File-Connector" config="Collector-Manager/<collector Id>/conf/file-connector.xml" />

Troubleshooting Methodologies - 2

  • Check if the /opt/DCF/Collecting/Stream-Collector/<Collector Id>/conf/requests/context-vco.xml has proper values (username, password, port & protocol) for VeloCloud access.
  • For specific discovery/monitoring issues use REST API to get the data directly from VCO to confirm VCO is returning proper value. The base URI for the REST API is https://<VCO IP>/portal/rest/

DCF Collector (Discovery & Monitoring)

Table 1. REST API used for Discovery and monitoring for different user types
Topology/Status REST API User Type
Operator MSP Enterprise
Login
Login /login/operatorLogin Yes NA NA
Login /login/enterpriseLogin Not Used Yes Yes
Discovery
Enterprises /network/getNetworkEnterprises Yes NA NA
Enterprises enterpriseProxy/getEnterpriseProxyEnterprises NA Yes NA
Enterprises enterprise/getEnterprise Not used Not used Yes
Edge

/network/getNetworkEnterprises

{ “with” : [“edges”]}

 Yes NA NA
Edge /enterprise/getEnterpriseEdges Not used Yes Yes
Edge interfaces (WAN and LAN) /edge/getEdgeConfigurationStack Yes Yes Yes
Edge links /enterprise/getEnterpriseEdges Yes Yes Yes
Gateway /network/getNetworkGatewayPools Yes Yes NA
GatewayPool /network/getNetworkGatewayPools Yes Yes NA
Non VeloCloud Site and the Tunnels

/enterprise/getEnterpriseDataCenters

 Yes Yes Yes
Edge and Gateway relationship

/network/getNetworkGatewayPools

(connectedEdgeList field in the response)

 Yes NA NA
Edge and Gateway relationship gateway/getGatewayEdgeAssignments Not used Yes NA
Edge and Gateway relationship /edge/getEdgeConfigurationStack Not used Not used Yes
Edge Clusters

/edge/getEdgeConfigurationStack

/enterprise/getEnterpriseEdgeServices

 Yes Yes Yes
Edge Hub and Spoke Relationship

/edge/getEdgeConfigurationStack

/enterprise/getEnterpriseEdgeServices

Yes Yes Yes
Edge and Gateway relationship edge/getEdge Yes Yes Yes
Monitoring
Edge Status enterprise/getEnterpriseEdges Yes Yes Yes
Edge link Status enterprise/getEnterpriseEdges Yes Yes Yes
Edge link Performance data monitoring/getAggregateEdgeLinkMetrics Yes Yes NA
Gateway Status network/getNetworkGatewayPools Yes Yes NA
Non VC site Tunnel Status enterprise/getEnterpriseDataCenters Yes Yes Yes
Edge performance data metrics/getEdgeStatusMetrics Yes Yes Yes

Troubleshooting Methodologies -3

Smarts
  • Logs and log locations:
    • Server logs are located at $INSTALL_DIR/smarts/local/logs
  • Logging and troubleshooting:
    • Enable Discovery logging:
      • Run dmctl –s <server> put ESM_Manager::ESM-Manager::EnableVeloCloudDiscoveryDebug TRUE
      • Run dmctl –s <server> put ESM_Manager::ESM-Manager::IsDebug TRUE
      • Run dmctl –s <server> put ESM_Manager::ESM-Manager::LogLevel DEBUG
        Note: This logs all the Kafka messages for all VCO.
  • Enable Monitoring Logging:
    • Run dmctl –s <server> put ESM_Manager::ESM-Manager:: EnableVeloCloudMonitoringDebug TRUE
    • Run dmctl –s <server> put ESM_Manager::ESM-Manager::LogLevel DEBUG
      Note: This logs all the Kafka messages for all VCO.
  • Make sure VeloCloud feature is enabled (default enabled):
    • Run dmctl –s <server> get ESM_Manager::ESM-Manager::IsVC
  • Smarts notifies DiscoveryFailure & MonitoringFailure event for the Orchestrator with the reason for failure set in the event details.
  • Log analysis:
    • Discovery Kafka message:
      • Sample:
        {"groupName":"group","discoveryID":null,"jobID":"9999","type":"Tenant","timestamp":1554472776,"value":0.0,"action":"r","properties":{"VelocloudOrchestrator":"10.106.124.199","Tenant-name":"testTenant_1","source":"velocloud-sdwan4dfefd0c-c8b6-41ac-a83f-96776c9001c4","type":"Tenant","Tenant-Id":"1","EnterpriseLogicalId":"c8cc3bc7-df6e-407c-ac30-352b4b3de93a"},"metrics":{"VCO-Req":{"properties":{"unit":"code","name":"VCO-Req"},"value":8080.0}},"relations":[{"type":"Orchestrator","element":"VCO-a.b.c.d","relationName":"SupervisedBy"}],"initialized":true,"forceRefresh":true,"name":"testTenant_1"}
Creates Tenant::testTenant_1
Creates Orchestrator::VCO-a.b.c.d
Establish relationship Tenant::testTenant_1 SupervisedBy Orchestrator::VCO-a.b.c.d
    • Monitoring Kafka message:
      • Sample: 
        {"groupName":"group","discoveryID":null,"jobID":"9999","type":"VGateway","timestamp":1554753805,"value":0.0,"action":"\u0000","properties":{"instanceName":"GatewayName","attributes":"Status,IsActivated","source":"velocloud-sdwan048048cb-701e-440d-85e9-e46110965181","GatewayName":"VGateway-OneCloud-GW1-Bangalore","type":"VGateway","attributevalues":"OK,TRUE"},"metrics":{"VCO-Req":{"properties":{"unit":"code","name":"VCO-Req"},"value":0.0}},"relations":[],"initialized":true,"forceRefresh":false,"name":"VGateway-OneCloud-GW1-Bangalore"}
VGateway::Vgateway-OneCloud-GW1-Bangalore  is the instance name
Status is OK
IsActivated is TRUE
  1. Smarts Domain Managers

    1. Getting Collector Id for a VCO:

      Run "dmctl -s <server name> invoke ICF_PersistentDataSet::VCOTopologyCollectorInstanceIds get"

      Returns two blocks for each VCO. Each block has a collector Id and a tag to indicate if it is discovery/monitoring collector

      <connector enabled="true" name="File" type="File-Connector" config="Collector-Manager/<collector Id>/conf/file-connector.xml" />

    2. DCF Logs and log locations:

      Collector logs are located at /opt/DCF/Collecting/Collector-Manager/<collector Id>/logs/

    3. DCF Logging and troubleshooting:

      Edit /opt/DCF/Collecting/Collector-Manager/<collector Id>/conf/logging.properties to increase log level

      Change the default INFO level to one of - CONFIG, FINE, FINER & FINEST (lowest value)

      The new log level takes effect in the next discovery.

    4. The kafka messages can also be configured to be written to file.

      Edit /opt/DCF/Collecting/Collector-Manager/<collector Id>/conf/collecting.xml.

      Change enabled=true for file collector.

    5. Check if the /opt/DCF/Collecting/Stream-Collector/<Collector Id>/velo/conf/requests/context-vco.xml has proper values (username, password, port & protocol) for VeloCloud access.
    6. For specific discovery/monitoring issues use REST API to get the data directly from VCO to confirm VCO is returning proper value. The base URI for the REST API is https://<VCO IP>/portal/rest/. A REST client (e.g Postman, Insomnia & SoapUI) can be used to issue these REST queries.
      Note: The various VCO REST APIs used for discovery and monitoring for different user types are documented at the table REST API used for Discovery and monitoring for different user types.
    7. Smarts Logging:
      1. Server logs are located at $INSTALL_DIR/smarts/local/logs
        1. Enable Discovery logging:
          1. Run dmctl -s <server> put ESM_Manager::ESM-Manager::EnableVeloCloudDiscoveryDebug TRUE
          2. Run dmctl -s <server> put ESM_Manager::ESM-Manager::IsDebug TRUE
          3. Run dmctl -s <server> put ESM_Manager::ESM-Manager::LogLevel DEBUG
        2. Enable Monitoring Logging:
          1. Run dmctl -s <server> put ESM_Manager::ESM-Manager:: EnableVeloCloudMonitoringDebug TRUE
          2. Run dmctl -s <server> put ESM_Manager::ESM-Manager::LogLevel DEBUG
            Note: This logs all the Kafka messages for all VCO.
    8. Smarts Log analysis.
      1. Discovery Kafka message:

        -Sample: {"groupName":"group","discoveryID":null,"jobID":"9999","type":"Tenant","timestamp":1554472776,"value":0.0,"action":"r","properties":{"VelocloudOrchestrator":"10.106.124.199","Tenant-name":"testTenant_1","source":"velocloud-sdwan4dfefd0c-c8b6-41ac-a83f-96776c9001c4","type":"Tenant","Tenant-Id":"1","EnterpriseLogicalId":"c8cc3bc7-df6e-407c-ac30-352b4b3de93a"},"metrics":{"VCO-Req":{"properties":{"unit":"code","name":"VCO-Req"},"value":8080.0}},"relations":[{"type":"Orchestrator","element":"VCO-a.b.c.d","relationName":"SupervisedBy"}],"initialized":true,"forceRefresh":true,"name":"testTenant_1"}

        Creates Tenant::testTenant_1

        Creates Orchestrator::VCO-a.b.c.d

        Establish relationship Tenant::testTenant_1 SupervisedBy Orchestrator::VCO-a.b.c.d

      2. Monitoring Kafka message:

        -Sample: {"groupName":"group","discoveryID":null,"jobID":"9999","type":"VGateway","timestamp":1554753805,"value":0.0,"action":"\u0000","properties":{"instanceName":"GatewayName","attributes":"Status,IsActivated","source":"velocloud-sdwan048048cb-701e-440d-85e9-e46110965181","GatewayName":"VGateway-OneCloud-GW1-Bangalore","type":"VGateway","attributevalues":"OK,TRUE"},"metrics":{"VCO-Req":{"properties":{"unit":"code","name":"VCO-Req"},"value":0.0}},"relations":[],"initialized":true,"forceRefresh":false,"name":"VGateway-OneCloud-GW1-Bangalore"}

        VGateway::Vgateway-OneCloud-GW1-Bangalore is the instance name

        Status is OK

        IsActivated is TRUE

    9. Feature is enabled (default enabled)

      Run dmctl -s <server> get ESM_Manager::ESM-Manager::IsVC

    10. Smarts ESM server raises a DiscoveryFailure/MonitoringFailure event on the Orchestrator instance when there is an internal error or communication failure between the components. The event has the reason for the failure set in the event details.