Data published by Kafka Connector in M&R are in raw value format as below. This data is converted to JSON as shown in the input section for Kafka Mapper to process.
| Input | Kafka Mapper | Output |
|---|---|---|
{
"value": -1.4765625,
"timestamp": 1666940399000,
"action": 114,
"group": "",
"properties": {
"collhost": "haas-bng-010-121.vmware.com",
"collinst": "Generic-SNMP",
"ip": "10.106.126.64",
"source": "Watch4NetSnmpCollector-1",
"w4ncert": "1.0",
"pollgrp": "APG-GENERATED-300.CISCO-ENHANCED-MEMORY-POOL.CISCO-IPSLA-HTTP-LATEST.CISCO-IPSLA-HTTP-STATSTABLE.CISCO-IPSLA-INVENTORY.CISCO-IPSLA-JITTER-LATEST.CISCO-IPSLA-JITTER-STATSTABLE.CISCO-IPSLA-STATSTABLE.CISCO-MEMORY.CISCO-MEMORY-OLD.CISCO-PROCESS-POST12.2(3.5).GENERIC-INTERFACES.GENERIC-SYSUPTIME",
"devdesc": "Cisco IOS Software. 3800 Software (C3845-ADVENTERPRISEK9-M). Version 12.4(15)T14. RELEASE SOFTWARE (fc2)",
"@polling": "2",
"devtype": "Router",
"altsrc": "Generic-SNMP",
"vendor": "Cisco Systems",
"contact": "",
"name": "bufferSmMiss",
"location": "",
"model": "3845",
"device": "10.106.126.64",
"datagrp": "CISCO-MEMORY-OLD"
}
} |
{
"metricType": "$.properties.source",
"instance": "$.properties.device",
"properties.entityName": "$.properties.device",
"properties.entityType": "$.properties.devtype",
"processedTimestamp": "$.timestamp",
"type": "$.properties.datagrp",
"properties.dataSource": "$.properties.source",
"properties.deviceName": "$.properties.device",
"metrics.$['properties']['name']": "$.value",
"timestamp": "$.timestamp",
"properties.deviceType": "$.properties.devtype",
"tags": "$.properties"
} |
{
"instance": "10.106.126.64",
"metricType": "Watch4NetSnmpCollector-1",
"timestamp": 1666940399000,
"processedTimestamp": 1666940399000,
"type": "CISCO-MEMORY-OLD",
"metrics": {
"bufferSmMiss": -1.4765625
},
"properties": {
"deviceType": "Router",
"entityName": "10.106.126.64",
"entityType": "Router",
"dataSource": "Watch4NetSnmpCollector-1",
"deviceName": "10.106.126.64"
},
"tags": {
"collhost": "haas-bng-010-121.vmware.com",
"collinst": "Generic-SNMP",
"ip": "10.106.126.64",
"source": "Watch4NetSnmpCollector-1",
"w4ncert": "1.0",
"pollgrp": "APG-GENERATED-300.CISCO-ENHANCED-MEMORY-POOL.CISCO-IPSLA-HTTP-LATEST.CISCO-IPSLA-HTTP-STATSTABLE.CISCO-IPSLA-INVENTORY.CISCO-IPSLA-JITTER-LATEST.CISCO-IPSLA-JITTER-STATSTABLE.CISCO-IPSLA-STATSTABLE.CISCO-MEMORY.CISCO-MEMORY-OLD.CISCO-PROCESS-POST12.2(3.5).GENERIC-INTERFACES.GENERIC-SYSUPTIME",
"devdesc": "Cisco IOS Software. 3800 Software (C3845-ADVENTERPRISEK9-M). Version 12.4(15)T14. RELEASE SOFTWARE (fc2)",
"@polling": "2",
"devtype": "Router",
"altsrc": "Generic-SNMP",
"vendor": "Cisco Systems",
"dcName": "_core",
"contact": "",
"name": "bufferSmMiss",
"location": "",
"model": "3845",
"device": "10.106.126.64",
"datagrp": "CISCO-MEMORY-OLD"
}
} |
The explanation for all the lines in the Kafka Mapper is covered in the previous example section. This example provides more information on the generic definition for ingesting M&R data as all the properties are copied to the tags section. The M&R data contains only one metric per message and is assigned through the following line:
"metrics.$['properties']['name']": "$.value"
Here the name is assigned by looking up the value name within the properties bucket from input, which evaluates to bufferSmMiss, and value from value which evaluates to -1.4765625.
From the Kafka Mapper definition you can see that input must contain the properties - device, devtype, datagrp, and source. Any of these property values if missing or empty, then it leads to the metric/message to be dropped. The Kafka Mapper definition must be updated in accordance with the input, so that the mandatory properties always contains valid value.
