Working with Streaming databus

The databus is a framework for sharing the data stored in VMware Aria Operations for Networks with other applications using APIs. With the databus, you can share high volumes of data with low latency in near real-time.

VMware Aria Operations for Networks databus provides the following benefits:

• Provides a way to programmatically consume high-volume data in near real-time.
• Works on a change-based consumption model which gives databus a distinct advantage over traditional API calls, especially in the context of high-volume data like flows and metrics.

Use Cases

With VMware Aria Operations for Networks databus, you can use the data for various purposes. Following are a couple of examples of how you can use the databus:

Use case	Description	Steps
Retain flows for a longer duration	In general, flow retention in VMware Aria Operations for Networks is one month. With the databus, you can retain flows for a longer duration.	Set up a subscriber URL to receive HTTPS/HTTP data. Ensure you are aware of the required security information. See Know about the Security and the SSL Certificate. Register a subscriber to subscribe flows message group. Data starts streaming. For details on message formatting, see Message Format Received from the Databus. To learn more about the databus capabilities, see Understanding the Databus Terminologies and Capabilities.
Sending alerts to various destinations	By defaults, you can send alerts to SNMP / SMTP destinations. With databus, you can send alerts to various other destinations, such as Slack, ServiceNow, PagerDuty, and so on. A few examples of alerts that you can get using databus are system alerts for password expiry, VMware vCenter server disconnected, and so on.	Set up a subscriber URL to receive HTTPS/HTTP data. Ensure you are aware of the required security information. See Know about the Security and the SSL Certificate. Register a subscriber to subscribe problem message group. Data starts streaming. For details on the message formatting, see Message Format Received from the Databus. To learn more about the databus capabilities, see Understanding the Databus Terminologies and Capabilities.
Streaming metrics out of VMware Aria Operations for Networks	Polling APIs for metrics is a multi-step process that includes getting entity IDs, periodically calling the APIs, and filter out duplicate information. The databus makes this entire process very efficient; you subscribe to the metrics group or sub-group of your choice, and the metrics directly gets stream out to the endpoint as the metrics get processed.	Set up a subscriber URL to receive HTTPS/HTTP data. Ensure you are aware of the required security information. See Know about the Security and the SSL Certificate. Register a subscriber to subscribe metrics message group. Data starts streaming. For details on message formatting, see Message Format Received from the Databus. To learn more about the databus capabilities, see Understanding the Databus Terminologies and Capabilities.

Supported Message Group

The databus streaming is a better alternative to APIs if you are looking to poll information periodically. And to receive information from databus, you must subscribes to message group(s). A message group is a specific set of data that you can subscribe from the databus. Each message group can have up to 10 subscribers, except for Flows and Metrics message groups which can have only one subscriber per message group. Also, the Metrics message group can have only one subscriber per message group.

VMware Aria Operations for Networks supports the following message groups:

Message Group	Sub-message Group	Message Details
Metrics	vms-metrics	Metric, interval, time stamp, unit, entity type, and points.
	hosts-metrics	Metric, interval, time stamp, unit, entity type, and points.
	flows-metrics	Metric, interval, time stamp, unit, entity type, and points.
	nics-metrics	Metric, interval, time stamp, unit, entity type, and points.
	switchports-metrics	Metric, interval, time stamp, unit, entity type, and points.
	nsxt-metrics	Metric, interval, time stamp, unit, entity type, and points.
Applications	N/A	Entity ID, entity type, name, created by, creation time, last modified by, last modified time, source, tiers, and discovery info.
Flows	N/A	Source VM, destination VM, source IP, destination IP, port, source folders, destination folders, protocol, source IP sets, destination IP sets, source security groups, destination security groups, traffic type, source security tags, destination security tags, source VM tags, destination VM tags, within host, firewall action, and flow tag.
Problems	N/A	Entity ID, entity type, name, anchor entities, message, event tags, admin state, archived, event time epoch ms, event type, and severity.

Table 1. Configuration-related Message Group

Message Group	Message Details
VMs	Entity ID, name, entity type, IP addresses, default gateway, default gateways,VNICs, source firewall rules, destination firewall rules, VM UUID, manager UUID, cluster, resource pool, host, vendor ID, vCenter manager, folders, datastore, data center, source inversion rules, destination inversion rules, CPU count, memory, OS full name, and HCX info.
Hosts	Entity ID, name, entity type, VMKNICs, cluster, vCenter manager, VM count, datastore, service tag, vendor ID, maintenance mode, connection state, CPU count, memory, and manager UUID.
NICs	Entity Id, name, entity type, VLAN, and VM.
Switchports	Entity ID, name, entity type, operational status, administrative status, sub type, device, and manager.

Tip: A sub-message group provides more fine-grained information. All subscribers to a particular message group receive the same information.

The data published over databus is categorized into different message groups. As a user you can subscribe to one or more message groups, and data related to those message groups starts streaming to HTTP or HTTPS endpoints.

Use the following steps to configure the databus:

1. Know about the Security and the SSL Certificate

While using the databus, you must be aware of the following security information:

Environment	Details
On-prem	HTTP/HTTPS as a transport protocol - When using databus for on-prem environment, you can get data from the databus over HTTP / HTTPS endpoint. To ensure that the data is encrypted during transport and verification of subscriber authenticity, use HTTPS. Firewall Rules - To ensure only VMware Aria Operations for Networks can send data to the subscriber, you need to enable the firewall rules. All the VMware Aria Operations for Networks platform nodes in a cluster send the data to the subscriber, so you must configure the firewall rules to allow traffic from all the platform nodes. SSL Certificates - The SSL certificates used by the subscribers must be signed by well-known CAs. If the SSL certificates are self-signed or signed by internal CAs of the organization and not trusted by Java, you must import the CA certificate to the Java trust store of VMware Aria Operations for Networks. Contact VMware support to get help on how to import the Java trust store. Note: The CA certificate that is imported to the trust store will be trusted for all outgoing SSL traffic from VMware Aria Operations for Networks platforms.
SaaS	HTTPS as a transport protocol - When using databus for a SaaS environment, you can get data from the databus only over HTTPS endpoints. CSP token for message authentication - Each databus message from VMware Aria Operations for Networks (SaaS) contains an authorization header with a JWT token issued by CSP. You as a subscriber can verify the authenticity of the token by validating the token with the public key from CSP (https://console.cloud.vmware.com/csp/gateway/am/api/auth/token-public-key). You can verify the authenticity of the token to ensure that the client ID in the token is the client ID of theVMware Aria Operations for Networks databus application registered on CSP. You get the client ID details from VMware Aria Operations for Networks Databus developers. SSL Certificates - The SSL certificates used by the subscribers must be signed by well-known CAs. By default the SSL certificates that VMware Aria Operations for Networks (SaaS) issues are signed by JDK. VMware Aria Operations for Networks (SaaS) doesn’t support the import of custom CA certificate.

2. Register, Update, or Delete a Subscriber

• To register a subscriber, you must do an API call using the POST method. For details, see the VMware Aria Operations for Networks API Reference.

  If the registration is successful, the message response contains a subscriber ID. You can use the subscriber ID to delete a subscription or update a subscription. After successful subscriber registration, it might take up to five minutes for the subscriber to start receiving messages.

• To update a subscriber, you must do an API call using the PUT method. For details, see the VMware Aria Operations for Networks API Reference.
  Note: You can only update the Subscriber URL. You cannot update the message group.
• To delete a subscriber, you must do an API call using the DELETE method. For details, see VMware Aria Operations for Networks API Reference.

  After successful subscriber deletion, it may take up to five minutes for the data publishing to stop.

3. Message Format Received from the Databus

The messages received as part of various message groups are a list of multiple messages and have a common high-level structure. Each message in the list is a cloud-event formatted object. To learn more about cloud events, see cloudevents.

All the message has two sets of specific fields:

• Common structure - which is common across all the message group. For example:

  "id": "14cfd27c-76d8-43e3-9fe9-f53bf69c0c5e", 
      "type": "Application", 
      "specversion": "1.0", 
      "source": "CAESBkFCQ0RFRg==", 
      "messageGroup": "applications", 
      "status": "CREATE", 

• Message group-specific structure - which is unique for every message group. For example, the following is the data object within an application message which is unique to the application message group:

      "data": { 
        "entity_id": "1:561:0", 
        "entity_type": "APPLICATION", 
        "name": "app1", 
        "created_by": "[email protected]", 
        "creation_time": 1637303575039, 
        "last_modified_by": "[email protected]", 
        "last_modified_time": 1637303575039, 
        "source": "MANUAL", 
        "tiers": [ 
          { 
            "entity_id": "1:562:1", 
            "entity_type": "TIER", 
            "name": "app1_t1" 
         } 
        ] 
      } 

The structure of the objects within databus messages is in line with the schema of public API responses, except for the metric where a subset is in use. The message structure is backward compatible across different versions of VMware Aria Operations for Networks. In the later versions, new fields may get added. For the detailed structure of the messages for every message group, see VMware Aria Operations for Networks API Guide.

The following table lists the names of the different types of messages for every message group:

Message groups	Type or Schema of the message
Alerts (problem events only)	Entity Message List. For details see VMware Aria Operations for Networks API Reference
Applications	Application Message List. For details see VMware Aria Operations for Networks API Reference
flows	Flow Message. For details see VMware Aria Operations for Networks API Reference
metrics	The structure of the metrics data is different from the public APIs to make the messages compact. [ { "id": "string", "type": "string", "specversion": "string", "source": "string", "message_group": "metrics", "data": { "metric": "string", "interval": 0, "timestamp": 0, "unit": "string", "entity_type": "string", "points": [ { "entity_id": "string", "value": 0 } ] } } ]