Infrastructure Metrics Analysis on AWS

A set of metrics that show the system and the container includes the Replica and Client node infrastructure resources used in the VMware Blockchain network.

System Metrics

The system metrics are collected from the overall system status of each VM in the blockchain. These metrics are displayed in the Wavefront dashboard for monitoring.

System Metrics	Description
Number of vCPU	Displays the number of vCPUs used per VM.
Total memory allocated	Displays the memory allocation for each VM.
Percentage memory available	Displays the percentage of available memory per VM. In Tanzu Observability by Wavefront, the X-axis represents the metrics time, and Y-axis denotes the percentage available. You can identify each VM in the graph by its color. Point to the graph to view the detailed memory use information of the VM in a pop-up window. Scroll through the graph legend to view the VM information. You can configure alerts to be triggered if the memory use percentage drops to, for example, 20% in Tanzu Observability by Wavefront.
Percentage memory used	Displays the percentage of memory used per VM. In Tanzu Observability by Wavefront, the vertical axis indicates the metrics time, and the horizontal axis denotes the percentage used. Each VM in the graph can be identified by its color. Hover over the graph to view the detailed memory usage information of the VM in a pop-up window. You can also view the same information in the legend. Scroll through the graph legend to view the VM information. In Tanzu Observability by Wavefront, you can configure alerts to be triggered if the memory use percentage exceeds, for example, 90%. You can find the VM IP address in the graph legend and open the VM to investigate the spike in memory usage.
Total disk space	Displays the disk space allocated to the primary hard disk for each VM.
Disk usage percentage	Displays the percentage of disk space used per VM per directory. In Tanzu Observability by Wavefront, the graph shows the disk usage percentage for each color-coded VM. The graph legend also shows the disk partition, disk path, and read and write disk type.
Disk-io read time	Displays the disk I/O read speed with which the data transfer occurs between the hard disk drive and RAM. In Tanzu Observability by Wavefront, you can configure alerts if the read time reaches a certain threshold.
Disk-io write time	Displays the disk I/O write speed with which the data transfer occurs between the hard disk drive and RAM. In Tanzu Observability by Wavefront, you can configure alerts if the write time reaches a certain threshold.
Disk-io read bytes	Displays the disk I/O bytes read to the disk partition.
Disk-io write bytes	Displays the disk I/O bytes written to the disk partition.

Container Metrics

Metrics are collected from the container infrastructure to show resource use and displayed in the Wavefront dashboard for monitoring. The dashboard displays the container status, usage, and health metrics.

Each set of VMs in a blockchain has the same blockchain ID. The metrics are listed based on blockchain ID, VM type, a Replica or Client node, VM IP address, and VM value. All containers and VMs have a unique color designated when applicable for simplified identification. Each container has a name assigned to it for easy detection.

Container Metrics	Description
Number of running containers per node	Lists the number of containers actively running in each VM.
Number of container images	Lists the number of available container images in each VM. In Tanzu Observability by Wavefront, the CPU usage percentage section looks for a VM with 0% usage, which indicates that the VM is either paused or stopped. Use the IP address to access the VM and investigate why the problem occurred.
Number of paused containers per node	Lists the number of paused containers per VM.
Number of stopped containers per node	Lists the number of stopped containers per VM.
vCPU Count	Lists the number of vCPU available per VM to the container.
Memory usage percentage	Displays the VM memory used in percentage per container. In Tanzu Observability by Wavefront, the vertical axis indicates the metrics time, and the horizontal axis denotes the percentage used. Each container and VM in the line graph can be identified by its color. Hover over the graph to view the detailed memory usage information of the VM in a pop-up window. Scroll through the graph legend to view the VM information.
CPU usage percentage	Displays the VMCPU used in percentage per container. The usage maximum is the number of container cores multiplied by 100. In Tanzu Observability by Wavefront, the X-axis represents the metrics time, and Y-axis denotes the percentage used. Each VM in the graph can be identified by its color. Hover over the graph to view the detailed CPU usage information of the container in a pop-up window. Scroll through the graph legend to view the VM information. In Tanzu Observability by Wavefront, you can configure alerts to be triggered if the CPU percentage exceeds, for example, 100%. Spikes in CPU usage might be due to a large workload being processed or an attempt to execute an infinite loop process.

Health Metrics	Description
vmware.blockchain.agent.daml.health.gauge	Indicates the Daml component health within a blockchain. You can view the Daml execution engine health status for Replica nodes. For Client nodes, you can view the Daml Ledger API health status. The Daml index database health directly affects the Daml Ledger API health status. For example, if the Daml index database shows an unhealthy state, the Daml Ledger API shows the same status. Metric tags can be used to differentiate between the Replica and the Client node types. The following values indicate the Daml component health status. 1 indicates a healthy state. 0 indicates an unhealthy state. -1 indicates that an exception occurred. The most common reason for the exception is that the Daml component is not running.
vmware.blockchain.agent.concord.health.gauge	Indicates the Concord container health of every Replica node within a blockchain. The Client node does not show Concord container health. The following values indicate the Concord container health status. 1 indicates a healthy state. 0 indicates an unhealthy state. -1 indicates that an exception occurred. The most common reason for the exception is that the Concord container is not running.

Network Metrics

Metrics are collected to show the network use of all the VMware Blockchain node VMs. The network metrics are displayed in the Tanzu Observability by Wavefront dashboard for monitoring. The dashboard displays the network status and usage metrics.

Network Metrics	Description
Bytes sent rate	Displays the rate at which bytes are sent from the network interface. Tracking the rate of bytes sent for a prolonged period can help diagnose performance degradation.
Bytes received rate	Displays the rate at which bytes are received from the network interface. Tracking the rate of bytes received for a prolonged period can help diagnose performance degradation.
Packets sent rate	Displays the number of packets sent from the network interface.
Packets received rate	Displays the number of packets received from the network interface.
Receive error rate	Displays the number of received errors detected from the network interface.
Transmit error rate	Displays the number of transmit errors detected from the network interface. Transmit error rate increase might indicate transmission delays.
Received packet drop rate	Displays the number of received packets dropped from the network interface. Received packet drop rate surge might indicate large processing requests or a network congestion problem.
Transmitted packet drop rate	Displays the number of transmit errors detected from the network interface. Significant packet loss during the transmission process can affect performance.