Configuring Healthwatch

This topic describes how to manually configure and deploy the Healthwatch™ for VMware Tanzu^® (Healthwatch) tile.

To install, configure, and deploy Healthwatch through an automated pipeline, see Installing, Configuring, and Deploying a Tile Through an Automated Pipeline.

Overview of Configuring and Deploying Healthwatch

The Healthwatch tile monitors metrics across one or more Ops Manager foundations by scraping metrics from Healthwatch Exporter tiles installed on each foundation. For more information about the architecture of the Healthwatch tile, see Healthwatch Tile in Healthwatch Architecture.

After installing Healthwatch, you configure Healthwatch component VMs, including the configuration files associated with them, through the tile UI. You can also configure errands and system logging, as well as scale VM instances up or down and configure load balancers for multiple VM instances.

To configure and deploy the Healthwatch tile:

Notes: To quickly deploy the Healthwatch tile to ensure that it deploys successfully before you fully configure it, you only need to configure the Assign AZ and Networks pane.

Navigate to the Healthwatch tile in the Ops Manager Installation Dashboard. For more information, see Navigate to the Healthwatch Tile below.
Assign jobs to your Availability Zones (AZs) and networks. For more information, see Assign AZs and Networks below.
Configure the Prometheus pane. For more information, see Configure Prometheus below.
(Optional) Configure the Alertmanager pane. For more information, see (Optional) Configure Alertmanager below.
(Optional) Configure the Grafana pane. For more information, see (Optional) Configure Grafana below.
(Optional) Configure the Grafana Authentication pane. For more information, see (Optional) Configure Grafana Authentication below.
(Optional) Configure the Grafana Dashboards pane. For more information, see (Optional) Configure Grafana Dashboards below.
(Optional) Configure the Canary URLs pane. For more information, see (Optional) Configure Canary URLs below.
(Optional) Configure the Remote Write pane. For more information, see (Optional) Configure Remote Write below.
(Optional) Configure the TKGI Cluster Discovery pane. For more information, see (Optional) Configure TKGI Cluster Discovery below.
(Optional) Configure the Errands pane. For more information, see (Optional) Configure Errands below.
(Optional) Configure the Syslog pane. For more information, see (Optional) Configure Syslog below.
(Optional) Configure the Resource Config pane. For more information, see (Optional) Configure Resources below.
(Optional) Configure for OpenTelemetry. For more information, see (Optional) Configure for OpenTelemetry below.
Deploy Healthwatch. For more information, see Deploy Healthwatch below.

After you have configured and deployed the Healthwatch tile, you can configure and deploy the Healthwatch Exporter tiles for the Ops Manager foundations you want to monitor. For more information, see Next Steps below.

Navigate to the Healthwatch Tile

To navigate to the Healthwatch tile:

Navigate to the Ops Manager Installation Dashboard.
Click the Healthwatch tile.

Assign AZs and Networks

In the Assign AZ and Networks pane, you assign jobs to your AZs and networks.

To configure the Assign AZ and Networks pane:

Select Assign AZs and Networks.
Under Place singleton jobs in, select the first AZ. Ops Manager runs any job with a single instance in this AZ.
Under Balance other jobs in, select one or more other AZs. Ops Manager balances instances of jobs with more than one instance across the AZs that you specify.
From the Network dropdown, select the runtime network that you created when configuring the BOSH Director tile.
Click Save.

Configure Prometheus

In the Prometheus pane, you configure the Prometheus instance in the Healthwatch tile to scrape metrics from the Healthwatch Exporter tiles installed on each Ops Manager foundation, as well as any external services or databases from which you want to collect metrics.

The values that you configure in the Prometheus pane also configure their corresponding properties in the Prometheus configuration file. For more information, see Overview of Configuration Files in Healthwatch in Configuration File Reference Guide, Prometheus in Configuration File Reference Guide, and the Prometheus documentation.

To configure the Prometheus pane:

Select Prometheus.
For Scrape interval, specify the frequency at which you want the Prometheus instance to scrape Prometheus exposition endpoints for metrics. The Prometheus instance scrapes all Prometheus exposition endpoints at once through a global scrape. You can enter a value string that specifies ns, us, µs, ms, s, m, or h. To scrape detailed metrics without consuming too much storage, VMware recommends using the default value of 15s, or 15 seconds.
(Optional) To configure the Prometheus instance to scrape metrics from the Healthwatch Exporter tiles installed on other Ops Manager foundations or from external services or databases, configure additional scrape jobs under Additional scrape jobs. You can configure scrape jobs for any app or service that exposes metrics using a Prometheus exposition format, such as Concourse CI. For more information about Prometheus exposition formats, see the Prometheus documentation.
Note: The Prometheus instance automatically discovers and scrapes Healthwatch Exporter tiles that are installed on the same Ops Manager foundation as the Healthwatch tile. You do not need to configure scrape jobs for these Healthwatch Exporter tiles. You only need to configure scrape jobs for Healthwatch Exporter tiles that are installed on other Ops Manager foundations.
1. Click Add.
2. For Scrape job configuration parameters, provide the configuration YAML for the scrape job you want to configure. This job can use any of the properties defined by Prometheus except those in the tls_config section. Do not prefix the configuration YAML with a dash. For example:
```
job_name: foundation-1
metrics_path: /metrics
scheme: https
static_configs:
 - targets:
   - "1.2.3.4:9090"
   - "5.6.7.8:9090"
```
  For more information, see the Prometheus documentation.
  Important: For the job_name property, do not use the following job names:
  - Healthwatch-view-pas-exporter
  - Healthwatch-view-pks-exporter
  - tsdb
  - grafana
  - pks-master-kube-scheduler
  - pks-master-kube-controller-manager
3. (Optional) To allow the Prometheus instance to communicate with the server for your external service or database over TLS:
  1. For Certificate and private key for TLS, provide a certificate and private key for the Prometheus instance to use for TLS connections to the server for your external service or database.
  2. For CA certificate for TLS, provide a certificate for the certificate authority (CA) that the server for your external service or database uses to verify TLS certificates.
  3. For Target server name, enter the name of the server for your external service or database as it appears on the server’s TLS certificate.
  4. If the certificate you provided in Certificate and private key for TLS is signed by a self-signed CA certificate or a certificate that is signed by a self-signed CA certificate, activate the Skip TLS certificate verification checkbox. When this checkbox is activated, the Prometheus instance does not verify the identity of the server for your external service or database. This checkbox is deactivated by default.
4. For Chunk size to calculate Diego_AvailableFreeChunksDisk SVM, enter in MB the size that you want to specify for free chunks of disk. The default value is 6144. Healthwatch uses this free chunk size to calculate the available free disk chunks super value metric (SVM), which it then uses to calculate the Diego_AvailableFreeChunksDisk metric. If you configure Healthwatch Exporter for TAS for VMs to deploy the SVM Forwarder VM, the SVM Forwarder VM sends theDiego_AvailableFreeChunksDisk metric back into the Loggregator Firehose so third-party nozzles can send it to external destinations, such as a remote server or external aggregation service. For more information about SVMs, see SVM Forwarder VM - Platform Metrics and SVM Forwarder VM - Healthwatch Component Metrics in Healthwatch Metrics. For more information about deploying the SVM Forwarder VM, see (Optional) Configure Resources in Configuring Healthwatch Exporter for TAS for VMs.
  Note: If you are using the OpenTelemetry Collector this step does not apply.
5. For Chunk size to calculate Diego_AvailableFreeChunksMemory SVM, enter in MB the size that you want to specify for free chunks of memory. The default value is 4096. Healthwatch uses this free chunk size to calculate the available free memory chunks SVM, which it then uses to calculate the Diego_AvailableFreeChunksMemory metric. If you configure Healthwatch Exporter for TAS for VMs to deploy the SVM Forwarder VM, the SVM Forwarder VM sends the Diego_AvailableFreeChunksMemory metric back into the Loggregator Firehose so third-party nozzles can send it to external destinations, such as a remote server or external aggregation service. For more information about SVMs, see SVM Forwarder VM - Platform Metrics and SVM Forwarder VM - Healthwatch Component Metrics in Healthwatch Metrics. For more information about deploying the SVM Forwarder VM, see (Optional)Configure Resources in Configuring Healthwatch Exporter for TAS for VMs.
  Note: If you are using the OpenTelemetry Collector this step does not apply.
(Optional) For Static IP addresses for Prometheus VMs, enter a comma-separated list of valid static IP addresses that you want to reserve for the Prometheus instance. You must enter a separate IP address for each VM in the Prometheus instance. These IP addresses must not be within the reserved IP ranges you configured in the BOSH Director tile. To find the IP addresses of the Prometheus VMs:
1. Select the Status tab.
2. In the TSDB row, record the IP addresses of each Prometheus VM from the IPs column.
  Note: The Prometheus instance includes two VMs by default. For more information about viewing or scaling your VMs, see Healthwatch Components and Resource Requirements.
Click Save.

(Optional) Configure Alertmanager

In the Alertmanager pane, you configure alerting for Healthwatch. To configure alerting for Healthwatch, you configure the alerting rules that Alertmanager follows and the alert receivers to which Alertmanager sends alerts.

To configure the Alertmanager pane, see Configuring Alerting.

(Optional) Configure Grafana

In the Grafana pane, you configure the route for the Grafana UI. You can also configure email alerts and HTTP and HTTPS proxy request settings for the Grafana instance.

The values that you configure in the Grafana pane also configure their corresponding properties in the Grafana configuration file. For more information, see Overview of Configuration Files in Healthwatch in Configuration File Reference Guide, Grafana in Configuration File Reference Guide, and the Grafana documentation.

To configure the Grafana pane:

Select Grafana.
Under Grafana UI route, configure the route used to access the Grafana UI by selecting one of the following options:
- Automatically configure in TAS for VMs: If you are installing Healthwatch on an Ops Manager foundation with TAS for VMs installed, Healthwatch automatically configures a route for the Grafana UI in TAS for VMs. VMware recommends selecting this option when available. You access the Grafana UI by navigating to https://grafana.sys.DOMAIN in a browser window, where DOMAIN is the system domain you configured in the Domains pane of the TAS for VMs tile. For more information, see the TAS for VMs documentation.
- Manually configure: Reveals the configuration fields described in the following steps, where you manually configure the URL and TLS settings for the Grafana UI. To manually configure the URL and TLS settings for the Grafana UI:
  1. For Grafana root URL, enter the URL used to access the Grafana UI. Configuring this field allows a generic OAuth provider or UAA to redirect users to the Grafana UI. Alertmanager also uses this URL to generate links to the Grafana UI in alert messages.
    Note: Healthwatch does not automatically assign a default root URL to the Grafana UI. You must manually configure a root URL for the Grafana UI in the Grafana root URL field.
    After you deploy the Healthwatch tile for the first time, you must configure a DNS entry for the Grafana instance in the console for your IaaS using this root URL and the IP address of either the Grafana VMs or the load balancer associated with the Grafana instance. The Grafana instance listens on either port 443 or 80, depending on whether you provide a TLS certificate in the Certificate and private key for HTTPS fields below. For more information about configuring DNS entries for the Grafana instance, see Configuring DNS for the Grafana Instances.
  2. (Optional) To allow HTTPS connections to one or more Grafana instances, you must provide a certificate and private key for the Grafana instance to use for TLS connections in Certificate and private key for HTTPS.
    
    VMware recommends also providing a certificate signed by a trusted third-party CA in CA certificate for HTTPS. You can generate a self-signed certificate using the Ops Manager root CA, but doing so causes your browser to warn you that your CA is invalid every time you access the Grafana UI.
    - To use a certificate signed by a trusted third-party CA:
      1. In Certificate and private key for HTTPS, provide a certificate and private key for the Grafana instance to use for TLS connections.
      2. In CA certificate for HTTPS, provide the certificate of the third-party CA that signs the certificate you provided in the previous step.
    - To generate a self-signed certificate from the Ops Manager root CA:
      1. Under Certificate and private key for HTTPS, click Change.
      2. Click Generate RSA Certificate.
      3. In the Generate RSA Certificate pop-up window, enter *.DOMAIN, where DOMAIN is the domain of the DNS entry that you configured for the Grafana instance. For example, if the DNS entry you configured for the Grafana instance is grafana.example.com, enter *.example.com. For more information about configuring a DNS entry for the Grafana instance, see Configuring DNS for the Grafana Instance.
      4. Click Generate.
      5. Navigate to the Ops Manager Installation Dashboard.
      6. From the dropdown in the upper-right corner of the Ops Manager Installation Dashboard, click Settings.
      7. Select Advanced Options.
      8. Click DOWNLOAD ROOT CA CERT.
      9. Return to the Ops Manager Installation Dashboard.
      10. Click the Healthwatch tile.
      11. Select Grafana.
      12. For Certificate and private key for HTTPS, provide the Ops Manager root CA certificate that you downloaded in a previous step.
  3. (Optional) To configure an additional cipher suite for TLS connections to the Grafana instance, enter a comma-separated list of ciphers in Additional ciphers for TLS. For a list of supported cipher suites, see cipher_suites.go in the Go repository on GitHub.
Under Grafana email alerts, choose whether to configure email alerts from the Grafana instance. VMware recommends using Alertmanager to configure and manage alerts in Healthwatch. If you require additional or alternative alerts, you can configure the SMTP server for the Grafana instance to send email alerts.
- To allow email alerts from the Grafana instance:
  1. Select Configure.
  2. For SMTP server host name, enter the host name of your SMTP server.
  3. For SMTP server port, enter the port of your SMTP server.
  4. For SMTP server username, enter your SMTP authentication username.
  5. For SMTP server password, enter your SMTP authentication password.
  6. (Optional) To allow the Grafana instance to skip TLS certificate verification when communicating with your SMTP server over TLS, activate the Skip TLS certificate verification checkbox. When this checkbox is activated, the Grafana instance does not verify the identity of your SMTP server. This checkbox is deactivated by default.
  7. For From address, enter the sender email address that appears on outgoing email alerts.
  8. For From name, enter the sender name that appears on outgoing email alerts.
  9. For EHLO client ID, enter the name for the client identity that your SMTP server uses when sending EHLO commands.
  10. For Certificate and private key for TLS, enter a certificate and private key for the Grafana instance to use for TLS connections to your SMTP server.
- To disallow email alerts from the Grafana instance, select Do not configure. Email alerts are disallowed by default.
  
  For more information, see the Grafana documentation.
Under HTTP and HTTPS proxy request settings, choose whether to allow the Grafana instance to make HTTP and HTTPS requests through proxy servers:
Note: You only need to configure proxy settings if you are deploying Healthwatch in an air-gapped environment and want to configure alert channels to external addresses, such as the external Slack webhook.
- To allow the Grafana instance to make HTTP and HTTPS requests through a proxy server:
  1. Select Configure.
  2. For HTTP proxy URL, enter the URL for your HTTP proxy server. The Grafana instance sends all HTTP and HTTPS requests to this URL, except those from hosts you configure in the HTTPS proxy URL and Excluded hosts fields below.
  3. For HTTPS proxy URL, enter the URL for your HTTPS proxy server. The Grafana instance sends all HTTPS requests to this URL, except those from hosts you configure in the Excluded hosts field below.
  4. For Excluded hosts, enter a comma-separated list of the hosts you want to exclude from proxying. VMware recommends including *.bosh and the range of your internal network IP addresses so the Grafana instance can still access the Prometheus instance without going though the proxy server. For example, *.bosh,10.0.0.0/8,*.example.com allows the Grafana instance to access all BOSH DNS addresses and all internal network IP addresses containing 10.0.0.0/8 or *.example.com directly, without going though the proxy server.
- To disallow the Grafana instance from making HTTP and HTTPS requests through proxy servers, select Do not configure. HTTP and HTTPS proxy requests are disallowed by default.
(Optional) For Static IP addresses for Grafana VMs, enter a comma-separated list of valid static IP addresses that you want to reserve for the Grafana instance. These IP addresses must not be within the reserved IP ranges you configured in the BOSH Director tile.
(Optional) If you want to use Grafana legacy alerting instead of new Grafana Alerting, select the Opt out of Grafana Alerting checkbox. Please note that this will delete any alerts and changes made in Grafana Alerting.
(Optional) If you want to disable the gravatar, select the Disable gravatar checkbox.
(Optional) To log all access to Grafana, select the Enable router logging checkbox. This will allow auditing of all traffic into the system.
Click Save.

(Optional) Configure Grafana Authentication

In the Grafana Authentication pane, you configure how users log in to the Grafana UI.

To configure the Grafana Authentication pane, see Configuring Grafana Authentication.

(Optional) Configure Grafana Dashboards

In the Grafana Dashboards pane, you configure which dashboards the Grafana instance creates in the Grafana UI. The Grafana instance can create dashboards for metrics from TAS for VMs, TKGI, VMware Tanzu SQL with MySQL for VMs (Tanzu SQL for VMs), and VMware Tanzu RabbitMQ for VMs (Tanzu RabbitMQ). For more information about these dashboards, see Default Dashboards in the Grafana UI in Using Healthwatch Dashboards in the Grafana UI.

To configure the Grafana Dashboards pane:

Select Grafana Dashboards.
Under TAS for VMs, select one of the following options:
- Include: The Grafana instance creates dashboards in the Grafana UI for metrics from TAS for VMs. To specify the version of TAS for VMs for which you want the Grafana instance to create dashboards:
  1. From the Version dropdown, select one of the following options:
    - The version of TAS for VMs that is installed on your Ops Manager foundation.
    - Auto-detect: The Grafana instance automatically discovers and creates dashboards for the version of TAS for VMs that is installed on your Ops Manager foundation.
      Note: If you choose to include TAS for VMs dashboards, you must configure TAS for VMs to forward system metrics to the Loggregator Firehose. Otherwise, no metrics appear in the Router dashboard in the Grafana UI. For more information, see Troubleshooting Missing Router Metrics in Troubleshooting Healthwatch.
      
      Important: If you have installed Ops Manager 3.0 you need to select version 3.0 from the list of dashboards on the Grafana Dashboards tab of the Healthwatch tile.
- Exclude: The Grafana instance does not create dashboards in the Grafana UI for metrics from TAS for VMs.
Under TKGI, select one of the following options:
- Include: The Grafana instance creates dashboards in the Grafana UI for metrics from TKGI. To specify the version of TKGI for which you want the Grafana instance to create dashboards:
  1. From the Version dropdown, select one of the following options:
    - The version of TKGI that is installed on your Ops Manager foundation.
    - Auto-detect: The Grafana instance automatically discovers and creates dashboards for the version of TKGI that is installed on your Ops Manager foundation.
- Exclude: The Grafana instance does not create dashboards in the Grafana UI for metrics from TKGI.
Under Tanzu SQL for VMs, select one of the following options:
- Include: The Grafana instance creates a dashboard in the Grafana UI for metrics from Tanzu SQL for VMs.
- Exclude: The Grafana instance does not create a dashboard in the Grafana UI for metrics from Tanzu SQL for VMs.
Under Tanzu RabbitMQ, select one of the following options:
- Include: The Grafana instance creates dashboards in the Grafana UI for metrics from Tanzu RabbitMQ.
  Note: If you choose to include Tanzu RabbitMQ dashboards, set the Metrics polling interval field in the Tanzu RabbitMQ tile to -1. This prevents the Tanzu RabbitMQ tile from sending duplicate metrics to the Loggregator Firehose. To configure this field, see the Tanzu RabbitMQ documentation.
- Exclude: The Grafana instance does not create dashboards in the Grafana UI for metrics from Tanzu RabbitMQ.

(Optional) Configure Canary URLs

In the Canary URLs pane, you configure target URLs to which the Blackbox Exporters in the Prometheus instance sends canary tests. Testing a canary target URL allows you to gauge the overall health and accessibility of an app, runtime, or deployment.

The Canary URLs pane configures the Blackbox Exporters in the Prometheus instance. For more information, see the Blackbox exporter repository on GitHub.

The Blackbox Exporters in the Prometheus instance run canary tests on the fully-qualified domain name (FQDN) of your Ops Manager deployment by default. The results from these canary tests appear in the Ops Manager Health dashboard in the Grafana UI.

To configure the Canary URLs pane:

Select Canary URLs.
For Port, specify the port that the Blackbox Exporter exposes to the Prometheus instance. The default port is 9115. You do not need to specify a different port unless port 9115 is already in use on the Prometheus instance.
(Optional) Under Additional target URLs, you can configure additional canary target URLs. The Prometheus instance runs continuous canary tests to these URLs and records the results. To configure additional canary target URLs:
1. Click Add.
2. For URL, enter the URL to which you want the Prometheus instance to send canary tests.
  Note: The Prometheus instance automatically creates scrape jobs for these URLs. You do not need to create additional scrape jobs for them in the Prometheus pane.
Click Save.

(Optional) Configure Remote Write

In the Remote Write pane, you can configure the Prometheus instance to write to remote storage, in addition to its local time series database (TSDB). Healthwatch stores monitoring data for six weeks before deleting it. Configuring remote write allows Healthwatch to store data that is older than six weeks in a remote database or storage endpoint. For a list of compatible remote databases and storage endpoints, see the Prometheus documentation.

The values that you configure in the Remote Write pane also configure their corresponding properties in the Prometheus configuration file. For more information, see Overview of Configuration Files in Healthwatch in Configuration File Reference Guide, Remote Write in Configuration File Reference Guide, and the Prometheus documentation.

To configure the Remote Write pane:

Select Remote Write.
Under Remote write destinations, click Add.
For Remote storage URL, enter the URL for your remote storage endpoint. For example, https://REMOTE-STORAGE-FQDN, where REMOTE-STORAGE-FQDN is the FQDN of your remote storage endpoint.
In Remote timeout, enter in seconds the amount of time that the Prometheus VM tries to make a request to your remote storage endpoint before the request fails.
If your remote storage endpoint requires a username and password to log in to it, configure the following fields:
1. For Remote storage username, enter the username that the Prometheus instance uses to log in to your remote storage endpoint.
2. For Remote storage password, enter the password that the Prometheus instance uses to log in to your remote storage endpoint.
  Note: If you configure a username and password for the Prometheus instance to use when logging in to your remote storage endpoint, you cannot also configure a bearer token.
If your remote storage endpoint requires a bearer token to log in to it, enter the bearer token that the Prometheus instance uses to log in to your remote storage endpoint in Bearer token.
Note: If you configure a bearer token for the Prometheus instance to use when logging in to your remote storage endpoint, you cannot also configure a username and password.
(Optional) To allow the Prometheus instance to communicate with the server for your remote storage endpoint over TLS:
1. For Certificate and private key for TLS, provide a certificate and private key for the Prometheus instance to use for TLS connections to your remote storage endpoint.
2. For CA certificate for TLS, provide the certificate for the CA that the server for your remote storage endpoint uses to verify TLS certificates.
3. For Remote storage server name, enter the name of the server for your remote storage endpoint as it appears on the server’s TLS certificate.
4. If the certificate you provided in Certificate and private key for TLS is signed by a self-signed CA certificate or a certificate that is signed by a self-signed CA certificate, activate the Skip TLS certificate verification checkbox. When this checkbox is activated, the Prometheus instance does not verify the identity of the server for your remote storage endpoint. This checkbox is deactivated by default.
(Optional) To allow the Prometheus instance to make HTTP or HTTPS requests to your remote storage endpoint through a proxy server, enter the URL for your proxy server in Proxy URL.
You can configure more granular settings for writing to your remote storage endpoint by specifying additional parameters for the shards containing in-memory queues that read from the write-ahead log in the Prometheus instance. To configure additional parameters for these shards:
1. For Queue capacity, enter how many samples your remote storage endpoint can queue in memory per shard before the Prometheus instance blocks the queue from reading from the write-ahead log.
2. For Minimum shards per queue, enter the minimum number of shards the Prometheus instance can use for each remote write queue. This number is also the number of shards the Prometheus instance uses when remote write begins after each deployment of the Healthwatch tile.
3. For Maximum shards per queue, enter the maximum number of shards the Prometheus instance can use for each remote write queue.
4. For Maximum samples per send, enter the maximum number of samples the Prometheus instance can send to a shard at a time.
5. For Maximum batch wait time, enter in seconds the maximum amount of time the Prometheus instance can wait before sending a batch of samples to a shard, whether that shard has reached the limit configured in Maximum samples per send or not.
6. For Minimum backoff time, enter in milliseconds the minimum amount of time the Prometheus instance can wait before retrying a failed request to your remote storage endpoint.
7. For Maximum backoff time, enter in milliseconds the maximum amount of time the Prometheus instance can wait before retrying a failed request to your remote storage endpoint.
  
  For more information about configuring these queue parameters, see the Prometheus documentation.
Click Save.

(Optional) Configure TKGI Cluster Discovery

In the TKGI Cluster Discovery pane, you configure TKGI cluster discovery for Healthwatch. You only need to configure this pane if you have Ops Manager foundations with TKGI installed.

To configure TKGI cluster discovery, see Configuring TKGI Cluster Discovery.

(Optional) Configure Errands

Errands are scripts that Ops Manager runs automatically when it installs or uninstalls a product, such as a new version of Healthwatch. There are two types of errands: post-deploy errands run after the product is installed, and pre-delete errands run before the product is uninstalled. However, there are no pre-delete errands for Healthwatch.

By default, Ops Manager always runs all errands.

In the Errands pane, you can select On to always run an errand or Off to never run it.

For more information about how Ops Manager manages errands, see the Ops Manager documentation.

To configure the Errands pane:

Select Errands.
(Optional) Choose whether to always run or never run the following errands:
- Smoke Test Errand: Verifies that the Grafana and Prometheus instances are running.
- Update Grafana Admin Password: Updates the administrator password for the Grafana UI.
Click Save.

(Optional) Configure Syslog

In the Syslog pane, you can configure system logging in Healthwatch to forward log messages from Healthwatch component VMs to an external destination for troubleshooting, such as a remote server or external syslog aggregation service.

To configure the Syslog pane:

Select Syslog.
Under Do you want to configure Syslog forwarding?, select one of the following options:
- No, do not forward Syslog: Disallows syslog forwarding.
- Yes: Allows syslog forwarding and allows you to edit the configuration fields described below.
For Address, enter the IP address or DNS domain name of your external destination.
For Port, enter a port on which your external destination listens.
For Transport Protocol, select TCP or UDP from the dropdown. This determines which transport protocol Healthwatch uses to forward system logs to your external destination.
(Optional) To transmit logs over TLS:
1. Activate the Enable TLS checkbox. This checkbox is deactivated by default.
2. For Permitted Peer, enter either the name or SHA1 fingerprint of the remote peer.
3. For SSL Certificate, enter the TLS certificate for your external destination.
(Optional) For Queue Size, specify the number of log messages Healthwatch can hold in a buffer at a time before sending them to your external destination. The default value is 100000.
(Optional) To forward debug logs to your external destination, activate the Forward Debug Logs checkbox. This checkbox is deactivated by default.
(Optional) To specify a custom syslog rule, enter it in Custom rsyslog configuration in RainerScript syntax. For more information about custom syslog rules, see the TAS for VMs documentation. For more information about RainerScript syntax, see the rsyslog documentation.
Click Save Syslog Settings.

(Optional) Configure Resources

In the Resource Config pane, you can scale Healthwatch component VMs up or down according to the needs of your deployment, as well as associate load balancers with a group of VMs. For example, you can scale the persistent disk size of the Prometheus instance to allow longer data retention.

To configure the Resource Config pane:

Select Resource Config.
(Optional) To scale a job, select an option from the dropdown for the resource you want to modify:
- Instances: Configures the number of instances each job has.
- VM Type: Configures the type of VM used in each instance.
- Persistent Disk Type: Configures the amount of persistent disk space to allocate to the job.
(Optional) To add a load balancer to a job:
1. Click the icon next to the job name.
2. For Load Balancers, enter the name of your load balancer.
3. Ensure that the Internet Connected checkbox is deactivated. Activating this checkbox gives VMs a public IP address that allows outbound Internet access.
Click Save.

(Optional) Configure for OpenTelemetry

Follow these steps to configure HealthWatch and receive data through the OpenTelemetry Collector for Tanzu Application Service 6.0 and above:

Navigate to the Healthwatch Exporter for TAS for VMs tile in the Ops Manager Installation Dashboard.
1. Select Resource Config under the Settings tab.
  1. Set the TAS Configure Exporter and TAS Gauge Exporter configurations to O because they are only used by FireHose.
  2. Click Save.
2. Click the Credentials tab:
  1. Click Link to Credential next to Healthwatch Otel Mtls.
  2. Save the certificate content. The cert and key will be used when configuring the Open Telemetry collector.
Navigate to the Ops Manager Installation Dashboard.
1. From the dropdown in the upper-right corner of the Ops Manager Installation Dashboard, click Settings.
2. Select Advanced Options.
3. Click DOWNLOAD ROOT CA CERT. The Ops Manager root CA will be used when configuring the Open Telemetry collector
Open the VMware Tanzu Application Service for VMs tile.
1. Click Settings and select System Logging.
  1. Deselect the Enable V1 Firehose and Enable V2 Firehose configurations.
  2. Scroll down to the OpenTelemetry Collector Metric Exporters (beta) configuration.
  3. Scroll to the bottom of the Open Telemetry configuration and add a collector for Healthwatch.
  4. For TAS 6.0, Healthwatch expects a prometheus Open Telemetry collector the supports mTLS and sends data on port 65331. For example:
```
prometheus/healthwatch:
 endpoint: ":65331"
 add_metric_suffixes: false
 tls:
   ca_pem: "CA-CERT"
   cert_pem: "CERT_PEM"
   key_pem: "PRIVATE_KEY_PEM"
```
    Where:
    - CA-CERT is the Ops Manager root CA.
    - CERT_PEM is the cert_pem of the Healthwatch Otel Mtls credential.
    - PRIVATE_KEY_PEM is the private_key_pem of the Healthwatch Otel Mtls credential.
  5. For Tanau Platform for Cloud Foundry (formerly TAS for VMs) 10.0, you can configure certificates under OpenTelemetry Collector Secrets and refer them in OTel configuration. For example:
```
exporters:
  prometheus/healthwatch:
    endpoint: ":65331"
    add_metric_suffixes: false
    tls:
      ca_pem: '{{ .healthwatch.ca }}'
      cert_pem: '{{ .healthwatch.cert }}'
      key_pem: '{{ .healthwatch.key }}'
service:
  pipelines:
    metrics:
      exporters:
        - prometheus/healthwatch
```
    Add Secrets: 1. Click Add next to OpenTelemetry Collector Secrets. 2. In Name, enter healthwatch. 3. For Certificate Authority enter CA-CERT from the Operations Manager root CA. 4. The Client Certificate PEM is the cert_pem of the Healthwatch OTel mtls credential. 5. The Client Ceritificate Private Key PEM is the private_key_pem of the Healthwatch Otel Mtls credential. 6. Remove the newline character (\n) from the certificates you copy: awk '{gsub(/\n/,"\n")}1' <file_name> or printf -- "<CERT_DATA>"
  6. Click Save.
If you made changes to the Healthwatch Exporter for Tanzu Application Service tile configuration in Settings > TAS for VMs Metric Exporter VMs > Filter out custom application metrics, deploy your changes to Healthwatch as explained in the next section.

Deploy Healthwatch

To complete your installation of the Healthwatch tile:

Return to the Ops Manager Installation Dashboard.
Click Review Pending Changes.
Click Apply Changes.

For more information, see the Ops Manager documentation.

Next Steps

After you have successfully installed the Healthwatch tile, continue to one of the following topics to configure and deploy the Healthwatch Exporter tiles for the Ops Manager foundations you want to monitor:

If you have TAS for VMs installed on an Ops Manager foundation you want to monitor, see Configuring Healthwatch Exporter for TAS for VMs.
If you have TKGI installed on an Ops Manager foundation you want to monitor, see Configuring Healthwatch Exporter for TKGI.