Use RabbitMQ queue depth as a scaling metric with App Autoscaler

You can configure App Autoscaler to use the RabbitMQ queue depth metric to scale apps in your VMware TAS for VMs deployment.

Overview of RabbitMQ queue depth

RabbitMQ is a message broker that communicates using message queues. Queue depth is the number of messages in a RabbitMQ queue that are waiting to be dequeued. You can configure Autoscaler to use RabbitMQ queue depth as the scaling metric for apps that consume messages from RabbitMQ queues for processing. When Autoscaler scales these apps up, they can process these RabbitMQ queues more quickly.

You can configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app in the following ways:

Through the Cloud Foundry Command-Line Interface (cf CLI). For more information, see Configuring RabbitMQ queue depth as the scaling metric for an app through the cf CLI.
Through Apps Manager. For more information, see Configure RabbitMQ queue depth as the scaling metric for an app through Apps Manager.

You can also configure Autoscaler to query queues on an external RabbitMQ cluster by creating a user-provided service that represents the external RabbitMQ cluster. For more information, see Use an external RabbitMQ cluster.

To monitor when Autoscaler scales an app based on changes in RabbitMQ queue depth, see Reviewing Autoscaling events for changes in RabbitMQ queue depth.

For information about use cases that may complicate or prevent you from configuring RabbitMQ queue depth as the scaling metric for an app, see Special considerations for using RabbitMQ queue depth as a scaling metric.

VMware recommends that you load-test your app to verify that the autoscaling rules you configured are effective. For more information, see Load-testing your app in Using Autoscaler in production.

For more information about RabbitMQ queues, see the RabbitMQ documentation.

Configuring RabbitMQ queue depth as the scaling metric for an app through the cf CLI

The procedures in this section describe how to configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app through the cf CLI.

You can configure configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app in the following ways:

Using a manifest file. For more information, see Configure an Autoscaling rule using a manifest file.
Using CLI commands. For more information, see Configure an Autoscaling rule using CLI commands.

For the procedures in this section, you must use the App Autoscaler CLI plug-in. To download and install the App Autoscaler CLI plug-in, see Install the App Autoscaler CLI plug-in in Using the App Autoscaler CLI.

Configure an autoscaling rule using a manifest file

You can configure autoscaling rules through a manifest file. This manifest file only configures Autoscaler, and does not interfere with any other existing app manifest files in your TAS for VMs deployment.

To configure an autoscaling rule that defines RabbitMQ queue depth as its scaling metric using a manifest file:

In a terminal window, target the space in which the app you want to scale is deployed by running:
```
cf target -o ORG-NAME -s SPACE-NAME
```
Where:
- ORG-NAME is the name of the org containing the space in which the app you want to scale is deployed.
- SPACE-NAME is the name of the space in which the app you want to scale is deployed.
If the space in which the app you want to scale is deployed does not already have an Autoscaler service instance of Autoscaler deployed in it, create an Autoscaler service instance by running:
```
cf create-service app-autoscaler PLAN-NAME SERVICE-NAME
```
Where:
- PLAN-NAME is the name of the service plan you want to use for the Autoscaler service instance.
- SERVICE-INSTANCE-NAME is the name you want to give the Autoscaler service instance. For example, autoscaler.
If there is already an Autoscaler service instance in the space in which the app you want to scale is deployed, skip this step.
Bind the Autoscaler service instance you created in the previous step to the app you want to scale by running:
```
cf bind-service APP-NAME SERVICE-INSTANCE-NAME
```
Where:
- APP-NAME is the name of the app you want to scale.
- SERVICE-INSTANCE-NAME is the name of the Autoscaler service instance in the previous step.
To create a manifest file for Autoscaler that configures an autoscaling rule with RabbitMQ queue depth as its scaling metric, create a YAML file that includes the following configuration parameters:
```
---
instance_limits:
  min: LOWER-SCALING-LIMIT
  max: UPPER-SCALING-LIMIT
rules:
- rule_type: rabbitmq
  rabbitmq_service: RABBITMQ-SERVICE-INSTANCE-NAME
  rule_sub_type: QUEUE-NAME
  threshold:
    min: MINIMUM-QUEUE-DEPTH-THRESHOLD
    max: MAXIMUM-QUEUE-DEPTH-THRESHOLD
scheduled_limit_changes: []
```
Where:
- LOWER-SCALING-LIMIT is the minimum number of instances you want Autoscaler to create for the app.
- UPPER-SCALING-LIMIT is the maximum number of instances you want Autoscaler to create for the app.
- (Optional) RABBITMQ-SERVICE-INSTANCE-NAME is the name of the RabbitMQ service instance you want Autoscaler to query. Defining this parameter ensures that Autoscaler does not unnecessarily query other RabbitMQ service instances that are bound to the app.
- QUEUE-NAME is the name of the queue you want Autoscaler to monitor. This queue must be on a RabbitMQ service instance that is bound to the app you want to scale.
- MINIMUM-QUEUE-DEPTH-THRESHOLD is the minimum queue depth threshold. If the average queue depth falls below this number of messages, Autoscaler scales the number of app instances down.
- MAXIMUM-QUEUE-DEPTH-THRESHOLD is the maximum queue depth threshold. If the average queue depth rises above this number of messages, Autoscaler scales the number of app instances up.
The following example shows an Autoscaler manifest file monitoring the example_queue queue on the example-service RabbitMQ service instance, with a minimum queue depth threshold of 10 and a maximum queue depth threshold of 50:
```
---
instance_limits:
  min: 10
  max: 100
rules:
- rule_type: rabbitmq
  rabbitmq_service: example-service
  rule_sub_type: example_queue
  threshold:
    min: 10
    max: 50
scheduled_limit_changes: []
```
Apply the autoscaling rule you configured in the previous step to the app you want to scale by running:
```
cf configure-autoscaling APP-NAME MANIFEST-FILENAME
```
Where:
- APP-NAME is the name of the app.
- MANIFEST-FILENAME is the filename of the manifest file you created in the previous step. For example, autoscaler.yml.

Configure an autoscaling rule by using CLI commands

To configure an autoscaling rule that defines RabbitMQ queue depth as its scaling metric using CLI commands:

In a terminal window, target the space in which the app you want to scale is deployed by running:
```
cf target -o ORG-NAME -s SPACE-NAME
```
Where:
- ORG-NAME is the name of the org containing the space in which the app you want to scale is deployed.
- SPACE-NAME is the name of the space in which the app you want to scale is deployed.
If the space in which the app you want to scale is deployed does not already have a service instance of Autoscaler deployed in it, create an Autoscaler service instance by running:
```
cf create-service app-autoscaler PLAN-NAME SERVICE-INSTANCE-NAME
```
Where:
- PLAN-NAME is the name of the service plan you want to use for the Autoscaler service instance.
- SERVICE-INSTANCE-NAME is the name you want to give the Autoscaler service instance. For example, autoscaler.
If there is already an Autoscaler service instance in the space in which the app you want to scale is deployed, skip this step.
Bind the Autoscaler service instance you created in the previous step to the app you want to scale by running:
```
cf bind-service APP-NAME SERVICE-INSTANCE-NAME
```
Where:
- APP-NAME is the name of the app you want to scale.
- SERVICE-INSTANCE-NAME is the name of the Autoscaler service instance in the previous step.
Configure upper and lower scaling limits for the app by running:
```
cf update-autoscaling-limits APP-NAME LOWER-SCALING-LIMIT UPPER-SCALING-LIMIT
```
Where:
- APP-NAME is the name of the app.
- LOWER-SCALING-LIMIT is the minimum number of instances you want Autoscaler to create for the app.
- UPPER-SCALING-LIMIT is the maximum number of instances you want Autoscaler to create for the app.
Set Autoscaler to begin making scaling decisions for the app by running:
```
cf enable-autoscaling APP-NAME
```
Where APP-NAME is the name of the app.
Create a rabbitmq autoscaling rule by running:
```
cf create-autoscaling-rule APP-NAME rabbitmq MINIMUM-QUEUE-DEPTH-THRESHOLD MAXIMUM-QUEUE-DEPTH-THRESHOLD --rabbitmq-instance RABBITMQ-SERVICE-INSTANCE-NAME --subtype QUEUE-NAME
```
Where:
- APP-NAME is the name of the app for which you want to create an autoscaling rule.
- MINIMUM-QUEUE-DEPTH-THRESHOLD is the minimum queue depth threshold. If the average queue depth falls below this number of messages, Autoscaler scales the number of app instances down.
- MAXIMUM-QUEUE-DEPTH-THRESHOLD is the maximum queue depth threshold. If the average queue depth rises above this number of messages, Autoscaler scales the number of app instances up.
- (Optional) RABBITMQ-SERVICE-INSTANCE-NAME is the name of the RabbitMQ service instance you want Autoscaler to query. Defining this parameter ensures that Autoscaler does not unnecessarily query other RabbitMQ service instances that are bound to the app.
- QUEUE-NAME is the name of the queue you want Autoscaler to monitor. This queue must be on a RabbitMQ service instance that is bound to the app you want to scale.
The following example command configures a rabbitmq autoscaling rule for the example-app app that monitors the example_queue queue on the example-service RabbitMQ service instance, with a minimum queue depth threshold of 10 and a maximum queue depth threshold of 50 milliseconds:
```
cf create-autoscaling-rule example-app rabbitmq 10 50 --rabbitmq-instance example-service --subtype example_queue
```

Configure RabbitMQ queue depth as the scaling metric for an app through Apps Manager

To configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app using Apps Manager:

Log in to Apps Manager. For more information, see Logging in to Apps Manager.
Select the org that contains the space in which the app you want to scale is deployed.
Select the space in which the app you want to scale is deployed.
Under Under Processes and Instances, click Manage Autoscaling. The Manage Autoscaling window appears.
Next to Scaling Rules, click Edit. The Edit Scaling Rules window appears.
Click Add rule. The Select type dropdown appears.
From the Select type drop-down menu, select RabbitMQ. More configuration text boxes appear.
1. For Scale down if less than, enter the minimum queue depth threshold. If the average queue depth falls below this number of messages, Autoscaler scales the number of app instances down.
2. For Scale up if more than, enter the maximum queue depth threshold. If the average queue depth rises above this number of messages, Autoscaler scales the number of app instances up.
3. For Queue name, enter the name of the queue you want Autoscaler to monitor. This queue must be on a RabbitMQ service instance that is bound to the app you want to scale.
Click Save.

Use an external RabbitMQ cluster

If your RabbitMQ queue is hosted on an external RabbitMQ cluster, you can create a user-provided service that represents the external RabbitMQ cluster. Autoscaler can query this user-provided service to retrieve queue metadata.

To use a user-provided service for your external RabbitMQ cluster:

Activate the RabbitMQ management plug-in on your RabbitMQ cluster by following the procedures in the RabbitMQ documentation. Autoscaler can use the RabbitMQ HTTP Management API to discover the depth of the queues in the RabbitMQ cluster.
In a terminal window, run:
```
cf create-user-provided-service SERVICE-NAME -p '{"uri":"amqp://USERNAME:PASSWORD@RABBITMQ-FQDN","http_api_uri":"http://USERNAME:PASSWORD@RABBITMQ-FQDN:15672/api/","vhost":"VHOST"}' \ -t rabbitmq
```
Where:
- SERVICE-NAME is the name you want to give the user-provided service.
- USERNAME is the username you want to configure for the external RabbitMQ cluster.
- PASSWORD is the password you want to configure for the external RabbitMQ cluster.
- RABBITMQ-FQDN is the fully-qualified domain name (FQDN) of the RabbitMQ cluster.
- VHOST is the RabbitMQ virtual host you want the user-provided service to use.
  CautionIf you do not define the "vhost" parameter, Autoscaler requests queue metadata for all queues on the RabbitMQ cluster. This is a very expensive operation. For more information about RabbitMQ virtual hosts, see the RabbitMQ documentation.
Bind the user-provided service to the app you want to scale by running:
```
cf bind-service APP-NAME SERVICE-NAME
```
Where:
- APP-NAME is the name of the app you want to scale.
- SERVICE-NAME is the name of the user-provided service you created in the previous step.

Important Depending on the app to which you bind the user-provided service, you might need to configure the VCAP_SERVICES environment variable to approve accessing the user-provided service. For more information, see the VMware Tanzu RabbitMQ for VMs documentation.

After you bind the user-provided service to the app, configure Autoscaler to use RabbitMQ queue depth as the scaling metric for an app by following one of the procedures in Configuring RabbitMQ queue depth as the scaling metric for an app through the cf CLI or Configuring RabbitMQ queue depth as the scaling metric for an app through Apps Manager.

Reviewing autoscaling events for changes in RabbitMQ queue depth

When Autoscaler scales the number of app instances up after the RabbitMQ queue depth metric increases above the maximum queue depth threshold, Autoscaler records an autoscaling event.

You can monitor the autoscaling events that Autoscaler records for changes in RabbitMQ queue depth in the following ways:

Through the cf CLI. For more information, see Review Autoscaling events for changes in RabbitMQ queue depth through the cf CLI.
Through Apps Manager. For more information, see Review Autoscaling events for changes in RabbitMQ queue depth through Apps Manager.

Review autoscaling events for changes in RabbitMQ queue depth through the cf CLI

To review the autoscaling events that Autoscaler records for changes in RabbitMQ queue depth through the cf CLI:

In a terminal window, run:
```
cf autoscaling-events APP-NAME
```
Where APP-NAME is the name of the app for which you want to review autoscaling events.

If Autoscaler has scaled the number of app instances up due to increases in the RabbitMQ queue depth metric, the above command returns output that contains autoscaling events similar to the following example:
```
Time                   Description
2022-05-24T20:47:34Z   Scaled up from 10 to 11 instances. The RabbitMQ message depth in example_queue is at an average of 50.35, which is above the maximum threshold of 50.00.
```

Review events in RabbitMQ queue

To review the autoscaling events that Autoscaler records for changes in RabbitMQ queue depth through Apps Manager:

Log in to Apps Manager. For more information, see Logging in to Apps Manager.
Select the org that contains the space in which the app you want to scale is deployed.
Select the space in which the app you want to scale is deployed.
Under Under Processes and Instances, click Manage Autoscaling.
Under Event History, click View More. A list of autoscaling events appears. If Autoscaler has scaled the number of app instances up due to increases in the RabbitMQ queue depth metric, the list of autoscaling events includes events similar to the following example:
```
Scaled up from 10 to 11 instances. The RabbitMQ message depth in example_queue is at an average of 50.35, which is above the maximum threshold of 50.00.
```

Special considerations for using RabbitMQ queue depth as a scaling metric

This section describes use cases that may complicate or prevent you from configuring RabbitMQ queue depth as the scaling metric for an app. For more information, see the following sections.

RabbitMQ service binding credentials in runtime CredHub

To receive RabbitMQ queue depth metrics, Autoscaler must retrieve service binding credentials for the RabbitMQ HTTP Management API from Cloud Controller. However, when a TAS for VMs deployment is configured to store credentials in runtime CredHub, Autoscaler cannot retrieve service binding credentials from Cloud Controller.

For more information, see Autoscaler does not support runtime CredHub in Troubleshooting App Autoscaler.

Multiple RabbitMQ service instances

If an app is bound to more than one RabbitMQ service instance, and the autoscaling rules you configure for the app do not specify the name of a particular RabbitMQ service instance name, Autoscaler queries all RabbitMQ service instances that are bound to the app for the specified queue. To avoid this, specify the name of the RabbitMQ service instance that hosts the queue you want to monitor when you configure autoscaling rules for the app.