Horizontal Pod Autoscaling with a VMware Tanzu GemFire Cluster

This topic describes how to enable a horizontal pod autoscaler using VMware Tanzu GemFire for Kubernetes.

When you configure an autoscaler, the Custom Resource Definition (CRD) entry for server replicas is used to scale the initial cluster to the specified size before the autoscaler is configured and enabled.

After the autoscaler is running, additional updates to the server replicas field are no longer applied to the underlying statefulset because the autoscaler manages updating the number of replicas.

Before you delete a cluster with autoscaling enabled, record the number of running servers. You can also find the number of servers running in the cluster in the controller log. Set the number of server replicas to this number that you recorded when recreating the cluster. This allows the cluster to recover by initially scaling to the last known size.

Prerequisites

Create a GemFireCluster Configured for Autoscaling

To create a GemFireCluster configured for autoscaling:

Create an image pull secret in the namespace default with credentials authorized to pull images from VMware Tanzu Network:
```
kubectl create secret docker-registry image-pull-secret --docker-server=registry.tanzu.vmware.com --docker-username='USERNAME' --docker-password='PASSWORD'
```
Where:
- USERNAME is the user name of an account with credentials authorized to pull images from VMware Tanzu Network.
- PASSWORD is the password for the USERNAME account.

Save the following GemFireCluster definition to my-gemfire-cluster.yaml.

---
apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
  name: my-gemfire-cluster
spec:
  autoscaler:
    minReplicaCount: 2
    maxReplicaCount: 4
  image: registry.tanzu.vmware.com/pivotal-gemfire/vmware-gemfire:10.1.0
  servers:
  resources:
    requests:
      cpu: 1

To function, the default, out-of-the-box autoscaler requires that you set the CPU resource request for servers. The autoscaler created by default has the following metrics configuration:
```
metrics:
- resource:
    name: cpu
    target:
      averageUtilization: 80
      type: Utilization
  type: Resource
```
You can, override, and set completely new metrics. If you override the metrics, you may need to set the corresponding resource requests if you are not using custom metric resources.

Create the GemFireCluster:

kubectl apply -f my-gemfire-cluster.yaml

Confirm HorizontalPodAutoScaler Creation

To confirm that a HorizontalPodAutoScaler has been created, run:

kubectl get hpa my-gemfire-cluster-horizontal-pod-autoscaler

For example:

$ kubectl get hpa my-gemfire-cluster-horizontal-pod-autoscaler

NAME                                REFERENCE                    TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
gemfire-horizontal-pod-autoscaler   StatefulSet/gemfire-server   5%/75%          2         3         3          13m

The default autoscaler is configured with the following:

  behavior:
    scaleDown:
      policies:
      - periodSeconds: 1800
        type: Pods
        value: 1
      selectPolicy: Max
      stabilizationWindowSeconds: 3600
    scaleUp:
      policies:
      - periodSeconds: 15
        type: Pods
        value: 4
      - periodSeconds: 15
        type: Percent
        value: 100
      selectPolicy: Max
      stabilizationWindowSeconds: 0
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 80
        type: Utilization
    type: Resource

You can modify the scaleUp and scaleDown settings to better fit your workload.