The Custom Resource Definition

In VMware Tanzu GemFire for Kubernetes, the Custom Resource Definition (CRD) is a YAML file that defines a VMware Tanzu GemFire cluster named GemFireCluster and specifies its characteristics.

CRD Template

The CRD file specifies fields and values that define the Tanzu GemFire cluster API. The template below lists the fields that you can set for a Tanzu GemFire cluster. Details are provided after the template.

apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
  name: <string>
  namespace: <string>
spec:
  serviceAccountName: <string>
  image: <string>
  imagePullSecrets: <list of references>
  antiAffinityPolicy: <string>
  autoscaler:
    minReplicaCount: <integer>
    maxReplicaCount: <integer>
    metrics: <list of kubernetes k8s.io/api/autoscaling/v2/MetricSpec>
    behavior: a kubernetes *k8s.io/api/autoscaling/v2/HorizontalPodAutoscalerBehavior 
  metrics:
    emission: <string>
    interval: <string>
  ingress:
    gatewayName: <string>
  resourceManager:
    criticalHeapPercentage: <integer>
    evictionHeapPercentage: <integer>
  sysctls: <list of sysctls name/value pairs>
  persistentVolumeClaim:
    resources:
      requests:
        storage: <storage units>
    storageClassName: <string>
    deleteOnShutdown: <boolean>
  serialization:
    pdx:            
      readSerialized: <boolean>
  security:
    tls:
      secretName: <string>
      clientAuthenticationRequired: <boolean>
    mgmtSvcCredentialsSecretName: <string>
  locators:
    replicas: <integer>
    labels: <dictionary of key/value pairs>
    annotations: <dictionary of key/value pairs>
    libraries: <list of objects>
      - name: <string>
        container:
          image: <string>
          path: <string>
          imagePullSecretRef: <reference>
    resources:
      requests:
        memory: <memory units>
        cpu: <cpu units>
      limits:
        memory: <memory units>
        cpu: <cpu units>
    persistentVolumeClaim:     
      resources:               
        requests:
          storage: <storage units>        
      storageClassName: <string>
      deleteOnShutdown: <boolean>
    overrides:
      jvmOptions: <list of strings>
      gemFireProperties: <list of GemFireProperties name/value pairs>
      statefulSet:
        spec:
          template:
            metadata:
              name: <string>
              namespace: <string>
              labels: <list of strings>
              annotations: <list of strings>
            spec: <Kubernetes pod template spec>
  servers:
    replicas: <integer>
    labels: <dictionary of key/value pairs>
    annotations: <dictionary of key/value pairs>
    libraries: <list of objects>
      - name: <string>
        container:
          image: <string>
          path: <string>
          imagePullSecretRef: <reference>
    resources:
      requests:
        memory: <memory units>
        cpu: <cpu units>
      limits:
        memory: <memory units>
        cpu: <cpu units>
    persistentVolumeClaim:     
      resources:               
        requests:
          storage: <storage units>        
      storageClassName: <string>
      deleteOnShutdown: <boolean>
    overrides:
      jvmOptions: <list of strings>
      gemFireProperties: <list of GemFireProperties name/value pairs>
      statefulSet:
        spec:
          template:
            metadata:
              name: <string>
              namespace: <string>
              labels: <list of strings>
              annotations: <list of strings>
            spec: <Kubernetes pod template spec>

Fields and Values

The first two fields establish the nature of the CRD and its name. For this release of Tanzu GemFire, these should be identical:

apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster

Instance Metadata

The fields in this section apply metadata to the Tanzu GemFire cluster.

metadata.name

metadata:
  name: <string>

(Required) Sets the name of the GemFireCluster cluster. This value cannot be changed on an existing cluster.

metadata.namespace

metadata:
  namespace: <string>

(Optional) Sets the Kubernetes namespace in which the GemFireCluster cluster will be deployed. If not specified, the current kubectl context’s namespace will be used. This value cannot be changed on an existing cluster.

Cluster Configuration

The fields in this section apply configuration to the entire Tanzu GemFire cluster.

spec.serviceAccountName

spec:
  serviceAccountName: <string>

(Optional) Specifies the Kubernetes Service Account to be configured by the gemfire-operator within all pods for the Tanzu GemFire cluster. When not specified, the default Service Account for the namespace is automatically configured by Kubernetes.

spec.image

spec:
  image: <string>

(Required) Specifies the image used by the gemfire-operator when creating pods for the Tanzu GemFire cluster. This specifies a cluster version. Updating this value triggers an upgrade to a new version of the Tanzu GemFire cluster. Downgrades to a previous version are not supported, and will result in a failure of the first locator to restart.

spec.imagePullSecrets

spec:
  imagePullSecrets: <list of references>

(Optional) Specifies the list of names for image pull secrets in this Tanzu GemFire cluster’s namespace. Required if more than one registry holds images that are needed for creating a Tanzu GemFire cluster. When not specified, the name of the single image pull secret used during the application of the CRD for creating a Tanzu GemFire cluster is image-pull-secret.

Include each image pull secret’s name in this list of references, as in this example with two image pull secrets:

spec:
  imagePullSecrets:
  - name: "secret-1"
  - name: "secret-2"

spec.antiAffinityPolicy

spec:
  antiAffinityPolicy: <string>

(Optional) Configures the anti-affinity policy for the Tanzu GemFire cluster. There are implications for data availability when multiple pods reside within a single node. Three possible case-sensitive string values exist:

None allows each node to host multiple server pods and multiple locator pods. This is the policy used if the antiAffinityPolicy is not set.
Cluster allows each node to host multiple server pods and multiple locator pods, but no more than one of each from a given Tanzu GemFire cluster.
Full allows each node to host at most one server pod and at most one locator pod.

If there are not enough nodes available to satisfy the anti-affinity policy, pods will remain in the pending state.

spec.autoscaler.minReplicaCount

  autoscaler: 
    minReplicaCount

The minimum number of servers that the autoscaler can scale down to. The default value is 2, and the minimum value is 1.

spec.autoscaler.maxReplicaCount

  autoscaler: 
    maxReplicaCount

The maximum number of servers that the autoscaler can scale up to. The default value is 2, and the minimum value is 1.

spec.autoscaler.metrics

  autoscaler: 
    metrics

Metrics used to trigger autoscaling. Overrides the defaults set by VMware Tanzu GemFire for Kubernetes.

spec.autoscaler.behavior

  autoscaler: 
    behavior

Configures the behavior of pod autoscaling. Overrides the defaults set by VMware Tanzu GemFire for Kubernetes.

spec.metrics.emission

metrics:
  emission: <string>

(Optional) Specifies the quantity of metrics available to be pulled from the Prometheus /metrics endpoint for both locators and servers on port 4321. When not specified, the default value is Default. Three possible case-sensitive string values exist:

None deactivates metrics availability.
Default makes approximately 200 metrics available for collection at 1-minute intervals.
All makes all approximately 1600 metrics available for collection at 2-second intervals.

spec.metrics.interval

metrics:
  interval: <string>

(Optional) Specifies a time interval at which metrics are available for collection. When set, this value overrides the default interval defined for the metrics.emission field. Accepted value consist of a positive integer followed by one of the unit characters: s for seconds, m for minutes, h for hours, d for days, or w for weeks.

For example, 4m is four minutes, and 90s is 90 seconds.

spec.offPlatform

ingress:
    gatewayName: <string>

(Optional) Specifies the name of the Gateway the operator will use to create TLSRoute objects.

spec.resourceManager

  resourceManager:            
    criticalHeapPercentage: <integer>                    
    evictionHeapPercentage: <integer>

(Optional) Sets the eviction-heap-percentage and critical-heap-percentage properties used by the Tanzu GemFire Resource Manager. The default values are 75 (75%) for eviction-heap-percentage and 90 (90%) for critical-heap-percentage.

A value of 0 deactivates the feature. Each values can be set independently. If both values are set, the eviction-heap-percentage must always be a smaller value than the critical-heap-percentage.

For more information about the Tanzu GemFire Resource Manager, see Using the Tanzu GemFire Resource Manager in Managing Heap Memory in the VMware Tanzu GemFire product documentation.

spec.sysctls

sysctls: <list of sysctls name/value pairs>

(Optional) Configures the sysctls set on GemFireCluster pods. These options will override any options with the same name that are set by default. The configured sysctls must be able to be set independently for each pod on a node (namespaced), and unsafe sysctls must be explicitly allowed. For more information, see Using sysctls in a Kubernetes Cluster. The default setting (shown below) deactivates TCP SYN cookies. While SYN cookies might be used to protect the system against malicious attacks that flood TCP SYN packets, this feature is not compatible with a stable and busy Tanzu GemFire cluster.

spec:
  sysctls:
    - name: "net.ipv4.tcp_syncookies"
      value: "0"

spec.serialization

spec:
  serialization:            
    pdx:                    
      readSerialized: <boolean>

(Optional) When true, enables the readSerialized behavior of cluster members. The default value is true. This value cannot be changed on an existing cluster. When set to false, some operations will require that domain JAR files are deployed to the Tanzu GemFire cluster.

spec.security.tls

spec:
  security:
    tls:
      secretName: <string>
      clientAuthenticationRequired: <boolean>

(Optional) Configures TLS for the Tanzu GemFire cluster. The default value is

spec:
  security:
    tls:
      secretName: <CLUSTER-NAME>-cert
      clientAuthenticationRequired: false

Where CLUSTER-NAME is the name of the Tanzu GemFire cluster (value used for field metadata.name).

To deactivate TLS, set the TLS configuration to the empty struct as in this example:

spec:
  security:
    tls: {}

spec.security.tls.secretName

spec:
  security:
    tls:
      secretName: <string>

(Required if TLS configuration is not empty) Sets the name of the Secret containing tls.crt, tls.key, ca.crt, keystore.p12, truststore.p12, and password keys. If a Secret with this name does not exist in the same namespace as the Tanzu GemFire cluster, a new Secret is created with an auto-generated certificate and trust anchor. This value cannot be changed on an existing cluster.

spec.security.tls.clientAuthenticationRequired

spec:
  security:
    tls:
      clientAuthenticationRequired: <boolean>

(Required if TLS configuration is not empty) Activates or deactivates TLS client authentication on the Tanzu GemFire cluster. This value cannot be changed on an existing cluster.

spec.security.mgmtSvcCredentialsSecretName

spec:
  security:
    mgmtSvcCredentialsSecretName: <string>

(Optional) Configures the name of the Secret containing the credentials that will be used by the gemfire-operator and lifecycle hooks when connecting to the Tanzu GemFire cluster. The field is mandatory when Authentication and Authorization are enabled for the Tanzu GemFire cluster. Since both basic authentication and token based authentication are supported, the Secret must have either the username and password keys, or the token key.

spec.persistentVolumeClaim.resources

persistentVolumeClaim:
  resources:
    requests:
      storage: <storage units>

(Optional) Sets the amount of persistent storage requested for both locators and servers. If unset, defers to spec.locators.persistentVolumeClaim.resources for locators which has a default value of 2Gi and and spec.servers.persistentVolumeClaim.resources for servers which has a default value of 4Gi. spec.persistentVolumeClaim.resources is overwritten by spec.locators.persistentVolumeClaim.resources and spec.servers.persistentVolumeClaim.resources. This value cannot be changed on an existing Tanzu GemFire cluster.

spec.persistentVolumeClaim.storageClassName

persistentVolumeClaim:
  storageClassName: <string>

(Optional) The name of the Kubernetes StorageClass to use when allocating the persistent volume claim. To allocate a persistent volume claim, as needed for Tanzu GemFire cluster servers and locators, the Kubernetes cluster must have a defined StorageClass name. Overwrites option set in spec.persistentVolumeClaim.storageClassName. If your Kubernetes cluster does not define a default StorageClass name, then your CRD must define one. spec.persistentVolumeClaim.storageClassName is overwritten by spec.locators.persistentVolumeClaim.storageClassName and spec.servers.persistentVolumeClaim.storageClassName.

spec.persistentVolumeClaim.deleteOnShutdown

persistentVolumeClaim:
  deleteOnShutdown: <boolean>

(Optional) When true, deletes the persistent volume claims when the Tanzu GemFire cluster is shut down. By default, persistent volume claims will not be deleted on shut down. This value cannot be changed on an existing Tanzu GemFire cluster. spec.persistentVolumeClaim.deleteOnShutdown is overwritten by spec.locators.persistentVolumeClaim.deleteOnShutdown and spec.servers.persistentVolumeClaim.deleteOnShutdown.

Member Configuration

The fields in this section apply configuration to the set of locators or the set of servers. These fields are defined under spec.locators and spec.servers.

replicas

replicas: <integer>

(Optional) Sets the number of locators and servers in the cluster. The number must be greater than 0. The default number of locators is 1, and the default number of servers is 2. The number locators (spec.locators.replicas) can not be changed on an existing cluster.

labels

labels: <dictionary of key/value pairs>

(Optional) Adds specified labels to locators and servers in the cluster without risk of overriding others. There is no default defined.

For example:

apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
  name: <CLUSTER-NAME>
spec:
  image: <IMAGE-NAME>
  locators:
    labels:
      environment: production
      tier: premium
  servers:
    labels:
      environment: production
      tier: premium

annotations

annotations: <dictionary of key/value pairs>

(Optional) Adds specified annotations to locators and servers in the cluster, allowing for any existing annotations to be overridden. There is no default defined.

For example:

apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
  name: <CLUSTER-NAME>
spec:
  image: <IMAGE-NAME>
  locators:
    annotations:
      issue-tracking: "https://my-issue-tracking-system/"
      last-modified: 2021-07-01
  servers:
    annotations:
      issue-tracking: "https://my-issue-tracking-system/"
      last-modified: 2021-07-01

libraries

libraries: <list of objects>
  - name: <string>
    container:
      image: <string>
      path: <string>
      imagePullSecretRef: <reference>

(Optional) Specifies a list of custom jars to be added to the member’s classpath.

The table below describes the fields in more detail.

Field	Required	Description
`name`	Yes	Arbitrary ID. Must follow the syntax rules for a DNS name and be unique per member.
`image`	Yes	Image reference, from a public or private image registry. The image must have a system shell executable through `/bin/sh`.
`path`	Yes	Location, within the image, from which to copy the libraries. Can be a single file or an expression resolvable by the system shell.
`imagePullSecretRef`	No	Secret reference to be used as the `imagePullSecret` when pulling the image. Required only when the `image` is hosted in a private registry that requires authentication.

For example:

apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
  name: <CLUSTER-NAME>
spec:
  image: <IMAGE-NAME>
  locators:
    libraries:
      - name: my-custom-security-manager
        container:
          image: my.company.com/myImage:1.0.0
          path: "/build/libs/*.jar"
          imagePullSecretRef:
            name: pull-secret-name
  servers:
    libraries:
      - name: my-custom-security-manager
        container:
          image: my.company.com/myImage:1.0.0
          path: "/build/libs/*.jar"
          imagePullSecretRef:
            name: pull-secret-name

resources.requests.memory

resources:
  requests:
    memory: <memory units>

(Optional) Specifies the amount of memory a locator or a server needs. The default and minimum allowed value is 1Gi. This value affects how Kubernetes schedules and runs the pods for a Tanzu GemFire cluster, and it also determines the quantity of memory that will be specified for garbage collection memory spaces within the JVM. To improve performance, follow the recommendations in Tuning the JVM’s Garbage Collection Parameters in Managing Heap and Off-heap Memory in the VMware Tanzu GemFire documentation.

In general, the total heap size, Xmx/Xms, is set to 80% of the specified resources.requests.memory, and the new generation space size, NewSize/MaxNewSize, is set to 20% of the total heap size. The non-heap size must be a minimum of 350Mi, and the new generation space size is capped to a maximum value of 4Gi. This table lists how Xmx/Xms and NewSize/MaxNewSize are configured based on the value of resources.requests.memory:

`resources.requests.memory`	Value set for `Xmx/Xms`	Value set for `NewSize/MaxNewSize`
Less than `1Gi`	N/A (Error)	N/A (Error)
At least `1750Mi` but less than `20Gi`	80% of `resources.requests.memory`	20% of `Xmx/Xms`
`20Gi` or more	80% of `resources.requests.memory`	`4Gi`

For more information about how Kubernetes schedules and runs pods, see the following in the Kubernetes documentation:

For information about the format of the memory units, see Resource units in Kubernetes in the Kubernetes documentation.

resources.limits.memory

resources:
  limits:
    memory: <memory units>

(Optional) Specifies the memory limit of each container. This value affects how Kubernetes schedules and runs pods for a Tanzu GemFire cluster. If not specified, no limits is set on the containers.

For more information about how Kubernetes schedules and runs pods, see the following in the Kubernetes documentation:

For information about the format of the memory units, see Resource units in Kubernetes in the Kubernetes documentation.

resources.requests.cpu and resources.limits.cpu

resources:
  requests:
    cpu: <cpu units>
  limits:
    cpu: <cpu units>

(Optional) Specifies the CPU resource needs of each container. The value affects how Kubernetes schedules and runs the pods for a Tanzu GemFire cluster. If not specified, no CPU requests or limits are set on the containers.

For information about how Kubernetes schedules and runs pods, see the following in the Kubernetes documentation:

For information about the format of the CPU units, see Resource units in Kubernetes in the Kubernetes documentation.

persistentVolumeClaim.resources

persistentVolumeClaim:
  resources:
    requests:
      storage: <storage units>

(Optional) Sets the amount of persistent storage requested for each member. The default value for locators is 2Gi and the default value for servers is 4Gi. Overwrites option set in spec.persistentVolumeClaim.resources. This value cannot be changed on an existing Tanzu GemFire cluster.

persistentVolumeClaim.storageClassName

persistentVolumeClaim:
  storageClassName: <string>

persistentVolumeClaim.deleteOnShutdown

persistentVolumeClaim:
  deleteOnShutdown: <boolean>

(Optional) When true, deletes the persistent volume claims when the Tanzu GemFire cluster is shut down. By default, persistent volume claims are not deleted on shut down. Overwrites option set in spec.persistentVolumeClaim.deleteOnShutdown. This value cannot be changed on an existing Tanzu GemFire cluster.

overrides.gemFireProperties

overrides:
  gemFireProperties: <list of GemFireProperties name/value pairs>

(Optional) Sets additional Tanzu GemFire properties that will be applied to locators and servers at start up. These options will override any options with the same name that are set by default.

For example, to enable the Tanzu GemFire Developer REST API, set the start-dev-rest-api property for servers:

overrides:
  gemFireProperties:
    - name: "start-dev-rest-api"
      value: "true"

Note: The following properties cannot be overridden as they are automatically configured when Transport Layer Security is enabled; the deployment will be rejected when trying do so:

ssl-keystore
ssl-keystore-password
ssl-truststore
ssl-truststore-password
ssl-enabled-components
ssl-require-authentication
ssl-web-require-authentication
ssl-endpoint-identification-enabled

Note: The following properties cannot be overridden on locators as they are automatically configured and required by the gemfire-operator; the deployment will be rejected when trying do so:

http-service-port
http-service-bind-address
jmx-manager-start
jmx-manager-port
jmx-manager-hostname-for-clients

overrides.jvmOptions

overrides:
  jvmOptions: <list of strings>

(Optional) If defined, the specified JVM options will be appended to the end of the JVM options list used during member startup. If multiple values for the same JVM option are specified in the list, the last specified value takes precedence over the former values.

This example overrides the initial amount of memory allocated:

apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
  name: <CLUSTER-NAME>
spec:
  image: <IMAGE-NAME>
  locators:
    overrides:
      jvmOptions:
        - "-Xms170m"
  servers:
    overrides:
      jvmOptions:
        - "-Xms170m"

Note: Using this field to configure Tanzu GemFire properties is not recommended. Instead, use the overrides.gemFireProperties.

overrides.statefulSet

overrides:
  statefulSet:
    spec:
      template:
        metadata:
          name: <string>
          namespace: <string>
          labels: <list of strings>
          annotations: <list of strings>
        spec: <Kubernetes pod template spec>

(Optional) Represents a set of pods with consistent identities. The StatefulSet consists of a subset of fields that define a set of values that override defaults.

This example imposes an override for pod node affinity:

apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
  name: <CLUSTER-NAME>
spec:
  image: <IMAGE-NAME>
  locators:
    overrides:
      statefulSet:
        spec:
          template:
            spec:
              affinity:
                nodeAffinity:
                  preferredDuringSchedulingIgnoredDuringExecution:
                  - weight: 1
                    preference:
                      matchExpressions:
                      - key: disktype
                        operator: In
                        values:
                        - mechanical
              containers: []
  servers:
    overrides:
      statefulSet:
        spec:
          template:
            spec:
              affinity:
                nodeAffinity:
                  preferredDuringSchedulingIgnoredDuringExecution:
                  - weight: 1
                    preference:
                      matchExpressions:
                      - key: disktype
                        operator: In
                        values:
                        - ssd
              containers: []

Warning: It is possible to override fields in a manner that leaves the Tanzu GemFire cluster in a broken state.

Note: The following volume names cannot be used when adding extra Kubernetes Volumes to the overridden statefulSet. Attempts to add any of the names below will result in the deployment being rejected.

init
data
certs
security
extensions

Note: The following paths cannot be used when mounting extra resources into locator or server container. Attempts to add any of the paths below will result in the deployment being rejected.

“/init”
“/data”
“/certs”
“/security”
“/gemfire/extensions”