In VMware GemFire for Kubernetes, the Custom Resource Definition (CRD) is a YAML file that defines a VMware GemFire cluster named GemFireCluster and specifies its characteristics.
CRD Template
The CRD file specifies fields and values that define the VMware GemFire cluster API. The template below lists the fields that you can set for a VMware GemFire cluster.
Details are provided after the template.
apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
name: <string>
namespace: <string>
spec:
serviceAccountName: <string>
image: <string>
imagePullSecrets: <list of references>
antiAffinityPolicy: <string>
metrics:
emission: <string>
interval: <string>
resourceManager:
criticalHeapPercentage: <integer>
evictionHeapPercentage: <integer>
sysctls: <list of sysctls name/value pairs>
serialization:
pdx:
readSerialized: <boolean>
security.tls
secretName: <string>
clientAuthenticationRequired: <boolean>
security:
tls:
secretName: <string>
clientAuthenticationRequired: <boolean>
mgmtSvcCredentialsSecretName: <string>
locators:
replicas: <integer>
labels: <dictionary of key/value pairs>
annotations: <dictionary of key/value pairs>
libraries: <list of objects>
- name: <string>
container:
image: <string>
path: <string>
imagePullSecretRef: <reference>
resources:
requests:
memory: <memory units>
cpu: <cpu units>
limits:
memory: <memory units>
cpu: <cpu units>
persistentVolumeClaim:
resources:
requests:
storage: <storage units>
storageClassName: <string>
overrides:
jvmOptions: <list of strings>
gemFireProperties: <list of GemFireProperties name/value pairs>
statefulSet:
spec:
template:
metadata:
name: <string>
namespace: <string>
labels: <list of strings>
annotations: <list of strings>
spec: <Kubernetes pod template spec>
servers:
replicas: <integer>
labels: <dictionary of key/value pairs>
annotations: <dictionary of key/value pairs>
libraries: <list of objects>
- name: <string>
container:
image: <string>
path: <string>
imagePullSecretRef: <reference>
resources:
requests:
memory: <memory units>
cpu: <cpu units>
limits:
memory: <memory units>
cpu: <cpu units>
persistentVolumeClaim:
resources:
requests:
storage: <storage units>
storageClassName: <string>
overrides:
jvmOptions: <list of strings>
gemFireProperties: <list of GemFireProperties name/value pairs>
statefulSet:
spec:
template:
metadata:
name: <string>
namespace: <string>
labels: <list of strings>
annotations: <list of strings>
spec: <Kubernetes pod template spec>
Fields and Values
The first two fields establish the nature of the CRD and its name. For this release of GemFire for Kubernetes, these should always be the same:
apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
Instance Metadata
The fields in this section apply metadata to the GemFire for Kubernetes cluster.
metadata.name
metadata:
name: <string>
(Required) Sets the name of the GemFireCluster cluster. This value cannot be changed on an existing cluster.
metadata.namespace
metadata:
namespace: <string>
(Optional) Sets the Kubernetes namespace in which the GemFireCluster cluster will be deployed. If not specified, the current kubectl context’s namespace will be used. This value cannot be changed on an existing cluster.
Cluster Configuration
The fields in this section apply configuration to the entire GemFire for Kubernetes cluster.
spec.serviceAccountName
spec:
serviceAccountName: <string>
(Optional) Specifies the Kubernetes Service Account to be configured by the gemfire-operator within all pods for the GemFire for Kubernetes cluster. When not specified, the default Service Account for the namespace is automatically configured by Kubernetes itself.
spec.image
spec:
image: <string>
(Required) Specifies the image to be used by the gemfire-operator when creating pods for the GemFire for Kubernetes cluster; therefore, it specifies a cluster version. Updating this value will trigger an upgrade to a new version of the GemFire for Kubernetes cluster. Downgrades to a previous version are not supported, and will result in a failure of the first locator to restart.
spec.imagePullSecrets
spec:
imagePullSecrets: <list of references>
(Optional) Specifies the list of names for image pull secrets in this GemFire for Kubernetes cluster’s namespace. Required if more than one registry holds images that are needed for creating a GemFire for Kubernetes cluster. When not specified, the name of the single image pull secret used during the application of the CRD for creating a GemFire for Kubernetes cluster is image-pull-secret.
Include each image pull secret’s name in this list of references, as in this example with two image pull secrets:
spec:
imagePullSecrets:
- name: "secret-1"
- name: "secret-2"
spec.antiAffinityPolicy
spec:
antiAffinityPolicy: <string>
(Optional) Configures the anti-affinity policy for the GemFire for Kubernetes cluster. There are implications for data availability when multiple pods reside within a single node. There are three possible case-sensitive string values:
Noneallows each node to host multiple server pods and multiple locator pods. This is the policy used if theantiAffinityPolicyis not set.Clusterallows each node to host multiple server pods and multiple locator pods, but no more than one of each from a given GemFire for Kubernetes cluster.Fullallows each node to host at most one server pod and at most one locator pod.
If there are not enough nodes available to satisfy the anti-affinity policy, pods will remain in the pending state.
spec.metrics.emission
metrics:
emission: <string>
(Optional) Specifies the quantity of metrics available to be pulled from the Prometheus /metrics endpoint for both locators and servers on port 4321. When not specified, the default value is Default. There are three possible case-sensitive string values:
Nonedeactivates metrics availability.Defaultmakes approximately 200 metrics available for collection at 1-minute intervals.Allmakes all approximately 1600 metrics available for collection at 2-second intervals.
spec.metrics.interval
metrics:
interval: <string>
(Optional) Specifies a time interval at which metrics are available for collection. When set, this value overrides the default interval defined for the metrics.emission field. An accepted value is a positive integer followed by one of the unit characters: s for seconds, m for minutes, h for hours, d for days, or w for weeks.
For example, 4m is four minutes, and 90s is 90 seconds.
spec.resourceManager
resourceManager:
criticalHeapPercentage: <integer>
evictionHeapPercentage: <integer>
(Optional) Sets the eviction-heap-percentage and critical-heap-percentage properties used by the GemFire for Kubernetes Resource Manager . The default values are 75 (75%) for eviction-heap-percentage and 90 (90%) for critical-heap-percentage. A value of zero deactivates the feature. Both values may be set independently; if both are set, the eviction-heap-percentage must always be a smaller value than the critical-heap-percentage.
spec.sysctls
sysctls: <list of sysctls name/value pairs>
(Optional) Configures the sysctls set on GemFireCluster pods. These options will override any options with the same name that are set by default. The configured sysctls must be able to be set independently for each pod on a node (namespaced), and unsafe sysctls must be explicitly allowed. For more information, see Using sysctls in a Kubernetes Cluster. The default setting (shown below) deactivates TCP SYN cookies. While SYN cookies might be used to protect the system against malicious attacks that flood TCP SYN packets, this feature is not compatible with a stable and busy GemFire for Kubernetes cluster.
spec:
sysctls:
- name: "net.ipv4.tcp_syncookies"
value: "0"
spec.serialization
spec:
serialization:
pdx:
readSerialized: <boolean>
(Optional) When true, enables the readSerialized behavior of cluster members. The default value is true. This value cannot be changed on an existing cluster. When false, some operations will require that domain JAR files are deployed to the GemFire for Kubernetes cluster.
spec.security.tls
spec:
security:
tls:
secretName: <string>
clientAuthenticationRequired: <boolean>
(Optional) Configures TLS for the GemFire for Kubernetes cluster. The default value is
spec:
security:
tls:
secretName: <CLUSTER-NAME>-cert
clientAuthenticationRequired: false
where CLUSTER-NAME is the name of the GemFire for Kubernetes cluster (value used for field metadata.name).
To deactivate TLS, set the TLS configuration to the empty struct as in this example:
spec:
security:
tls: {}
spec.security.tls.secretName
spec:
security:
tls:
secretName: <string>
(Required if TLS configuration is not empty) Sets the name of the Secret containing tls.crt, tls.key, ca.crt , keystore.p12, truststore.p12, and password keys. If a Secret with this name does not exist in the same namespace as the GemFire for Kubernetes cluster, a new Secret will be created with an auto-generated certificate and trust anchor. This value cannot be changed on an existing cluster.
spec.security.tls.clientAuthenticationRequired
spec:
security:
tls:
clientAuthenticationRequired: <boolean>
(Required if TLS configuration is not empty) Turns on or off TLS client authentication on the GemFire for Kubernetes cluster. This value cannot be changed on an existing cluster.
spec.security.mgmtSvcCredentialsSecretName
spec:
security:
mgmtSvcCredentialsSecretName: <string>
(Optional) Configures the name of the Secret containing the credentials that will be used by the gemfire-operator and lifecycle hooks when connecting to the GemFire for Kubernetes cluster. The field is mandatory when Authentication and Authorization are enabled for the GemFire for Kubernetes cluster. Since both basic authentication and token based authentication are supported, the Secret must have either the username and password keys, or the token key.
Member Configuration
The fields in this section apply configuration to the set of locators or the set of servers. These fields are defined under spec.locators and spec.servers.
replicas
replicas: <integer>
(Optional) Sets the quantity of locators and servers in the cluster. The quantity must be greater than zero. The default quantity of locators is one, and the default quantity of servers is two. The amount of locators (spec.locators.replicas) can not be changed on an existing cluster.
labels
labels: <dictionary of key/value pairs>
(Optional) Adds specified labels to locators and servers in the cluster without risk of overriding others. There is no default defined.
For example:
apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
name: <CLUSTER-NAME>
spec:
image: <IMAGE-NAME>
locators:
labels:
environment: production
tier: premium
servers:
labels:
environment: production
tier: premium
annotations
annotations: <dictionary of key/value pairs>
(Optional) Adds specified annotations to locators and servers in the cluster, allowing for any existing annotations to be overridden. There is no default defined.
For example:
apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
name: <CLUSTER-NAME>
spec:
image: <IMAGE-NAME>
locators:
annotations:
issue-tracking: "https://my-issue-tracking-system/"
last-modified: 2021-07-01
servers:
annotations:
issue-tracking: "https://my-issue-tracking-system/"
last-modified: 2021-07-01
libraries
libraries: <list of objects>
- name: <string>
container:
image: <string>
path: <string>
imagePullSecretRef: <reference>
(Optional) Specifies a list of custom jars to be added to the member’s classpath.
The table below describes the fields in more detail.
| Field | Required | Description |
|---|---|---|
name |
Yes | Arbitrary ID, must follow the syntax rules for a DNS name and be unique per member |
image |
Yes | Image reference, from a public or private image registry. The image must have the Bourne-again SHell (bash) installed and executable through /bin/bash |
path |
Yes | Location, within the image, from which to copy the libraries. Can be a single file or an expression resolvable by the Bourne-again SHell (bash) |
imagePullSecretRef |
No | Secret reference to be used as the imagePullSecret when pulling the image. Required only when the image is hosted in a private registry that requires authentication |
For example:
apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
name: <CLUSTER-NAME>
spec:
image: <IMAGE-NAME>
locators:
libraries:
- name: my-custom-security-manager
container:
image: my.company.com/myImage:1.0.0
path: "/build/libs/*.jar"
imagePullSecretRef:
name: pull-secret-name
servers:
libraries:
- name: my-custom-security-manager
container:
image: my.company.com/myImage:1.0.0
path: "/build/libs/*.jar"
imagePullSecretRef:
name: pull-secret-name
resources.requests.memory
resources:
requests:
memory: <memory units>
(Optional) Specifies the amount of memory a locator or a server needs. The default and minimum allowed value is 1Gi. This value affects how Kubernetes schedules and runs the pods for a GemFire for Kubernetes cluster, and it also determines the quantity of memory that will be specified for garbage collection memory spaces within the JVM. To improve performance, follow the recommendations in Tuning the JVM’s Garbage Collection Parameters in Managing Heap and Off-heap Memory in the VMware GemFire documentation.
In general, the total heap size, Xmx/Xms, is set to 80% of the specified resources.requests.memory, and the new generation space size, NewSize/MaxNewSize, is set to 20% of the total heap size. The non-heap size must be a minimum of 350Mi, and the new generation space size is capped to a maximum value of 4Gi. This table lists how Xmx/Xms and NewSize/MaxNewSize are configured based on the value of resources.requests.memory:
resources.requests.memory |
Value set for Xmx/Xms |
Value set for NewSize/MaxNewSize |
|---|---|---|
Less than 1Gi |
N/A (Error) | N/A (Error) |
At least 1750Mi but less than 20Gi |
80% of resources.requests.memory |
20% of Xmx/Xms |
20Gi or more |
80% of resources.requests.memory |
4Gi |
For more information on how Kubernetes schedules and runs pods, see How Pods with resource requests are scheduled and How Pods with resource limits are run. For information about the format of the memory units, see Resource units in Kubernetes.
resources.limits.memory
resources:
limits:
memory: <memory units>
(Optional) Specifies the memory limit of each container. This value affects how Kubernetes schedules and runs pods for a GemFire for Kubernetes cluster. If not specified, no limits will be set on the containers. For more information about how Kubernetes schedules and runs pods, see How Pods with resource requests are scheduled and How Pods with resource limits are run. For information about the format of the memory units, see Resource units in Kubernetes.
resources.requests.cpu and resources.limits.cpu
resources:
requests:
cpu: <cpu units>
limits:
cpu: <cpu units>
(Optional) Specifies the CPU resource needs of each container. The value affects how Kubernetes schedules and runs the pods for a GemFire for Kubernetes cluster. If not specified, no CPU requests or limits will be set on the containers. For more information about how Kubernetes schedules and runs pods, see How Pods with resource requests are scheduled and How Pods with resource limits are run. For information about the format of the CPU units, see Resource units in Kubernetes.
persistentVolumeClaim.resources
persistentVolumeClaim:
resources:
requests:
storage: <storage units>
(Optional) Sets the amount of persistent storage requested for each member. The default value for locators is 2G and the default value for servers is 4G. This value cannot be changed on an existing GemFire for Kubernetes cluster.
persistentVolumeClaim.storageClassName
persistentVolumeClaim:
storageClassName: <string>
(Optional) The name of the Kubernetes StorageClass to use when allocating the persistent volume claim. To allocate a persistent volume claim, as needed for GemFire for Kubernetes cluster servers and locators, the Kubernetes cluster must have a defined StorageClass name. If your Kubernetes cluster does not define a default StorageClass name, then your CRD must define one.
overrides.gemFireProperties
overrides:
gemFireProperties: <list of GemFireProperties name/value pairs>
(Optional) Sets additional GemFire properties that will be applied to locators and servers at start up. These options will override any options with the same name that are set by default.
For example, to enable the GemFire for Kubernetes Developer REST API, set the start-dev-rest-api property for servers:
overrides:
gemFireProperties:
- name: "start-dev-rest-api"
value: "true"
Note: The following properties cannot be overridden as they are automatically configured when Transport Layer Security is enabled; the deployment will be rejected when trying do so:
- ssl-keystore
- ssl-keystore-password
- ssl-truststore
- ssl-truststore-password
- ssl-enabled-components
- ssl-require-authentication
- ssl-web-require-authentication
- ssl-endpoint-identification-enabled
Note: The following properties cannot be overridden on locators as they are automatically configured and required by the gemfire-operator; the deployment will be rejected when trying do so:
- http-service-port
- http-service-bind-address
- jmx-manager-start
- jmx-manager-port
- jmx-manager-hostname-for-clients
overrides.jvmOptions
overrides:
jvmOptions: <list of strings>
(Optional) If defined, the specified JVM options will be appended to the end of the JVM options list used during member startup. If multiple values for the same JVM option are specified in the list, the last value will take precedence over the former value(s).
This example overrides the initial amount of memory allocated:
apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
name: <CLUSTER-NAME>
spec:
image: <IMAGE-NAME>
locators:
overrides:
jvmOptions:
- "-Xms170m"
servers:
overrides:
jvmOptions:
- "-Xms170m"
Note: Using this field to configure GemFire for Kubernetes properties is not recommended; use the overrides.gemFireProperties field instead.
overrides.statefulSet
overrides:
statefulSet:
spec:
template:
metadata:
name: <string>
namespace: <string>
labels: <list of strings>
annotations: <list of strings>
spec: <Kubernetes pod template spec>
(Optional) Represents a set of pods with consistent identities. The StatefulSet consists of a subset of fields that define a set of values that override defaults.
This example imposes an override for pod node affinity:
apiVersion: gemfire.vmware.com/v1
kind: GemFireCluster
metadata:
name: <CLUSTER-NAME>
spec:
image: <IMAGE-NAME>
locators:
overrides:
statefulSet:
spec:
template:
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: disktype
operator: In
values:
- mechanical
containers: []
servers:
overrides:
statefulSet:
spec:
template:
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: disktype
operator: In
values:
- ssd
containers: []
Note: It is possible to override fields in a manner that leaves the GemFire for Kubernetes cluster in a broken state.
Note: The following volume names cannot be used when adding extra Kubernetes Volumes to the overridden statefulSet; the deployment will be rejected when trying do so:
- init
- data
- certs
- security
- extensions
Note: The following paths cannot be used when mounting extra resources into locator or server container; the deployment will be rejected when trying do so:
- “/init”
- “/data”
- “/certs”
- “/security”
- “/gemfire/extensions”
