Work with a VMware GemFire Cluster

This topic describes how to work with a VMware GemFire cluster when using VMware GemFire for Kubernetes.

Once a GemFire for Kubernetes cluster has been created, gfsh commands aid the administration and maintenance of the cluster.

Many gfsh commands require that a session be connected to the GemFire for Kubernetes cluster before the operation can be completed.

Connect to the VMware GemFire Cluster

Use one of three ways to connect to a locator in order to issue gfsh commands to the Kubernetes-deployed GemFire for Kubernetes cluster.

The three ways to connect:

Use an interactive gfsh session. Requires Kubernetes cluster exec privilege.
Use kubectl to invoke a gfsh command. Requires Kubernetes cluster exec privilege.
Use a Kubernetes LoadBalancer service to permit gfsh GemFire for Kubernetes cluster access via the Management API. The exposed endpoint enables an interactive gfsh session.

TLS prerequisites

Some of the gfsh commands below require a truststore and keystore password if TLS is enabled on the GemFire for Kubernetes cluster. Under the default TLS configuration, the truststore and keystore passwords are the same, and may be obtained using the following command:

kubectl -n NAMESPACE-NAME get secret NAME-cert -o=jsonpath='{.data.password}' | base64 --decode

where NAMESPACE-NAME is your chosen name for the GemFire for Kubernetes cluster namespace, and the cluster NAME is the metadata: name field from the deployment YAML.

Interactive `gfsh` session

With a Kubernetes cluster for which you have exec privilege, use kubectl to start an interactive gfsh session with the pod running the locator. Use a command of the form:

kubectl -n NAMESPACE-NAME exec -it NAME-locator-0 -- gfsh

where NAMESPACE-NAME is your chosen name for the GemFire for Kubernetes cluster namespace, and the cluster NAME is the metadata: name field from the deployment YAML.

For example, if the GemFire for Kubernetes cluster namespace is gemfire-cluster and the cluster NAME is gemfire1:

$ kubectl -n gemfire-cluster exec -it gemfire1-locator-0 -- gfsh
    _________________________     __
   / _____/ ______/ ______/ /____/ /
  / /  __/ /___  /_____  / _____  / 
 / /__/ / ____/  _____/ / /    / /  
/______/_/      /______/_/    /_/ 

Monitor and Manage VMware GemFire
gfsh>

Connect with the GemFire for Kubernetes cluster:

gfsh>connect --locator=NAME-locator-0.NAME-locator.NAMESPACE-NAME.svc.DOMAIN[10334] --trust-store=/certs/truststore.p12 --trust-store-password=TLS_PASSWORD --key-store=/certs/keystore.p12 --key-store-password=TLS_PASSWORD

where NAMESPACE-NAME is your chosen name for the GemFire for Kubernetes cluster namespace, NAME is the metadata: name field from the deployment YAML, DOMAIN is the domain of the Kubernetes cluster (defaults to “cluster.local”), and TLS_PASSWORD is the password obtained from TLS prerequisites.

Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

Note: The --trust-store and --trust-store-password flags are required only if TLS is enabled. The --key-store and --key-store-password flags are required only if TLS is enabled with client authentication.

Once connected, issue gfsh commands as desired.

Exit the interactive gfsh session with exit:

gfsh>exit
Exiting...

Use kubectl to Invoke a gfsh Command

With a Kubernetes cluster for which you have exec privilege, use kubectl to invoke a single gfsh command on the GemFire for Kubernetes cluster. This method of executing gfsh commands can be useful for scripting cluster operations. The form of the kubectl command is:

kubectl -n NAMESPACE-NAME exec -it NAME-locator-0 -- gfsh -e 'connect --locator=NAME-locator-0.NAME-locator.NAMESPACE-NAME.svc.DOMAIN[10334] --trust-store=/certs/truststore.p12 --trust-store-password=TLS_PASSWORD --key-store=/certs/keystore.p12 --key-store-password=TLS_PASSWORD' -e "GFSH-COMMAND"

For example, if the GemFire for Kubernetes cluster namespace is gemfire-cluster:

the cluster NAME is gemfire1
the cluster DOMAIN is cluster.local
the username USERNAME is user
the password PASSWORD is pass
the key/trust store TLS_PASSWORD is password
the desired gfsh command is list members

the kubectl command to list the members of the GemFire for Kubernetes cluster is:

$ kubectl -n gemfire-cluster exec -it gemfire1-locator-0 -- gfsh -e 'connect --locator=gemfire1-locator-0.gemfire1-locator.gemfire-cluster.svc.cluster.local[10334] --trust-store=/certs/truststore.p12 --trust-store-password=password --key-store=/certs/keystore.p12 --key-store-password=password' -e "list members"

Use a LoadBalancer and Target the Management API

The GemFire for Kubernetes Management API can be used to issue gfsh commands to the GemFire for Kubernetes cluster.

Note: If TLS is enabled on a production GemFire for Kubernetes cluster, the cluster must use a custom TLS certificate which contains the LoadBalancer’s external IP as a Subject Alternative Name (SAN) IP Address entry, or a DNS name which resolves to that external IP address as a SAN DNS entry. For non-production environments, it is possible to skip TLS certificate validation. (See step 5 below)

Follow these steps to connect to the GemFire for Kubernetes cluster:

Create a LoadBalancer service as directed in Create a LoadBalancer Service for the Management API.
Acquire and install gfsh for the machine that will issue the gfsh commands. The gfsh executable is located in the bin directory of expanded VMware GemFire binary downloaded from VMware Tanzu Network. If more than one version of VMware GemFire is available, see the Release Notes to determine the appropriate version of VMware GemFire to acquire.
Acquire the EXTERNAL-IP value to use in the gfsh connect command in step 5. Run the command:
```
kubectl get service LB-SVC-MGMT-API
```
where LB-SVC-MGMT-API is the metadata: name field from the LoadBalancer service’s configuration YAML. The value is the EXTERNAL-IP field from the command’s output.
Run gfsh.
At the gfsh prompt, connect to the GemFire for Kubernetes cluster. Include a -use-http option in the connect command:
```
gfsh>connect --locator --use-http=true --url=https://EXTERNAL-IP:7070/gemfire/v1
```
where EXTERNAL-IP is the acquired value from step 3.

Note: If TLS is deactivated, change https to http in the above command.

Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

Note: When working with a non-production, development environment with TLS enabled, a developer may choose to work in a less secure manner by passing the --skip-ssl-validation flag to the gfsh connect command. When this flag is passed, the keystore path, keystore password, truststore path, and truststore password may be left blank.

Note: If prompted, enter the KEYSTORE_PATH, KEYSTORE_PASSWORD, TRUSTSTORE_PATH, and TRUSTSTORE_PASSWORD as in this example:
```
key-store: KEYSTORE_PATH
key-store-password: KEYSTORE_PASSWORD
key-store-type(default: JKS):
trust-store: TRUSTSTORE_PATH
trust-store-password: TRUSTSTORE_PASSWORD
trust-store-type(default: JKS):
ssl-ciphers(default: any):
ssl-protocols(default: any):
ssl-enabled-components(default: all):
```

Create a Region

Create regions once connected through gfsh. Specify the name of the region and the type of the region in the command. For example:

gfsh>create region --name=customers --type=PARTITION_REDUNDANT

For details, see the gfsh create region command in the VMware GemFire documentation.

Note: If SecurityManager is enabled, the user must have the DATA:MANAGE Resource Permissions assigned.

Obtain Logs and VMware GemFire Cluster State

Kubernetes Pod Logs

Kubernetes pod logs include the logged events from the GemFire for Kubernetes cluster and the startup of the pod. To print a Kubernetes pod log to standard output, use a command of the form:

$ kubectl -n NAMESPACE-NAME logs POD-NAME

where NAMESPACE-NAME is your chosen name for the GemFire for Kubernetes cluster namespace, and POD-NAME is composed as pod/NAME-locator-N or pod/NAME-server-N. The NAME is the metadata: name field, and N identifies which locator or server is desired.

VMware GemFire Cluster Logs

If the GemFire cluster is healthy, follow the instructions in Healthy Cluster Logs. If the GemFire cluster is unhealthy, the gfsh export logs command may hang or not work. Follow the instructions in Unhealthy Cluster Logs.

Healthy Cluster Logs

The gfsh export logs command places the GemFire for Kubernetes cluster logs on the locator-0 pod. The created ZIP file should be copied to the local machine for viewing the artifacts. The locator-0 pod must have sufficient disk space to capture the logs and statistics files from all the GemFire for Kubernetes locators and servers.

Use gfsh to Connect to the Instance with an interactive gfsh session.
Run the export logs command:
```
gfsh>export logs
Logs exported to the connected member's file system: /data/exportedLogs_XXX.zip
gfsh>exit
```
Note: If SecurityManager is enabled, the user must have the CLUSTER:READ Resource Permissions assigned.

The logs will be in a file named similar to exportedLogs_XXX.zip. Output from the command will provide the exact name of the file; use the exact name in the following commands.
Copy the logs from the locator-0 pod to the local machine:
```
$ kubectl -n NAMESPACE-NAME cp NAME-locator-0:exportedLogs_XXX.zip exportedLogs_XXX.zip
```
where NAMESPACE-NAME is your chosen name for the GemFire for Kubernetes cluster namespace, and NAME is the metadata: name field from the deployment YAML.
Unzip the logs:
```
$ unzip exportedLogs_XXX.zip
```

Unhealthy Cluster Logs

If the GemFire cluster is unhealthy, the gfsh export logs command may hang or not work. Follow these instructions to manually gather logs.

For each member of the cluster for which you want to gather logs, run:
```
kubectl -n NAMESPACE_NAME cp NAME-MEMBER_TYPE-X:logsAndStats LOCAL_FOLDER
```
Where:
- NAMESPACE-NAME is the namespace on which the GemFire cluster is deployed.
- NAME is the name of the cluster name.
- MEMBER_TYPE is the type of member. Valid values are locator and server.
- X is the member index, an integer value between 0 and memberType.replicas - 1.
- logsAndStats is created by default on every member and contains the logs, statistics, and gc-logs for that particular member.
- LOCAL_FOLDER is the local folder to which copy the files.
Examples:
```
kubectl -n default cp gemfire-cluster-locator-0:logsAndStats hang-debug-locator0
kubectl -n gemfire-system cp sample-cluster-server-3:logsAndStats logsStats-server3
```

Collect and View Metrics

Use VMware Tanzu Observability by Wavefront

The VMware Tanzu Observability by Wavefront product integrates seamlessly with GemFire for Kubernetes clusters to both collect GemFire for Kubernetes cluster metrics and view those metrics via a dashboard.

There are three parts to the integration.

You can install the Wavefront proxy and Kubernetes Metrics Collector before or after the creation of a GemFire for Kubernetes cluster.

Install the Proxy and Kubernetes Metrics Collector by following the instructions at Kubernetes Setup. The installation will be either with Tanzu Mission Control or with Helm.
- For a Tanzu Mission Control installation:
  1. Enable the use of Tanzu Observability for your organization by following the directions at Enable Observability for Your Organization.
  2. Install a configured Wavefront proxy and Kubernetes Metrics Collector on your Kubernetes cluster by following the directions at Add a Cluster to Observability.
- For a Helm installation:
  1. Acquire an API token that will allow Tanzu Observability to authenticate communication from the Wavefront Proxy. Follow the directions at Generating an API Token.
  2. Use the API token as you install the Wavefront proxy and Kubernetes Metrics Collector using Helm following the directions at Kubernetes Quick Install Using Helm.
When you create the GemFire for Kubernetes cluster, it emits metrics by default. Optionally, you can specify a much higher quantity of metrics to be emitted at a higher frequency by setting spec.metrics within the CRD.
Create or integrate a Wavefront dashboard. Sample dashboards are available in the https://github.com/vmware-samples/tanzu-gemfire-observability-sample repository.