This topic describes how to work with a Tanzu GemFire cluster when using VMware Tanzu GemFire for Kubernetes.

Once a Tanzu GemFire cluster has been created, gfsh commands aid the administration and maintenance of the cluster.

Many gfsh commands require that a session be connected to the Tanzu GemFire cluster before the operation can be completed.

Connect to the Tanzu GemFire Cluster

Use one of three ways to connect to a locator in order to issue gfsh commands to the Kubernetes-deployed Tanzu GemFire cluster.

The three ways to connect:

  • Use an interactive gfsh session. Requires Kubernetes cluster exec privilege.
  • Use kubectl to invoke a gfsh command. Requires Kubernetes cluster exec privilege.
  • Use a Kubernetes LoadBalancer service to permit gfsh Tanzu GemFire cluster access via the Management API. The exposed endpoint enables an interactive gfsh session.

TLS prerequisites

Some of the gfsh commands below require a truststore and keystore password if TLS is enabled on the Tanzu GemFire cluster. Under the default TLS configuration, the truststore and keystore passwords are the same, and may be obtained using the following command:

kubectl -n NAMESPACE-NAME get secret NAME-cert -o=jsonpath='{.data.password}' | base64 --decode

Where:

  • NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace.
  • The cluster NAME is the metadata: name field from the deployment YAML.

Interactive gfsh session

With a Kubernetes cluster for which you have exec privilege, use kubectl to start an interactive gfsh session with the pod running the locator. Use a command of the form:

kubectl -n NAMESPACE-NAME exec -it NAME-locator-0 -- gfsh

Where:

  • NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace.
  • The cluster NAME is the metadata: name field from the deployment YAML.

For example, if the Tanzu GemFire cluster namespace is gemfire-cluster and the cluster NAME is gemfire1:

$ kubectl -n gemfire-cluster exec -it gemfire1-locator-0 -- gfsh
    _________________________     __
   / _____/ ______/ ______/ /____/ /
  / /  __/ /___  /_____  / _____  / 
 / /__/ / ____/  _____/ / /    / /  
/______/_/      /______/_/    /_/ 

Monitor and Manage Tanzu GemFire
gfsh>

Connect with the Tanzu GemFire cluster:

gfsh>connect --locator=NAME-locator-0.NAME-locator.NAMESPACE-NAME.svc.DOMAIN[10334] --trust-store=/certs/truststore.p12 --trust-store-password=TLS_PASSWORD --key-store=/certs/keystore.p12 --key-store-password=TLS_PASSWORD

Where:

  • NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace.
  • NAME is the metadata: name field from the deployment YAML.
  • DOMAIN is the domain of the Kubernetes cluster. This defaults to “cluster.local”.
  • TLS_PASSWORD is the password obtained from TLS prerequisites.

Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

Note: The --trust-store and --trust-store-password flags are required only if TLS is enabled. The --key-store and --key-store-password flags are required only if TLS is enabled with client authentication.

Once connected, issue gfsh commands as desired.

Exit the interactive gfsh session with exit:

gfsh>exit
Exiting...

Use kubectl to Invoke a gfsh Command

With a Kubernetes cluster for which you have exec privilege, use kubectl to invoke a single gfsh command on the Tanzu GemFire cluster. This method of executing gfsh commands can be useful for scripting cluster operations. The form of the kubectl command is:

kubectl -n NAMESPACE-NAME exec -it NAME-locator-0 -- gfsh -e 'connect --locator=NAME-locator-0.NAME-locator.NAMESPACE-NAME.svc.DOMAIN[10334] --trust-store=/certs/truststore.p12 --trust-store-password=TLS_PASSWORD --key-store=/certs/keystore.p12 --key-store-password=TLS_PASSWORD' -e "GFSH-COMMAND"

Where:

  • NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace.
  • NAME is the metadata: name field from the deployment YAML.
  • DOMAIN is the domain of the Kubernetes cluster. This defaults to “cluster.local”.
  • TLS_PASSWORD is the password obtained from TLS prerequisites.

Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

Note: The --trust-store and --trust-store-password flags are required only if TLS is enabled. The --key-store and --key-store-password flags are required only if TLS is enabled with client authentication.

For example, if the Tanzu GemFire cluster namespace is gemfire-cluster:

  • the cluster NAME is gemfire1
  • the cluster DOMAIN is cluster.local
  • the username USERNAME is user
  • the password PASSWORD is pass
  • the key/trust store TLS_PASSWORD is password
  • the desired gfsh command is list members

  • the kubectl command to list the members of the Tanzu GemFire cluster is:

    $ kubectl -n gemfire-cluster exec -it gemfire1-locator-0 -- gfsh -e 'connect --locator=gemfire1-locator-0.gemfire1-locator.gemfire-cluster.svc.cluster.local[10334] --trust-store=/certs/truststore.p12 --trust-store-password=password --key-store=/certs/keystore.p12 --key-store-password=password' -e "list members"

Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

Use a LoadBalancer and Target the Management API

The Tanzu GemFire Management API can be used to issue gfsh commands to the Tanzu GemFire cluster.

Note: If TLS is enabled on a production Tanzu GemFire cluster, the cluster must use a custom TLS certificate which contains the LoadBalancer’s external IP as a Subject Alternative Name (SAN) IP Address entry, or a DNS name which resolves to that external IP address as a SAN DNS entry. For non-production environments, it is possible to skip TLS certificate validation. (See step 5 below)

Follow these steps to connect to the Tanzu GemFire cluster:

  1. Create a LoadBalancer service for the Management API.

    apiVersion: v1
kind: Service
metadata:
name: lb-svc-mgmt-api
spec:
selector:
    gemfire.vmware.com/app: NAME-locator
ports:
    - name: management
    port: 7070
    targetPort: 7070
sessionAffinity: ClientIP
sessionAffinityConfig:
    clientIP:
        timeoutSeconds: 10800
type: LoadBalancer

  2. Acquire and install gfsh for the machine that will issue the gfsh commands. The gfsh executable is located in the bin directory of expanded Tanzu GemFire binary downloaded from VMware Tanzu Network. If more than one version of Tanzu GemFire is available, see the Release Notes to determine the appropriate version of Tanzu GemFire to acquire.

  3. Acquire the EXTERNAL-IP value to use in the gfsh connect command in step 5. Run the command:

    kubectl get service lb-svc-mgmt-api

    Where lb-svc-mgmt-api is the metadata: name field from the LoadBalancer service’s configuration YAML. The value is the EXTERNAL-IP field from the command’s output.

  4. Run gfsh.

  5. At the gfsh prompt, connect to the Tanzu GemFire cluster. Include a -use-http option in the connect command:

    gfsh>connect --locator --use-http=true --url=https://EXTERNAL-IP:7070/gemfire/v1

    Where EXTERNAL-IP is the acquired value from step 3.

    Note: If TLS is deactivated, change https to http in the above command.

    Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

    Note: When working with a non-production, development environment with TLS enabled, a developer may choose to work in a less secure manner by passing the --skip-ssl-validation flag to the gfsh connect command. When this flag is passed, the keystore path, keystore password, truststore path, and truststore password may be left blank.

    Note: If prompted, enter the KEYSTORE_PATH, KEYSTORE_PASSWORD, TRUSTSTORE_PATH, and TRUSTSTORE_PASSWORD as in this example:

    key-store: KEYSTORE_PATH
key-store-password: KEYSTORE_PASSWORD
key-store-type(default: JKS):
trust-store: TRUSTSTORE_PATH
trust-store-password: TRUSTSTORE_PASSWORD
trust-store-type(default: JKS):
ssl-ciphers(default: any):
ssl-protocols(default: any):
ssl-enabled-components(default: all):

Create a Region

Create regions once connected through gfsh. Specify the name of the region and the type of the region in the command. For example:

gfsh>create region --name=customers --type=PARTITION_REDUNDANT

For details, see the gfsh create region command in the Tanzu GemFire documentation.

Note: If SecurityManager is enabled, the user must have the DATA:MANAGE Resource Permissions assigned.

Obtain Logs and Tanzu GemFire Cluster State

Kubernetes Pod Logs

Kubernetes pod logs include the logged events from the Tanzu GemFire cluster and the startup of the pod. To print a Kubernetes pod log to standard output, use a command of the form:

$ kubectl -n NAMESPACE-NAME logs POD-NAME

Where:

  • NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace.
  • POD-NAME is composed as pod/NAME-locator-N or pod/NAME-server-N.
  • NAME is the metadata: name field, and N identifies which the desired locator or server.

Tanzu GemFire Cluster Logs

If the Tanzu GemFire cluster is healthy, follow the instructions in Healthy Cluster Logs. If the Tanzu GemFire cluster is unhealthy, the gfsh export logs command may hang or not work. Follow the instructions in Unhealthy Cluster Logs.

Healthy Cluster Logs

The gfsh export logs command places the Tanzu GemFire cluster logs on the locator-0 pod. The created ZIP file should be copied to the local machine for viewing the artifacts. The locator-0 pod must have sufficient disk space to capture the logs and statistics files from all the Tanzu GemFire locators and servers.

  1. Use gfsh to Connect to the Instance with an interactive gfsh session.

  2. Run the export logs command:

    gfsh>export logs
Logs exported to the connected member's file system: /data/exportedLogs_XXX.zip
gfsh>exit

    Note: If SecurityManager is enabled, the user must have the CLUSTER:READ Resource Permissions assigned.

    The logs will be in a file named similar to exportedLogs_XXX.zip. Output from the command will provide the exact name of the file; use the exact name in the following commands.

  3. Copy the logs from the locator-0 pod to the local machine:

    $ kubectl -n NAMESPACE-NAME cp NAME-locator-0:exportedLogs_XXX.zip exportedLogs_XXX.zip

    Where:

    • NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace.
    • NAME is the metadata: name field from the deployment YAML.

  4. Unzip the logs:

    $ unzip exportedLogs_XXX.zip

Unhealthy Cluster Logs

If the Tanzu GemFire cluster is unhealthy, the gfsh export logs command may hang or not work. Follow these instructions to manually gather logs.

  1. For each member of the cluster for which you want to gather logs, run:

    kubectl -n NAMESPACE_NAME cp NAME-MEMBER_TYPE-X:logsAndStats LOCAL_FOLDER

    Where:

    • NAMESPACE-NAME is the namespace on which the Tanzu GemFire cluster is deployed.
    • NAME is the name of the cluster name.
    • MEMBER_TYPE is the type of member. Valid values are locator and server.
    • X is the member index, an integer value between 0 and memberType.replicas - 1.
    • logsAndStats is created by default on every member and contains the logs, statistics, and gc-logs for that particular member.
    • LOCAL_FOLDER is the local folder to which copy the files.

    Examples:

    kubectl -n default cp gemfire-cluster-locator-0:logsAndStats hang-debug-locator0
kubectl -n gemfire-system cp sample-cluster-server-3:logsAndStats logsStats-server3

Collect and View Metrics

Using Tanzu Observability (formerly known as VMware Aria Operations for Applications)

Tanzu Observability (formerly known as VMware Aria Operations for Applications) integrates seamlessly with Tanzu GemFire clusters both to collect Tanzu GemFire cluster metrics and to view those metrics on a dashboard.

Set Up

To setup metrics collection for Operations for Applications, follow the instructions for VMware Tanzu GemFire for Kubernetes Setup. This will guide you to install the Wavefront proxy and Kubernetes Metrics Collector to your Kubernetes cluster.

Configuration

When you create the Tanzu GemFire cluster, approximately 200 metrics are available for collection at a one-minute interval by default. You can change these settings by modifying spec.metrics within the Tanzu GemFire Custom Resource Definition (CRD).

check-circle-line exclamation-circle-line close-line
Scroll to top icon