Once a Tanzu GemFire cluster has been created, gfsh commands aid the administration and maintenance of the cluster.

Many gfsh commands require that a session be connected to the Tanzu GemFire cluster before the operation can be completed.

Connect to the Tanzu GemFire Cluster

Use one of three ways to connect to a locator in order to issue gfsh commands to the Kubernetes-deployed Tanzu GemFire cluster.

The three ways to connect:

  • Use an interactive gfsh session. Requires Kubernetes cluster exec privilege.
  • Use kubectl to invoke a gfsh command. Requires Kubernetes cluster exec privilege.
  • Use a Kubernetes LoadBalancer service to permit gfsh Tanzu GemFire cluster access via the Management API. The exposed endpoint enables an interactive gfsh session.

TLS prerequisites

Some of the gfsh commands below require a truststore and keystore password if TLS is enabled on the Tanzu GemFire cluster. Under the default TLS configuration, the truststore and keystore passwords are the same, and may be obtained using the following command:

kubectl -n NAMESPACE-NAME get secret NAME-cert -o=jsonpath='{.data.password}' | base64 -D

where NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace, and the cluster NAME is the metadata: name field from the deployment YAML.

Interactive gfsh session

With a Kubernetes cluster for which you have exec privilege, use kubectl to start an interactive gfsh session with the pod running the locator. Use a command of the form:

kubectl -n NAMESPACE-NAME exec -it NAME-locator-0 -- gfsh

where NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace, and the cluster NAME is the metadata: name field from the deployment YAML.

For example, if the Tanzu GemFire cluster namespace is gemfire-cluster and the cluster NAME is gemfire1:

$ kubectl -n gemfire-cluster exec -it gemfire1-locator-0 -- gfsh
    _________________________     __
   / _____/ ______/ ______/ /____/ /
  / /  __/ /___  /_____  / _____  / 
 / /__/ / ____/  _____/ / /    / /  
/______/_/      /______/_/    /_/ 

Monitor and Manage VMware Tanzu GemFire
gfsh>

Connect with the Tanzu GemFire cluster:

gfsh>connect --locator=NAME-locator-0.NAME-locator.NAMESPACE-NAME.svc.DOMAIN[10334] --trust-store=/certs/truststore.p12 --trust-store-password=TLS_PASSWORD --key-store=/certs/keystore.p12 --key-store-password=TLS_PASSWORD

where NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace, NAME is the metadata: name field from the deployment YAML, DOMAIN is the domain of the Kubernetes cluster (defaults to “cluster.local”), and TLS_PASSWORD is the password obtained from TLS prerequisites.

Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

Note: The --trust-store and --trust-store-password flags are required only if TLS is enabled. The --key-store and --key-store-password flags are required only if TLS is enabled with client authentication.

Once connected, issue gfsh commands as desired.

Exit the interactive gfsh session with exit:

gfsh>exit
Exiting...

Use kubectl to Invoke a gfsh Command

With a Kubernetes cluster for which you have exec privilege, use kubectl to invoke a single gfsh command on the Tanzu GemFire cluster. This method of executing gfsh commands can be useful for scripting cluster operations. The form of the kubectl command is:

kubectl -n NAMESPACE-NAME exec -it NAME-locator-0 -- gfsh -e 'connect --locator=NAME-locator-0.NAME-locator.NAMESPACE-NAME.svc.DOMAIN[10334] --trust-store=/certs/truststore.p12 --trust-store-password=TLS_PASSWORD --key-store=/certs/keystore.p12 --key-store-password=TLS_PASSWORD' -e "GFSH-COMMAND"

where NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace, NAME is the metadata: name field from the deployment YAML, DOMAIN is the domain of the Kubernetes cluster (defaults to “cluster.local”), and TLS_PASSWORD is the password obtained from TLS prerequisites.

Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

Note: The --trust-store and --trust-store-password flags are required only if TLS is enabled. The --key-store and --key-store-password flags are required only if TLS is enabled with client authentication.

For example, if the Tanzu GemFire cluster namespace is gemfire-cluster:

  • the cluster NAME is gemfire1
  • the cluster DOMAIN is cluster.local
  • the username USERNAME is user
  • the password PASSWORD is pass
  • the key/trust store TLS_PASSWORD is password
  • the desired gfsh command is list members
  • the kubectl command to list the members of the Tanzu GemFire cluster is:

    $ kubectl -n gemfire-cluster exec -it gemfire1-locator-0 -- gfsh -e 'connect --locator=gemfire1-locator-0.gemfire1-locator.gemfire-cluster.svc.cluster.local[10334] --trust-store=/certs/truststore.p12 --trust-store-password=password --key-store=/certs/keystore.p12 --key-store-password=password' -e "list members"
    

Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

Use a LoadBalancer and Target the Management API

The Tanzu GemFire Management API can be used to issue gfsh commands to the Tanzu GemFire cluster.

Note: If TLS is enabled on a production Tanzu GemFire cluster, a custom TLS certificate must be provided which contains the LoadBalancer’s external IP address or a DNS name which resolves to that external IP address. For non-production environments, it is possible to skip TLS certificate validation. (See step 5 below.)

Follow these steps to connect to the Tanzu GemFire cluster:

  1. Create a LoadBalancer service as directed in Create a LoadBalancer Service for the Management API.
  2. Acquire and install gfsh for the machine that will issue the gfsh commands. The gfsh executable is located in the bin directory of expanded VMware Tanzu GemFire binary downloaded from VMware Tanzu Network. If more than one version of VMware Tanzu GemFire is available, see the Release Notes to determine the appropriate version of VMware Tanzu GemFire to acquire.
  3. Acquire the EXTERNAL-IP value to use in the gfsh connect command in step 5. Run the command:

    kubectl get service LB-SVC-MGMT-API
    

    where LB-SVC-MGMT-API is the metadata: name field from the LoadBalancer service’s configuration YAML. The value is the EXTERNAL-IP field from the command’s output.

  4. Run gfsh.
  5. At the gfsh prompt, connect to the Tanzu GemFire cluster. Include a -use-http option in the connect command:

    gfsh>connect --locator --use-http=true --url=https://EXTERNAL-IP:7070/gemfire/v1
    

    where EXTERNAL-IP is the acquired value from step 3.

    Note: If TLS is disabled, change https to http in the above command.

    Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD or --token=TOKEN, depending the authentication scheme used.

    Note: When working with a non-production, development environment with TLS enabled, a developer may choose to work in a less secure manner by passing the --skip-ssl-validation flag to the gfsh connect command. When this flag is passed, the keystore path, keystore password, truststore path, and truststore password may be left blank.

    Note: If prompted, enter the KEYSTORE_PATH, KEYSTORE_PASSWORD, TRUSTSTORE_PATH, and TRUSTSTORE_PASSWORD as in this example:

    key-store: KEYSTORE_PATH
    key-store-password: KEYSTORE_PASSWORD
    key-store-type(default: JKS):
    trust-store: TRUSTSTORE_PATH
    trust-store-password: TRUSTSTORE_PASSWORD
    trust-store-type(default: JKS):
    ssl-ciphers(default: any):
    ssl-protocols(default: any):
    ssl-enabled-components(default: all):
    

Create a Region

Create regions once connected through gfsh. Specify the name of the region and the type of the region in the command. For example:

gfsh>create region --name=customers --type=PARTITION_REDUNDANT

For details, see the gfsh create region command in the VMware Tanzu GemFire documentation.

Note: If SecurityManager is enabled, the user must have the DATA:MANAGE Resource Permissions assigned.

Obtain Logs and Tanzu GemFire Cluster State

Kubernetes Pod Logs

Kubernetes pod logs include the logged events from the Tanzu GemFire cluster and the startup of the pod. To print a Kubernetes pod log to standard output, use a command of the form:

$ kubectl -n NAMESPACE-NAME logs POD-NAME

where NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace, and POD-NAME is composed as pod/NAME-locator-N or pod/NAME-server-N. The NAME is the metadata: name field, and N identifies which locator or server is desired.

Tanzu GemFire Cluster Logs

If the GemFire cluster is healthy, follow the instructions in Healthy Cluster Logs. If the GemFire cluster is unhealthy, the gfsh export logs command may hang or not work. Follow the instructions in Unhealthy Cluster Logs.

Healthy Cluster Logs

The gfsh export logs command places the Tanzu GemFire cluster logs on the locator-0 pod. The created ZIP file should be copied to the local machine for viewing the artifacts. The locator-0 pod must have sufficient disk space to capture the logs and statistics files from all the Tanzu GemFire locators and servers.

  1. Use gfsh to Connect to the Instance with an interactive gfsh session.

  2. Run the export logs command:

    gfsh>export logs
    Logs exported to the connected member's file system: /data/exportedLogs_XXX.zip
    gfsh>exit
    

    Note: If SecurityManager is enabled, the user must have the CLUSTER:READ Resource Permissions assigned.

    The logs will be in a file named similar to exportedLogs_XXX.zip. Output from the command will provide the exact name of the file; use the exact name in the following commands.

  3. Copy the logs from the locator-0 pod to the local machine:

    $ kubectl -n NAMESPACE-NAME cp NAME-locator-0:exportedLogs_XXX.zip exportedLogs_XXX.zip
    

    where NAMESPACE-NAME is your chosen name for the Tanzu GemFire cluster namespace, and NAME is the metadata: name field from the deployment YAML.

  4. Unzip the logs:

    $ unzip exportedLogs_XXX.zip
    

Unhealthy Cluster Logs

If the GemFire cluster is unhealthy, the gfsh export logs command may hang or not work. Follow these instructions to manually gather logs.

  1. For each member of the cluster for which you want to gather logs, run:

    kubectl -n NAMESPACE_NAME cp NAME-MEMBER_TYPE-X:logsAndStats LOCAL_FOLDER
    

    where:

    • NAMESPACE-NAME is the namespace on which the GemFire cluster is deployed.
    • NAME is the name of the cluster name.
    • MEMBER_TYPE is the type of member. Valid values are locator and server.
    • X is the member index, an integer value between 0 and memberType.replicas - 1.
    • logsAndStats is created by default on every member and contains the logs, statistics, and gc-logs for that particular member.
    • LOCAL_FOLDER is the local folder to which copy the files.

    Examples:

    kubectl -n default cp gemfire-cluster-locator-0:logsAndStats hang-debug-locator0
    kubectl -n gemfire-system cp sample-cluster-server-3:logsAndStats logsStats-server3
    

Collect and View Metrics

Use VMware Tanzu Observability by Wavefront

The VMware Tanzu Observability by Wavefront product integrates seamlessly with Tanzu GemFire clusters to both collect Tanzu GemFire cluster metrics and view those metrics via a dashboard.

There are three parts to the integration. The installation of the Wavefront Collector and Proxy may be done before or after the creation of a Tanzu GemFire cluster.

  • Install the Wavefront Collector and Proxy by following the instructions at Kubernetes Setup. The installation will be either with Tanzu Mission Control or with Helm.

    • For a Tanzu Mission Control installation:
      1. Enable the use of Tanzu Observability for your organization by following the directions at Enable Observability for Your Organization.
      2. Install a configured Wavefront Collector and a Proxy on your Kubernetes cluster by following the directions at Add a Cluster to Observability.
    • For a Helm installation:
      1. Acquire an API token that will allow Tanzu Observability to authenticate communication from the Wavefront Proxy. Follow the directions at Generating an API Token.
      2. Use the API token as you install the Wavefront Collector and Proxy via Helm following the directions at Kubernetes Quick Install Using Helm.
  • When you create the Tanzu GemFire cluster, it emits metrics by default. If desired, you can specify a much higher quantity of metrics to be emitted at a higher frequency by setting spec.metrics within the CRD.

  • Create or integrate a Wavefront dashboard. There are sample dashboards available in the https://github.com/vmware-samples/tanzu-gemfire-observability-sample repository.

check-circle-line exclamation-circle-line close-line
Scroll to top icon