This topic describes how to work with a Tanzu GemFire cluster when using VMware Tanzu GemFire for Kubernetes.
Once a Tanzu GemFire cluster has been created, gfsh
commands aid the administration and maintenance of the cluster.
Many gfsh
commands require that a session be connected to the Tanzu GemFire cluster before the operation can be completed.
Use one of three ways to connect to a locator in order to issue gfsh
commands to the Kubernetes-deployed Tanzu GemFire cluster.
The three ways to connect:
gfsh
session. Requires Kubernetes cluster exec privilege.kubectl
to invoke a gfsh
command. Requires Kubernetes cluster exec privilege.gfsh
Tanzu GemFire cluster access via the Management API. The exposed endpoint enables an interactive gfsh
session.Some of the gfsh
commands below require a truststore and keystore password if TLS is enabled on the Tanzu GemFire cluster. Under the default TLS configuration, the truststore and keystore passwords are the same, and may be obtained using the following command:
kubectl -n NAMESPACE-NAME get secret NAME-cert -o=jsonpath='{.data.password}' | base64 --decode
Where:
NAMESPACE-NAME
is your chosen name for the Tanzu GemFire cluster namespace.NAME
is the metadata: name
field from the deployment YAML.gfsh
sessionWith a Kubernetes cluster for which you have exec privilege, use kubectl
to start an interactive gfsh
session with the pod running the locator. Use a command of the form:
kubectl -n NAMESPACE-NAME exec -it NAME-locator-0 -- gfsh
Where:
NAMESPACE-NAME
is your chosen name for the Tanzu GemFire cluster namespace.NAME
is the metadata: name
field from the deployment YAML.For example, if the Tanzu GemFire cluster namespace is gemfire-cluster
and the cluster NAME
is gemfire1
:
$ kubectl -n gemfire-cluster exec -it gemfire1-locator-0 -- gfsh
_________________________ __
/ _____/ ______/ ______/ /____/ /
/ / __/ /___ /_____ / _____ /
/ /__/ / ____/ _____/ / / / /
/______/_/ /______/_/ /_/
Monitor and Manage Tanzu GemFire
gfsh>
Connect with the Tanzu GemFire cluster:
gfsh>connect --locator=NAME-locator-0.NAME-locator.NAMESPACE-NAME.svc.DOMAIN[10334] --trust-store=/certs/truststore.p12 --trust-store-password=TLS_PASSWORD --key-store=/certs/keystore.p12 --key-store-password=TLS_PASSWORD
Where:
NAMESPACE-NAME
is your chosen name for the Tanzu GemFire cluster namespace.NAME
is the metadata: name
field from the deployment YAML.DOMAIN
is the domain of the Kubernetes cluster. This defaults to “cluster.local”.TLS_PASSWORD
is the password obtained from TLS prerequisites.Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD
or --token=TOKEN
, depending the authentication scheme used.
Note: The --trust-store
and --trust-store-password
flags are required only if TLS is enabled. The --key-store
and --key-store-password
flags are required only if TLS is enabled with client authentication.
Once connected, issue gfsh
commands as desired.
Exit the interactive gfsh
session with exit
:
gfsh>exit
Exiting...
With a Kubernetes cluster for which you have exec privilege, use kubectl
to invoke a single gfsh
command on the Tanzu GemFire cluster. This method of executing gfsh
commands can be useful for scripting cluster operations. The form of the kubectl
command is:
kubectl -n NAMESPACE-NAME exec -it NAME-locator-0 -- gfsh -e 'connect --locator=NAME-locator-0.NAME-locator.NAMESPACE-NAME.svc.DOMAIN[10334] --trust-store=/certs/truststore.p12 --trust-store-password=TLS_PASSWORD --key-store=/certs/keystore.p12 --key-store-password=TLS_PASSWORD' -e "GFSH-COMMAND"
Where:
NAMESPACE-NAME
is your chosen name for the Tanzu GemFire cluster namespace.NAME
is the metadata: name
field from the deployment YAML.DOMAIN
is the domain of the Kubernetes cluster. This defaults to “cluster.local”.TLS_PASSWORD
is the password obtained from TLS prerequisites.Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD
or --token=TOKEN
, depending the authentication scheme used.
Note: The --trust-store
and --trust-store-password
flags are required only if TLS is enabled. The --key-store
and --key-store-password
flags are required only if TLS is enabled with client authentication.
For example, if the Tanzu GemFire cluster namespace is gemfire-cluster
:
NAME
is gemfire1
DOMAIN
is cluster.local
USERNAME
is user
PASSWORD
is pass
TLS_PASSWORD
is password
gfsh
command is list members
the kubectl
command to list the members of the Tanzu GemFire cluster is:
$ kubectl -n gemfire-cluster exec -it gemfire1-locator-0 -- gfsh -e 'connect --locator=gemfire1-locator-0.gemfire1-locator.gemfire-cluster.svc.cluster.local[10334] --trust-store=/certs/truststore.p12 --trust-store-password=password --key-store=/certs/keystore.p12 --key-store-password=password' -e "list members"
Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD
or --token=TOKEN
, depending the authentication scheme used.
The Tanzu GemFire Management API can be used to issue gfsh
commands to the Tanzu GemFire cluster.
Note: If TLS is enabled on a production Tanzu GemFire cluster, the cluster must use a custom TLS certificate which contains the LoadBalancer’s external IP as a Subject Alternative Name (SAN) IP Address entry, or a DNS name which resolves to that external IP address as a SAN DNS entry. For non-production environments, it is possible to skip TLS certificate validation. (See step 5 below)
Follow these steps to connect to the Tanzu GemFire cluster:
Create a LoadBalancer service for the Management API.
apiVersion: v1
kind: Service
metadata:
name: lb-svc-mgmt-api
spec:
selector:
gemfire.vmware.com/app: NAME-locator
ports:
- name: management
port: 7070
targetPort: 7070
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
type: LoadBalancer
Acquire and install gfsh
for the machine that will issue the gfsh
commands. The gfsh
executable is located in the bin
directory of expanded Tanzu GemFire binary downloaded from Broadcom Support. If more than one version of Tanzu GemFire is available, see the Release Notes to determine the appropriate version of Tanzu GemFire to acquire.
Acquire the EXTERNAL-IP
value to use in the gfsh
connect command in step 5. Run the command:
kubectl get service lb-svc-mgmt-api
Where lb-svc-mgmt-api
is the metadata: name
field from the LoadBalancer service’s configuration YAML. The value is the EXTERNAL-IP
field from the command’s output.
gfsh
.At the gfsh
prompt, connect to the Tanzu GemFire cluster. Include a -use-http
option in the connect command:
gfsh>connect --locator --use-http=true --url=https://EXTERNAL-IP:7070/gemfire/v1
Where EXTERNAL-IP
is the acquired value from step 3.
Note: If TLS is deactivated, change https
to http
in the above command.
Note: If Authentication and Authorization are enabled for the cluster, include the credentials as --user=USERNAME --password=PASSWORD
or --token=TOKEN
, depending the authentication scheme used.
Note: When working with a non-production, development environment with TLS enabled, a developer may choose to work in a less secure manner by passing the --skip-ssl-validation
flag to the gfsh connect
command. When this flag is passed, the keystore path, keystore password, truststore path, and truststore password may be left blank.
Note: If prompted, enter the KEYSTORE_PATH
, KEYSTORE_PASSWORD
, TRUSTSTORE_PATH
, and TRUSTSTORE_PASSWORD
as in this example:
key-store: KEYSTORE_PATH
key-store-password: KEYSTORE_PASSWORD
key-store-type(default: JKS):
trust-store: TRUSTSTORE_PATH
trust-store-password: TRUSTSTORE_PASSWORD
trust-store-type(default: JKS):
ssl-ciphers(default: any):
ssl-protocols(default: any):
ssl-enabled-components(default: all):
Create regions once connected through gfsh
. Specify the name of the region and the type of the region in the command. For example:
gfsh>create region --name=customers --type=PARTITION_REDUNDANT
For details, see the gfsh
create region command in the Tanzu GemFire documentation.
Note: If SecurityManager
is enabled, the user must have the DATA:MANAGE
Resource Permissions assigned.
Kubernetes pod logs include the logged events from the Tanzu GemFire cluster and the startup of the pod. To print a Kubernetes pod log to standard output, use a command of the form:
$ kubectl -n NAMESPACE-NAME logs POD-NAME
Where:
NAMESPACE-NAME
is your chosen name for the Tanzu GemFire cluster namespace.POD-NAME
is composed as pod/NAME-locator-N
or pod/NAME-server-N
.NAME
is the metadata: name
field, and N
identifies which the desired locator or server.If the Tanzu GemFire cluster is healthy, follow the instructions in Healthy Cluster Logs. If the Tanzu GemFire cluster is unhealthy, the gfsh export logs
command may hang or not work. Follow the instructions in Unhealthy Cluster Logs.
The gfsh export logs
command places the Tanzu GemFire cluster logs on the locator-0 pod. The created ZIP file should be copied to the local machine for viewing the artifacts. The locator-0 pod must have sufficient disk space to capture the logs and statistics files from all the Tanzu GemFire locators and servers.
Use gfsh
to Connect to the Instance with an interactive gfsh
session.
Run the export logs
command:
gfsh>export logs
Logs exported to the connected member's file system: /data/exportedLogs_XXX.zip
gfsh>exit
Note: If SecurityManager
is enabled, the user must have the CLUSTER:READ
Resource Permissions assigned.
The logs will be in a file named similar to exportedLogs_XXX.zip
. Output from the command will provide the exact name of the file; use the exact name in the following commands.
Copy the logs from the locator-0 pod to the local machine:
$ kubectl -n NAMESPACE-NAME cp NAME-locator-0:exportedLogs_XXX.zip exportedLogs_XXX.zip
Where:
NAMESPACE-NAME
is your chosen name for the Tanzu GemFire cluster namespace.NAME
is the metadata: name
field from the deployment YAML.Unzip the logs:
$ unzip exportedLogs_XXX.zip
If the Tanzu GemFire cluster is unhealthy, the gfsh export logs
command may hang or not work. Follow these instructions to manually gather logs.
For each member of the cluster for which you want to gather logs, run:
kubectl -n NAMESPACE_NAME cp NAME-MEMBER_TYPE-X:logsAndStats LOCAL_FOLDER
Where:
NAMESPACE-NAME
is the namespace on which the Tanzu GemFire cluster is deployed.NAME
is the name of the cluster name.MEMBER_TYPE
is the type of member. Valid values are locator
and server
.X
is the member index, an integer value between 0
and memberType.replicas - 1
.logsAndStats
is created by default on every member and contains the logs, statistics, and gc-logs for that particular member.LOCAL_FOLDER
is the local folder to which copy the files.Examples:
kubectl -n default cp gemfire-cluster-locator-0:logsAndStats hang-debug-locator0
kubectl -n gemfire-system cp sample-cluster-server-3:logsAndStats logsStats-server3
Tanzu Observability (formerly known as VMware Aria Operations for Applications) integrates seamlessly with Tanzu GemFire clusters both to collect Tanzu GemFire cluster metrics and to view those metrics on a dashboard.
To setup metrics collection for Operations for Applications, follow the instructions for VMware Tanzu GemFire for Kubernetes Setup. This will guide you to install the Wavefront proxy and Kubernetes Metrics Collector to your Kubernetes cluster.
When you create the Tanzu GemFire cluster, approximately 200 metrics are available for collection at a one-minute interval by default. You can change these settings by modifying spec.metrics
within the Tanzu GemFire Custom Resource Definition (CRD).