Using VM-Level Diagnostic Tool

The section describes the method for using VM-Level Diagnostic Tool in VMware Telco Cloud Operations cluster.

Prerequisites

The utility is part of the control plane node OVA and is available after the OVA installation. No additional set up is required.

Procedure

Log in to the control plane node VM as user clusteradmin.
Change into the deployment-logger directory found under /home/clusteradmin.
Before running the utility very first time, create a vms.txt file in the deployment-logger folder. In vms.txt, provide the IP address for each worker VM and the corresponding password for the clusteradmin user as displayed in the following snippet. After the first run, the vms.txt is not required and can be discarded. The utility configures key-based login and can be invoked subsequently without providing any credentials.
Note:
- The file name must be vms.txt.
- The file must be in the following format, that is valid IP address followed by a comma and the clusteradmin user password on that IP.
```
[clusteradmin@control_plane_node deployment-logger]$ cat vms.txt
10.182.136.200,clusteradmin_password
10.184.87.178,clusteradmin_password
10.186.9.43,clusteradmin_password
10.186.30.239,clusteradmin_password
```
To run the utility, execute the following command:
```
$ ./deployment-logger.sh
```
Enter the clusteradmin user password when prompted.

The utility tries to access all the VMs in the cluster, and collects the logs and system-level details. The collected contents are included in a support bundle file, present at /home/clusteradmin/tco as shown below:

[clusteradmin@sc-rdops-vm14-dhcp-167-51 tco]$ ls -al
total 356
drwxr-xr-x  2 clusteradmin clusteradmin     51 Feb 10 03:52 .
drwxr-xr-x 11 clusteradmin clusteradmin   4096 Feb 10 02:42 ..
-rw-r--r--  1 clusteradmin clusteradmin 358400 Feb 10 02:42 support-bundle-2021-Feb-10-020245.tar.gz

Extract the contents of the support bundle tar.gz file and change into the support-bundle folder to view the collected logs and system details as shown in the following code block:

[clusteradmin@sc-rdops-vm14-dhcp-167-51 tco]$ tar -xvf support-bundle-2021-Feb-10-020245.tar.gz
 (Details get extracted)
 ....
 ....
 
 
[clusteradmin@sc-rdops-vm14-dhcp-167-51 tco]$ ls -al
total 356
drwxr-xr-x  2 clusteradmin clusteradmin     51 Feb 10 03:52 .
drwxr-xr-x 11 clusteradmin clusteradmin   4096 Feb 10 02:42 ..
drwxr-xr-x 15 clusteradmin clusteradmin    277 Feb 10 02:42 support-bundle
-rw-r--r--  1 clusteradmin clusteradmin 358400 Feb 10 02:42 support-bundle-2021-Feb-10-020245.tar.gz
 
[clusteradmin@sc-rdops-vm14-dhcp-182-27 tco]$ cd support-bundle/
 
[clusteradmin@sc-rdops-vm14-dhcp-167-51 support-bundle]$ ls -al
total 360
drwxr-xr-x 15 clusteradmin clusteradmin    323 Feb 10 03:53 .
drwxr-xr-x 11 clusteradmin clusteradmin   4096 Feb 10 02:42 ..
drwxr-xr-x  4 clusteradmin clusteradmin    147 Feb 10 02:42 10.182.136.200
drwxr-xr-x  4 clusteradmin clusteradmin    147 Feb 10 02:42 10.184.87.178
drwxr-xr-x  4 clusteradmin clusteradmin    147 Feb 10 02:42 10.186.30.239
drwxr-xr-x  4 clusteradmin clusteradmin    147 Feb 10 02:42 10.186.9.43
drwxr-xr-x  2 clusteradmin clusteradmin     33 Feb 10 02:42 configmaps
drwxr-xr-x  4 clusteradmin clusteradmin    147 Feb 10 02:42 control-plane-node
drwxr-xr-x  2 clusteradmin clusteradmin     30 Feb 10 02:42 k8s-pod
drwxr-xr-x  2 clusteradmin clusteradmin      6 Feb 10 02:42 k8s-pod-log
drwxr-xr-x  2 clusteradmin clusteradmin     27 Feb 10 02:42 node
drwxr-xr-x  2 clusteradmin clusteradmin     83 Feb 10 02:42 persistent-volumes
drwxr-xr-x  2 clusteradmin clusteradmin     30 Feb 10 02:42 service-info
-rw-r--r--  1 clusteradmin clusteradmin 358400 Feb 10 02:42 support-bundle-2021-Feb-10-020245.tar
drwxr-xr-x  2 clusteradmin clusteradmin     30 Feb 10 02:42 tco-pod
drwxr-xr-x  2 clusteradmin clusteradmin      6 Feb 10 02:42 tco-pod-log
-rw-r--r--  1 clusteradmin clusteradmin    227 Feb 10 02:42 tco-release

Note:

When running the utility for the first time after VM deployment, when cluster is not yet deployed, you can receive messages:
```
2021-02-10 02:42:43.861  INFO 13092 --- [main] com.vmware.omega.runner.AbstractRunner   : Collecting nodes info ...
The connection to the server localhost:8080 was refused - did you specify the right host or port?
```
This message can be safely ignored. It is an indication that the Kuberenetes cluster has not yet been set up to collect information. The utility is set up to try and collect all the details required. Even if a particular log or system detail is unavailable, the utility continues to gather whatever is available to support troubleshooting the overall cluster.
The VMware Telco Cloud Operations user interface also has a Support Bundle collection mechanism. That feature continues to work as in previous versions. Currently it only collects all the VMware Telco Cloud Operations cluster-based details. It does not collect the deployment logs or system-level details for each VM.