Use this section as a reference to aid you while using VMware Cloud Director Container Service Extension as a service provider administrator.

Troubleshooting with Logs

This section details important log files and their locations for VMware Cloud Director Container Service Extension 4.0. You can refer to these log files as an aid when you troubleshoot or re-attempt operations on failed ephemeral virtual machine script errors.

Table 1. Log Bundles
Component Log Description Location Syslog Supported Notes
VMware Cloud Director Container Service Extension server cse.log VMware Cloud Director Container Service Extension server log that collects all standard output and errors at one location. This includes running status of cluster creation, deletion and heartbeat, events and errors that records in the RDE. /root/cse.log Yes This file location is in a virtual machine where the VMware Cloud Director Container Service Extension server OVA is deployed and CSE is operating.
VMware Cloud Director Container Service Extension server cse-wire.log Contains requests and responses that VMware Cloud Director SDK generates. Refer to this log when you debug unexpected failures in customer environment. /root/cse-wire.log No
To activate wire logging, complete the following steps:
  1. Log in to the VMware Cloud Director Container Service Extension OVA virtual machine.
  2. Open /root/config.toml.
  3. Enter the following line:wire_logging_enabled = true.
    Note: To deactivate wire logging, comment out this line.
  4. Use the following command to restart VMware Cloud Director Container Service Extension service: systemctl restart cse.service.
Bootstrap VM cloud-final.err Collects all the commands and statuses that are executed to create and delete clusters on ephemeral virtual machines and subsequent steps to ensure the target cluster is self-manageable. If cluster creation and deletion is successful, all commands are recorded. If there is a failure, the error message is recorded before the operation discontinues. /var/log/cloud-final.err
Note: This log is located in Bootstrap VM. It is necessary to log in to Bootstrap VM.
Yes Set autoRepairOnErrors: Off. This is a UI supported setting at cluster creation time. You can find this in the Debug Settings tab in the UI.
Bootstrap VM cloud-final.out This is a file on the ephemeral virtual machine that records all the standard outputs that occur during package downloads, clusterctl init, kubectl machines, apply. /var/log/cloud-final.out
Note: This log is located in Bootstrap VM. It is necessary to log in to Bootstrap VM.
Yes Set autoRepairOnErrors: Off. This is a UI supported setting at cluster creation time. You can find this in the Debug Settings tab in the UI.
Bootstrap VM, Kubernetes cluster bash shell:

Execute k8s-log-bundle.sh		 Collects logs from CAPVCD pods. If there are no errors on Ephemeral virtual machine script execution, and kubectl get machines continues to run a pending loop, the user can use this script to go to the next step of troubleshooting to perform the root cause analysis. https://github.com/vmware/cloud-provider-for-cloud-director/blob/main/scripts/generate-k8s-log-bundle.sh No This is a bash script file. Ensure you perform the following operations before you run the script:
  • Download the kubeconfig file of the cluster.
  • Export KUBECONFIG=<full-path-of-kube-config>
Bootstrap VM, Kubernetes cluster CAPVCD logs: 
export KUBECONFIG=/path/to/kubeconfig

kubectl get pods -A (copy CAPVCD pod name) 
kubectl logs -n kube-system CAPVCD-POD-NAME
 Logs for CAPVCD No Activate wire logs, such as HTTP requests and HTTP responses to and from VMware Cloud Director, on CAPVCD pod logs.
kubectl set env -n capvcd-system deployment/capvcd-controller-manager GOVCD_LOG_ON_SCREEN=true -oyaml.
To deactivate wire logging, use the following command:
kubectl set env -n capvcd-system deployment/capvcd-controller-manager GOVCD_LOG_ON_SCREEN-
Note: Activating wire logs pollutes the logs. It is only necessary to activate the wire logs for debugging failures.
Kubernetes cluster CPI output for CPI logs 
export KUBECONFIG=/path/to/kubeconfig
kubectl get pods -A (copy CPI pod name)
kubectl logs -n kube-system CPI-POD-NAME
 Logs for CPI pods No Activate wire logs, HTTP requests and HTTP responses to and from VMware Cloud Director, on CPI pod logs.

kubectl set env -n kube-system deployment/vmware-cloud-director-ccm GOVCD_LOG_ON_SCREEN=true -oyaml

To deactivate wire logging, use the following command: 
kubectl set env -n kube-system deployment/vmware-cloud-director-ccm GOVCD_LOG_ON_SCREEN-
Note: Activating wire logs pollutes the logs. It is only necessary to activate the wire logs for debugging failures.
Kubernetes cluster CSI logs 
export KUBECONFIG=/path/to/kubeconfig

kubectl get pods -A (copy CSI controller pod name or CSI node-plugin pod name) 
kubectl logs -n kube-system CSI-POD-NAME -c vcd-csi-plugin
 Logs for CSI No Activate wire logs, HTTP requests and HTTP responses to/from VMware Cloud Director, on CSI pod logs. 
kubectl set env -n kube-system StatefulSet/csi-vcd-controllerplugin -c vcd-csi-plugin GOVCD_LOG_ON_SCREEN=true -oyaml
kubectl set env -n kube-system DaemonSet/csi-vcd-nodeplugin -c vcd-csi-plugin GOVCD_LOG_ON_SCREEN=true -oyaml
To deactivate wire logging, use the following command: 
kubectl set env -n kube-system StatefulSet/csi-vcd-controllerplugin -c vcd-csi-plugin GOVCD_LOG_ON_SCREEN-
kubectl set env -n kube-system DaemonSet/csi-vcd-nodeplugin -c vcd-csi-plugin GOVCD_LOG_ON_SCREEN-
Note: Activating wire logs pollutes the logs. It is only necessary to activate the wire logs for debugging failures.
Kubernetes cluster RDE Projector Logs
export KUBECONFIG=/path/to/kubeconfig
kubectl get pods -A (copy CSI controller pod name or CSI node-plugin pod name)
kubectl logs -n kube-system CSI-POD-NAME -c vcd-csi-plugin
Logs for RDE Projector No Activate wire logs, HTTP requests and HTTP responses to and from VMware Cloud Director, on RDE projector pod logs. 
kubectl set env -n rdeprojector-system deployment/rdeprojector-controller-manager GOVCD_LOG_ON_SCREEN=true -oyaml

To deactivate wire logging, use the following command:

kubectl set env -n capvcd-system deployment/rdeprojector-controller-manager GOVCD_LOG_ON_SCREEN-
Kubernetes cluster through API Cluster RDE using Postman Tool 
GET https://{{base_url}}/cloudapi/1.0.0/entities/types/vmware/capvcdCluster/1.1.0
Discover the entity that matches cluster name.

Look for entity->status->VcdKe→errorSet : This is json array that contains errors that were encountered during cluster operation		 No HTTP GET