Various services generate logs automatically. You can generate log bundles in vRealize Automation. You can also configure your environment to send logs to vRealize Log Insight.

Information about how to use the vracli command line utility to generate log bundles is available by using the --help argument in the vracli command line (for example, vracli log-bundle --help).

For related information about using vRealize Log Insight, see How do I configure log forwarding to vRealize Log Insight.

Log bundle commands

You can create a log bundle to contain all the logs that are generated by the services that you run. A log bundle contains all your service logs and is needed for troubleshooting.

In a clustered environment (high availability mode), run the vracli log-bundle command on only one node. Logs are pulled from all nodes in the environment. However, in the event of a networking or other cluster issue, logs are pulled from as many nodes as can be reached. For example, if one node is disconnected in a cluster of three nodes, logs are only collected from the two healthy nodes. Output from the vracli log-bundle command contains information about any issues found and their workaround steps.

  • To create a log bundle, SSH to any node and run the following vracli command:

    vracli log-bundle

  • To change the timeout value for collecting logs from each node, run the following vracli command:

    vracli log-bundle --collector-timeout $CUSTOM_TIMEOUT_IN_SECONDS

    For example, if your environment contains large log files, slow networking, or high CPU usage, you can set the timeout to greater than the 1000 second default value.

  • To configure other options such as assembly timeout and buffer location, use the following vracli help command:

    vracli log-bundle --help

Log bundle structure

The log bundle is a timestamped tar file. The name of the bundle matches the patter log-bundle-<date>T<time>.tar file, for example log-bundle-20200629T131312.tar. Typically the log bundle contains logs from all nodes in the environment. In case of an error, it contains as many logs as possible. It minimally contains logs from the local node.

The log bundle consists of the following content:
  • Environment file

    The environment file contains the output of various Kubernetes maintenance commands. It supplies information about current resource usage per nodes and per pods. It also contains cluster information and description of all available Kubernetes entities.

  • Host logs and configuration

    The configuration of each host (for example its /etc directory) and the host-specific logs (for example journald) are collected in one directory for each cluster node or host. The name of the directory matches the host name of the node. The internal contents of the directory match the file system of the host. The number of such directories matches the number of cluster nodes.

  • Services logs
    Logs for Kubernetes services are located in the following folder structure:
    • <hostname>/services-logs/<namespace>/<app-name>/file-logs/<container-name>.log
    • <hostname>/services-logs/<namespace>/<app-name>/console-logs/<container-name>.log

    An example file name is my-host-01/services-logs/prelude/vco-app/file-logs/vco-server-app.log.

    • hostname is the host name of the node on which the application container is or was running. Typically, there is one instance for each node for each service. For example, 3 nodes = 3 instances.
    • namespace is the Kubernetes namespace in which the application is or was deployed. For user-facing services, this value is prelude.
    • app-name is the name of the Kubernetes application that produced the logs, for example provisioning-service-app.
    • container-name is the name of the container that produced the logs. Some apps consist of multiple containers. For example, vco-app contains the vco-server-app and vco-controlcenter-app containers.
  • (Legacy) Pod logs

    Prior to the logging architecture changes made in vRealize Automation 8.2, services logs (described in the previous bullet) were located in each pod's directory in the log bundle. While you can continue to generate pod logs in the bundle by using the vracli log-bundle --include-legacy-pod-logs command line, doing so is not advised as all log information already resides in each services' logs. Including pod logs can unnecessarily increase the time and space required to generate the log bundle.

Reducing the size of the log bundle

To generate a smaller log bundle, use either of the following vracli log-bundle commands:

  • vracli log-bundle --since-days n

    Use this command to collect only the log files that were generated over the past number of days. Otherwise, logs are retained and collected for 7 days. For example:

    vracli log-bundle --since-days 3

  • vracli log-bundle --services service_A,service_B,service_C

    Use this command to collect only the logs for the named provided services. For example:

    vracli log-bundle --services ebs-app,vco-app

  • vracli log-bundle --skip-heap-dumps

    Use this command to exclude all heap dumps from the generated log bundle.