You can run the Supportability and Serviceability (SoS) Utility on the VMware Cloud Builder appliance to generate a support bundle, which you can use to help debug a failed bring-up of VMware Cloud Foundation.

Note:

After a successful bring-up, you should only run the SoS Utility on the SDDC Manager VM. See Supportability and Serviceability (SoS) Tool in the VMware Cloud Foundation Operations and Administration Guide.

The SoS Utility is not a debug tool, but it does provide health check operations that can facilitate debugging a failed deployment.

To run the SoS Utility in VMware Cloud Builder, SSH in to the VMware Cloud Builder appliance using the admin administrative account, then enter su to switch to the root user, and navigate to the /opt/vmware/sddc-support directory and type ./sos followed by the options required for your desired operation.

./sos --option-1 --option-2 ... --option-n

SoS Utility Help Options

Use these options to see information about the SoS tool itself.

Option Description

--help

-h

Provides a summary of the available SoS tool options

--version

-v

Provides the SoS tool's version number.

SoS Utility Generic Options

These are generic options for the SoS Utility.

Option Description
--configure-sftp Configures SFTP for logs.
--debug-mode Runs the SoS tool in debug mode.
--force
Allows SoS operations from the VMware Cloud Builder appliance after bring-up.
Note: In most cases, you should not use this option. Once bring-up is complete, you can run the SoS Utility directly from the SDDC Manager VM.
--history Displays the last twenty SoS operations performed.
--log-dir LOGDIR Specifies the directory to store the logs.
--log-folder LOGFOLDER Specifies the name of the log directory.
--setup-json SETUP_JSON

Custom setup-json file for log collection.

SoS prepares the inventory automatically based on the environment where it is running. If you want to collect logs for a pre-defined set of components, you can create a setup.json file and pass the file as input to SoS. A sample JSON file is available on the VMware Cloud Builder in the /opt/vmware/sddc-support/ directory.
--skip-known-host-check Skips the specified check for SSL thumbprint for host in the known host.
--zip Creates a zipped tar file for the output.

SoS Utility Log File Options

Option Description
--api-logs

Collects output from APIs.

--cloud-builder-logs Collects Cloud Builder logs.
--esx-logs

Collects logs from the ESXi hosts only.

Logs are collected from each ESXi host available in the deployment.

--no-clean-old-logs

Use this option to prevent the tool from removing any output from a previous collection run.

By default, before writing the output to the directory, the tool deletes the prior run's output files that might be present. If you want to retain the older output files, specify this option.

--no-health-check

Skips the health check executed as part of log collection.

--nsx-logs

Collects logs from the NSX Manager instances only.

--rvc-logs

Collects logs from the Ruby vSphere Console (RVC) only. RVC is an interface for ESXi and vCenter.

Note: If the Bash shell is not enabled in vCenter, RVC log collection will be skipped .
Note: RVC logs are not collected by default with ./sos log collection.
--sddc-manager-logs

Collects logs from the SDDC Manager only.

--test

Collects test logs by verifying the files.

--vc-logs

Collects logs from the vCenter Server instances only.

Logs are collected from each vCenter server available in the deployment.

--vm-screenshots Collects screen shots from all VMs.

SoS Utility JSON Generator Options

The JSON generator options within the SoS Utility provide a method to execute the creation of the JSON file from a completed deployment parameter workbook. To run the JSON generator, you must provide, as a minimum, a path to the deployment parameter workbook and the design type using the following syntax:
./sos --jsongenerator --jsongenerator-input JSONGENERATORINPUT --jsongenerator JSONGENERATORDESIGN
Option Description
--jsongenerator Invokes the JSON generator utility.
--jsongenerator-input JSONGENERATORINPUT Specify the path to the input file to be used by the JSON generator utility. For example: /tmp/vcf-ems-deployment-parameter.xlsx.
--jsongenerator-design JSONGENERATORDESIGN Use vcf-public-ems for VMware Cloud Foundation.
--jsongenerator-supress Supress confirmation to force cleanup directory. (optional)
--jsongenerator-logs JSONGENERATORLOGS Set the directory to be used for logs. (optional)

SoS Utility Platform Audit Options

The Platform Audit options provide a method to execute validation tasks from the command line and offer the flexibility to validate individual tasks, or all tasks at once. Validating individual tasks helps accelerate the troubleshooting process, since you can validate that task repeatedly until it succeeds.

To run a platform audit, you must provide, as a minimum, a path to the JSON input file:
./sos --platformaudit --platformaudit-input FILE
Note: The platform audit capabilities of the SoS Utility have been deprecated and will be removed in the next release of VMware Cloud Foundation. All functionality has been ported over to the new bring-up validation service. When you initiate validation through the VMware Cloud Builder administration interface, the bring-up validation service performs the validation tasks. All logging is directed to the vcf-bringup.log and vcf-debug-bringup.log files. You can use the SoS Utility platform audit options in this release, but this method may not contain enhancements to validation tasks that are available through the bring-up validation service.
Option Description
--platformaudit

Invokes the platform audit operation.

--platformaudit-dependency

Executes audit tests with dependencies.

--platformaudit-input FILE

Specify the path to the JSON input file to be used by the platform audit utility.

--platformaudit-stop

Stops all running platform audit processes.

--platformaudit-modules MODULE1,MODULE2,MODULE3

Specify the specific audit tests to run. If specifying multiple tests, separate the modules with commas.

--platformaudit-output OUTPUT

Saves the output to the specified file.

--platformaudit-reason

Outputs reasons for failed or skipped tests.

--platformaudit-tree

Displays a list of available audit tests.

SoS Utility Health Check Options

The SoS Utility can be used to perform health checks on various components or services, including connectivity, compute, and storage.

Note: The health check options are primarily designed to run on the SDDC Manager VM. Running them on the VMware Cloud Builder appliance requires the --force parameter, which instructs the SoS Utility to identify the SDDC Manager VM deployed by VMware Cloud Builder during the bring-up process, and then execute the health check remotely. For example:
./sos --health-check --force
Option Description
--certificate-health

Verifies that the component certificates are valid (within the expiry date).

--connectivity-health

Performs a connectivity health check to inspect whether the different components of the system such as the ESXi hosts, vCenter Servers, NSX Manager VMs, and SDDC Manager VM can be pinged.

--compute-health

Performs a compute health check.

--general-health

Verifies ESXi entries across all sources, checks the Postgres DB operational status for hosts, checks ESXi for error dumps, and gets NSX Manager and cluster status.

--get-host-ips

Returns server information.

--health-check

Performs all available health checks.

--ntp-health

Verifies whether the time on the components is synchronized with the NTP server in the VMware Cloud Builder appliance.

--services-health

Performs a services health check to confirm whether services are running

--run-vsan-checks Runs proactive vSAN tests to verify the ability to create VMs within the vSAN disks.

Sample Output

The following text is a sample output from an --ntp-health operation.

root@cloud-builder [ /opt/vmware/sddc-support ]# ./sos --ntp-health --skip-known-host --force
Welcome to Supportability and Serviceability(SoS) utility!

User passed --force flag, Running SOS from Cloud Builder VM, although Bringup is completed and SDDC Manager is available. Please expe           ct failures with SoS operations.
Health Check : /var/log/vmware/vcf/sddc-support/healthcheck-2020-02-11-23-03-53-24681
Health Check log : /var/log/vmware/vcf/sddc-support/healthcheck-2020-02-11-23-03-53-24681/sos.log
SDDC Manager : sddc-manager.vrack.vsphere.local
NTP : GREEN
+-----+-----------------------------------------+------------+-------+
| SL# |                   Area                  |   Title    | State |
+-----+-----------------------------------------+------------+-------+
|  1  |    ESXi : esxi-1.vrack.vsphere.local    |  ESX Time  | GREEN |
|  2  |    ESXi : esxi-2.vrack.vsphere.local    |  ESX Time  | GREEN |
|  3  |    ESXi : esxi-3.vrack.vsphere.local    |  ESX Time  | GREEN |
|  4  |    ESXi : esxi-4.vrack.vsphere.local    |  ESX Time  | GREEN |
|  5  | vCenter : vcenter-1.vrack.vsphere.local | NTP Status | GREEN |
+-----+-----------------------------------------+------------+-------+

Legend:

 GREEN - No attention required, health status is NORMAL
 YELLOW - May require attention, health status is WARNING
 RED - Requires immediate attention, health status is CRITICAL


Health Check completed successfully for : [NTP-CHECK]

The following text is sample output from a --vm-screenshots log collection operation.

root@cloud-builder [ /opt/vmware/sddc-support ]# ./sos --vm-screenshots  
     --skip-known-host --force
Welcome to Supportability and Serviceability(SoS) utility!

User passed --force flag, Running SOS from Cloud Builder VM, although Bringup is completed 
and SDDC Manager is available. Please expect failures with SoS operations.
Logs : /var/log/vmware/vcf/sddc-support/sos-2018-08-24-10-50-20-8013
Log file : /var/log/vmware/vcf/sddc-support/sos-2018-08-24-10-50-20-8013/sos.log
Log Collection completed successfully for : [VMS_SCREENSHOT]