Advanced Tanzu Operations Manager troubleshooting with BOSH CLI

You can use the BOSH CLI to help diagnose and resolve issues with your VMware Tanzu Operations Manager deployment. Before using the information and techniques in this topic, review Diagnosing TAS for VMs deployment problems.

To follow the steps in this topic, you must log in to the BOSH Director VM. The BOSH Director runs on the virtual machine (VM) that Tanzu Operations Manager deploys on the first install of the BOSH Director tile.

After you authenticate in to the BOSH Director, you can run specific commands using the BOSH Command Line Interface (BOSH CLI). BOSH Director diagnostic commands have access to information about your entire Tanzu Operations Manager installation.

Before you run any BOSH CLI commands, verify that no BOSH Director tasks are running on the Tanzu Operations Manager VM. For more information, see Tasks in BOSH CLI commands in the BOSH documentation.

Gather credential and IP address information

Before you begin troubleshooting with the BOSH CLI, follow these instructions to collect the information you need from the Tanzu Operations Manager interface:

1. Open the Tanzu Operations Manager interface by going to the Tanzu Operations Manager fully qualified domain name (FQDN) in a web browser.

2. Click the BOSH Director tile and click the Status tab.

3. Record the IP address for the Director job. This is the IP address of the VM where the BOSH Director runs.

   

4. Click the Credentials tab.

5. Go to Director Credentials and click Link to Credential. Record these credentials.

   

6. Return to the Tanzu Operations Manager Installation Dashboard.

7. (Optional) To prepare to troubleshoot the job VM for any other product, click the product tile and repeat the previous procedure to record the IP address and VM credentials for that job VM.

8. Log out of Tanzu Operations Manager.

Ensure that there are no Tanzu Operations Managerinstallations or updates in progress while using the BOSH CLI.

Log in to the Tanzu Operations Manager VM with SSH

Use SSH to connect to the Tanzu Operations Manager VM. To log in to the Tanzu Operations Manager VM, go to the procedure for your IaaS:

• AWS
• Azure
• GCP
• OpenStack
• vSphere

AWS

To log in to the Tanzu Operations Manager VM with SSH in AWS, you need the key pair you used when you created the Tanzu Operations Manager VM. To see the name of the key pair, click the Tanzu Operations Manager VM and locate the key pair name in properties.

To log in to the Tanzu Operations Manager VM with SSH in AWS:

1. Locate the Tanzu Operations Manager FQDN on the AWS EC2 instances page.

2. Run chmod 600 ops_mgr.pem to change the permissions on the .pem file to be more restrictive. For example:

   chmod 600 ops_mgr.pem
   

3. Log in to the Tanzu Operations Manager VM with SSH.

   ssh -i ops_mgr.pem ubuntu@FQDN
   

   Where FQDN is the fully qualified domain name of Tanzu Operations Manager.

   For example:

   $ ssh -i ops_mgr.pem [email protected]
   

Azure

To log in to the Tanzu Operations Manager VM with SSH in Azure, you need the key pair you used when creating the Tanzu Operations Manager VM. If you need to reset the SSH key, locate the Tanzu Operations Manager VM in the Azure portal and click Reset Password.

To log in to the Tanzu Operations Manager VM with SSH in Azure:

1. From the Azure portal, locate the Tanzu Operations Manager FQDN by selecting the VM.

2. Change the permissions for your SSH private key by running the following command:

   chmod 600 PRIVATE-KEY
   

   Where PRIVATE-KEY is the name of your SSH private key.

3. SSH into the Tanzu Operations Manager VM and run:

   ssh -i PRIVATE-KEY ubuntu@FQDN
   

   Where:

   • FQDN is the FQDN for your Tanzu Operations Manager deployment.
   • PRIVATE-KEY is the name of your SSH private key.

GCP

To log in to the Tanzu Operations Manager VM with SSH in GCP:

1. Confirm that you have installed the Google Cloud SDK and CLI. For more information, see the Google Cloud Platform documentation.

2. Initialize Google Cloud CLI, using a user account with Owner, Editor, or Viewer permissions to access the project. Ensure that the Google Cloud CLI can log in to the project by running the command gcloud auth login.

3. From the GCP web console, go to Compute Engine.

4. Locate the Tanzu Operations Manager VM in the VM Instances list.

5. Under Remote access, click the SSH drop-down menu and click View gcloud command.

6. Copy the SSH command that appears in the pop-up window.

7. Paste the command into your terminal window to SSH to the VM. For example:

   $ gcloud compute ssh "YOUR-VM" --zone "YOUR-ZONE-ID"
   

8. Run sudo su - ubuntu to switch to the ubuntu user.

OpenStack

To log in to the Tanzu Operations Manager VM with SSH in OpenStack, you need the key pair that you created in Configure Security in Deploying Tanzu Operations Manager on OpenStack. If you must reset the SSH key, locate the Tanzu Operations Manager VM in the OpenStack console and boot it in recovery mode to generate a new key pair.

To log in to the Tanzu Operations Manager VM with SSH in OpenStack:

1. Locate the Tanzu Operations Manager FQDN on the Access & Security page.

2. Run chmod 600 ops_mgr.pem to change the permissions on the .pem file to be more restrictive. For example:

   chmod 600 ops_mgr.pem
   

3. Log in to the Tanzu Operations Manager VM with SSH.

   ssh -i ops_mgr.pem ubuntu@FQDN
   

   Where FQDN is the fully qualified domain name of Tanzu Operations Manager.

   For example:

   $ ssh -i ops_mgr.pem [email protected]
   

vSphere

To log in to the Tanzu Operations Manager VM with SSH in vSphere, you must have the public SSH key that imports the Tanzu Operations Manager .ova or .ovf file into your virtualization system.

You set the public SSH key in the Public SSH Key text box of the Customize template screen when you deployed Tanzu Operations Manager. For more information, see Deploy Tanzu Operations Manager in Deploying Tanzu Operations Manager on vSphere.

If you lose your SSH key, you must shut down the Tanzu Operations Manager VM in the vSphere UI and then reset the public SSH key. For more information, see Edit vApp Settings in the vSphere documentation.

To log in to the Tanzu Operations Manager VM with SSH in vSphere:

1. Run the following command:

   ssh ubuntu@FQDN
   

   Where FQDN is the fully qualified domain name of Tanzu Operations Manager.

   For example:

   $ ssh [email protected]
   

2. When you are prompted, enter the public SSH key.

Authenticate with the BOSH Director VM

To authenticate with BOSH, use one of the following methods:

• Set the BOSH Environment Variables on the Tanzu Operations Manager VM.
• Create a Local BOSH Director Alias.

Set the BOSH environment variables on the Tanzu Operations Manager VM

If you have access to the Tanzu Operations Manager VM, SSH into the Tanzu Operations Manager VM and do the following:

1. Record the Bosh Commandline Credentials from the Credentials tab of the BOSH Director tile.

2. SSH into the Tanzu Operations Manager VM. For more information, see Log in to the Tanzu Operations Manager VM with SSH.

3. Export all the environment variables by running the following command:

   export YOUR-ENV-VARIABLES
   

   Where YOUR-ENV-VARIABLES is the value for credential in the BOSH command line credentials that you recorded in a previous step.

   For example:

   $ export BOSH_CLIENT=ops_manager \
   BOSH_CLIENT_SECRET=some_secret \
   BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate \
   BOSH_ENVIRONMENT=10.0.0.5 bosh
   

4. Verify that BOSH access works by running the following command.

   bosh deployments
   

Create a local BOSH Director alias

To create a BOSH Director alias and log in to the BOSH Director VM:

1. Run the following command to create a local alias for the BOSH Director using the BOSH CLI:

   bosh alias-env MY-ENV -e DIRECTOR-IP-ADDRESS --ca-cert /var/tempest/workspaces/default/root_ca_certificate
   

   Where:

   • MY-ENV: Enter an alias for the BOSH Director, such as gcp.
   • DIRECTOR-IP-ADDRESS: Enter the IP address of your BOSH Director VM.

   For example:

   $ bosh alias-env gcp -e 10.0.0.3 --ca-cert /var/tempest/workspaces/default/root_ca_certificate
   

2. Log in to the BOSH Director VM using one of the following options:

   • Internal User Store Login through UAA: Log in to the BOSH Director VM using BOSH.
   • External User Store Login through SAML: Use an external user store to log in to the BOSH Director VM.

Log in to the BOSH Director VM with UAA

1. Retrieve the Director user name and password in one of the following ways:

   • In Tanzu Operations Manager, click the BOSH Director, select the Credentials tab, and click the link to Director Credentials.
   • Browse to https://FQDN/api/v0/deployed/director/credentials/director_credentials to obtain the password, where FQDN is the fully qualified domain name of Tanzu Operations Manager.

2. Log in to the BOSH Director VM:

   bosh -e MY-ENV log-in
   

   Where MY-ENV is the alias for your BOSH Director.

   For example:

   $ bosh -e gcp log-in
   

   Follow the BOSH CLI prompts and enter the BOSH Director credentials to log in to the BOSH Director VM.

Log in to the BOSH Director VM with SAML

1. Log in to your identity provider and use the following information to configure SAML Service Provider Properties:

   • Service Provider Entity ID: bosh-uaa
   • ACS URL: https://DIRECTOR-IP-ADDRESS:8443/saml/SSO/alias/bosh-uaa
   • Binding: HTTP Post
   • SLO URL: https://DIRECTOR-IP-ADDRESS:8443/saml/SSO/alias/bosh-uaa
   • Binding: HTTP Redirect
   • Name ID: Email Address

2. Log in to the BOSH Director VM:

   bosh -e MY-ENV log-in
   

   Where MY-ENV is the alias for your BOSH Director.

   For example:

   $ bosh -e gcp log-in
   

   Follow the BOSH CLI prompts and enter your SAML credentials to log in to the BOSH Director VM.

   Your browser must be able to reach the BOSH Director to log in with SAML.

3. Click Log in with organization credentials (SAML). A temporary authentication code appears.

4. Copy the Temporary Authentication Code that appears in your browser.

5. You see a login confirmation. For example:

   Logged in as [email protected]
   

SSH into the BOSH Director VM

Do the following steps to log in to the BOSH Director VM with SSH:

1. From Tanzu Operations Manager, open the BOSH Director tile.

2. Click the Credentials tab.

3. Next to Bbr Ssh Credentials, click Link to Credential. A tab opens containing a JSON credential structure.

4. Copy the RSA PRIVATE KEY and paste it into a file named bbr.pem. Include -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY-----.

   Keep the key secure. The key provides full access to the entire Tanzu Operations Manager environment.

5. Replace all \n characters in bbr.pem with a line break.

6. Copy bbr.pem to the ~/.ssh/ directory on your computer.

7. Run chmod 600 ~/.ssh/bbr.pem to modify the permissions of the file.

8. Log in to the BOSH Director VM with SSH from your machine.

   ssh bbr@BOSH-DIRECTOR-IP -i ~/.ssh/bbr.pem
   

   If you use GCP, ensure SSH port 22 is open for the BOSH Director VM in your GCP console. If the SSH port is not open, open it by creating a firewall rule.

9. Run sudo -i to get the root privilege.

Use the BOSH CLI for troubleshooting

This section describes five BOSH CLI commands commonly used during troubleshooting.

• VMs: Lists the VMs in a deployment.
• Cloud Check: Runs a cloud consistency check and interactive repair.
• Create Recovery Plan and Recover: Interactively create a plan to use in repairing resources.
• SSH: Starts an interactive session or executes commands with a VM.

BOSH VMs

The bosh vms command provides an overview of the virtual machines that BOSH manages.

To use this command, run bosh -e MY-ENV vms to see an overview of all virtual machines managed by BOSH, or bosh -e MY-ENV -d MY-DEPLOYMENT vms to see only the virtual machines associated with a particular deployment. Replace MY-ENV with your environment, and, if using the -d flag, also replace MY-DEPLOYMENT with the name of a deployment.

When you troubleshoot an issue with your deployment, bosh vms might show a VM in an unknown state. Run bosh cloud-check on a VM in an unknown state to instruct BOSH to diagnose problems with the VM.

You can also run bosh vms to identify VMs in your deployment, then use the bosh ssh command to log in to an identified VM with SSH for further troubleshooting.

bosh vms supports the following arguments:

• --dns: Report also includes the DNS A record for each VM.

• --vitals: Report also includes load, CPU, memory usage, swap usage, system disk usage, ephemeral disk usage, and persistent disk usage for each VM.

  The Status tab of the TAS for VMs tile shows information similar to the bosh vms output.

BOSH Cloud check

Run the bosh cloud-check command to instruct BOSH to detect differences between the VM state database maintained by the BOSH Director and the actual state of the VMs. For each difference detected, bosh cloud-check can offer the following repair options:

• Reboot VM: Instructs BOSH to reboot a VM. Rebooting can resolve many transient errors.

• Ignore problem: Instructs BOSH to do nothing. You might want to ignore a problem ands instead run bosh ssh in an attempt to troubleshoot directly on the machine.

• Reassociate VM with corresponding instance: Updates the BOSH Director state database. Use this option if you believe that the BOSH Director state database is in error and that a VM is correctly associated with a job.

• Recreate VM using last known apply spec: Instructs BOSH to destroy the server and recreate it from the deployment manifest that the installer provides. Use this option if a VM is corrupted.

• Delete VM reference: Instructs BOSH to delete a VM reference in the Director state database. If a VM reference exists in the state database, BOSH expects to find an agent running on the VM. Select this option only if you know that this reference is in error. Once you delete the VM reference, BOSH can no longer control the VM.

To use this command, run bosh -e MY-ENV -d MY-DEPLOYMENT cloud-check, where MY-ENV is your environment, and MY-DEPLOYMENT is your deployment.

Example scenarios

Unresponsive Agent

$ bosh -e example-env -d example-deployment cloud-check

ccdb/0 (vm-3e37133c-bc33-450e-98b1-f86d5b63502a) is not responding:

• Ignore problem
• Reboot VM
• Recreate VM using last known apply spec
• Delete VM reference (DANGEROUS!)

Missing VM

$ bosh -e example-env -d example-deployment cloud-check

VM with cloud ID `vm-3e37133c-bc33-450e-98b1-f86d5b63502a' missing:

• Ignore problem
• Recreate VM using last known apply spec
• Delete VM reference (DANGEROUS!)

Unbound Instance VM

$ bosh -e example-env -d example-deployment cloud-check 

VM `vm-3e37133c-bc33-450e-98b1-f86d5b63502a' reports itself as `ccdb/0' but does not have a bound instance:

• Ignore problem
• Delete VM (unless it has persistent disk)
• Reassociate VM with corresponding instance

Out of Sync VM

$ bosh -e example-env -d example-deployment cloud-check 

VM vm-3e37133c-bc33-450e-98b1-f86d5b63502a' is out of sync: expectedcf-d7293430724a2c421061: ccdb/0’, got `cf-d7293430724a2c421061: nats/0’:

• Ignore problem
• Delete VM (unless it has persistent disk)

BOSH Create Recovery Plan and Recover

If underlying infrastructure issues cause widespread problems with your deployment, run the bosh create-recovery-plan and bosh recover commands. The bosh create-recovery-plan command scans the deployment for problems and then prompts you to generate a recovery plan, which is saved to a file. The bosh recover command consumes the recovery plan file.

These commands together work similarly to bosh cloud-check, with the following differences:

• You select resolutions to problems by instance group and problem type. The bosh cloud-check command asks for a resolution for each particular problem.
• You can override max_in_flight per instance group, to speed deployment recovery. The bosh cloud-check command uses the max_in_flight values in the deployment manifest.

To use these commands:

1. Run bosh -e MY-ENV -d MY-DEPLOYMENT create-recovery-plan MY-RECOVERY-PLAN, where:

   • MY-ENV is your environment
   • MY-DEPLOYMENT is your deployment
   • MY-RECOVERY-PLAN is a filename for the recovery plan

   For example:

   $ bosh -e example-env -d example-deployment create-recovery-plan example-recovery-plan.yml
   

2. Run bosh -e MY-ENV -d MY-DEPLOYMENT recover MY-RECOVERY-PLAN, where:

   • MY-ENV is your environment
   • MY-DEPLOYMENT is your deployment
   • MY-RECOVERY-PLAN is the filename you provided to the bosh create-recovery-plan command

   For example:

   $ bosh -e example-env -d example-deployment recover example-recovery-plan.yml
   

For more information about the bosh create-recovery-plan and bosh-recover commands, see the BOSH documentation.

BOSH SSH

Use bosh ssh to log in to the VMs in your deployment with SSH.

To use bosh ssh:

1. Identify a VM to log in to with SSH. Run the following command to list the VMs in the given deployment:

   bosh -e MY-ENV -d MY-DEPLOYMENT vms
   

   Where:

   • MY-ENV is your environment alias.
   • MY-DEPLOYMENT is the deployment name.

2. Run:

   bosh -e MY-ENV -d MY-DEPLOYMENT ssh VM-NAME/GUID`
   

   For example:

   $ bosh -e example-env -d example-deployment ssh diego-cell/abcd0123-a012-b345-c678-9def01234567