Cloud Proxy Troubleshooting

Cloud proxy troubleshooting steps are provided to help you easily resolve issues that you may come across in vRealize Operations.

Before you proceed with troubleshooting, see the Cloud Proxy FAQ.

Installation and/or First Boot Failure

To verify the issue, check if /var/log/firstboot contains a file named "Succeeded".

If not, the following problems could result in vRealize Operations installation and/or first boot failure:

1. OTK used while deploying Cloud Proxy is invalid. To verify, check the cloud proxy console.

   Solution: Redeploy cloud proxy.

Cloud Proxy VM is running, but the status is Offline in vRealize Operations.

To verify the connection, use the following commands: (For the complete list of commands, please see Using the Cloud Proxy Command-Line Interface.

# Overall status of cloud proxy:cprc-cli -s

# Ping itself:
ip addr
ping <address>


# Ping gateway:
ip route
ping <gateway>

# Verify the connection outside the cloud proxy,
ping 8.8.8.8

Note: If you are using a network proxy, 
use the /opt/vmware/share/vami/vami_config_net option#5 command 
to ensure you have the correct configuration for the testings.

The following problems could result in vRealize Operations displaying the status of cloud proxy as offline.

1. Incorrect network proxy information in cloud proxy configuration. 
   To verify the connection via a network proxy, use the following:

   curl -vvv --proxy http(s)://proxy_user:proxy_pass@proxy_ip:proxy_port -H 'Accept: application/json' -H 'Content-Type: application/json' -X GET https://<gateway url>/casa/security/ping (gateway url example - 10238.gw.dev.vrops-ops.com)
    
   To ignore SSL validation for a proxy server, 
   use curl --proxy-insecure. With SSL validation the customer can provide Proxy Server certificate during cloud proxy deployment or re-configuration 
   so that provided certificate from customer can be used to check the connection with curl with SSL certificate validation.

   Solution:

   1. SSH to the Cloud Proxy VM and set the connectretry to 0 in /storage/db/vmware-vrops-cprc/configuration/cprc.configuration to ensure that the Cloud Proxy retries to connect.
   2. Shutdown the Cloud Proxy VM.
   3. Update the network proxy configurations from the vCenter Server VM options using the vApp options Edit OVF Details for a Virtual Machine.
   4. Boot the Cloud Proxy VM.
2. Required ports are not open.
   To verify:

   openssl s_client -showcerts -connect {address}:443
    
   curl -v telnet://{address}:443
   
   # Or, change the address to the machine you want to check: 
   python -c "import socket; print(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex(('127.0.0.1', 443)))"
   
   # If you get a !=0 response, the server is not listening to the port.

   Solution:

   1. SSH to the Cloud Proxy VM and set connectretry to 0 in /storage/db/vmware-vrops-cprc/configuration/cprc.configuration to ensure that the Cloud Proxy retries to connect.
   2. Provide port access as mentioned in the prerequisite section of Configuring Cloud Proxies in vRealize Operations
   3. Boot the Cloud Proxy VM.
3. Invalid certificate.
   To verify:

   openssl s_client -showcerts -connect {address}:443

   Solution:

   1. SSH to the Cloud Proxy VM and set connectretry to 0 in /storage/db/vmware-vrops-cprc/configuration/cprc.configuration to ensure that the Cloud Proxy retries to connect.
   2. Follow the steps mentioned in VMware KB Article, 83698.
4. The logs folder /storage/log is running out of partition space. 

   Solution: Remove log files to ensure that enough space is available. Note that this is an exceptional case. In normal conditions, log files are auto archived.

5. One or more of the following services are down: httpd-north.service, haproxy.service and collector.service.
   Solution:

   • Check service status by running the following command: systemctl status <service name>.
   • To start service, use the following command: systemctl start <service name>.
6. OTK expired.

   Solution: Redeploy Cloud Proxy with fresh OTK.

Cloud proxy is online, and state of Cloud Account is Collecting, but status is Object Down.  

The following problem could result in vRealize Operations displaying the state of Cloud Account as Collecting, while the status is, Object Down.

1. Incorrect account credentials. 

   Solution: Check and update the credentials used while setting up the cloud account.

Cloud proxy status is stuck in Going Online.

It can take up to 20 mins on first reboot, for the cloud proxy to be registered and come online. Wait for the specified time to see if cloud proxy comes online. If it still does not come online, one or more of the following services are down: httpd-north.service, haproxy.service, and collector.service.

Solution:

1. Check service status by running the following command: systemctl status <service name>
2. To start service, use the following command: systemctl start <service name>.

Cloud proxy does not upgrade automatically, after the upgrade of vRealize Operations  

There could be a few possible reasons why cloud proxy does not upgrade automatically after an upgrade of vRealize Operations.

1. High network latency leading to PAK download failure. Latency of >500ms is not supported.

   Solution: See the VMWare KB article 80590 on how to manually upgrade cloud proxy via CLI.

2. Upgrade status is stuck at Running since the previous upgrade had failed.
   Solution: Follow the steps given below to change the upgrade status.

   1. Stop the casa service: systemctl stop vmare-casa.service.
   2. Change the upgrade status from RUNNING to NONE in the following files:

      ./storage/db/vmware-vrops-cprc/status/cprc.upgrade.status
      ./storage/db/vmware-vrops-cprc/status/cprc.pak.status

   3. See the VMware KB article 80590 and run the manual upgrade.

Cloud proxy gets disconnected at regular intervals

There could be a few possible reasons why cloud proxy gets disconnected at regular intervals.

1. Check the network connectivity and latency.
2. Check if the cloud proxy VM can reach the DNS and use the NSlookup to validate the DNS connectivity.