Cloud proxy troubleshooting steps are provided to help you easily resolve issues that you may come across in vRealize Operations Cloud.

Before you proceed with troubleshooting, see the Cloud Proxy FAQ.

Installation and/or First Boot Failure

To verify the issue, check if /var/log/firstboot contains a file named "Succeeded".

If not, the following problems could result in vRealize Operations Cloud installation and/or first boot failure:

  1. OTK used while deploying Cloud Proxy is invalid. To verify, check the cloud proxy console.

    Solution: Redeploy cloud proxy.

  2. Some of the partitions in the Cloud Proxy VM ran out of space - Installation failed with the error message No space on partition.

    Solution: Contact VMware support.

Cloud Proxy VM is running, but the status is Offline in vRealize Operations Cloud.

To verify the connection, use the following commands: (For the complete list of commands, please see Using the Cloud Proxy Command-Line Interface.
# Overall status of cloud proxy:cprc-cli -s

# Ping itself:
ip addr
ping <address>


# Ping gateway:
ip route
ping <gateway>

# Verify the connection outside the cloud proxy,
ping 8.8.8.8

Note: If you are using a network proxy, 
use the /opt/vmware/share/vami/vami_config_net option#5 command 
to ensure you have the correct configuration for the testings.

The following problems could result in vRealize Operations Cloud displaying the status of cloud proxy as offline.

  1. Incorrect network proxy information in cloud proxy configuration. 
    To verify the connection via a network proxy, use the following:
    curl -vvv --proxy http(s)://proxy_user:proxy_pass@proxy_ip:proxy_port -H 'Accept: application/json' -H 'Content-Type: application/json' -X GET https://<gateway url>/casa/security/ping (gateway url example - 10238.gw.dev.vrops-ops.com)
     
    To ignore SSL validation for a proxy server, 
    use curl --proxy-insecure. With SSL validation the customer can provide Proxy Server certificate during cloud proxy deployment or re-configuration 
    so that provided certificate from customer can be used to check the connection with curl with SSL certificate validation.
    Solution:
    1. SSH to the Cloud Proxy VM and set the connectretry to 0 in /storage/db/vmware-vrops-cprc/configuration/cprc.configuration to ensure that the Cloud Proxy retries to connect.
    2. Shutdown the Cloud Proxy VM.
    3. Update the network proxy configurations from the vCenter Server VM options using the vApp options Edit OVF Details for a Virtual Machine.
    4. Boot the Cloud Proxy VM.
  2. Incorrect Firewall settings. 
    To verify access to outside URLs:
    # For URLs to whitelist, please refer to the prerequisite section in Configuring Cloud Proxies in vRealize Operations Cloud.
    
    # ping s3-us-west-2.amazonaws.com
    
    # The Amazon vRealize Operations Cloud instance can be found in this file: cat $VCOPS_BASE/user/conf/collector/collector.properties
    
    # Example row ttpRelayServerURL=https://10027.gw.staging.vrops-ops.com/suite-api
    wget <ttpRelayServerURL>
    Solution:
    1. SSH to the Cloud Proxy VM and set connectretry to 0 in /storage/db/vmware-vrops-cprc/configuration/cprc.configuration to ensure that the Cloud Proxy retries to connect.
    2. Shutdown the Cloud Proxy VM.
    3. Whitelist URLs mentioned in the prerequisite section of Configuring Cloud Proxies in vRealize Operations Cloud.
    4. Boot the Cloud Proxy VM.
  3. Required ports are not open.
    To verify:
    openssl s_client -showcerts -connect {address}:443
     
    curl -v telnet://{address}:443
    
    # Or, change the address to the machine you want to check: 
    python -c "import socket; print(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex(('127.0.0.1', 443)))"
    
    # If you get a !=0 response, the server is not listening to the port.
    Solution:
    1. SSH to the Cloud Proxy VM and set connectretry to 0 in /storage/db/vmware-vrops-cprc/configuration/cprc.configuration to ensure that the Cloud Proxy retries to connect.
    2. Provide port access as mentioned in the prerequisite section of Configuring Cloud Proxies in vRealize Operations Cloud
    3. Boot the Cloud Proxy VM.
  4. Invalid certificate.
    To verify:
    openssl s_client -showcerts -connect {address}:443
    Solution:
    1. SSH to the Cloud Proxy VM and set connectretry to 0 in /storage/db/vmware-vrops-cprc/configuration/cprc.configuration to ensure that the Cloud Proxy retries to connect.
    2. Follow the steps mentioned in VMware KB Article, 83698.
  5. The logs folder /storage/log is running out of partition space. 

    Solution: Remove log files to ensure that enough space is available. Note that this is an exceptional case. In normal conditions, log files are auto archived.

  6. One or more of the following services are down: httpd-north.service, haproxy.service and collector.service.
    Solution:
    • Check service status by running the following command: systemctl status <service name>.
    • To start service, use the following command: systemctl start <service name>.
  7. OTK expired.

    Solution: Redeploy Cloud Proxy with fresh OTK.

Cloud proxy is online, and state of Cloud Account is Collecting, but status is Object Down.  

The following problem could result in vRealize Operations Cloud displaying the state of Cloud Account as Collecting, while the status is, Object Down.

  1. Incorrect account credentials. 

    Solution: Check and update the credentials used while setting up the cloud account.

Cloud proxy status is stuck in Going Online.

It can take up to 20 mins on first reboot, for the cloud proxy to be registered and come online. Wait for the specified time to see if cloud proxy comes online. If it still does not come online, one or more of the following services are down: httpd-north.service, haproxy.service, and collector.service.

Solution:
  1. Check service status by running the following command: systemctl status <service name>
  2. To start service, use the following command: systemctl start <service name>.

Cloud proxy does not upgrade automatically, after the upgrade of vRealize Operations Cloud  

There could be a few possible reasons why cloud proxy does not upgrade automatically after an upgrade of vRealize Operations Cloud.

  1. High network latency leading to PAK download failure. Latency of >500ms is not supported.

    Solution: See the VMWare KB article 80590 on how to manually upgrade cloud proxy via CLI.

  2. Upgrade status is stuck at Running since the previous upgrade had failed.
    Solution: Follow the steps given below to change the upgrade status.
    1. Stop the casa service: systemctl stop vmare-casa.service.
    2. Change the upgrade status from RUNNING to NONE in the following files:
      ./storage/db/vmware-vrops-cprc/status/cprc.upgrade.status
      ./storage/db/vmware-vrops-cprc/status/cprc.pak.status
    3. See the VMware KB article 80590 and run the manual upgrade.

Cloud proxy gets disconnected at regular intervals

This is a known limitation related to DNS resolution failure. Contact VMware support.