Cloud proxy troubleshooting steps are provided to help you easily resolve issues that you may come across in vRealize Operations.
Before you proceed with troubleshooting, see the Cloud Proxy FAQ.
Installation and/or First Boot Failure
To verify the issue, check if /var/log/firstboot
contains a file named "Succeeded".
If not, the following problems could result in vRealize Operations installation and/or first boot failure:
- Unique Registration Key used while deploying Cloud Proxy is invalid. To verify, check the cloud proxy console.
Solution: Redeploy cloud proxy.
Cloud Proxy VM is running, but the status is Offline in vRealize Operations.
# Overall status of cloud proxy:cprc-cli -s # Ping itself: ip addr ping <address> # Ping gateway: ip route ping <gateway> # Verify the connection outside the cloud proxy, ping 8.8.8.8 Note: If you are using a network proxy, use the /opt/vmware/share/vami/vami_config_net option#5 command to ensure you have the correct configuration for the testings.
The following problems could result in vRealize Operations displaying the status of cloud proxy as offline.
- Incorrect network proxy information in cloud proxy configuration.
To verify the connection via a network proxy, use the following:
curl -vvv --proxy http(s)://proxy_user:proxy_pass@proxy_ip:proxy_port -H 'Accept: application/json' -H 'Content-Type: application/json' -X GET https://<gateway url>/casa/security/ping (gateway url example - 10238.gw.dev.vrops-ops.com) To ignore SSL validation for a proxy server, use curl --proxy-insecure. With SSL validation the customer can provide Proxy Server certificate during cloud proxy deployment or re-configuration so that provided certificate from customer can be used to check the connection with curl with SSL certificate validation.
Solution:- SSH to the Cloud Proxy VM and set the
connectretry
to0
in/storage/db/vmware-vrops-cprc/configuration/cprc.configuration
to ensure that the Cloud Proxy retries to connect. - Shutdown the Cloud Proxy VM.
- Update the network proxy configurations from the vCenter Server VM options using the vApp options Edit OVF Details for a Virtual Machine.
- Boot the Cloud Proxy VM.
- SSH to the Cloud Proxy VM and set the
- Required ports are not open.
To verify:
openssl s_client -showcerts -connect {address}:443 curl -v telnet://{address}:443 # Or, change the address to the machine you want to check: python -c "import socket; print(socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex(('127.0.0.1', 443)))" # If you get a !=0 response, the server is not listening to the port.
Solution:- SSH to the Cloud Proxy VM and set
connectretry
to0
in /storage/db/vmware-vrops-cprc/configuration/cprc.configuration to ensure that the Cloud Proxy retries to connect. - Provide port access as mentioned in the prerequisite section of Configuring Cloud Proxies in vRealize Operations
- Boot the Cloud Proxy VM.
- SSH to the Cloud Proxy VM and set
- Invalid certificate.
To verify:
openssl s_client -showcerts -connect {address}:443
Solution:- SSH to the Cloud Proxy VM and set
connectretry
to0
in /storage/db/vmware-vrops-cprc/configuration/cprc.configuration to ensure that the Cloud Proxy retries to connect. - Follow the steps mentioned in VMware KB Article, 83698.
- SSH to the Cloud Proxy VM and set
- The logs folder /storage/log is running out of partition space.
Solution: Remove log files to ensure that enough space is available. Note that this is an exceptional case. In normal conditions, log files are auto archived.
- One or more of the following services are down:
httpd-north.service
,haproxy.service
andcollector.service
.Solution:- Check service status by running the following command: systemctl status <service name>.
- To start service, use the following command: systemctl start <service name>.
- Unique Registration Key expired.
Solution: Redeploy Cloud Proxy with new Unique Registration Key.
Cloud proxy is online, and state of Cloud Account is Collecting
, but status is Object Down
.
The following problem could result in vRealize Operations displaying the state of Cloud Account as Collecting
, while the status is, Object Down
.
- Incorrect account credentials.
Solution: Check and update the credentials used while setting up the cloud account.
Cloud proxy status is stuck in Going Online
.
It can take up to 20 mins on first reboot, for the cloud proxy to be registered and come online. Wait for the specified time to see if cloud proxy comes online. If it still does not come online, one or more of the following services are down: httpd-north.service
, haproxy.service
, and collector.service
.
- Check service status by running the following command:
systemctl status <service name>
- To start service, use the following command:
systemctl start <service name>
.
Cloud proxy does not upgrade automatically, after the upgrade of vRealize Operations
There could be a few possible reasons why cloud proxy does not upgrade automatically after an upgrade of vRealize Operations.
- High network latency leading to PAK download failure. Latency of >500ms is not supported.
Solution: See the VMWare KB article 80590 on how to manually upgrade cloud proxy via CLI.
- Upgrade status is stuck at
Running
since the previous upgrade had failed.Solution: Follow the steps given below to change the upgrade status.- Stop the casa service: systemctl stop vmware-casa.service.
- Change the upgrade status from
RUNNING
toNONE
in the following files:./storage/db/vmware-vrops-cprc/status/cprc.upgrade.status ./storage/db/vmware-vrops-cprc/status/cprc.pak.status
- See the VMware KB article 80590 and run the manual upgrade.
Cloud proxy gets disconnected at regular intervals
- Check the network connectivity and latency.
- Check if the cloud proxy VM can reach the DNS and use the NSlookup to validate the DNS connectivity.