You can troubleshoot some common deployment issues for VMware Tunnel using these tools.

Troubleshooting VMware Tunnel Using the Tunnel_snap Utility

The tunnel_snap utility collects all the necessary diagnostic data required for troubleshooting your Tunnel deployment. This utility saves time by reducing the back-and-forth communication with support.

You must run this utility on each Tunnel server separately, on these folders:

  • awcm.dat

  • ca.pem

  • device.xml

  • dh2048.pem

  • server.conf

  • tunnel_init.log

  • tunnel.log

  • tunnel.log.1

  • version.info

  • vpn.dat

/opt/vmware/tunnel

To run the utility, use this command:

sudo ./tunnel_snap.sh

The utility collects the diagnostic data in:

_/opt/vmware/tunnel/tunnel_snap.tar _

Troubleshooting VMware Tunnel Using the UAG Web UI

The UAG Web UI offers a way to check the service availability and collect all the UAG log files including Tunnel and Proxy log files.

  1. Click the Read More section to assist with troubleshooting.

  2. Monitor the Edge Services Status. Expand the Edge Services section and find Tunnel. When Tunnel is running as expected, a green light on the left side of the service is shown. If any other color light is shown, either the service is not running or it is running with errors that requires further investigation. In the UAG Web UI, hover over the color light shown for more information.

  3. Collecting Logs from the Appliance. Download the .zip archive of logs from the Support Settings section of the Admin UI. These log files are collected from the /opt/vmware/gateway/logs directory on the appliance.

  4. Review the appliance-agent.log, making sure that both Tunnel and the Proxy services are installed correctly.

    Note:

    The log should display: [main] INFO c.a.a.a.s.i.tunnel.TunnelInstaller - VMware Tunnel Proxy installation SUCCESS!!!! and/or [main] INFO c.a.a.a.s.i.tunnel.TunnelInstaller - VMware Tunnel Per-App Tunnel installation SUCCESS!!!!

You can access the Tunnel logs from the UAG without logging into the appliance by accessing a specific URL based on your deployment. To download a ZIP file that contains your logs, enter the following URL in a browser:https://<virtual appliance domain name>:9443/rest/v1/monitor/support-archive

Troubleshooting Per-App Tunnel Component

Use these commands to troubleshoot the Per-App Tunnel component.

Function

Command

Example

Unified Access Gateway/CentOS/RHEL 7.x

Start the Service

systemctl start vpnd.services

Unified Access Gateway/CentOS/RHEL 7.x

Stop the Service

systemctl stop vpnd.service

Unified Access Gateway/CentOS/RHEL 7.x

Restart the Service

systemctl restart vpnd.service

Troubleshooting PAC Reader

If you have any issues with the VMware Tunnel PAC Reader, check the status and the logs of the PAC Reader. The logs are located in the home pacreader folder on the PAC Reader. Use these commands to troubleshoot the PAC Reader.

Function

Command

Start the PAC Reader

./pacreader.sh start

Stop the PAC Reader

./pacreader.sh stop

Check the PAC Reader status

/pacreader status

Run the PAC Reader in validation mode

./pacreader.sh validate

This command tells the PAC Reader to fetch and parse the PAC file but does not push the rules to the Workspace ONE UEM console.

Troubleshooting Common Errors While Working With VMware Tunnel

This section covers common error messages that you may encounter while working with Tunnel and the procedure to fix the root cause of the problem.

Device Configuration Error

When the VMware Tunnel VPN profile is not installed on the device, end users might see Device Not Configured when they try to open a Tunnel client.

  1. In the UEM console , go to the Device Detail page of the affected device and click the Profiles tab to confirm if the Tunnel VPN profile is installed.

  2. For all the Android devices, open the Workspace ONE Intelligent Hub and under the Profiles section, check if the Tunnel VPN profile exists.

  3. For all iOS devices, go to Settings > VPN and verify the VPN configuration details.

TLS Handshake Failure

You encounter this error if the SSL certificate present on the device does not match with the certificate on the server or if the certificate is not valid.

  1. In the UEM console, go to the Tunnel configuration page and verify the front-end Certificate Thumbprint under server Authentication.

  2. For all the Android devices, open the Workspace ONE Intelligent Hub and under the Profiles section, verify the certificate thumbprint for the Type.cer.

    1. For all the iOS devices, go to Settings > General > Device Management> Device Manager.

    2. Click More Details and under the Certificate section, click the certificate with the Tunnel hostname.

  3. Scroll down to the SHA-1 text box and verify the certificate thumbprint.

  4. On the server side, open/opt/vmware/tunnel/vpnd/server.confand search for ssl_thumbprint.

  5. Verify if the thumbprint on the device, server, and the UEM console is the same. If not, restart the vpnd service on the UEM console and republish the VPN profile.

DNS Resolution Failure

You might encounter DNS resolution error if the VMware Tunnel server FQDN does not get resolved to an IP address.

  1. From the device connected network, ensure that the Tunnel server FQDN resolves to an IP address.

  2. In the command prompt, enter the following command: nslookup <Tunnel_Server_FQDN>. Tunnel server FQDN resolves to an IP address.

Unable to Reach the Tunnel Gateway

If device is unable to communicate with the Tunnel server on the mentioned port, you may not be able to reach the Tunnel gateway.

  1. From the device connected network, ensure that the device connects to the Tunnel server on the port that is mentioned in the Tunnel configuration. The device must get connected and display the Tunnel server front-end SSL certificate.

  2. In the command prompt, enter the following command:

    openssl s_client -connect <dest_fqdn>:<port> -servername
              <server_fqdn>
  3. In the Tunnel server, enter the following command: netstat -tlpn

    The server must display the port that is mentioned in the tunnel configuration.

  4. In the Tunnel server, enter the following command: systemctl status vpnd. The service must be active and running.

    Note:
    • Verify the Firewall and the load balancer rules.

    • SSL Offloading and SSL Bridging are not supported for the Per-App Tunnel component.

Access Denied Error / Device Unknown to Gateway

You might encounter an "access denied error" or a "device unknown to Gateway" error if the device details are not present on the Tunnel server or when the device is non-compliant.

  1. Open the Workspace ONE Intelligent Hub and verify the compliance status.

  2. Go to the Device detail page for the affected device and verify the device compliance status.

  3. From the /opt/vmware/tunnel/vpnd directory, run ./vpnreport allowlist --udid=<Device_udid>.In the result xml, the ComplianceStatusId must be 3 or 5 for the affected device UUID.

    Note:

    The connection between the Tunnel server and the API server connection must be successful to achieve the expected result.

No Apps Assigned

You might encounter the "No Apps Assigned" error within the Workspace ONE Tunnel application when the managed application is not mapped with the VMware VPN profile.

  • Go to the internal or the public application under Apps & Books and check for the device in the assignment group where the App Tunneling is enabled.

Unable to Access Internal Sites From Managed Apps Through the VPN

Intranet websites are not accessible from the Tunnel Server.

  1. Ensure that you can access the internal websites from the tunnel server. If it is a Cascade mode, the internal site must be accessible from the Backend server.

  2. Ensure that all the application binaries are allowlisted for the VPN. For example, applications like VMware Horizon Client and Microsoft Outlook might have multiple binaries that must be allowlisted.

Device Traffic Rules is Not Sent to the Devices

Device Traffic Rules control how traffic is directed through the VMware Tunnel when using the Per-App Tunnel component. These rules allow you to tunnel, block, or bypass traffic as needed. In some scenarios, the updated Device Traffic Rules is not sent to the devices. When the administrator changes the Device Traffic Rules and click Save, the Device Traffic Rules gets mapped to the profile, but the updated Device Traffic Rules is not replaced for the devices where the VPN profile is already installed. Device Traffic Rules is only updated for the newly enrolled devices or for the devices that have the VPN profile reinstalled.

To send the updated Device Traffic Rules to the devices post modifying the Device Traffic Rules, administrators must click Save and Publish. Save and Publish adds a version to the VPN profile and republishes Device Traffic Rules to all the devices.

Note:
  • If the administrator changes the Android application in the Device Traffic Rules and clicks Save and Publish, the VPN profiles for both iOS, Android profiles gets a version update and the VPN profile installs are queued for all the assigned devices.

  • Reinstalling the profile reissues the client certificate to the device with a new thumbprint.

VPN-Managed Application Fails to Honor Device Traffic Rules on Overriding the Device Traffic Rules

VPN-managed application fails to honor the Device Traffic Rules on overriding the Device Traffic Rules for the Child OG. The VPN profile fails to map the correct Device Traffic Rules configuration.

  • Make sure that you create the application and the VPN profile at the OG level which has the traffic rules that are overridden.

Unable to View Internal and Public Applications Under the Device Traffic Rules Application List

Internal and public applications are not displayed under the Device Traffic Rules application list. You might encounter this issue if the VPN profile is not mapped with the correct Tunnel Configuration.

  1. Go to Profile > List View.

  2. Select the profile that is mapped to the application and click VPN Payload. Verify the Tunnel server configuration.

  3. If the Tunnel not configured message is displayed, click Add version and remove the VPN payload.

  4. Add a new VPN Payload.

Tunnel Server is Not Up to Update With Respect to the Compliance Change Events

Devices fail to honor compliance policy updates. You might encounter this issue if the device compliance change event fails to reach the Tunnel server.

  1. In the Workspace ONE UEM console, go to All Settings > System > Advanced > Site URL.

  2. Verify the AirWatch Cloud Messaging connection.

  3. Perform the Tunnel test connection from the Tunnel configuration page.

  4. From the Tunnel server, verify the service status by running the following commands:

    1. systemctl status vpnd.

    2. systemctl status vpnreportd.

      Note:

      If you have multiple AirWatch Cloud Messaging that uses implicit clustering, configure the load balancer to use the cookie persistence that routes the AirWatch Cloud Messaging traffic.

Tunnel Front-End Server Fails to Communicate With the Back-End Server

Due to the incorrect network configuration or usage of an incorrect certificate for the server-client authentication, you might experience a communication failure between the Tunnel front-end server and the back-end server.

  1. Ensure that the front-end server can communicate with the back-end Tunnel server on the port mentioned in the tunnel configuration.

  2. Run the following command in the Tunnel front-end server:

    openssl s_client -connect <dest_fqdn>:<port> -servername
              <backend_fqdn>

    Must display the Tunnel back-end server SSL certificate.

  3. In the server.conf file, verify the following:

    On the Tunnel, front-end server verifies if the c_r_t (that is, cascade_root_thumbprint ) has the thumbprint of the back-end server's SSL certificate.

    1. The c_r_t in the Tunnel front-end server is same as the cascade_back_end_thumbprint in the back-end server.

    On the Tunnel back-end server c_r_t should have the root CA's thumbprint of the Tunnel front-end server's SSL certificate.

    1. When the AirWatch certificate is used for Server Auth, the c_r_t in the back-end server is always same as the ssl_thumbprint in the Tunnel front-end server.

    2. When a third-party SSL certificate is used for Server Auth, the c_r_t in the back-end server is the third party's root CA's thumbprint.

  4. Verify if there are any firewall or load balancer rules blocking between the front-end server to back-end Tunnel Server.

    Note:
    • SSL Offloading and SSL Bridging are not supported for the Per-App Tunnel configuration.

    • If you are using Public certificate for the server authentication, the certificate must have a Server and Client authentication under Enhanced Key Usage field.

Unable to Load and Add Device Traffic Rules and Server Traffic Rules in the VMware Tunnel Configuration Page

When you load the Tunnel configuration page, "Tunnel Configuration doesn't exist" is displayed and you may not be able to add Device Traffic Rules or Server Traffic Rules.

To clear the IIS bindings hostname and keeping the hostname blank:

  1. From the Windows Start menu, click Administrative Tools > Internet Information Services (IIS) Manager to open it on the API server.

  2. In IIS Manager under Connections, expand your server's name.

  3. Then expand Sites.

  4. Right-click on a website, and click Edit Bindings.

  5. In the Site Bindings window, select the http/https binding for this website, and click Edit.

  6. In the Edit Site Binding window keep the hostname blank and click OK.

  7. Restart the IIS sites for the changes to take effect.

Alternatively, instead of clearing the IIS bindings hostname and keeping the hostname blank, you can complete the following steps:

  1. Update the Tunnel microservice appsettings. json's Host key under the AirWatchApiClient to include the hostname that is used in the IIS bindings.

    "AirWatchApiClient": {
        "Host": "acme.example.com",
        "ClientTimeoutInSeconds": 40,
        "HostDiscoveryTimeoutInSeconds": 30,
    	"Port": 8081
    Note:

    The port key will only be used if the customer is using a custom port.

  2. Restart the AirWatch Tunnel Service

  3. Refresh the browser if you are using the Tunnel configuration screen after the service restart.

Verify VMware Tunnel Microservice

You can use the Tunnel health endpoint to verify the upstream or downstream connectivity to the Tunnel microservice.

  1. Use the following REST API to get the Tunnel microservice health from Workspace ONE UEM API Explorer.

    GET {environment}/api/mdm/tunnel/health
    aw-tenant-code: API key configured
    Basic auth
    
  2. Verify the API response of VMware Tunnel health endpoint.

    200 ok
    { 
     "api_to_tunnel_microservice_connectivity": "True", 
     "tunnel_microservice _to_api_connectivity": "True", 
     "database_connectivity_status": "True" 
    }

Unable to Upload Third-Party SSL Certificate

You may not be able to upload third-party SSL certificate when the third-party library currently used for the encryption/decryption (BouncyCastle) fails to read the certificate password due to a pad block corrupted issue.

  1. Import the non-working certificate onto the windows certificate store on the app server of the console where this issue is seen.

  2. Once imported, export the certificate from the store with the same password if required. The exported certificate will be available on your local machine on the path you chose to save it.

  3. Use this exported certificate for uploading on the third-party server authentication tab of the Tunnel configuration. The certificated should upload successfully and the Tunnel config can be saved.

Tech Zone Resources for Tunnel

Have you heard of our Tech Zone site? They create more in depth articles to help our customers succeed. Here are a few articles they have created for Tunnel. For more information refer to: https://techzone.vmware.com/.

Topic Date Released Link
Compliance Integration with MS Office 365 using Graph APIs April 2023 https://techzone.vmware.com/resource/compliance-integration-ms-office-365-using-graph-apis
Migrating from Tunnel Proxy to Per-App Tunnel March 2023 https://techzone.vmware.com/migrating-tunnel-proxy-app-tunnel
Deploying VMware Workspace ONE Tunnel: Workspace ONE Operational Tutorial March 2023 https://techzone.vmware.com/deploying-vmware-workspace-one-tunnel-workspace-one-operational-tutorial
Deploying Traditional Win32 Applications to Windows Devices: Workspace ONE Operational Tutorial December 2022 https://techzone.vmware.com/deploying-traditional-win32-applications-windows-devices-workspace-one-operational-tutorial
Compliance Integration with MS Office 365 using Workspace ONE Tunnel December 2022 https://techzone.vmware.com/resource/compliance-integration-ms-office-365-using-workspace-one-tunnel
Understanding & troubleshoot Tunnel Connections March 2021 https://techzone.vmware.com/understand-and-troubleshoot-tunnel-connections#introduction-to-tunnel-connections