Troubleshooting app deployment and health

In this topic, you will learn how to diagnose and resolve common issues when you deploy and run apps on VMware Tanzu Application Service for VMs (TAS for VMs).

Common issues

The following sections describe common issues you might encounter when attempting to deploy and run your app, and suggest possible resolutions.

cf push times out

If your deployment times out during the upload or staging phase, you may receive one of the following error messages:

504 Gateway Timeout
Error uploading application
Timed out waiting for async job JOB-NAME to finish

If this happens:

Check your network speed. Depending on the size of your app, your cf push could be timing out because the upload is taking too long. VMware recommends an Internet connection speed of at least 768 KB/s, or 6 Mb/s, for uploads.
Make sure you are pushing only needed files. By default, cf push pushes all the contents of the current working directory. Make sure you are pushing only the directory for your app. If your app is too large, or if it has many small files, TAS for VMs may time out during the upload. To reduce the number of files you are pushing, ensure that you push only the directory for your app, and remove unneeded files or use the .cfignore file to specify excluded files. For more information, see Ignore Unnecessary Files When Pushing in Considerations for Designing and Running an App in the Cloud.
Set the CF_STAGING_TIMEOUT and CF_STARTUP_TIMEOUT environment variables. By default, your app has 15 minutes to stage and 5 minutes to start. You can increase these times by setting CF_STAGING_TIMEOUT and CF_STARTUP_TIMEOUT. For more information, run cf push -h.
If your app contains a large number of files, try pushing the app repeatedly. Each push uploads a few more files. Eventually, all files upload, and the push succeeds. This is less likely to work if your app has many small files.

App too large

If your app is too large, you may receive an error message on cf push. For example:

413 Request Entity Too Large
You have exceeded your organization's memory limit

If this happens:

Make sure your org and space have enough resources for all instances of your app. You cannot use more resources in a space than is allocated for your org. To view the resource quota name for your org, use cf org ORG-NAME, then cf quota QUOTA-NAME. To view resource quota name for your space, use cf space SPACE-NAME, then cf space-quota QUOTA-NAME. Your total organization resource usage is the sum of the resources used by all apps in all spaces within the org. Each app’s resource usage is the resources allocated to it multiplied by the number of instances. To view the resource usage of all the apps in a space, use cf apps, then cf app APP-NAME for each app.
Make sure your app is less than 1 GB. By default, cf push deploys all the contents of the current working directory. To reduce the number of files you are pushing, ensure that you push only the directory for your app, and remove unneeded files or use the .cfignore file to specify excluded files. The following limits apply:
- The app files to push cannot exceed 1 GB.
- The droplet that results from compiling those files cannot exceed 1.5 GB. Droplets are typically a third larger than the pushed files.
- The combined size of the app files, compiled droplet, and buildpack cache cannot total more than 4 GB of space during staging.

Unable to detect a supported app type

If TAS for VMs cannot identify an appropriate buildpack for your app, you see an Unable to detect a supported app type error.

You can view what buildpacks are available with the cf buildpacks command.

If you see a buildpack that you believe should support your app, see the buildpack documentation for details about how that buildpack detects apps it supports.

If you do not see a buildpack for your app, you may still be able to push your app with a custom buildpack using cf push -b with a path to your buildpack.

App deploy fails

Even when the deploy fails, the app might exist on TAS for VMs. Run cf apps to review the apps in the currently targeted org and space. You might be able to correct the issue using the CLI or About Apps Manager, or you might need to delete the app and re-deploy.

If you push multiple apps using a manifest and one fails to deploy, TAS for VMs does not attempt to push apps specified after the app that failed.

Common reasons deploying an app fails include:

The app does not successfully bind to a service, such as a PostgreSQL or MongoDB. The service may not exist, may be down, or may reject the binding.
The app does not have a unique URL. For more information, see App Requires Unique URL below.

App requires unique URL

TAS for VMs requires that each app that you deploy has a unique URL. Otherwise, the new app URL collides with an existing app URL and TAS for VMs cannot successfully deploy the app. You can resolve this issue by running cf push with either of the following flags to create a unique URL:

--random-route to create a URL that includes the app name and random words. Using this option might create a long URL, depending on the number of words that the app name includes.

App fails to start

After cf push stages the app and uploads the droplet, the app may fail to start, commonly with a pattern of starting and crashing similar to the following example:

-----> Uploading droplet (23M)
...
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 down
...
0 of 1 instances running, 1 failing
FAILED
Start unsuccessful

If this happens:

Find the reason app is failing and modify your code. Run cf events APP-NAME and cf logs APP-NAME --recent and look for messages similar to the following example:
```
2018-07-20T15:53:26.00-0700   app.crash      app-name   index: 2, reason: CRASHED, cell_id: d874ad05-d0ca-4c63-9f5a-0b1ddd90dd5d, instance: f406c53e-b1ca-4a0f-6140-dddd, exit_description: APP/PROC/WEB: Exited with status 1
```
These messages may identify a memory or port issue. If they do, take that as a starting point when you re-examine and fix your app code.
Make sure your app code uses the PORT environment variable. Your app may be failing because it is listening on the wrong port. Instead of hard-coding the port on which your app listens, use the PORT environment variable. For example, this Ruby snippet assigns the port value to the listen_here variable:
```
listen_here = ENV['PORT']
```
For more examples specific to your app framework, see the appropriate buildpack documentation for your app’s language.
Make sure your app adheres to the principles described in Twelve-Factor App and Prepare to Deploy an App. These texts explain how to prevent situations where your app builds locally but fails to build in the cloud.

Verify the timeout configuration of your app. App health checks use a timeout setting when an app starts up for the first time. For more information, see Using App Health Checks. If an app fails to start up due to health check timeout, you might see messages in the logs similar to the following example:

2017-01-30T14:07:20.39-0800 [CELL/0]     OUT Creating container
2017-01-30T14:07:20.65-0800 [CELL/0]     OUT Successfully created container
2017-01-30T14:07:22.30-0800 [CELL/0]     OUT Starting health monitoring of container
2017-01-30T14:08:23.52-0800 [CELL/0]     ERR Timed out after 1m0s:
health check never passed.
2017-01-30T14:08:23.57-0800 [CELL/0]     OUT Destroying container
2017-01-30T14:08:23.59-0800 [API/0]      OUT Process has crashed with type: "web"
2017-01-30T14:08:23.59-0800 [CELL/0]     OUT Creating container
2017-01-30T14:08:23.60-0800 [API/0]      OUT App instance exited with guid 91086440-bac0-44f0-808f-a034a1ec2ed0
payload: {"instance"=>"", "index"=>0, "reason"=>"CRASHED",
"exit_description"=>"2 error(s) occurred:\n\n* 1 error(s)
occurred:\n\n* Exited with status 6\n* 2 error(s)
occurred:\n\n* cancelled\n* cancelled", "crash_count"=>1,
"crash_timestamp"=>1485814103565763172,
"version"=>"3e6e4232-7e19-4168-9583-1176833d2c71"}
2017-01-30T14:08:23.83-0800 [CELL/0]     OUT Successfully destroyed container
2017-01-30T14:08:23.84-0800 [CELL/0]     OUT Successfully created container
2017-01-30T14:08:25.41-0800 [CELL/0]     OUT Starting health monitoring of container

App consumes too much memory, then crashes

An app that cf push has uploaded and started can crash later if it uses too much memory.

Make sure your app is not consuming more memory than it should. When you ran cf push and cf scale, that configured a limit on the amount of memory your app should use. Check your app’s actual memory usage. If it exceeds the limit, modify the app to use less memory.

Routing conflict

TAS for VMs allows multiple apps, or versions of the same app, to be mapped to the same route. This feature enables Blue-Green deployment. For more information see Using Blue-Green Deployment to Reduce Downtime and Risk.

Routing multiple apps to the same route may cause undesirable behavior in some situations by routing incoming requests randomly to one of the apps on the shared route.

If you suspect a routing conflict, run cf routes to check the routes in your TAS for VMs deployment.

If two apps share a route outside of a blue-green deployment strategy, choose one app to re-assign to a different route and follow the procedure below:

Remove the existing route from that app by running:
```
cf unmap-route APP-NAME OLD-ROUTE
```
Map the app to a new, unique route by running:
```
cf map-route APP-NAME NEW-ROUTE
```

Gather diagnostic information

This section describes how to gather diagnostic information and troubleshoot app deployment issues.

Examine environment variables

cf push deploys your app to a container on the server. The environment variables in the container govern your app.

You can set environment variables in a manifest created before you deploy. For more information, see Deploying with App Manifests.

You can also set an environment variable by running cf set-env, followed by cf push. You must run cf push for the variable to take effect in the container environment.

Run cf env APP-NAME to view the environment variables that you have set using the cf set-env command and the variables in the container environment:

Getting env variables for app example-app in org example-org / space development as admin...
OK

System-Provided:
{
 "VCAP_SERVICES": {
   "p-mysql-n/a": [
     {
       "credentials": {
         "uri":"postgres://lrra:[email protected]:5432/lraa
       },
       "label": "p-mysql-n/a",
       "name": "p-mysql",
       "syslog_drain_url": "",
       "tags": ["postgres","postgresql","relational"]
     }
   ]
 }
}

User-Provided:
my-env-var: 100
my-drain: http://drain.example.com

View logs

To view app logs streamed in real-time, run cf logs APP-NAME.

To aggregate your app logs to view log history, bind your app to a syslog drain service. For more information, see Streaming App Logs to Log Management Services.

Important The Diego architecture does not support the cf files command, and as of cf CLI v7, the cd files command is deprecated for all uses.

Trace Cloud Controller REST API calls

If a command fails or produces unexpected results, re-run it with HTTP tracing enabled to view requests and responses between the Cloud Foundry Command Line Interface (cf CLI) and the Cloud Controller REST API.

For example:

Re-run cf push with -v:
```
cf push APP-NAME -v
```
Re-run cf push while appending API request diagnostics to a log file:
```
CF_TRACE=PATH-TO-TRACE.LOG cf push APP-NAME
```

These examples enable HTTP tracing for a single command only. To enable it for an entire shell session, set the variable first:

export CF_TRACE=true
export CF_TRACE=PATH-TO-TRACE.LOG

Important CF_TRACE is a local environment variable that modifies the behavior of the cf CLI. Do not confuse CF_TRACE with the variables in the container environment where your apps run. For more information about these container variables, see Examine Environment Variables.

Analyze Zipkin trace IDs

When the Zipkin feature is enabled in TAS for VMs, the Gorouter adds or forwards Zipkin trace IDs and span IDs to HTTP headers. For more information about what the Gorouter provides in the HTTP header, see HTTP Headers in HTTP Routing.

After adding Zipkin HTTP headers to app logs, developers can run cf logs APP-NAME to correlate the trace and span IDs logged by the Gorouter with the trace IDs logged by their app. To correlate trace IDs for a request through multiple apps, each app must forward appropriate values for the headers with requests to other apps.

Use troubleshooting commands

You can investigate app deployment and health using the cf CLI.

Some cf CLI commands may return connection credentials. Remove credentials and other sensitive information from command output before you post the output a public forum.

cf apps: Returns a list of the apps deployed to the current space with deployment options, including the name, current state, and routes for each app.
cf app APP-NAME: Returns the health and status of each instance of a specific app in the current space, including instance ID number, current state, and how long it has been running.

Important CPU values returned by cf app show the total usage of each app instance on all CPU cores on a host VM, where each core contributes 100%. For example, the CPU of a single-threaded app instance on a Diego cell with one core cannot exceed 100%, and four instances sharing the cell cannot exceed an average CPU of 25%. A multi-threaded app instance running alone on a cell with eight cores can draw up to 800% CPU.
cf env APP-NAME: Returns environment variables set using cf set-env and variables existing in the container environment.
cf events APP-NAME: Returns information about app crashes, including error codes. Shows that an app instance exited. For more detail, look in the app logs. For a list of TAS for VMs errors, see the cloudfoundry-attic repository on GitHub.
cf logs APP-NAME --recent: Dumps recent logs. For more information, see Viewing Logs in the Command Line Interface in App Logging in TAS for VMs.
cf logs APP-NAME: Returns a real-time stream of the app STDOUT and STDERR. Press Ctrl+C or Command+C on your keyboard to exit the real-time stream.

Commands available in cf CLI v6, but deprecated in cf CLI v7:

cf files APP-NAME: Lists the files in an app directory. Given a path to a file, outputs the contents of that file. Given a path to a subdirectory, lists the files within. Use this to explore individual logs.

Important Your app should direct its logs to STDOUT and STDERR. The cf logs command also returns messages from any Log4j facility that you configure to send logs to STDOUT. For more information, see the Log4j website.

Access apps with SSH

If you need to troubleshoot an instance of an app, you can gain SSH access to the app with the SSH proxy and daemon. For more information, see App SSH Overview.

Send requests to app instance

To obtain debug data without SSH, you can make HTTP requests to a specific instance of an app by using the X-CF-APP-INSTANCE HTTP header. For more information, see App Instance Routing in HTTP Routing.