Troubleshooting Anti-Virus for VMware Tanzu

This topic provides instructions for troubleshooting Anti-Virus for VMware Tanzu and verifying that it is protecting your Ops Manager deployment.

Installation Issues

Tanzu Operations Manager etcd_server Not Running after Update

Symptom

Applying changes in Ops Manager fails. The bottom of the changelog contains an error message similar to:

Started updating job nats > nats/0 (12bfae02-b4af-4104-b2bd-227ff07b2d92) (canary). Done (00:02:31)

Failed updating job etcd_server > etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11) (canary): 'etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)' is not running after update. Review logs for failed jobs: clamd (00:05:53)
Error 400007: 'etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)' is not running after update. Review logs for failed jobs: clamd

Explanation

The Anti-Virus Mirror for VMware Tanzu server was unavailable during initial deployment.

Solution

Review the manifest file, and replace the database_mirror key with the address of a stable mirror server. The official supported mirror is database.clamav.net.

Tanzu Operations Manager Antivirus Job Fails to Start

Symptom

Applying changes in Ops Manager fails. The bottom of the changelog contains an error message similar to:

Error: Action Failed get_task: Task d5b87522-c8b2-4870-7855-73d50bff0748 result: 1 of 6 pre-start scripts failed. Failed Jobs: antivirus. Successful Jobs: bpm, syslog_forwarder, bosh-dns, ipsec, pxc-mysql.

Explanation

The antivirus job can fail to start because it does not get the virus definitions from the antivirus-mirror. The antivirus-mirror fails to supply the virus definitions if it has failed to correctly obtain the following files: main.cvd, bytecode.cvd, and daily.cvd. If you manually get the ClamAV Virus Database, using curl or similar tools can return a file with an error instead of the virus definitions. For example:

$ curl -L -O database.clamav.net/main.cvd
$ cat main.cvd
error code: 1020

Solution

Configure the tile to use either the official mirror or an existing mirror. For information, see Configure Anti-Virus Mirror in Installing and Configuring Anti-Virus Mirror.

For use cases where CVD files are manually obtained, a supported method must be used. For information about error codes and supported methods, see ClamAV Documentation.

Virus Database Update Issues

Invalid Database Definitions

Symptom

Updating virus definitions writes an error like the following to the Anti-Virus Mirror for Tanzu log destination:

2019/07/03 20:28:30 file /var/vcap/data/antivirus-mirror/unvalidated/main.cvd rejected: /var/vcap/data/antivirus-mirror/unvalidated/main.cvd is an invalid cvd file: exit status 1

Explanation

The Anti-Virus Mirror for Tanzu database verifier detected that a virus database file downloaded from the external database is invalid.

Solution

Check that the database files downloaded properly and re-download if necessary.

Old Database Definitions

Symptom

Updating virus definitions writes an error like the following to the Anti-Virus Mirror for Tanzu log destination:

2019/07/03 20:35:34 file /var/vcap/data/antivirus-mirror/unvalidated/daily.cvd rejected: /var/vcap/data/antivirus-mirror/unvalidated/daily.cvd is not newer than /var/vcap/store/antivirus-mirror/validated/daily.cvd

Explanation

The Anti-Virus Mirror for Tanzu database verifier detected that a virus database file downloaded from the external database is older than the one most recently processed by the internal mirror.

Solution

Check that the latest version of the database files were downloaded. If the internal Anti-Virus Mirror for Tanzu has the latest files, no action is required.

Runtime Issues

Anti-Virus for Tanzu Restarting Everyday

Symptom

Observed in logs…

Explanation

Anti-Virus for Tanzu updates its virus database twice a day. To ensure no downtime there needs to be enough memory allocated to hold the old and new databases in memory for a short period. If there is insufficient memory a restart is needed.

The minimum recommended memory required by Anti-Virus for Tanzu may have changed since the product was installed.

Solution

Ensure that the tile config reserves a minimum of 3Gb of memory for AV (4GB preferred).

Memory limit (in bytes)

Solution

Set the Memory limit (in bytes) to a minimum value of 3221225472 in the Anti-Virus tile. For instructions, see Configure Anti-Virus.

Anti-Virus for Tanzu Is Not Detecting Malware

Symptom

Malware signature or sample malware is not detected, even though the ClamAV daemon is properly configured.

Explanation

Virus signatures are not up-to-date.

Solution

To resolve this issue, verify that:

The mirror server is correctly configured.
The mirror server is available within the private subnet.
At least one hour has elapsed. One hour is the default scan schedule interval.

If the local mirror is up-to-date and Anti-Virus for Tanzu is still failing to detect a malware sample, you might have encountered a new threat. VMware recommends alerting the community using existing channels and reporting the suspicious file directly to the ClamAV team.

Note VMware does not provide support for ClamAV detection failures, mirror coordination, or threat tracking activity.

Anti-Virus for Tanzu Reports False Positives

Symptom

Anti-Virus for Tanzu reports a false positive result such as non-malicious file is reported to be a virus.

Explanation

Anti-Virus for Tanzu compares files to its database of known malicious patterns. Anti-Virus for Tanzu might detect a non-malicious file as a virus due to a coincidental similarity to those patterns.

Solution

Submit false positive reports to ClamAV. You can also subscribe to the ClamAV email list to be kept up-to-date with ClamAV database changes. It takes about a week for ClamAV to verify and publish a new database.

CPU Spikes while Using Anti-Virus for Tanzu

Symptom

Anti-Virus for Tanzu is taking more CPU resources than assigned in its configuration.

Explanation

Anti-Virus for Tanzu resource consumption is restricted using cgroups. Anti-Virus for Tanzu is resource-limited whenever other processes are active. However, cgroups enables Anti-Virus for Tanzu to occupy more CPU resources when all other processes are idle, because it does not impact their performance.

Solution

Set the Enforce CPU limit field to Always in the Anti-Virus for Tanzu tile. For instructions, see Configure Anti-Virus.

Out of memory while using Anti-Virus for Tanzu

Symptom

Anti-Virus for Tanzu fails to start and /var/log/syslog reports Memory cgroup out of memory: Kill process on the clamd process similar to:

2019-02-20T19:35:40.249205+00:00 localhost kernel: [  254.669948] Memory cgroup out of memory: Kill process 7493 (clamd) score 586 or sacrifice child
2019-02-20T19:35:40.249205+00:00 localhost kernel: [  254.679053] Killed process 7527 (clamd) total-vm:786136kB, anon-rss:626692kB, file-rss:1592kB

Explanation

Anti-Virus for Tanzu resource consumption is restricted by cgroups. The clamd process is terminated if the memory usage limit is exceeded. When memory swapping is deactivated by other BOSH jobs, the Anti-Virus for Tanzu resource requires a larger memory limit.

Solution

This is expected behavior from cgroups. To configure the memory limit, configure Memory limit (in bytes) in the Anti-Virus for Tanzu tile.

Caution When updating the memory limit, ensure that all VMs, including errand VMs, have sufficient memory resources.

Insufficient CPU limit while using Anti-Virus for Tanzu

Symptom

Anti-Virus for Tanzu fails to start during deployment. However, the clamd and freshclam processes eventually run.

The deployment failure log looks similar to:

Task 1071 | 19:40:49 | Updating instance clamav_1: clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0) (canary) (00:05:26)
                L Error: 'clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0)' is not running after update. Review logs for failed jobs: clamd, freshclam

When you run bosh -d DEPLOYMENT instances --ps, you see that the the clamd and freshclam processes are running successfully after the failed deployment.

For example:

$ bosh -d clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c instances --ps

Instance                                       Process    Process State  AZ  IPs
clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c  -          running        z1  10.0.0.7
~                                              clamd      running        -   -
~                                              freshclam  running        -   -

Explanation

Anti-Virus for Tanzu startup is CPU intensive and, if restricted, can prevent Anti-Virus for Tanzu from starting up correctly.

Solutions

Ensure cpu_limit is set high enough for Anti-Virus for Tanzu to execute normally. If the limit is too strict, Anti-Virus for Tanzu fails to start. To make changes to this limit, configure CPU limit (percentage) in the Anti-Virus for Tanzu tile.
Set enforce_cpu_limit to false. This allocates more CPU cycles to ClamAV if other processes are not using CPU resources.
To deactivate this limit, set the Enforce CPU limit field to When other processes are using CPU resources in the Anti-Virus for Tanzu tile.
From the Ops Manager Installation Dashboard, navigate to the tile with the failing antivirus job. On Resource Config, adjust the VM Type for the Anti-Virus for Tanzu job to have sufficient CPU resources.

Too many open files error while using Anti-Virus Mirror for Tanzu

Symptom

The Anti-Virus Mirror for Tanzu log reports that too many log files are open:

2019/07/29 20:02:41 10.0.0.72 is requesting main.cvd
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 10ms
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 20ms
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 40ms
2019/07/29 20:02:42 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 5ms

Explanation

Anti-Virus Mirror for Tanzu opens files when a database is requested. There is a limit to how many files it can open at a time.

Solution

Increase the number of Anti-Virus Mirror for Tanzu instances. VMware recommends that there is one Anti-Virus Mirror for Tanzu for every 250 instances where Anti-Virus for Tanzu is installed. For more information, see Scale the Number of Deployed Mirrors.

Restoring with BOSH backup and restore fails

Symptom

When using Anti-Virus Mirror for Tanzu, errors occur when you redeploy VMware Tanzu Application Service for VMs (TAS for VMs) while restoring with BOSH Backup and Restore (BBR). For information about redeploying TAS for VMs, see Redeploy TAS for VMs in restoring deployments from backup with BBR in the Tanzu Operations Manager documentation.

Explanation

Anti-Virus Mirror for Tanzu must be running before you install Anti-Virus for Tanzu on other VMs in your deployment. Otherwise, Anti-Virus Mirror for Tanzu might not deploy before other tiles and dependencies deploy.

If Anti-Virus Mirror for Tanzu is not running, VMs with Anti-Virus for Tanzu installed cannot download the required database signature files. If this happens, errors and failed deployments occur.

Solution

To resolve this issue, you must ensure that Anti-Virus Mirror for Tanzu is deployed before restoring your deployment.

To do this:

Follow the procedures before Redeploy TAS for VMs in restoring deployments from backup with BBR in the Tanzu Operations Manager documentation. Do not apply changes.
Exclude Anti-Virus Mirror for Tanzu from the Anti-Virus for Tanzu deployment by following the procedure in Exclude Anti-Virus Mirror for Tanzu during Apply Changes. This ensures that Anti-Virus for Tanzu is not deployed on the Anti-Virus Mirror for Tanzu
Remove the Anti-Virus Mirror for Tanzu exclusion from the Anti-Virus for Tanzu configurations by following the procedure in Remove the Exclusion.
Continue to restore your deployment by following the remaining procedures in Restoring deployments from backup with BBR in the Tanzu Operations Manager documentation.

CPU Spikes while Enforcing a CPU Limit in Anti-Virus for Tanzu

Symptom

Anti-Virus for Tanzu is using more CPU resources than assigned in its configuration, even with the Enforce CPU limit field set to Always.

Explanation

Anti-Virus for Tanzu resource consumption is restricted using cgroups. If the VM does not have enough CPU or memory resources, the clamd PID is removed from the cgroup.procs file. This causes Anti-Virus for Tanzu to ignore the Enforce CPU limit setting.

Solution

Increase the VM size. VMware recommends a minimum VM size of micro.cpu using 2 CPU and 2 GB RAM.

Freshclam Logs Show “can’t query *.ping.clamav.net”

Symptom

The freshclam logs show the following warning messages:

Can't query main.IP-ADDRESS.ping.clamav.net
Can't query daily.IP-ADDRESS.ping.clamav.net
Can't query bytecode.IP-ADDRESS.ping.clamav.net

Explanation

Freshclam is the process which downloads virus definitions. Freshclam queries these endpoints to give ClamAV information about the current definitions being used and the version of the ClamAV binary. Failure to query these endpoints indicates one of the ClamAV servers is experiencing network difficulty, but this is unrelated to downloading virus definitions and does not affect updates.

Solution

No action is required. This issue does not impact the functionality of Anti-Virus for Tanzu.

<a id=“on-access-scan-degradation” VM status becomes `failing` or `unresponsive agent` for

unrelated Ubuntu VMs

Symptom

When On-Access Scan is enabled, status for VMs running on Ubuntu stemcells becomes unresponsive agent or failing, especially on unrelated VMs across different deployments, and the behavior disappears when On-Access Scan is disabled.

Explanation

A few issues were uncovered in ClamAV On-Access Scanner causing infinite loops under varying circumstances. As a result, the On-Access Scanner has a large number of files opened, causing resource starvation on the VM, causing the failures.

Solution

The described issue is fixed in ClamAV 1.3.0. Upgrade to the Anti-Virus for Tanzu to a version containing ClamAV 1.3.0 or later, once it becomes available. Until then, the possible workarounds are either to enable the “VM Resurrector Plugin” in the BOSH tile, or deactivate the On-Access Scanner.

Installation Issues

Tanzu Operations Manager etcd_server Not Running after Update

Symptom

Explanation

Solution

Tanzu Operations Manager Antivirus Job Fails to Start

Symptom

Explanation

Solution

Virus Database Update Issues

Invalid Database Definitions

Symptom

Explanation

Solution

Old Database Definitions

Symptom

Explanation

Solution

Runtime Issues

Anti-Virus for Tanzu Restarting Everyday

Symptom

Explanation

Solution

Solution

Anti-Virus for Tanzu Is Not Detecting Malware

Symptom

Explanation

Solution

Anti-Virus for Tanzu Reports False Positives

Symptom

Explanation

Solution

CPU Spikes while Using Anti-Virus for Tanzu

Symptom

Explanation

Solution

Out of memory while using Anti-Virus for Tanzu

Symptom

Explanation

Solution

Insufficient CPU limit while using Anti-Virus for Tanzu

Symptom

Explanation

Solutions

Too many open files error while using Anti-Virus Mirror for Tanzu

Symptom

Explanation

Solution

Restoring with BOSH backup and restore fails

Symptom

Explanation

Solution

CPU Spikes while Enforcing a CPU Limit in Anti-Virus for Tanzu

Symptom

Explanation

Solution

Freshclam Logs Show “can’t query *.ping.clamav.net”

Symptom

Explanation

Solution

<a id=“on-access-scan-degradation” VM status becomes failing or unresponsive agent for

Symptom

Explanation

Solution

<a id=“on-access-scan-degradation” VM status becomes `failing` or `unresponsive agent` for