This topic provides instructions for troubleshooting Anti-Virus for VMware Tanzu and verifying that it is protecting your Ops Manager deployment.
Applying changes in Ops Manager fails. The bottom of the changelog contains an error message similar to:
Started updating job nats > nats/0 (12bfae02-b4af-4104-b2bd-227ff07b2d92) (canary). Done (00:02:31)
Failed updating job etcd_server > etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11) (canary): ‘etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)’ is not running after update. Review logs for failed jobs: clamd (00:05:53)
Error 400007: ‘etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)’ is not running after update. Review logs for failed jobs: clamd
The Anti-Virus Mirror for VMware Tanzu server was unavailable during initial deployment.
Review the manifest file, and replace the database_mirror key with the address of a stable mirror server. The official supported mirror is database.clamav.net.
Applying changes in Ops Manager fails. The bottom of the changelog contains an error message similar to:
Error: Action Failed get_task: Task d5b87522-c8b2-4870-7855-73d50bff0748 result: 1 of 6 pre-start scripts failed. Failed Jobs: antivirus. Successful Jobs: bpm, syslog_forwarder, bosh-dns, ipsec, pxc-mysql.
The antivirus job can fail to start because it does not get the virus definitions from the antivirus-mirror. The antivirus-mirror fails to supply the virus definitions if it has failed to correctly obtain the following files: main.cvd, bytecode.cvd, and daily.cvd. If you manually get the ClamAV Virus Database, using curl or similar tools can return a file with an error instead of the virus definitions. For example:
$ curl -L -O database.clamav.net/main.cvd
$ cat main.cvd
error code: 1020
Configure the tile to use either the official mirror or an existing mirror. For information, see Configure Anti-Virus Mirror in Installing and Configuring Anti-Virus Mirror.
For use cases where CVD files are manually obtained, a supported method must be used. For information about error codes and supported methods, see ClamAV documentation.
Updating virus definitions writes an error like the following to the Anti-Virus Mirror log destination:
2019/07/03 20:28:30 file /var/vcap/data/antivirus-mirror/unvalidated/main.cvd rejected: /var/vcap/data/antivirus-mirror/unvalidated/main.cvd is an invalid cvd file: exit status 1
The Anti-Virus Mirror database verifier detected that a virus database file downloaded from the external database is invalid.
Check that the database files downloaded properly and re-download if necessary.
Updating virus definitions writes an error like the following to the Anti-Virus Mirror log destination:
2019/07/03 20:35:34 file /var/vcap/data/antivirus-mirror/unvalidated/daily.cvd rejected: /var/vcap/data/antivirus-mirror/unvalidated/daily.cvd is not newer than /var/vcap/store/antivirus-mirror/validated/daily.cvd
The Anti-Virus Mirror database verifier detected that a virus database file downloaded from the external database is older than the one most recently processed by the internal mirror.
Check that the latest version of the database files were downloaded. If the internal Anti-Virus Mirror has the latest files, no action is required.
Malware signature or sample malware is not detected, even though the ClamAV daemon is properly configured.
Virus signatures are not up-to-date.
To resolve this issue, verify that:
If the local mirror is up-to-date and Anti-Virus is still failing to detect a malware sample, you might have encountered a new threat. VMware recommends alerting the community using existing channels and reporting the suspicious file directly to the ClamAV team.
Note: VMware does not provide support for ClamAV detection failures, mirror coordination, or threat tracking activity.
Anti-Virus reports a false positive result such as non-malicious file is reported to be a virus.
Anti-Virus compares files to its database of known malicious patterns. Anti-Virus might detect a non-malicious file as a virus due to a coincidental similarity to those patterns.
Submit false positive reports to ClamAV. You can also subscribe to the ClamAV email list to be kept up-to-date with ClamAV database changes. It takes about a week for ClamAV to verify and publish a new database.
Anti-Virus is taking more CPU resources than assigned in its configuration.
Anti-Virus resource consumption is restricted using cgroups. Anti-Virus is resource-limited whenever other processes are active. However, cgroups enables Anti-Virus to occupy more CPU resources when all other processes are idle, because it does not impact their performance.
Set the Enforce CPU limit field to Always in the Anti-Virus tile. For instructions, see Configure Anti-Virus.
Anti-Virus fails to start and /var/log/syslog reports Memory cgroup out of memory: Kill process on the clamd process similar to:
2019-02-20T19:35:40.249205+00:00 localhost kernel: [ 254.669948] Memory cgroup out of memory: Kill process 7493 (clamd) score 586 or sacrifice child 2019-02-20T19:35:40.249205+00:00 localhost kernel: [ 254.679053] Killed process 7527 (clamd) total-vm:786136kB, anon-rss:626692kB, file-rss:1592kB
Anti-Virus resource consumption is restricted by cgroups. The clamd process is terminated if the memory usage limit is exceeded. When memory swapping is disabled by other BOSH jobs, the Anti-Virus resource requires a larger memory limit.
This is expected behavior from cgroups. To configure the memory limit, configure Memory limit (in bytes) in the Anti-Virus tile.
Warning: When updating the memory limit, ensure that all VMs, including errand VMs, have sufficient memory resources.
Anti-Virus fails to start during deployment. However, the clamd and freshclam processes eventually run.
The deployment failure log looks similar to:
Task 1071 | 19:40:49 | Updating instance clamav_1: clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0) (canary) (00:05:26)
L Error: ‘clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0)’ is not running after update. Review logs for failed jobs: clamd, freshclam
bosh -d DEPLOYMENT instances --ps, you see that the the
clamd and
freshclam processes are running successfully after the failed deployment.
For example:
$ bosh -d clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c instances –ps
Instance Process Process State AZ IPs
clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c - running z1 10.0.0.7
~ clamd running - -
~ freshclam running - -
Anti-Virus startup is CPU intensive and, if restricted, can prevent Anti-Virus from starting up correctly.
Ensure cpu_limit is set high enough for Anti-Virus to execute normally. If the limit is too strict, Anti-Virus fails to start. To make changes to this limit, configure CPU limit (percentage) in the Anti-Virus tile.
Set enforce_cpu_limit to false. This allocates more CPU cycles to ClamAV if other processes are not using CPU resources.
To disable this limit, set the Enforce CPU limit field to When other processes are using CPU resources in the Anti-Virus tile.
From the Ops Manager Installation Dashboard, navigate to the tile with the failing antivirus job. On Resource Config, adjust the VM Type for the Anti-Virus job to have sufficient CPU resources.
The Anti-Virus Mirror log reports that too many log files are open:
2019/07/29 20:02:41 10.0.0.72 is requesting main.cvd 2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 10ms
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 20ms
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 40ms
2019/07/29 20:02:42 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 5ms
Anti-Virus Mirror opens files when a database is requested. There is a limit to how many files it can open at a time.
Increase the number of Anti-Virus Mirror instances. VMware recommends that there is one Anti-Virus Mirror for every 250 instances where Anti-Virus is installed. For more information, see Scale the Number of Deployed Mirrors.
When using Anti-Virus Mirror, errors occur when you redeploy Tanzu Application Service (TAS) while restoring with BOSH Backup and Restore (BBR). For information about redeploying TAS, see Redeploy TAS for VMs in Restoring Deployments from Backup with BBR in the Ops Manager documentation.
Anti-Virus Mirror must be running before you install Anti-Virus on other VMs in your deployment. Otherwise, Anti-Virus Mirror might not deploy before other tiles and dependencies deploy.
If Anti-Virus Mirror is not running, VMs with Anti-Virus installed cannot download the required database signature files. If this happens, errors and failed deployments occur.
To resolve this issue, you must ensure that Anti-Virus Mirror is deployed before restoring your deployment.
To do this:
Follow the procedures before _Redeploy TAS for VMs](https://docs.pivotal.io/ops-manager/install/backup-restore/restore-pcf-bbr.html#redeploy-ert) in Restoring Deployments from Backup with BBR in the Ops Manager documentation. Do not apply changes.
Exclude Anti-Virus Mirror from the Anti-Virus deployment by following the procedure in Exclude Anti-Virus Mirror during Apply Changes. This ensures that Anti-Virus is not deployed on the Anti-Virus Mirror
Remove the Anti-Virus Mirror exclusion from the Anti-Virus configurations by following the procedure in Remove the Exclusion.
Continue to restore your deployment by following the remaining procedures in Restoring Deployments from Backup with BBR.
Anti-Virus is using more CPU resources than assigned in its configuration, even with the Enforce CPU limit field set to Always.
Anti-Virus resource consumption is restricted using cgroups. If the VM does not have enough CPU or memory resources, the clamd PID is removed from the cgroup.procs file. This causes Anti-Virus to ignore the Enforce CPU limit setting.
Increase the VM size. VMware recommends a minimum VM size of micro.cpu using 2 CPU and 2 GB RAM.
The freshclam logs show the following warning messages:
Can’t query main.IP-ADDRESS.ping.clamav.net
Can’t query daily.IP-ADDRESS.ping.clamav.net
Can’t query bytecode.IP-ADDRESS.ping.clamav.net
Freshclam is the process which downloads virus definitions. Freshclam queries these endpoints to give ClamAV information about the current definitions being used and the version of the ClamAV binary. Failure to query these endpoints indicates one of the ClamAV servers is experiencing network difficulty, but this is unrelated to downloading virus definitions and does not affect updates.
No action is required. This issue does not impact the functionality of Anti-Virus.
