Troubleshooting Compliance Scanner

This topic provides instructions for troubleshooting Compliance Scanner for VMware Tanzu.

scan_results Issues

scan_results Completed with Error (Exit Code 1)

Symptom

Running a scan results in an exit code 1 error similar to the following:

...
Instance   oscap_store/43aba653-4d0a-4cfc-bc72-a136ce33e3e0
Exit Code  1
Stdout     2019/07/05 21:11:04 Starting store on port 28894
           2019/07/05 21:11:05 [10.0.8.6] <nil>

           2019/07/05 21:11:05 Starting scan on 10.0.8.6
           2019/07/05 21:11:05 Get https://10.0.8.6:28893/run: x509: certificate has expired or is not yet valid

           Received no scan results

Stderr     -

1 errand(s)
Errand 'scan_results' completed with error (exit code 1)
Exit code 1

Explanation

Your certificates have expired.

Solution

Rotate your certificates. For instructions, see Rotating Certificates.

scan_results Received Signal Terminated (Exit Code 124)

Symptom

Running a scan results in an exit code 124 error similar to the following:

...
Instance   oscap_store/43aba653-4d0a-4cfc-bc72-a136ce33e3e0
Exit Code  124
Stdout    2019/06/24 15:43:30 Starting store on port 28894
          2019/06/24 15:43:30 [10.0.8.5] <nil>

          2019/06/24 15:43:30 Starting scan on 10.0.8.5
          2019/06/24 15:43:31 Received signal terminated
          Scanner timed out, did not receive all the scan results in 1200 second(s)

Explanation

Your scan has reached the configured timeout period. This may occur if a VM lacks sufficient memory or CPU to finish its scan in time.

Solution

Do one of the following:

Increase the scan timeout for all VMs. For more information, see Scanner Timeout under Configure Scans.
Increase the memory or CPU resources for any failing scanned VMs.

scan_results Does Not Include Scans for VMs in the Deployment

Symptom

Only scan results for the oscap_store VM are being generated. VMs in the deployment are not being scanned.

Explanation

Compliance Scanner is a BOSH add-on, as well as a tile. When a scan is run, the oscap_store VM verifies each VM’s OSCAP release version. Because of this, the latest OSCAP release must be deployed on all VMs to function properly.

Solution

When installing or upgrading Compliance Scanner, you must Apply Changes on TAS for VMs (and any other tiles with VMs you want to scan) after applying changes to the Compliance Scanner tile. This is so that the version of Compliance Scanner on the VMs match the version being used by the scanner.

Unable to resolve store domain: no such host

Symptom

Running a scan results in an exit code 1 error similar to the following:

...
Instance   oscap_store/43aba653-4d0a-4cfc-bc72-a136ce33e3e0
Exit Code  1
Stdout    2019/08/05 09:14:01 Starting store on port 28894
          2019/08/05 09:14:01 [172.15.0.3 172.15.0.4 172.15.0.5 172.15.1.5 172.15.1.4] <nil>

          2019/08/05 09:14:01 Starting scan on 172.15.0.3
          2019/08/05 09:14:01 Unable to scan 172.15.0.3: 500
          2019/08/05 09:14:01 Starting scan on 172.15.0.4
          2019/08/05 09:14:01 Unable to scan 172.15.0.4: 500
          2019/08/05 09:14:01 Starting scan on 172.15.0.5
          2019/08/05 09:14:01 Unable to scan 172.15.0.5: 500
          2019/08/05 09:14:01 Shutdown
          Received no scan results

and produces a scanner_web_ctl.log similar to the following:

/var/vcap/sys/log/config_scanner/scanner_web_ctl.log:

2019/08/05 10:21:34 Lookup store
2019/08/05 10:21:34 Unable to resolve store domain: lookup q-s4.oscap-store.NETWORK-NAME.p-compliance-scanner-5f51f48abb5fbf23b617.bosh on 169.254.0.2:53: no such host

Where NETWORK-NAME is the name given to the network being used for the Compliance Scanner tile.

Explanation

BOSH DNS cannot resolve the oscap_store VM URL if there is any capitalization in the network’s Name.

Solution

Verify that the network you selected inside the Assign AZs and Networks pane of your Compliance Scanner tile is not capitalized.

If this is the case, you must edit the name of the network being used to remove the capitalization:

On the Ops Manager Installation Dashboard, click BOSH Director.
Click Create Networks.
Change the network Name to a name without capital letters, and click Save.
Click Review Pending Changes, and then Apply Changes to all tiles.

This updates the network name property on all VMs.

oscap_store receives ReportFailed (Exit code 1)

Symptom

Running a scan results in an exit code 1 error similar to the following:

...
Instance   oscap_store/43aba653-4d0a-4cfc-bc72-a136ce33e3e0
Exit Code  1
Stdout    2020/03/12 16:51:12 Starting store on port 28894
          2020/03/12 16:51:12 [10.0.0.8 10.0.0.10 10.0.0.9 10.0.0.11] 
 
  
          2020/03/12 16:51:12 Starting scan on 10.0.0.8
          2020/03/12 16:51:12 Starting scan on 10.0.0.10
          2020/03/12 16:51:12 Starting scan on 10.0.0.9
          2020/03/12 16:51:12 Starting scan on 10.0.0.11
          2020/03/12 16:51:36 ReportFailed
          2020/03/12 16:51:36 Received  from 10.0.0.11:59566
          2020/03/12 16:51:36 10.0.0.11:59566 failed: Error from daemon: exit status 124 (check the logs at: /var/vcap/sys/log/config_scanner/daemon.log)
          2020/03/12 16:51:36 Attempting to cancel scan on host 10.0.0.9
          2020/03/12 16:51:36 Attempting to cancel scan on host 10.0.0.8
          2020/03/12 16:51:36 Attempting to cancel scan on host 10.0.0.10
          2020/03/12 16:51:36 Failed to cancel scan on host 
  
   : Get https://10.0.0.10:28893/cancel: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2020/03/12 16:51:36 Failed to cancel scan on host 
   
    : Get https://10.0.0.8:28893/cancel: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2020/03/12 16:51:36 Failed to cancel scan on host 
    
     : Get https://10.0.0.9:28893/cancel: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2020/03/12 16:51:36 Shutdown Scanner exited with 1 code

Explanation

The oscap_store receives a ReportFailed event from one of the VMs being scanned. The error message 10.0.0.11:59566 failed: Error from daemon: exit status 124 indicates that the scan timed out on that particular VM.

This can happen if a timeout increase was only applied to the oscap_store VM.

Solution

Increase the scan timeout for all VMs. For more information, see Scanner Timeout.

Scan Successful with POST Errors

This problem with POST errors only applies to Compliance Scanner v1.2.32 and earlier. Compliance Scanner v1.2.48 and later does not try to scan Windows VMs and any Linux VMs where Compliance Scanner is not deployed.

Symptom

Linux VMs without Compliance Scanner and all Windows VMs show POST errors during a successful scan.

For example:

...
Instance   oscap_store/bcb225db-f29f-4356-9d6a-525cbc99f6e1
Exit Code  0
Stdout    2020-05-13 14:02:42 Starting scan
          2020/05/13 14:02:42 Starting store on port 28894
          2020/05/13 14:02:42 [10.0.4.6 10.0.4.7 10.0.4.5] <nil>
          2020/05/13 14:02:42 Starting scan on 10.0.4.6
          2020/05/13 14:02:42 Starting scan on 10.0.4.7
          2020/05/13 14:02:47 Post https://10.0.4.7:28893/run: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
          2020/05/13 14:02:47 Starting scan on 10.0.4.5
          2020/05/13 14:02:47 Post https://10.0.4.5:28893/run: dial tcp 10.0.4.5:28893: connect: connection refused
          2020/05/13 14:03:31 ReportFinished
          2020/05/13 14:03:31 Received oscap_store-bcb225db-f29f-4356-9d6a-525cbc99f6e1.tgz from 10.0.4.6:33670
          2020/05/13 14:03:31 Shutdown
            adding: benchmarks/ (stored 0%)
            adding: benchmarks/CIS-Level-2.xml (deflated 90%)
            adding: benchmarks/Base-Xenial.xml (deflated 93%)
...

Explanation

Scan results include errors for VMs that Compliance Scanner has skipped:

For Windows VMs:
net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
For Linux VMs without Compliance Scanner installed and running:
dial tcp 10.0.4.5:28893: connect: connection refused

The above errors do not prevent the remainder of the scans from being completed.

Solution

There are a few situations to consider:

If the hosts that provide the errors are Windows VMs:
This is the expected behavior. Compliance Scanner does not currently support Windows.

If the hosts that provide the errors are VMs that you expect to have Compliance Scanner installed and running:
Perform an “Apply Changes” on the deployment and the Compliance Scanner for VMware Tanzu tile to ensure that the VMs have the Compliance Scanner installed and running.

To verify that Compliance Scanner is installed and running, you can run bosh instances --ps. Ensure that each VM shows that the scanner_daemon and scanner_web processes are in a running state.

For example:

$ bosh instances --ps

Instance                                          Process               Process State  AZ          IPs       Deployment
vm/bcb225db-f29f-4356-9d6a-525cbc99f6e1           -                     running        us-east1-b  10.0.4.6  p-compliance-scanner-a7c5cc7f126925f45180
~                                                 bosh-dns              running        -           -         -
~                                                 bosh-dns-healthcheck  running        -           -         -
~                                                 bosh-dns-resolvconf   running        -           -         -
~                                                 scanner_daemon        running        -           -         -
~                                                 scanner_web           running        -           -         -
~                                                 system-metrics-agent  running        -           -         -