Troubleshooting

This topic contains troubleshooting and known issues for Supply Chain Security Tools - Store.

Persistent volume retains data

Symptom

If Supply Chain Security Tools - Store is deployed, deleted, redeployed, and the database password is changed during the redeployment, the metadata-store-db pod fails to start. This is caused by the persistent volume used by postgres retaining old data, even though the retention policy is set to DELETE.

Solution

Caution: Changing the database password deletes your Supply Chain Security Tools - Store data.

To redeploy the app, either use the same database password or follow the steps below to erase the data on the volume:

  1. Deploy metadata-store app by using kapp.
  2. Verify that the metadata-store-db-* pod fails.
  3. Run:

    kubectl exec -it metadata-store-db-<some-id> -n metadata-store /bin/bash
    

    Where <some-id> is the ID generated by Kubernetes and appended to the pod name.

  4. Run rm -rf /var/lib/postgresql/data/* to delete all database data.

    Where /var/lib/postgresql/data/* is the path found in postgres-db-deployment.yaml.

  5. Delete the metadata-store app by using kapp.

  6. Deploy the metadata-store app by using kapp.

Missing persistent volume

Symptom

After Store is deployed, metadata-store-db pod might fail for missing volume while postgres-db-pv-claim pvc is in PENDING state. This is because the cluster where Store is deployed does not have storageclass defined. storageclass’s provisioner is responsible for creating the persistent volume after metadata-store-db attaches postgres-db-pv-claim.

Solution

  1. Verify that your cluster has storageclass by running kubectl get storageclass.
  2. Create a storageclass in your cluster before deploying Store. For example:

    # This is the storageclass that Kind uses
    kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
    
    # set the storage class as default
    kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
    

Multicluster Support: Error sending results to SCST - Store running in a different cluster

Symptom

The Store Ingress and multicluster support document instructs you on how to create SecretExports to share secrets for communicating with the Store. During installation, Supply Chain Security Tools - Scan (Scan) creates the SecretImport for ingesting the TLS CA certificate secret, but misses the SecretImport for the RBAC Auth token.

Solution

Follow the AWS documentation to install the Amazon EBS CSI Driver before installing Store or before upgrading to Kubernetes v1.23.

Certificate Expiries

Symptom

The Insight CLI or the Scan Controller fails to connect to the Store.

The logs of the metadata-store-app pod show the following error:

$ kubectl logs deployment/metadata-store-app -c metadata-store-app -n metadata-store
...
2022/09/12 21:22:07 http: TLS handshake error from 127.0.0.1:35678: write tcp 127.0.0.1:9443->127.0.0.1:35678: write: broken pipe
...

or

The logs of metadata-store-db show the following error:

$ kubectl logs statefulset/metadata-store-db -n metadata-store
...
2022-07-20 20:02:51.206 UTC [1] LOG:  database system is ready to accept connections
2022-09-19 18:05:26.576 UTC [13097] LOG:  could not accept SSL connection: sslv3 alert bad certificate
...

Explanation

cert-manager rotates the certificates, but the metadata-store and the PostgreSQL db are unaware of the change, and are using the old certificates.

Solution

If you see TLS handshake error in the metadata-store-app logs, delete the metadata-store-app pod and wait for it to come back up.

kubectl delete pod metadata-store-app-xxxx -n metadata-store

If you see could not accept SSL connection in the metadata-store-db logs, delete the metadata-store-db pod and wait for it to come back up.

kubectl delete pod metadata-store-db-0 -n metadata-store
check-circle-line exclamation-circle-line close-line
Scroll to top icon