This topic describes upgrading issues and resolutions.
To prevent issues with the metadata store database, such as the ones described in this topic, the database deployment is
If you have scripts searching for a
metadata-store-db deployment, edit the scripts to instead search for
When using Tanzu to upgrade to a new version of the store, there is occasionally data corruption. Here is an example of how this shows up in the log:
PostgreSQL Database directory appears to contain a database; Skipping initialization 2022-01-21 21:53:38.799 UTC  LOG: starting PostgreSQL 13.5 (Ubuntu 13.5-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit 2022-01-21 21:53:38.799 UTC  LOG: listening on IPv4 address "0.0.0.0", port 5432 2022-01-21 21:53:38.799 UTC  LOG: listening on IPv6 address "::", port 5432 2022-01-21 21:53:38.802 UTC  LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2022-01-21 21:53:38.807 UTC  LOG: database system was shut down at 2022-01-21 21:21:12 UTC 2022-01-21 21:53:38.807 UTC  LOG: invalid record length at 0/1898BE8: wanted 24, got 0 2022-01-21 21:53:38.807 UTC  LOG: invalid primary checkpoint record 2022-01-21 21:53:38.807 UTC  PANIC: could not locate a valid checkpoint record 2022-01-21 21:53:39.496 UTC  LOG: startup process (PID 14) was terminated by signal 6: Aborted 2022-01-21 21:53:39.496 UTC  LOG: aborting startup due to startup process failure 2022-01-21 21:53:39.507 UTC  LOG: database system is shut down
The log shows a database pod in a failure loop. For steps to fix the issue so that the upgrade can proceed, see the SysOpsPro documentation.
Because the default access mode in the PVC is
ReadWriteOnce, if you are deploying in an environment with multiple nodes then each pod might be on a different node. This causes the upgraded pod to spin up but then get stuck initializing because the original pod does not stop. To resolve this issue, find and delete the original pod so that the new pod can attach to the persistent volume:
Discover the name of the app pod that is not in a pending state by running:
kubectl get pods -n metadata-store
Delete the pod by running:
kubectl delete pod METADATA-STORE-APP-POD-NAME -n metadata-store