This topic provides information that can help troubleshoot problems you may encounter using Postgres for Kubernetes.

Monitor Deployment Progress

Use watch kubectl get all to monitor the progress of the Postgres operator deployment. The deployment is complete when the postgres operator pod is in the Running state. For example:

watch kubectl get all
NAME                                           READY       STATUS       RESTARTS      AGE
pod/postgres-operator-567dbc67b9-nrq5t         1/1         Running      0             57s
NAME                                           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes                             ClusterIP   10.96.0.1    <none>        443/TCP   2d4h
NAME                                           READY       UP-TO-DATE   AVAILABLE     AGE
deployment.apps/postgres-operator              1/1         1            1             57s
NAME                                           DESIRED     CURRENT      READY         AGE
replicaset.apps/postgres-operator-567dbc67b9   1           1            1             57s

Viewing Postgres Operator Logs

Check the logs of the operator to ensure that it is running properly.

kubectl logs -l app=postgres-operator
2019-08-05T17:24:16.182Z	INFO	controller-runtime.controller	Starting EventSource{"controller": "postgres", "source": "kind source: /, Kind="}
2019-08-05T17:24:16.182Z	INFO	setup	starting manager
2019-08-05T17:24:16.285Z	INFO	controller-runtime.controller	Starting Controller	{"controller": "postgres"}
2019-08-05T17:24:16.386Z	INFO	controller-runtime.controller
Starting workers	{"controller": "postgres", "worker count": 1}

List All Postgres Instances in the Cluster

When you create Postgres instances, each instance is created in its own namespace. To see all Postgres instances in the cluster, add the -all-namespaces option to the kubectl get command.

kubectl get postgres --all-namespaces
NAMESPACE   NAME               STATUS    AGE
default     postgres-sample    Running   19d
default     postgres-sample2   Running   15d
test        my-postgres        Failed    15d
test        my-postgres3       Failed    15d

Find the Versions of the Deployed Postgres Operator and Instances

To find the currently deployed version of the Postgres operator, use the helm command:

helm ls
NAME             	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART                        APP VERSION
postgres-operator	default  	1       	2021-10-11 13:26:00.769535 -0500 CDT	deployed	postgres-operator-v1.3.0	 v1.3.0

The version is in the chart name and the APP VERSION column.

To find the version of a Postgres instance, use the kubectl command to describe the instance's pod.

kubectl get pods
kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
postgres-sample-0                    1/1     Running   0          9s
postgres-operator-85f777b9db-wbj9b   1/1     Running   0          4m15s
Name:           postgres-sample-0
Namespace:      default
Priority:       0
Node:           minikube/192.168.64.32
Start Time:     Mon, 11 Oct 2021 14:10:38 -0500
Labels:         app=postgres
                controller-revision-hash=postgres-sample-5fc8fb8b4b
                headless-service=postgres-sample
                postgres-instance=postgres-sample
                role=read
                statefulset.kubernetes.io/pod-name=postgres-sample-0
                type=data
Annotations:    <none>
Status:         Running
IP:             172.17.0.8
Controlled By:  StatefulSet/my-postgres
Containers:
  pg-container:
    Container ID:  docker://6c651d690a6fdb6d1c0d3644ad8225037d31da1c33fd3f88f1625bdfd45cea3a
    Image:         postgres-instance:v1.3.0
    Image ID:      docker://sha256:00359ca344dd96eb05f2bd430430c97a6d46a40996c395fca44c209cb954a6e7
    Port:          5432/TCP
    Host Port:     0/TCP

The Tanzu Postgres version can be found in the image name of the pg-container entry.

Cannot Reduce Instance Data Size After Deployment

When deploying an instance using a specific storage size in the instance yaml deployment file, you cannot reduce the instance data storage size at a later stage. For example, after creating an instance and setting the storage size to 100M:

kubectl create -f postgres.yaml

Verify the storage size using a command similar to:

kubectl get postgres.sql.tanzu.vmware.com/postgres-sample -o jsonpath='{.spec.storageSize}'
100M

If you later patch the instance to decrease the storage size from 100M to 2M:

kubectl patch postgres.sql.tanzu.vmware.com/postgres-sample --type merge -p '{"spec":{"storageSize": "2M"}}'

the operation returns an error similar to:

Error from server (spec.storageSize: Invalid Value: "2M" spec.storageSize cannot be reduced for an existing instance
spec.storageSize: Invalid Value: "2M" spec.storageSize needs to be at least 250M): admission webhook "vpostgres.kb.io" denied the request: spec.storageSize: Invalid Value: "2M" spec.storageSize cannot be reduced for an existing instance
spec.storageSize: Invalid Value: "2M" spec.storageSize needs to be at least 250M

To reduce the instance data size, create a new instance and migrate the source data over. Ensure that the source data fits in the reduced data size allocation of the newly created instance.

Errors during Backup of two Different Instances on the Same Bucket

This scenario occurs when you have two separate Kubernetes clusters with matching instance and namespace names. This scenario requires the following conditions:

  • Each cluster has a matching namespace name; for example cluster 1 has a namespace called my-namespace, and cluster 2 has a namespace called my-namespace.
  • Each cluster has a Postgres instance with the same name, for example my-instance.
  • Both clusters share the same S3 bucket for backups.

During backup, the first Postgres instance creates a backup stanza using the format my-instance-my-namespace. That stanza is encrypted with a randomly-generated backup cipher. During backup configuration for the second instance, the instance detects that a backup stanza with the same name already exists in the bucket. However, the second instance cannot decrypt the backup information because it uses a different cipher. The error is similar to: :

ERROR: [043]: WAL segment to get required 2021-09-02 15:55:35.615 P00 INFO: archive-get command end: aborted with exception [043] command terminated with exit code 43 or FormatError: key/value found outside of section at line 1: ▒▒▒H▒t=֠O@▒Y▒.

Workaround: Use different instance names, or different namespace names, or different buckets for backups.

check-circle-line exclamation-circle-line close-line
Scroll to top icon