VMware Telco Cloud Service Assurance pods are not coming up

After restart of seven worker nodes, postgres pods are not coming up in VMware Telco Cloud Service Assurance.

Root Cause

The HA cluster gets into an inconsistent state, and must be recovered manually to select a new master and secondary nodes.

Solution

Run the following command to recover postgres:

postgres@postgres-monitor-0:/$ pg_autoctl drop node --hostname postgres-1.postgres-agent.default.svc.cluster.local --pgport 5432