From VMware Telco Cloud Automation 3.1 release onward, a new micro service called tca-database-admin-service is introduced. This service periodically tracks table size of the configured database. It also optionally runs vacuum full on tables when bloat size and overall table size exceed the configured thresholds.
Following services are added:
A new PostgreSQL job is added that does vacuum full on all databases.
A new script to explicitly perform vacuum full on all databases is added.
VMware Telco Cloud Automation Database Admin Service
The TCA Database Admin Service is a generic database admin service that currently provides TCA Database Vacuum Service only. In future releases, other database based add-ons can be added. Currently, this service is configured to do vacuum full on tca database. By default, the vacuum full is DISABLED by the service, which means the service only reports if thresholds are exceeded. This is a Golang based service that runs in a container which in turn runs in a Kubernetes pod. Currently it is configured to run only 1 replica of this service per appliance. It is ran on both TCA and TCA-CP. On TCA 3.1.0 CN in VM appliance, deployed setup, you can check if the service was deployed successfully by kapp as shown in the following example:
admin@10 [ ~ ]$ kubectl get app tca-database-admin-service -n tca-mgr NAME DESCRIPTION SINCE-DEPLOY AGE tca-database-admin-service Reconcile succeeded 50s 3d21h
As mentioned above the service runs as a pod, you can check if the pod is running as shown in the following example:
admin@10 [ ~ ]$ kubectl get pod -n tca-mgr | greptca-database-admin-service
tca-database-admin-service-67c796dd7c-9tjx5 1/1Running
0 24h
You can get the logs of the service as shown in the following example:
admin@10 [ ~ ]$ kubectl logs -n tca-mgr tca-database-admin-service-67c796dd7c-9tjx5 time="2024-02-06T00:09:18Z"level=info msg="Starting TCA database admin service" time="2024-02-06T00:09:18Z"level=info msg="Reading config..." time="2024-02-06T00:09:18Z"level=info msg="Starting Vacuum Service" time="2024-02-06T00:09:18Z"level=info msg="Registering for database tca" time="2024-02-06T00:09:18Z"level=info msg="open pg connection, host: postgres.tca-mgr.svc.cluster.local" time="2024-02-06T00:09:18Z"level=info msg="Starting ticker for database tca"
You can execute into this container to get the config, as described in the following example:
admin@10 [ ~ ]$ kubectl exec-it -n tca-mgr tca-database-admin-service-67c796dd7c-9tjx5 -- cat/config.yamlServices: - Name: Vacuum Service
Description: TCA Database Vacuum Service Configuration
Databases:
- Name: tca
Period: 3600
Tables:
- Name: "Job"
TableToLogicalDatabasePercentage: 30
TableBloatPercentage: 95
- Name: "objectstore.files"
TableToLogicalDatabasePercentage: 30
TableBloatPercentage: 95
As you can see that explicit thresholds are configured (bloat and table size relative to database size). For other tables for the configured database, the default bloat percentage threshold is set as 99% and table size to database size percentage threshold is set to 33.3%. Currently the PVC assigned to PostgreSQL has a capacity of 30 GB.
As mentioned above, the database admin service by default does not have vacuum full enabled on the configured database, because vacuum full is disruptive, it locks the table with access exclusive lock. The config map looks as shown in the following example:
admin@10 [ ~ ]$ kubectl get cm database-admin-configmap -n tca-mgr -oyaml
apiVersion: v1
data:
enableVacuumFull: "no"
kind: ConfigMap
metadata:
annotations:
kapp.k14s.io/disable-original: ""
kapp.k14s.io/identity: v1;tca-mgr//ConfigMap/database-admin-configmap;v1
creationTimestamp: "2024-02 03T02:33:46Z"
labels:
kapp.k14s.io/app: "1706927626409117159"
kapp.k14s.io/association:
v1.feb5250d0dd2d5ca12dc063fb7575a41
name: database-admin-configmap
namespace: tca-mgr
resourceVersion: "561042"
uid: 6886efab-fe36-48b0-859c-e9c183839b48
To enable the vacuum full flag:
Execute the following command. After executing this command, restart the TCA database admin service pod. You can change the value of enableFullVacuum to
yesusingkubectl editcommand.kubectl edit cm database-admin-configmap -n tca-mgr
An editor opens. Make the change from
notoyes.Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 data: enableVacuumFull: "yes" kind: ConfigMap metadata: annotations: kapp.k14s.io/disable-original: "" kapp.k14s.io/identity: v1;tca-mgr//ConfigMap/database-admin-configmap;v1 creationTimestamp: "2024-02-03T02:33:46Z" labels: kapp.k14s.io/app: "1706927626409117159" kapp.k14s.io/association: v1.feb5250d0dd2d5ca12dc063fb7575a41 name: database-admin-configmap namespace: tca-mgr resourceVersion: "561042" uid: 6886efab-fe36-48b0-859c-e9c183839b48Save the file and restart the
tca-database-admin-servicepod as follows:admin@10 [ ~ ]$ kubectl delete pod -n tca-mgr tca-database-admin-service-67c796dd7c-9tjx5pod "tca-database-admin-service-67c796dd7c-9tjx5"deleted
Once the service gets restarted, you can check the logs which says the vacuumFull is enabled.
admin@10 [ ~ ]$ kubectl logs tca-database-admin-service-67c796dd7c-pfp46 -n tca-mgr time="2024-02-07T21:56:00Z"level=info msg="Starting TCA database admin service" time="2024-02-07T21:56:00Z"level=info msg="Reading config..." time="2024-02-07T21:56:00Z"level=info msg="Starting Vacuum Service" time="2024-02-07T21:56:00Z"level=info msg="Registering for database tca" time="2024-02-07T21:56:00Z"level=info msg="open pg connection, host: postgres.tca-mgr.svc.cluster.local" time="2024-02-07T21:56:01Z"level=info msg="Starting ticker for database tca"
All the steps mentioned above can be performed in TCA-CP too under tca-cp-cn namespace.
