Backup-and-restore and crds-tcx apps are not deleted from default namespace during upgrade

While upgrading VMware Telco Cloud Service Assurance to 2.4, backup-and-restore and crds-tcx apps are not deleted from the default namespace. It shows the following error.

Removing finalizer from Backup CR : tcsa-pre-upgrade-backup
+ kubectl patch Backup tcsa-pre-upgrade-backup --type=json '-p=[{"op": "remove","path": "/metadata/finalizers"}]'
The Backup "tcsa-pre-upgrade-backup" is invalid:
* spec.components.postgres.dbs[0]: Invalid value: "adminservice": spec.components.postgres.dbs[0] in body should match '^(grafana|keycloak|analyticsservice|collector|alarmservice|airflow|remediation)$'
* spec.components.postgres.dbs[5]: Invalid value: "collector_manager": spec.components.postgres.dbs[5] in body should match '^(grafana|keycloak|analyticsservice|collector|alarmservice|airflow|remediation)$'
* spec.components.postgres.dbs[8]: Invalid value: "kpiservice": spec.components.postgres.dbs[8] in body should match '^(grafana|keycloak|analyticsservice|collector|alarmservice|airflow|remediation)$'
* spec.components.postgres.dbs[10]: Invalid value: "spe": spec.components.postgres.dbs[10] in body should match '^(grafana|keycloak|analyticsservice|collector|alarmservice|airflow|remediation)$'
* spec.components.postgres.dbs[11]: Invalid value: "svix_server": spec.components.postgres.dbs[11] in body should match '^(grafana|keycloak|analyticsservice|collector|alarmservice|airflow|remediation)$'

The following apps were still stuck in default ns because of this: [root@wdc-10-214-161-247 scripts]# kubectl get apps
NAME                 DESCRIPTION                                                                       SINCE-DEPLOY   AGE
backup-and-restore   Deleting                                                                          63s            43h
br-operator          Reconcile succeeded                                                               6s             43h
crds-tcx             Reconcile failed: Deploying: Error (see .status.usefulErrorMessage for details)   39m            44h
kube-arangodb        Reconcile succeeded                                                               50s            43h

Workaround

You must force delete backup-and-restore and crds-tcx apps from the default namespace.

To force delete backup-and-restore and crds-tcx apps from the default namespace, run the following command.

kubectl patch tcxproduct ${tcsa_name} --type='json' -p='[{"op": "replace","path": "/metadata/finalizers", "value" : [ "finalizer.tcxproduct.tcx.vmware.com" ] }]'