Cloud-Native VMware Telco Cloud Automation Upgrade Troubleshooting

General troubleshooting methods for upgrading cloud-native VMware Telco Cloud Automation.

Getting kubeconfig of VMware Telco Cloud Automation cluster

Use the appliance manager REST API to get all the clusters of a bootstrapper virtual machine.

curl -XGET --user "bootstrapperVMUsername:bootstrapperVMPassword" "https://{bootstrapperVMIP}:9443/api/admin/clusters?clusterType=MANAGEMENT"

API returns JSON response, use clusterName to get the name of the VMware Telco Cloud Automation cluster. Use the appliance manager REST API to get the kubeconfig.

curl -XGET --user "bootstrapperVMUsername:bootstrapperVMPassword" "https://{bootstrapperVMIP}:9443/api/admin/clusters/{clusterName}/kubeconfig?clusterType=MANAGEMENT"

API returns JSON response, use kubeconfig to get the base64 encoded kubeconfig. Perform a bas64 decode of the kubeconfig and use decoded value for the kubectl, helm commands.

Getting logs of Cloud-Native VMware Telco Cloud Automation upgrade

To view upgrade logs of upgrade, use the following steps:

Obtain the names of VMware Telco Cloud Automation upgrade pods using the following command:

Note: tca-mgr is the namespace for VMware Telco Cloud Automation Manager and tca-system is the namespace for VMware Telco Cloud Automation Control Plane.

kubectl get pods -n tca-mgr/tca-system | grep upgrade-tca

Example

kubectl get pods -n tca-mgr | grep upgrade-tca
-----------------------------------------------
 upgrade-tca-manager-agent-5bc47f79cb-t2c7v 2/2 Running 0 16h
 upgrade-tca-manager-helm-service-84464bdbd4-sqdc4 2/2 Running 0 16h

View the logs of VMware Telco Cloud Automation upgrade pods, use the following command for each of the pod:
```
kubectl logs <name of the pod> -n tca-mgr/tca-system
```

Upgrade failed with missing VMware Telco Cloud Automation cluster details

If upgrade fails with the error message TCA Cluster details not found. Please configure it in Platform Manager.

Follow steps to store the VMware Telco Cloud Automation cluster kubeconfig.

Save the base64 encoded kubeconfig and VMware Telco Cloud Automation cluster name in the below format in a json file as below:

$ cat tca_kubeconfig.json
{
   "data":{
      "items":[
         {
            "config":{
               "url":"https://<TCA cluster controlPlaneEndpointIP>:6443",
               "clusterName":"<TCA Cluster name>",
               "kubeconfig":"<base64 encoded kubeconfig>"
             }
         }
      ]
   }
}

Store kubeconfig in mongodb of VMware Telco Cloud Automation Manager.

curl -k -XPOST --user "<username:password>" "https://<TCA_IP>:9443/api/admin/global/config/kubernetes?isInternal=true" -H "Content-Type: application/json" -H "Accept: application/json" -d @tca_kubeconfig.json

Store kubeconfig in mongodb of VMware Telco Cloud Automation Control Plane.

curl -k -XPOST --user "<username:password>" "https://<TCA_CP_IP>:9443/api/admin/global/config/kubernetes?isInternal=true" -H "Content-Type: application/json" -H "Accept: application/json" -d @tca_kubeconfig.json

Unable to login to VMware Telco Cloud Automation due to upgrade failure

If upgrade fails and you are not able to login to the VMware Telco Cloud Automation, restart the VMware Telco Cloud Automation API pod using the following command:

kubectl rollout restart deployment/tca-api -n tca-system/tca-mgr

Note: tca-mgr is the namespace for VMware Telco Cloud Automation Manager and tca-system is the namespace for VMware Telco Cloud Automation Control Plane.

After the restart, you can retry to upgrade the VMware Telco Cloud Automation upgrade.

Upgrade failure while setting up upgrade agent

If upgrade fails with the error message failure while setting upgrade agent.

Follow the steps before retrying the upgrade.

Restart the VMware Telco Cloud Automation helm service pod using the following command:
```
kubectl rollout restart deployment/tca-helm-service -n tca-system/tca-mgr
```
Uninstall any upgrade related helm services.
1. Obtain the upgrade helm services, use the following command:
```
helm list -n tca-mgr/tca-system | grep tca-manager/tca-cp
```
2. Uninstall upgrade helm services using the following command:
```
helm uninstall <serviceName> -n tca-mgr/tca-system 
```
Note: tca-mgr is the namespace for VMware Telco Cloud Automation Manager and tca-system is the namespace for VMware Telco Cloud Automation Control Plane.

Upgrade failure due to Read Timed Out

If upgrade fails with the error Failure while setting upgrade agent: Helm API failed: Read timed out, check KB for detailed steps.

Reinstalling service which failed to upgrade

If any service fails to upgrade during VMware Telco Cloud Automation upgrade, you can retry to upgrade the VMware Telco Cloud Automation.

However, if the upgrade fails after multiple retries, you can follow the steps to uninstall and reinstall the services.

Save the current override values of the service to a yaml file.

helm get values {serviceName}  -n  {namespace}  >  {service}.yaml

Uninstall the helm service.

helm uninstall {serviceName}  -n  {namespace}

Add VMware Telco Cloud Automation helm repo to be able fetch helm charts.

helm repo add jfrog https://vmwaresaas.jfrog.io/artifactory/helm-registry
 
helm repo update

Verify the added helm repo.

 helm repo list
-----------------
Response:
  NAME URL
   frog https://vmwaresaas.jfrog.io/artifactory/helm-registry

Install the helm service.
```
helm install {serviceName} {chartName} --version {version} -f {service}.yaml  -n {namespace}
```
Note: VMware Telco Cloud Automation services are deployed in the namespaces: tca-mgr, tca-system, istio-system, metallb-system, tca-services, postgres-operator-system, and fluent-system.

Example for reinstalling VMware Telco Cloud Automation helm chart

helm get values tca -n tca-mgr > tca-mgr.yaml
 
helm uninstall tca -n tca-mgr
 
helm repo add jfrog https://vmwaresaas.jfrog.io/artifactory/helm-registry
 
helm repo update
 
helm install tca jfrog/tca --version 2.1.0 -f tca-mgr.yaml -n tca-mgr

Note: You can use helm search repo option to search for VMware Telco Cloud Automation helm charts.