Restart a Failed TKG Cluster Upgrade

If the update of a TKG cluster fails, you can restart the update job and try the update again.

Problem

The update of a TKG cluster fails and results in cluster status of upgradefailed.

Cause

There can be several reasons for a failed cluster update, such as insufficient storage. To restart a failed update job and try the cluster update again, complete the following procedure.

Solution

1. Log in to the Supervisor as an administrator.
2. Look up the update_job_name.

   kubectl get jobs -n vmware-system-tkg -l "run.tanzu.vmware.com/cluster-namespace=${cluster_namespace},cluster.x-k8s.io/cluster-name=${cluster_name}"

3. Run kubectl proxy so that curl can be used to issue requests.

   kubectl proxy &

   You should see Starting to serve on 127.0.0.1:8001.

   Note: You cannot use kubectl to patch or update the .status of a resource.
4. Using curl issue the following patch command to raise the .spec.backoffLimit.

   curl -H "Accept: application/json" -H "Content-Type: application/json-patch+json" 
   --request PATCH --data '[{"op": "replace", "path": "/spec/backoffLimit", "value": 8}]' 
   http://127.0.0.1:8001/apis/batch/v1/namespaces/vmware-system-tkg/jobs/${update_job_name}

5. Using curl issue the following patch command to clear the .status.conditions so that the Job controller can create new pods.

   $ curl -H "Accept: application/json" -H "Content-Type: application/json-patch+json" 
   --request PATCH --data '[{"op": "remove", "path": "/status/conditions"}]' 
   http://127.0.0.1:8001/apis/batch/v1/namespaces/vmware-system-tkg/jobs/${update_job_name}/status