Velero namespace stuck Terminating

You may encounter this issue when Data Protection is disabled on a cluster or when a cluster is detached from Tanzu Mission Control.

Problem

The Velero namespace is stuck in the Terminating phase either when Data Protection is disabled on a cluster or when a cluster is detached from Tanzu Mission Control.

Cause

Starting with Velero 1.12, certain resources like restores now make use of finalizers. In order for the Velero namespace to finish deleting, all finalizers for resources within the namespace must be removed. During disablement of Data Protection on a cluster or during the detach of a cluster from TMC, in rare situations, the Velero pods as well as the TMC data protection pods may be cleaned up before getting a chance to remove all the finalizers.

Solution

• ♦ Use one of the following solutions:
  • Remove finalizers from each resource. Look for resources still present in the Terminating namespace.:

    kubectl -n velero get restores
    kubectl -n velero get datauploads
    kubectl -n velero get datadownloads
    

    Remove the finalizers from each resource using kubectl edit or this one line command:

    kubectl -n velero patch restore <restore name> -p '{"metadata":{"finalizers":null}}' --type=merge
    

  • Force delete the namespace:

    kubectl get ns velero -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/velero/finalize" -f -