This article helps in finding and deleting long-running finalizers that are waiting on other resources and can cause deletions to appear stuck in the terminating state.
When Kubernetes attempts to delete the cluster object that has finalizers "dist.kio.kasten.io/debootstrap or dist.kio.kasten.io/manual-debootstrap" specified, the Kubernetes API marks the object for deletion by populating .metadata.deletionTimestamp, and returns a 202-status code (HTTP βAcceptedβ). The cluster object remains in a terminating/Pending state while the control plane, or other components, take the actions defined by thefinalizers.
Long-running finalizers that are waiting on other resources can cause deletions to appear stuck in the terminating state. This is a result of the ServiceAccount (or its corresponding token/bindings) being deleted on the secondary cluster, and 401 or 500 error shown in the Veeam Kasten for Kubernetes dashboard.
The Veeam Kasten for Kubernetes multi-cluster dashboard displays 'Removal Pending' for primary and secondary clusters, as shown below:
The traditional way to resolve this involves determining which API service is not responding and then debugging that issue.
An alternative workaround for this issue is to remove the clusters by manually deleting the finalizers which are restricting deletion of the cluster.
Here are the steps to remove an object's finalizers:
kubectl get clusters.dist.kio.kasten.io -n kasten-io-mc
Copy
kubectl delete clusters.dist.kio.kasten.io ${CLUSTER_NAME} -n kasten-io-mc
Copy
kubectl get cluster.dist.kio.kasten.io -n kasten-io -o yaml
Copy
Example output:
-- apiVersion: dist.kio.kasten.io/v1alpha1
kind: Cluster
metadata:
creationTimestamp: "2022-04-26T11:02:22Z"
deletionGracePeriodseconds: 0
** deletionTimestamp: "2022-04-26T11:18:54Z"
finalizers:
-- dist.kio.kasten.io/manual--debootstrap**
generation: 2
labels:
Β dist.kio.kasten.io/cluster--type: secondary
name: secondary--cluster
namespace: kasten--io--mc
resourceVersion: "3630125"
uid: 8eae79ca-0a3e-454e-938e-7ac3c281ba4c
kubectl patch --namespace=kasten-io-mc clusters.dist.kio.kasten.io ${CLUSTER_NAME} -p '{"metadata":{"finalizers":null}}' --type=merge type=merge
Copy
The following command can be used to delete all clusters at once:
for x in $(kubectl get clusters.dist.kio.kasten.io -n kasten-io-mc | awk 'NR>1 {print $1}') ; do kubectl patch --namespace=kasten-io-mc clusters.dist.kio.kasten.io $x -p '{"metadata":{"finalizers":null}}' --type=merge ; done
Copy
To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.