Delete a namespace stuck in the Terminating state - Alibaba Cloud Service Mesh

After you run kubectl delete ns, the namespace may remain in the Terminating state indefinitely instead of being removed:

$ kubectl delete ns <namespace>
Error from server (Conflict): Operation cannot be fulfilled on namespaces "<namespace>":
The system is ensuring all content is removed from this namespace.
Upon completion, this namespace will automatically be purged by the system.

To confirm the namespace status, run the following command:

kubectl describe ns <namespace>

If the output shows Status: Terminating, follow the steps below to resolve the issue. Sample output:

Name:         <namespace>
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"<namespace>","namespace":""}}

Status:       Terminating

Why namespaces get stuck

When you delete a namespace, Kubernetes sets a deletionTimestamp on the namespace and waits for its finalizers to complete. The default kubernetes finalizer instructs the namespace controller to delete all namespaced resources before the namespace itself can be removed.

The namespace gets stuck in Terminating when the controller cannot finish this cleanup. Common causes include:

Unavailable API services. If a Custom Resource Definition (CRD)-based API service, such as Istio custom resources in ASM, becomes unavailable, the controller cannot list or delete resources managed by that API. This is the most common cause in Service Mesh (ASM) environments.
Residual resources with their own finalizers. Resources within the namespace may have finalizers that their respective controllers have not yet processed.
Missing or broken controllers. If the controller responsible for a finalizer has been removed or is not running, the finalizer is never cleared.

Diagnose the root cause

Before you force-remove finalizers, identify what is blocking the deletion. Force-removing finalizers is a last resort that can leave orphaned resources in the cluster.

List all resources in the namespace. Replace <namespace> with the name of the stuck namespace.

   kubectl api-resources --verbs=list --namespaced -o name \
     | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

If the command returns an error like unable to retrieve the complete list of server APIs, an API service is unavailable. Identify the problematic API service.
```
   kubectl get apiservice | grep False
```
Inspect the unavailable API service. Replace <version.api-group> with the API service name from the previous output, for example v1beta1.metrics.k8s.io.
```
   kubectl describe apiservice <version.api-group>
```
Fix or remove the unavailable API service. For example, if an Istio-related API service is unavailable because the ASM control plane has been removed, delete the API service registration.
```
   kubectl delete apiservice <version.api-group>
```
After you fix the API service issue, verify that the namespace has been deleted. If the namespace no longer appears, the issue is resolved. If the namespace is still in Terminating, proceed to force-remove the finalizers.
```
   kubectl get namespace <namespace>
```

Force-remove namespace finalizers

Warning

Removing finalizers forces Kubernetes to delete the namespace without confirming that all resources have been cleaned up. This can leave orphaned resources in the cluster. To recover orphaned resources, re-create the namespace with the same name. The orphaned resources may reappear under the re-created namespace, allowing manual cleanup.

Option 1: Use kubectl patch (recommended)

Patch the namespace to clear its finalizers.

kubectl patch namespace <namespace> -p '{"spec":{"finalizers":null}}' --type=merge

Verify that the namespace has been deleted.

kubectl get namespace <namespace>

If this command fails due to cluster API restrictions, use Option 2.

Option 2: Use the Kubernetes API directly

Start a kubectl proxy session. Expected output:

   kubectl proxy &

   Starting to serve on 127.0.0.1:8001

Open a new shell terminal. Define environment variables to connect to the Kubernetes cluster, then run a curl command to verify connectivity and authorization.
Note
If you started the proxy with kubectl proxy, authentication is handled automatically and the TOKEN header is optional. The TOKEN-based approach provides an additional layer of verification.
```
   export TOKEN=$(kubectl describe secret $(kubectl get secrets | grep default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d '\t')
   curl http://localhost:8001/api/v1/namespaces --header "Authorization: Bearer $TOKEN" --insecure
```

Send a PUT request to the namespace finalize endpoint with an empty finalizers list. Replace both instances of <namespace> with the actual namespace name, for example istio-system.

   cat <<EOF | curl -s -X PUT localhost:8001/api/v1/namespaces/<namespace>/finalize \
     -H "Content-Type: application/json" \
     --header "Authorization: Bearer $TOKEN" --insecure \
     --data-binary @-
   {
     "kind": "Namespace",
     "apiVersion": "v1",
     "metadata": {
       "name": "<namespace>"
     },
     "spec": {
       "finalizers": []
     }
   }
   EOF

Verify that the namespace has been deleted.
```
   kubectl get namespace <namespace>
```

Clean up orphaned resources

After you force-remove the namespace, check for orphaned resources that may still exist in the cluster.

Re-create the namespace.
```
   kubectl create namespace <namespace>
```

Check whether any orphaned resources reappear.

   kubectl api-resources --verbs=list --namespaced -o name \
     | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

If orphaned resources are found, delete them manually.

   kubectl delete <resource-type> <resource-name> -n <namespace>

Delete the namespace again.
```
   kubectl delete namespace <namespace>
```