How to fix Kubernetes namespaces stuck in the terminating state

A Kubernetes namespace isolates specific system resources usually visible to all processes. Each namespace has its own services, pods, and deployments in the cluster. A namespace in Kubernetes is essentially the same as a project in OpenShift.

These days, most products are containerized and can easily be deployed on Kubernetes or OpenShift. This requires continuous application deployment and testing. I recently worked on test automation that involves the creation of namespaces, resources, and data in the namespace, followed by running the test suite, and then removing the data and namespace once the tests are complete.

While frequently creating and deleting namespaces in Kubernetes, I stumbled upon an issue where a namespace got stuck in the Terminating state and just refused to delete.

I use minikube to run Kubernetes locally, but you can use the following steps and commands in any Kubernetes or OpenShift environment.

Note: The example namespace in my screenshots and commands is tackle-operator.

Problem: Deleting a namespace

Sometimes the process to delete namespaces gets stuck, and the command never completes. While the command returns a message showing that the namespace was deleted, querying it indicates that it's actually in a Terminating state:

$ kubectl delete namespace tackle-operator
namespace "tackle-operator" deleted

$ kubectl get namespace
NAME               	   STATUS    	   AGE
default            	                Active    	   148d
ingress-nginx      	    Active    	   148d
kube-node-lease    	    Active    	   148d
kube-public        	    Active    	   148d
kube-system        	    Active    	   148d
kubernetes-dashboard   Active    	   148d
tackle-operator    	   Terminating       10d

The result shows the namespace was deleted, but its status reveals that it's not quite gone yet, and you can still see the namespace in the minikube UI.

Open the minikube dashboard in the Select namespace dropdown on the top, and you can still select the namespace you thought you deleted.

Image

You might try to delete it from the user interface (UI). To do so, click on the three dots shown at the right of the namespace and select Delete.

Check after a few minutes (or days, months, years). It still shows up as terminating.

[ Learning path: Getting started with Red Hat OpenShift Service on AWS (ROSA) ] 

Why do some namespaces never delete?

Kubernetes stores each namespace as a YAML or JSON file.

$ kubectl get namespace ${NAMESPACE} -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
	kubectl.kubernetes.io/last-applied-configuration: |
	kubernetes.io/metadata.name: tackle-operator
spec:
  finalizers:
  - kubernetes
status:
  conditions:
 - lastTransitionTime: "2022-01-19T19:05:31Z"
	message: 'Some content in the namespace has finalizers remaining: tackles.tackle.io/finalizer in 1 resource instances'
	reason: SomeFinalizersRemain
	status: "True"
	type: NamespaceFinalizersRemaining
  phase: Terminating

Notice the inclusion of the finalizers field in the above JSON. Some namespaces have a finalizer defined under spec.

A finalizer is a special metadata key that tells Kubernetes to wait until a specific condition is met before it fully deletes a resource.

So when you run a command like kubectl delete namespace abcd, Kubernetes checks for a finalizer in the metadata.finalizers field. If the resource defined in the finalizer cannot be deleted for any reason, then the namespace is not deleted either. This puts the namespace into a terminating state awaiting the removal of the resource, which never occurs.

When an object has been terminating for an excessive time, check its finalizers by inspecting the metadata.finalizers field in its YAML.

[ Deploy an application with Red Hat OpenShift on AWS ]

Deleting a namespace stuck in a terminating state

Once you understand the cause of the problem, you can understand why the solution is to remove the finalizer from the YAML. Only by eliminating the constraint preventing a namespace from being deleted can you remove the namespace successfully.

There are subtleties in this task, though. For example, you can't just edit the namespace YAML from the Kubernetes UI to remove the finalizer because the UI doesn't update the namespace. To see this, edit the namespace to remove the finalizer, and then update. Edit the YAML again, and you'll notice that the finalizer still exists.

Here's the right way to do it.

Step 1: Dump the contents of the namespace in a temporary file called tmp.json:

$ kubectl get namespace ${NAMESPACE} -o json > tmp.json

Step 2: Edit the temporary file in your favorite text editor (mine is Vi):

$ vi tmp.json

Step 3: Remove kubernetes from the finalizer array, and save the file. You can skip to step 4 now unless you're on OpenShift or OKD. For those, you must set up a temporary proxy. Keep this terminal open until the namespace is deleted. To start a proxy server at http://127.0.0.1:8001, use the proxy subcommand:

$ oc proxy

From a separate window, run this command:

$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/${PROJECT_NAME}/finalize

Step 4: Call the Kubernetes API application/json against the /finalize endpoint for the namespace to update the JSON. Use the port number appropriate for your instance. I'm using 44315 because that's where my instance of minikube is running:

$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:44315/api/v1/namespaces/${NAMESPACE}/finalize

If you don't know what port your Kubernetes instance is using, look in your browser's URL bar when you navigate to the UI:

$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:44315/api/v1/namespaces/${NAMESPACE}/finalize
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
	"name": "tackle-operator",
	"creationTimestamp": "2022-01-10T19:13:58Z",
	"deletionTimestamp": "2022-01-19T19:05:18Z",
  "status": {
	"phase": "Terminating",
	"conditions": [
  	{
    	"type": "NamespaceDeletionDiscoveryFailure",
    	"status": "False",
    	"lastTransitionTime": "2022-01-21T04:51:31Z",
    	"reason": "ResourcesDiscovered",
    	"message": "All resources successfully discovered"
  	},
  	{
    	"type": "NamespaceDeletionContentFailure",
    	"status": "False",
    	"lastTransitionTime": "2022-01-19T19:05:31Z",
    	"reason": "ContentDeleted",
    	"message": "All content successfully deleted, may be waiting on finalization"
  	},
}

And that's it. The minikube dashboard no longer displays the namespace:

$ kubectl get namespace ${NAMESPACE}
Error from server (NotFound): namespaces "${NAMESPACE}" not found

Stop terminating

In my testing, I create and delete namespaces frequently. During this, I discovered an issue where namespaces became stuck in the terminating state. I developed a straightforward process for entirely deleting the namespaces. I hope that this process will help anyone else who encounters the same issue.

[ Learn 16 steps for building production-ready Kubernetes clusters. ]