Creating a pod-watcher Operator for Kubernetes

All of the code written for this post is available on my GitHub.

A new engineer on my team was recently asked to write a simple operator that would log pod creation events in a specific namespace as part of their onboarding process. I offered to help them with this task as I had prior experience working on operators.

As I began to think about how I would build such an operator, it occurred to me that I had never actually written one from scratch. I decided to step through the entire creation of such an operator on my own. Here I’ll share the steps I arrived at to accomplish that goal.

1. Prerequisites

• A running Kubernetes cluster
  • If you don’t have one, you can start one locally with KinD (Kubernetes in Docker)
• Install kubectl
• Install operator-sdk
• A working development environment for the Go programming language

2. Creating a project

Fortunately, the operator-sdk command makes it trivial to bootstrap new operators. First, create a working directory for your operator. From the directory where you keep your Go projects:

$ mkdir pod-watcher && cd $_
$ operator-sdk init --repo github.com/<your_github_username>/pod-watcher

The first command here creates your working directory and moves you into it. The second command initializes your project by creating a main.go , Makefile, go.mod, and many other configuration files. While I won’t cover all of those here, you can learn more by looking at the Kubebuilder project layout docs.

3. Creating a controller

Now you need to create a controller. A controller is a component of an operator that watches for changes to specified resources. For this example, you need a controller to watch pods. Yet again, the operator-sdk makes this extremely easy. From inside  the project directory, run this command:

$ operator-sdk create api --version=v1 --kind=Pod --controller --resource=false

This creates a new controller at /controllers/pod_controller.go, and invokes the SetupWithManager method in the main function. Let’s go over the flags used in this command.

• version: The API version of a resource. For the `Pod` type, this is `v1`.
• kind: The resource you're creating an API for.
• controller: Creates a controller for the API. Controllers are covered more in the next section.
• resource: Whether or not you want to create a Custom Resource Definition (CRD) for your API. In this case, you don’t need to create a new resource since Pods are already a core part of the Kubernetes API.

4. Implementing the controller functionality

Controllers are created with two methods: Reconcile and SetupWithManager. Both  methods have a pointer receiver, which has a type of the reconciler struct  created for the controller. Taking a look at /controllers/pod_controller.go, you can see that these two methods still need to be implemented.

SetupWithManager

The SetupWithManager method tells the operator’s manager how to build the specified controller. At first, it looks like this:

func (r *PodReconciler) SetupWithManager(mgr ctrl.Manager) error {
  return ctrl.NewControllerManagedBy(mgr).
      // Uncomment the following line adding a pointer to an instance of the controlled resource as an argument
      // For().
      Complete(r)
}

For your controller to do anything, you must follow the instructions in the comments. This effectively turns the method into the following:

func (r *PodReconciler) SetupWithManager(mgr ctrl.Manager) error {
  return ctrl.NewControllerManagedBy(mgr).
      For(&v1.Pod{}).
      Complete(r)
}

This step is crucial, as it tells your controller what resource type to manage. In this case, you’re looking at Pods. Note that you must import `k8s.io/api/core/v1` to resolve the Pod type in code.

You only want your controller to handle events for Pod creations. You can do this by applying an Event Filter to the reconciler. This is handled with the WithEventFilter builder method. Update the method:

func (r *PodReconciler) SetupWithManager(mgr ctrl.Manager) error {
  return ctrl.NewControllerManagedBy(mgr).
      For(&v1.Pod{}).
      WithEventFilter(predicate.Funcs{
        DeleteFunc:  func(deleteEvent event.DeleteEvent) bool { return false },
        UpdateFunc:  func(updateEvent event.UpdateEvent) bool { return false },
        GenericFunc: func(genericEvent event.GenericEvent) bool { return false },
      }).
      Complete(r)
}

By default, all four event types (Delete, Update, Create, and Generic) are handled by the controller. To ignore certain event types, you must explicitly implement the filters for them as demonstrated above.

At this point, you have a controller that watches for the creation of all Pods on your cluster. You could run this operator, however it wouldn’t actually do anything. Let’s change that.

Reconcile

The Reconcile method is where the functionality of your controller lives.

You can find more details in the generated method comment, but for now start off with the basics:

func (r *PodReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  _ = log.FromContext(ctx)

  // TODO(user): your logic here

  return ctrl.Result{}, nil
}

The goal is to log any pod creation events that occur. You've already filtered out any other event types in the previous section, so you can just do the logging here. This requires just a few lines of code:

func (r *PodReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  logger := log.FromContext(ctx)
  logger.Info(fmt.Sprintf("Created pod %v", req.NamespacedName))
  return ctrl.Result{}, nil
}

At this point, you can run your operator and it logs any time a pod is created on your cluster. You’re just one small step from being finished.

5. Restrict Namespace

One of the original requirements of this exercise is to watch for pod creation events in a specific namespace. By default, operators watch for events in all namespaces. Fortunately, it’s trivial to restrict the namespace that your operator watches.

In main.go, find where ctrl.NewManager gets called. In the ctrl.Options struct being passed to the function, set a Namespace field with the desired namespace to watch. For this example, name it podwatcher-test:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
  Scheme:                 scheme,
  MetricsBindAddress:     metricsAddr,
  Port:                   9443,
  HealthProbeBindAddress: probeAddr,
  LeaderElection:         enableLeaderElection,
  LeaderElectionID:       "271a6ba3.my.domain",
  Namespace:              "podwatcher-test",
})

6. Run it!

At this point, you’re ready to run your pod-watcher operator. All you need to do is log in to your cluster and run make run from a terminal.

Once your operator is up, open a second terminal and create a pod in the podwatcher-test namespace:

$ kubectl create ns podwatcher-test
$ kubectl run --image nginx --namespace podwatcher-test my-pod

If you look back at the terminal where your operator is running, you see logged events. Note the text Created pod podwatcher-test/my-pod.

2023-12-01T15:20:09-05:00       INFO    Created pod podwatcher-test/my-pod      {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"my-pod","namespace":"podwatcher-test"}, "namespace": "podwatcher-test", "name": "my-pod", "reconcileID": "1ec45b28-64d3-4d9a-b0ae-7193624fcb6a"}

You've written a simple operator that watches for all pods created in a specific namespace. With a bit of Go knowledge and basic Kubernetes understanding, you can build operators to perform an endless number of tasks in the future.