Make systemd better for Podman with Quadlet

One of the best features of Podman is how well it works with systemd. Podman uses the standard fork/exec model, which is easily adaptable to the systemd system and service manager model.

[ Download now: Podman basics cheat sheet ]

In Podman at the edge, Valentin Rothberg writes:

"Running containerized workloads in systemd is a simple yet powerful means for reliable and rock-solid deployments. The systemd integration further lays the foundation for more advanced Podman features such as auto updates and rollbacks or running Kubernetes workloads in systemd. This integration allows systemd to manage service dependencies, monitor the lifecycle and service state, and possibly also restart services in case of failure."

Once administrators realized that Podman works well in systemd unit files, the Podman upstream started getting questions on best practices for running Podman as a systemd service. We studied the issues and worked with the upstream systemd team to template these best practices. Podman added the podman generate systemd command to create a unit file directly from a running container.

[ Get the systemd commands cheat sheet ] 

Alex Larsson created the Quadlet project after figuring out a better way to use this templating using systemd generators to regenerate services on the fly using a generic systemd unit file definition. Quadlet hid the complexity of running containers under systemd from users, making writing unit files from scratch much easier to maintain. Think of it as a Compose or Kubernetes file but for systemd. Once written, you can deploy it anywhere.

What is Quadlet?

The original Quadlet repository describes Quadlet this way:

What do you get if you squash a Kubernetes kubelet?
A quadlet

Quadlet is a tool for running Podman containers under systemd in an optimal way by allowing containers to run under systemd in a declarative way. It has been merged into Podman 4.4.

As the (now frozen) Quadlet repository says:

"Containers are often used in a cloud context, and they are then used in combination with an orchestrator like Kubernetes. They are also commonly used during development and testing to manually manage containers on an ad-hoc basis.

"However, there are also use cases where you want some kind of automatic container management, but on a smaller, single-node scale, and often more tightly integrated with the rest of the system. Typical examples of this can be embedded or automotive use, where there is no system administrator, or disconnected or edge servers.

"The recommended way to do this is to use systemd to orchestrate the containers, since this is an already running process manager, and since podman containers are just child processes. There are many documents that describe how to use podman with systemd directly, but the end result are generally large, hard to maintain systemd config files. And often the container setup isn't optimal."

Basically, anywhere you want to run a containerized system service without requiring human intervention, it's wise to use systemd to manage your locally running Podman containers.

[ Check out new container events and auditing features in Podman 4.4. ]

Using Quadlet in Podman

With Quadlet in Podman 4.4, you can create CTRNAME.container unit files that can be placed in one of the following directories:

/usr/share/containers/systemd/
/etc/containers/systemd/

For rootless users:

$HOME/.config/containers/systemd/

The following example creates a mysleep.container unit file that runs the sleep 1000 command on a ubi9-minimal container image:

cat $HOME/.config/containers/systemd/mysleep.container
[Unit]
Description=The sleep container
After=local-fs.target

[Container]
Image=registry.access.redhat.com/ubi9-minimal:latest
Exec=sleep 1000

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

Notice the [Container] section. You just need to specify the Image you want to run and the command (Exec) that you want to run, and then you can use all of the other fields normally specified in a systemd unit file.

Now you need to inform systemd about the new unit file. Type:

$ systemctl --user daemon-reload

This creates a mysleep.service file based on the mysleep.container file.

$ systemctl --user status mysleep.service
○ mysleep.service - The sleep container
 	Loaded: loaded (/home/dwalsh/.config/containers/systemd/mysleep.container; generated)
 	Active: inactive (dead)

Now enable this service or start it up.

$ systemctl --user start mysleep.service

The podman command inside the generated service will download the ubi9-init image and start the specified command, sleep 1000.

[ Get hands on with Podman in this tutorial. ]

You can check the status of the service with:

$ systemctl --user status mysleep.service
● mysleep.service - The sleep container
 	Loaded: loaded (/home/dwalsh/.config/containers/systemd/mysleep.container; generated)
 	Active: active (running) since Thu 2023-02-09 18:07:23 EST; 2s ago
   Main PID: 265651 (conmon)
  	Tasks: 3 (limit: 76815)
 	Memory: 1.6M
    	CPU: 94ms
 	CGroup: /user.slice/user-3267.slice/user@3267.service/app.slice/mysleep.service
         	├─libpod-payload-421c8293fc1ba7b6b08263e6895ed8187abb5ff136a7dfb073ed931883a68491
         	│ └─265653 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 1000
         	└─runtime
           	├─265648 /bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp >
           	└─265651 /usr/bin/conmon --api-version 1 -c 421c8293fc1ba7b6b08263e6895ed8187abb5ff136a7dfb073ed>

Feb 09 18:07:23 fedora systemd[2261]: Starting mysleep.service - The sleep container...
Feb 09 18:07:23 fedora podman[265630]: 2023-02-09 18:07:23.504068408 -0500 EST m=+0.044907849 container create >
Feb 09 18:07:23 fedora podman[265630]: 2023-02-09 18:07:23.542799094 -0500 EST m=+0.083638534 container init 42>
Feb 09 18:07:23 fedora systemd[2261]: Started mysleep.service - The sleep container.

When Podman is installed, it registers a systemd-generator tool that looks for files in the above directories and runs the /usr/libexec/podman/quadlet executable.

You can examine the service file that Quadlet creates using the Quadlet command line:

$ /usr/libexec/podman/quadlet -dryrun -user
quadlet-generator[265731]: Loading source unit file /home/dwalsh/.config/containers/systemd/mysleep.container
---mysleep.service---
[Unit]
Description=The sleep container
Before=local-fs.target
SourcePath=/home/dwalsh/.config/containers/systemd/mysleep.container
RequiresMountsFor=%t/containers

[X-Container]
Image=registry.access.redhat.com/ubi9-minimal:latest
Exec=sleep 1000

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-rm -f %t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=systemd-%N --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --sdnotify=conmon -d registry.access.redhat.com/ubi9-minimal:latest sleep 1000

The Quadlet-generated service file uses the same libraries as podman generate systemd –new but hides the complexity from the user.

One benefit is if a newer version of Podman is released with fixes or enhancements to the generator, your service is updated with the enhanced version the next time systemctl daemon-reload executes, such as upon reboot.

The container descriptions focus on the relevant container details, with no technical details about how the Podman integration works. This means they are straightforward to write and maintain, and integration can automatically improve as new Podman features become available. Now you can use the other cool features of Podman in systemd, like auto-update and rollback, for hands-free management of your containerized service's lifecycle.

Quadlet supports other advanced unit files, in addition to .container.

• .kube allows you to specify a Kubernetes.yaml file, which tells Quadlet to create a service file to run pods and containers under systemd services based on Kubernetes.
• .network tells Quadlet to create a service file that defines a Podman container network device. These network devices can then be used within .container and .kube unit files.
• .volume tells Quadlet to create a service file that defines a Podman volume. These volumes can then be used within .container unit files.

Find more information in the podman-systemd.unit(8) file.

Future articles will describe advanced use cases for Quadlet.

Wrapping up

Quadlet allows running containers under systemd in a declarative way. Similar to Compose or Kubernetes files, you can declare what you want to run without having to deal with all the complexities of running the workload.

Podman has a long history of integrating well into a modern Linux system, and Quadlet pushes the integration to the next level. Podman 4.4 has already been released to many Linux distributions, such as CentOS Stream and Fedora. Try out Quadlet and let us know what you think.

[ Kubernetes: Everything you need to know ]