Configure a container to start automatically as a systemd service

Use Podman and systemd integration to automatically start a containerized service with the operating system so that it persists across reboots.

Posted: February 21, 2023 by Alexon Oliveira (Sudoer, Red Hat)

Keys in old, red truck ignition — ^{Image by Kristin Heismeyer from Pixabay}

In early 2022, I had the privilege of writing a five-part series about Podman (see my Enable Sysadmin profile to find the articles). Podman is a daemonless engine for developing, managing, and running Open Container Initiative (OCI)-compliant containers. Containers aren't just a developer thing anymore; they're increasingly an intrinsic part of a sysadmin's work. Therefore it's important to know how to manage them in a practical and automated way.

[ Get the Podman basics cheat sheet. ]

This article shows one way you can use the power of Podman and systemd to create a container solution that starts and stops automatically with your operating system. I'll begin with why you might need something like this.

Use case scenario

Podman can run rootless containers, so you can run a container to do whatever you want, from completing a system task to running a full application solution, such as web servers or databases.

Say you want containers that run these services indefinitely. In a typical sysadmin environment, a privileged user configures these services to run at system boot and manages them with the systemctl command. You can enable and start a service with the systemctl command as a regular user. These system services start when the system boots and stop when the system shuts down.

When you invoke Podman at runtime from the command line, it runs only as long as you are in the session (graphical interface, text console, or SSH), and it stops when you close the session. This functionality is not ideal for your need to have a Podman-hosted service persist across reboots. Here's where integrating Podman and systemd is handy.

[ Get the systemd commands cheat sheet. ]

According to RHEL 9 documentation, with systemd unit files, you can:

Set up a container or pod to start as a systemd service.
Define the order in which the containerized services run and check for dependencies (for example, making sure another service is running, a file is available, or a resource is mounted).
Control the state of the systemd system using the systemctl command.
Generate portable descriptions of containers and pods using systemd unit files.

Those are sample use cases. Here are the configuration steps.

[ Read how Podman 4.4 lets you make systemd better for Podman with Quadlet ]

Prepare the environment

First, as a privileged user, create a basic user in the system to act as a "service user" for the application with systemd:

# useradd webuser

# passwd webuser
Changing password for user webuser.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.

Keep this terminal open. In a separate terminal, SSH to the same server with the newly created user and verify it can run Podman commands:

$ ssh webuser@fedora
webuser@fedora's password:
Last login: Wed Feb  1 15:52:26 2023

$ podman version
Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.18.7
Built:        Fri Nov 11 12:24:13 2022
OS/Arch:      linux/amd64

Now, allow this "service user" account to start a service at system start that persists over logouts. Use the loginctl command to configure the systemd user service to persist after the last user session of the configured service closes. In the privileged user's terminal window, do the following:

# loginctl show-user webuser | grep ^Linger
Linger=no

# loginctl enable-linger webuser

# loginctl show-user webuser | grep ^Linger
Linger=yes

Return to the basic user's terminal window, and pull the container image for your application. For this example, I will use the httpd container image because it's easier to demonstrate:

$ podman pull docker.io/library/httpd
Trying to pull docker.io/library/httpd:latest...
Getting image source signatures
Copying blob 70698c657149 done  
Copying blob 00df85967755 done  
Copying blob 8b4456c99d44 done  
Copying blob ec2ee6bdcb58 done  
Copying blob 8740c948ffd4 done  
Copying config 6e794a4832 done  
Writing manifest to image destination
Storing signatures
6e794a4832588ca05865700da59a3d333e7daaaf0544619e7f326eed7e72c903

$ podman image ls
REPOSITORY               TAG         IMAGE ID      CREATED      SIZE
docker.io/library/httpd  latest      6e794a483258  2 weeks ago  149 MB

Run the container and mount an external volume in the system. If you've read my article on how to Create fast, easy, and repeatable containers with Podman and shell scripts (if you didn't, check it now), you're aware that I've already created this external volume with its contents in the /var/local/httpd/ directory:

$ ls /var/local/httpd/
index.html

$ cat /var/local/httpd/index.html
<html>
  <header>
    <title>Enable SysAdmin</title>
  </header>
  <body>
    <p>Hello World!</p>
  </body>
</html>

To make it easier (and since I'm a bit lazy), I will use the same content for this httpd container. Run it with Podman to check whether it works as intended:

$ podman run --name=httpd --hostname=httpd -p 8081:80 -v /var/local/httpd:/usr/local/apache2/htdocs:Z -d docker.io/library/httpd
f6f3d836bde9a75c6d745f481c8e83f8c7c342db2bbeac901347c535eaef03d9

$ podman ps
CONTAINER ID  IMAGE                           COMMAND           CREATED        STATUS            PORTS                 NAMES
f6f3d836bde9  docker.io/library/httpd:latest  httpd-foreground  5 seconds ago  Up 4 seconds ago  0.0.0.0:8081->80/tcp  httpd

$ curl -v http://localhost:8081
*   Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.82.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 01 Feb 2023 19:02:28 GMT
< Server: Apache/2.4.55 (Unix)
< Last-Modified: Fri, 07 Oct 2022 21:34:58 GMT
< ETag: "74-5ea7894e36d30"
< Accept-Ranges: bytes
< Content-Length: 116
< Content-Type: text/html
<
<html>
  <header>
    <title>Enable SysAdmin</title>
  </header>
  <body>
    <p>Hello World!</p>
  </body>
</html>
* Connection #0 to host localhost left intact

OK, it works. Stop the container:

$ podman stop httpd && podman rm -a && podman volume prune
httpd
f6f3d836bde9a75c6d745f481c8e83f8c7c342db2bbeac901347c535eaef03d9
WARNING! This will remove all volumes not used by at least one container. The following volumes will be removed:
No dangling volumes found

$ curl -v http://localhost:8081
*   Trying 127.0.0.1:8081...
* connect to 127.0.0.1 port 8081 failed: Connection refused
*   Trying ::1:8081...
* connect to ::1 port 8081 failed: Connection refused
* Failed to connect to localhost port 8081 after 1 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8081 after 1 ms: Connection refused

The prerequisites are complete. It's time to move on to the next part.

[ Download now: A sysadmin's guide to Bash scripting. ]

Configure containers with systemd

First, start your container again with Podman:

$ podman run --name=httpd --hostname=httpd -p 8081:80 -v /var/local/httpd:/usr/local/apache2/htdocs:Z -d docker.io/library/httpd
16df315d3b23f41a70d1d3ffc11315c967cd213a107961df970327e85e62286f

$ podman ps
CONTAINER ID  IMAGE                           COMMAND           CREATED        STATUS            PORTS                 NAMES
16df315d3b23  docker.io/library/httpd:latest  httpd-foreground  4 seconds ago  Up 4 seconds ago  0.0.0.0:8081->80/tcp  httpd

If you are familiar with creating systemd service units, you can do it "handcrafted." But why trouble yourself? Use the power of Podman to do it for you. It looks like this:

$ podman generate systemd --new --files --name httpd
/home/webuser/container-httpd.service

$ ls
container-httpd.service

$ cat container-httpd.service
# container-httpd.service
# autogenerated by Podman 4.3.1
# Wed Feb  1 16:06:04 -03 2023

[Unit]
Description=Podman container-httpd.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm \
	-f %t/%n.ctr-id
ExecStart=/usr/bin/podman run \
	--cidfile=%t/%n.ctr-id \
	--cgroups=no-conmon \
	--rm \
	--sdnotify=conmon \
	--replace \
	--name=httpd \
	--hostname=httpd \
	-p 8081:80 \
	-v /var/local/httpd:/usr/local/apache2/htdocs:Z \
	-d docker.io/library/httpd
ExecStop=/usr/bin/podman stop \
	--ignore -t 10 \
	--cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
	-f \
	--ignore -t 10 \
	--cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target

Cool, isn't it? You can see all the parameters available with a command by running podman generate systemd --help.

Now that you've generated the systemd service file with Podman, you won't run it manually anymore. Stop it like this:

$ podman stop httpd && podman rm -a && podman volume prune
httpd
16df315d3b23f41a70d1d3ffc11315c967cd213a107961df970327e85e62286f
WARNING! This will remove all volumes not used by at least one container. The following volumes will be removed:
No dangling volumes found

The next step is to create a ~/.config/systemd/user/ directory to hold the container-httpd.service file and reload the systemd daemon. You could also use the /etc/systemd/system directory, but only a privileged user can copy the systemd service file to this directory. Here are the commands:

$ mkdir -p ~/.config/systemd/user/

$ cp -Z container-httpd.service ~/.config/systemd/user/

$ ls ~/.config/systemd/user/
container-httpd.service

$ systemctl --user daemon-reload

It's almost done. Start the container with systemctl using the newly created systemd service and check if it's working properly:

$ systemctl --user start container-httpd.service

$ systemctl --user status container-httpd.service
● container-httpd.service - Podman container-httpd.service
     Loaded: loaded (/home/webuser/.config/systemd/user/container-httpd.service; disabled; vendor preset: disabled)
     Active: active (running) since Wed 2023-02-01 16:11:28 -03; 1min 13s ago
       Docs: man:podman-generate-systemd(1)
    Process: 4564 ExecStartPre=/bin/rm -f /run/user/1001/container-httpd.service.ctr-id (code=exited, status=0/SUCCESS)
   Main PID: 4600 (conmon)
      Tasks: 14 (limit: 4649)
     Memory: 4.6M
        CPU: 139ms
     CGroup: /user.slice/user-1001.slice/user@1001.service/app.slice/container-httpd.service
             ├─ 4585 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-t>
             ├─ 4587 rootlessport
             ├─ 4592 rootlessport-child
             └─ 4600 /usr/bin/conmon --api-version 1 -c 81b6c1c3125bd9a10ad3bb5f62534cc01caf6e070a948be6ce0afa924d1b2b1b -u 81b6c1c3125bd9a10ad3bb5f>

…output omitted…

It seems to be working. To prove that, check the running container with Podman and try to curl your application again:

$ podman ps
CONTAINER ID  IMAGE                           COMMAND           CREATED        STATUS            PORTS                 NAMES
81b6c1c3125b  docker.io/library/httpd:latest  httpd-foreground  2 minutes ago  Up 2 minutes ago  0.0.0.0:8081->80/tcp  httpd

$ curl -v http://localhost:8081
*   Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.82.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 01 Feb 2023 19:14:32 GMT
< Server: Apache/2.4.55 (Unix)
< Last-Modified: Fri, 07 Oct 2022 21:34:58 GMT
< ETag: "74-5ea7894e36d30"
< Accept-Ranges: bytes
< Content-Length: 116
< Content-Type: text/html
<
<html>
  <header>
    <title>Enable SysAdmin</title>
  </header>
  <body>
    <p>Hello World!</p>
  </body>
</html>
* Connection #0 to host localhost left intact

Bingo! You can now configure it to start and stop with your system by enabling it with systemctl:

$ systemctl --user enable container-httpd.service
Created symlink /home/webuser/.config/systemd/user/default.target.wants/container-httpd.service → /home/webuser/.config/systemd/user/container-httpd.service.

Put it to the test and reboot the system. If it's working correctly, when the system boots, the httpd container service should start using systemd. It should be active, enabled, and responsive. Confirm this with curl:

$ uptime
 16:16:47 up 36 min,  1 user,  load average: 0,00, 0,03, 0,16

# reboot

$ systemctl --user is-active container-httpd.service
active

$ systemctl --user is-enabled container-httpd.service
enabled

$ curl -v http://localhost:8081
*   Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.82.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 01 Feb 2023 19:20:18 GMT
< Server: Apache/2.4.55 (Unix)
< Last-Modified: Fri, 07 Oct 2022 21:34:58 GMT
< ETag: "74-5ea7894e36d30"
< Accept-Ranges: bytes
< Content-Length: 116
< Content-Type: text/html
<
<html>
  <header>
    <title>Enable SysAdmin</title>
  </header>
  <body>
    <p>Hello World!</p>
  </body>
</html>
* Connection #0 to host localhost left intact

$ podman ps
CONTAINER ID  IMAGE                           COMMAND           CREATED        STATUS            PORTS                 NAMES
c8a15a2853ac  docker.io/library/httpd:latest  httpd-foreground  3 minutes ago  Up 3 minutes ago  0.0.0.0:8081->80/tcp  httpd

It worked! You have a container configured with a web application running as if it were a standard systemd service. It remains persistent across reboots or terminal exits. The same procedure works for pods with a few minor changes, too.

But what if you no longer need that service? How do you remove its persistence? As a systemd service, you can stop and disable it like this:

$ systemctl --user stop container-httpd.service

$ podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

$ systemctl --user disable container-httpd.service
Removed /home/webuser/.config/systemd/user/default.target.wants/container-httpd.service.

As a final test, reboot the system again and ensure the application is not up:

$ uptime
 16:21:34 up 4 min,  1 user,  load average: 0,02, 0,10, 0,06

# reboot

$ uptime
 16:22:06 up 0 min,  1 user,  load average: 0,29, 0,06, 0,02

$ systemctl --user is-active container-httpd.service
inactive

$ systemctl --user is-enabled container-httpd.service
disabled

$ podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

$ curl -v http://localhost:8081
*   Trying 127.0.0.1:8081...
* connect to 127.0.0.1 port 8081 failed: Connection refused
*   Trying ::1:8081...
* connect to ::1 port 8081 failed: Connection refused
* Failed to connect to localhost port 8081 after 0 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8081 after 0 ms: Connection refused

Gone with the wind! I hope I demonstrated clearly how the configuration and administration of persistent containers with systemd works.

I recommend you read these articles from my Enable Sysadmin colleagues that expand the ideas in this article:

Wrap up

The powerful integration between Podman and systemd allows sysadmins to configure a container with a complete application solution that starts and stops automatically with the operating system.

Managing containers based on systemd units is mainly useful for basic and small deployments that do not need to scale. For more sophisticated scaling and orchestration of many container-based applications and services, consider an enterprise orchestration platform based on Kubernetes, such as Red Hat OpenShift Container Platform. I hope this article helps you understand this topic, supports your Linux certification path, and adds to your general sysadmin knowledge.

How to use new container events and auditing features in Podman 4.4

Knowing which containers are executed on a machine, what was done to them, and who did it is an important cornerstone of auditing.

Clay pot vases stacked inside of each other

Make systemd better for Podman with Quadlet

Quadlet, a tool merged into Podman 4.4, hides the complexity of running containers under systemd to make it easier to maintain unit files written from scratch.