Extending Red Hat Unified Kernel Images More Securely By Using Addons

With the advent of Confidential Virtual Machines (CVMs) in RHEL, a new challenge has emerged: Extending the Red Hat UKI (Unified Kernel Image) more safely and without compromising its security footprint. Starting with Red Hat 9.4, the systemd package (252-31 and onwards) supports UKI addons, which aim to solve this issue.

In this blog, I explore the addons that enable safer extension of the UKI kernel command line.

What is the Unified Kernel Image (UKI)?

The linux kernel is the core of any Linux operating system. It's the interface between the hardware and the processes running on it, providing mediation between the two while making it all invisible to the user. It is also responsible for scheduling processes for allocation of CPU and memory between processes such that it guarantees isolation, security, and so on.

Initramfs is part of the Linux startup process. It contains all the software required to prepare an environment to load the root filesystem (the place containing all the user data). For example, when the root disk is encrypted, the user must enter a password to decrypt it. The program and modules to ask for the password and to decrypt the disk can't be in the rootfs (because it's encrypted), so they're instead in the initramfs..

A UKI is a UEFI binary containing the Linux kernel image, initramfs, kernel command line, and Red Hat signature. By shipping all these components together, we extend the SecureBoot process to also cover initramfs and the kernel command line.

A UKI is not and must not be limited to confidential virtual machines (CVM). In a traditional system, Red Hat ships the kernel, and initramfs has to be built on the target system. Shipping a pre-built initramfs provided by the Red Hat in an UKI has the following benefits:

• Faster build process: Target system does not need to build its own initramfs, which means no tool needs to be installed, so kernel installation is faster
• Better initramfs support: The pre-built initramfs contained in the shipped UKI is identical in all installations, making it easier for Red Hat to provide support for it

As a consequence, the benefits of UKI addons aren't restrained to CVMs.

While UKI effectively helps solve the issue of proving that no objects in the boot chain were altered by a malicious actor, the concept brings some limitations as the flexibility of the boot process is reduced. In particular:

• Initramfs must be created by the operating system vendor at the time when the UKI is built. This implies that the list of kernel modules included in the initramfs is fixed and constant, so it's not possible to add new ones at the end system
• Kernel command line is also made constant. For example, it becomes impossible to set the desired root volume by specifying the root= option on the kernel command line. This requires a different way to discover the root volume

This article aims to explain the current solution to the above limitations deployed upstream and offered by Red Hat Enterprise Linux 9.4: UKI Addons.

I'm focusing on kernel command line addons. Development for initramfs extensions isn't yet complete, so that's a topic for a future article.

I'm specifically focusing on the following boot scenario: 

Starting from UEFI firmware, we use the shim bootloader, which in turn calls systemd-stub to unpack and extract the relevant sections from the UKI, and then runs the kernel.

Extending the kernel command line

The Linux kernel offers a rich list of features and options that can be optionally enabled and tuned. The best way to define what the user or vendor wants to enable for the next boot is through the kernel command line.

The Linux kernel offers the possibility to inspect the current kernel command line by reading the file /proc/cmdline. This is a plain text file that contains the current command line being used by the kernel.

According to the Linux documentation, if a command is not recognized by the kernel and doesn’t contain a dot (.) or is provided after a dash (-), then the parameters are passed to init and ignored by the kernel.

Therefore, care must be taken to construct the kernel command line, because it's directly passed to the kernel, and potentially to the init process. An attacker can take advantage or insert custom kernel parameters in a UKI system to open backdoors and gain access.

The root kernel command line example

As an example, consider the root= kernel command-line option. As explained in kernel documentation, it's used to set up the root filesystem.

Assume that an attacker manages to change the root= option and plugs in an additional volume with a malicious OS image (loaded with malicious software, or simply setup with backdoors or a known root password). When booting, the UKI kernel receives the root= exploited option, resulting in it booting the malicious OS image. The Measured boot phase that checks the authenticity of the boot chain is tricked and fails to notice this change.

This is especially feasible if only the PCR7 is being used, because PCR7 records only the certificates used in the boot process.  A change in volume isn't detected by this measurement. For more information about PCRs and Measured boot, read this previous article by Vitaly Kuznetsov.

How the kernel cmdline works in a UKI

A UKI is simply a PE binary, composed of multiple sections. The main sections in an UKI are:

• .linux: Contains the Linux kernel
• .cmdline: Contains the kernel command line
• .initrd: Contains the initramfs

There are additional sections like .sbat, but for now I'll focus on the above list to keep things simple. I will discuss .sbat later. The full list of sections can be found in the UAPI group specifications.

The role of .cmdline in a UKI is simply to carry the plain text command line, and systemd-stub unpacks it and feeds it to the kernel. It's pretty clear that in the context of a CVM, the kernel command line must be carefully formulated to avoid unwanted parameters, and to prevent an attacker from exploiting those parameters to gain access to the UKI system.

Once a UKI is built and signed, there is no way to alter its content, and that includes the command line. Therefore, it's common to have a very minimal UKI .cmdline section, usually containing only console options. This provides maximum security at the expense of flexibility, because there is no way left to change the kernel command line.

But having a static UKI is basically useless. Think about the various parameters required for different instance sizes provided on the public cloud, the debug options that a kernel developer needs to add to debug an issue, and so on.

And that's why UKI addons are so important.

Who creates UKI addons

Addons can be created at different levels in the virtualization stack:

• Vendors: To provide optional features or debug tools for their UKI. The advantage here is that the key is already added to the SecureBoot database, so there are no additional steps once the addon is installed. On the other hand, an addon must be carefully formulated to avoid breakages or security issues
• UKI users: To add custom options, creating their own addons to tune the hardware and software available in the UKI system. In order to be trusted, a user must add a custom key to the SecureBoot database. This can be achieved using MOK (Machine Owner Key) and mokutil
• Virt host: If trusted, the virt host administrator can also add custom options. In a public cloud environment, this is not allowed because the cloud provider is not trusted, so the key is not added to the system SecureBoot database

There are existing solutions to extend the kernel command line, such as using SMBIOS parameters, EFI Shell, and so on. However, these parameters can easily be emulated by the hypervisor, which is not trusted.

For example, SMBIOS data is provided by the hypervisor, so it cannot be trusted.
In QEMU, it is very easy to feed SMBIOS table entries to the virtual machine:

qemu-kvm [...] -smbios 
type=11,value=io.systemd.stub.kernel-cmdline-extra=MY_CMDLINE

EFI shell is instead an app that runs within the guest context. However, it interferes with a user on an emulated console device.

As a result,  they have all been disabled by systemd-stub, and are automatically ignored. For example, see https://github.com/systemd/systemd/pull/28763  and https://github.com/systemd/systemd/issues/27604.

Safely extending a UKI kernel command line with UKI addons

Starting from upstream systemd v254 and RHEL systemd-252-31, systemd-stub is capable of reading UKI addons and using them to extend the .cmdline section of an UKI.

How does it do it? It leverages SecureBoot UEFI mechanism, which enables the caller to verify a PE section by verifying its signature against trusted keys stored in the SecureBoot DB of the system. If the signature is part of the trusted DB, then verification passes. Otherwise, it fails. In this way, we effectively extend the UKI kernel command line without losing anything in terms of security.

But what is a UKI addon? Simply put, it is a PE binary containing a very restricted number of sections. It can contain all the same sections of a UKI, except for .linux and .initrd (otherwise it would actually become a UKI). The most important section in an addon is the .cmdline section.

Systemd-stub is instructed to look for addons in specific locations of the EFI System Partition (ESP), where the used UKI and addons should reside. The ESP mount location is defined as $BOOT/EFI. The $BOOT definition is provided by the UAPI group specifications (in RHEL $BOOT is /boot/efi).

The criteria to identify an addon:

• The addon must be a PE binary
• The addon’s filename must end with .addon.efi
• The addon must be located in $BOOT/EFI/$UKI_NAME.extra.d/ if the addon has to be applied only when the UKI called $UKI_NAME is booted (all addons targeting a specific UKI are referred as local addons). Otherwise, the addon is located in $BOOT/loader/addons/ and is applied for any UKI booted (all addons targeting all UKIs installed are referred as global addons)

If all those conditions are satisfied, the addon is selected by systemd-stub and verified. If the signature key is valid, then the addon is then read, and the .cmdline section is appended to the UKI .cmdline content.

All addons provided in RHEL are signed with the same RHEL key used to sign the UKI. For more information about which key RHEL uses, see this previous blog post.

Using ukify to generate UKI addons

Ukify was officially added to the RHEL systemd package starting from v252-20. This tool is useful when it comes to creating a UKI because it's much more versatile than traditional tools like objcopy and dracut. In addition to creating a UKI addon, ukifycan also inspect them, enabling the user to see the contents of the text sections in a UKI and addon.

The ukifyapplication is capable of creating a signed UKI or UKI addon by using either pesign (already included in RHEL) or sbsign.  Additionally, it provides the opportunity to inspect the addon, which is useful because a PE binary is not plain text. Refer to ukify documentation to learn more about its features, and how to use it.

The tools specific to RHEL are kernel-bootcfg and kernel-addons, both part of the virt-firmware package. The former is capable of adding, removing, and updating a UKI, while the latter performs the same operations at the UKI addons level.

Secure Boot Advanced Targeting (SBAT)

Suppose that a vendor ships a new UKI addon, but later realizes that it contains an error, or that the command line is unsafe and prone to attacks. There are several possibilities for solving this problem:

1. Change the key used to sign the addons: For Red Hat, this is impractical and infeasible. Using the same key for UKI and addons would invalidate all UKIs and addons generated so far.
2. Add the addon hash to a blocklist: Again, not feasible for Red hat, and prone to error.
3. Modify the attestation process: Filter the hash matching the blocklisted addon, and reject it. This is unfeasible and prone to error.
4. Use SBAT rules: Identify the accepted generation number, and automatically reject all addons with a lower generation number. This is the only practical way to go, because it allows Red Hat to reject addons without the need to change the signing key, or hardcode anything related to the addon content.

A .sbat section is present in both a UKI and UKI addons. It's a text section and, as defined in the SBAT docs, each line is in the form component,generation,vendor,package,package version,url. The important part is the component,generation tuple.

sbat,1,SBAT
Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md
systemd,1,The systemd Developers,systemd,252,https://systemd.io/
systemd.rhel,1,Red Hat Enterprise Linux,systemd,252-22.el9,mailto:secalert@redhat.com
linux,1,Red Hat,linux,5.14.0-407.el9.x86_64,mailto:secalert@redhat.com
linux.rhel,1,Red Hat,linux,5.14.0-407.el9.x86_64,mailto:secalert@redhat.com
kernel-uki-virt.rhel,1,Red Hat,kernel-uki-virt,5.14.0-407.el9.x86_64,mailto:secalert@redhat.com

Note the various components:

• sbat: The SBAT mechanism itself. The version is increased when the mechanism itself has bugs or security issues
• systemd: Upstream systemd
• systemd.rhel: Downstream RHEL systemd
• linux: The upstream Linux kernel
• linux.rhel: The downstream RHEL kernel
• kernel-uki-virt.rhel: The RHEL UKI (there is no upstream equivalent)

It's easy to map. In the systemd.rhel component in the example above:

• Component = systemd.rhel
• Generation = 1
• Vendor = Red Hat Enterprise Linux
• Package = systemd
• Package version = 252-22.el9
• Url = mailto:secalert@redhat.com

In order for the mechanism to work, the same tuples have to be present in the shim, and are verified at startup against the UKI components, just after signature verification. If the shim generation for a matching component is higher, then the binary is discarded, even if it is being signed by a key present in the SecureBoot DB.

As a simple example, suppose that the shim has the following SBAT variables:

sbat,1,SBAT Version
my_addon,1

And an addon is discovered at boot with the following .sbat:

sbat,1,SBAT Version, ……
my_addon,1, ……

In this case, all (component, version) tuples match, so the addon is considered as long as it’s signed by a trusted key. Suppose that after a while, an issue with the addon is discovered, and so the shim version is increased to 2:

sbat,1,SBAT Version
my_addon,2

At this point, any other addon with my_addon version equal to 1 is automatically discarded, regardless of the signature being trusted or not.

How to ship RHEL UKI addons

RHEL currently ships UKIs in the kernel-uki-virt sub-RPM. In addition to that, a new sub-RPM kernel-uki-virt-addons is shipped containing the RHEL signed and supported addons. Upon installing, the addons are automatically placed into /usr/lib/modules/(uname -r)/ folder, the designated root filesystem location to store the not-installed UKIs and addons. The reason for storing addons and UKIs there is to avoid polluting the ESP with unnecessary addons and reducing the already limited free space.

Tools like kernel-addons helps the user install the chosen addons in the right ESP locations so that they are used at the next boot.

For example, to install the FIPS enabler addon (assume the kernel-uki-virt-addons.rpm is already installed):

# kernel-addons --install-addon /usr/lib/modules/(uname -r)/vmlinuz-virt.efi.extra.d/fips-enable-virt.fedora.x86_64.addon.efi --uki-title 'My_UKI'

In this example, /usr/lib/modules/(uname -r)/vmlinuz-virt.efi.extra.d/fips-enable-virt.fedora.x86_64.addon.efi is the addon path, and 'My_UKI' is the UKI title as defined when the UKI was created. The kernel-addons tool has many more options to install, delete, update, and list addons. The best way to see all flags is to install it and call kernel-addons --help

Summary

RHEL UKIs provide security benefits by embedding kernel, initramfs, and cmdline. UKI addons signed by RHEL can be used to preserve these benefits, and to provide the flexibility of a traditional approach where artifacts can be altered by the end user after installation. There's lots of tooling to make it easy to create, inspect, and install addons in the ESP, and even revoke dangerous addons (even ones signed by a trusted certificate). In short, RHEL UKIs are an essential part of SecureBoot and Confidential Computing.