Red Hat Enterprise Linux and Secure Boot in the cloud

Secure Boot technology is part of Unified Extensible Firmware Interface (UEFI) specification. It is a useful and powerful tool which can be used to improve boot time security of an operating system by only allowing trusted code to be executed on that system. The technology is not new—it was part of UEFI specification since v2.0 (2006), and it is extensively used by x86 hardware vendors today. In the cloud world, however, the technology only became available fairly recently:

• Google made Shielded VMs generally available in April, 2019
• Microsoft announced Trusted Launch general availability in Nov, 2021
• AWS added support for SecureBoot in May, 2022
• Red Hat Enterprise Linux has supported Secure Boot technology since the RHEL 7 release in June, 2014

In this article, we will focus on how to enable and configure Secure Boot for various Red Hat Enterprise Linux (RHEL) image types in AWS, Google Cloud and Microsoft Azure.

Why use Secure Boot?

When enabled, Secure Boot technology improves boot time security of an operating system. In particular, it verifies that the bootloader and the kernel of the operating system are signed by a trusted vendor. Additionally, the RHEL kernel works in a more secure "lockdown" mode verifying that all kernel modules loaded later are also signed by a trusted vendor.

Secure Boot is also a crucial piece of Confidential Virtual Machines (CVM) enablement. In particular, CVM implementation must guarantee that only trusted bits are present in the boot chain. This is normally done by using a combination of Secure Boot and Trusted boot technology, we’ve discussed this in a previous article.

Understanding Secure Boot chain in RHEL

Secure Boot technology is based on public key cryptography and includes the following main parts:

• Platform Key (PK): This is a self-signed key which establishes trust between your platform and your platform’s firmware
• Key Exchange Key (KEK): These keys are used to establish trust between your operating system and your platform’s firmware. Specifically, these keys are used to update Allowed Signature database ('db') and Forbidden Signature database ('dbx')
• Allowed Signature database ('db'). This is a list of certificates or binary hashes which is used to check whether the particular binary is allowed to boot on the system or not. Additionally, all certificates from ‘db’ are imported into kernel “.platform” keyring, this allows, for example, for having signed third party kernel modules which can be loaded in kernel ‘lockdown’ mode
• Forbidden Signature database ('dbx'). This is a list of ‘bad’ certificates or binary hashes which are forbidden to boot on the system

When RHEL boots on a cloud instance in Secure Boot enabled mode, the following chain of events happens:

• When the VM is launched, it starts booting system firmware which is supplied by the cloud provider
• The firmware initializes UEFI variables from a variable store. When the VM boots for the first time, the store is initialized from the VM image’s default values
• When booted, the firmware tries loading the operating system boot loader. For RHEL VM in a x86 UEFI environment, the first stage bootloader is ‘shim’
• Shim binary in RHEL is signed by Microsoft’s “Microsoft Corporation UEFI CA 2011” so RHEL can boot with Secure Boot enabled on various hardware and virtualized platforms where Allowed Signature database ('db') is pre-populated with the default Microsoft certificates
• Shim extends the list of trusted certificates with Red Hat Secure Boot CA (can be found in ‘redhat-sb-certs’ package in Red Hat CodeReady Linux Builder repository)
• Optionally, shim can extend the list of trusted certificates with Machine Owner Key ('MOK'). This option is frequently used with bare metal platforms where updating Secure Boot variables requires cooperation with the OEM vendor

After that the boot path differs depending on which kernel layout is being used: ‘traditional’ or ‘Unified Kernel Image’ (UKI). In case of booting the traditional RHEL kernel:

• shim loads GRUB bootloader binary which is signed by the corresponding certificate (e.g. ‘Red Hat Secure Boot Signing 502’ on x86_64 architecture, can also be found in ‘redhat-sb-certs’ package). This certificate is signed by Red Hat Secure Boot CA and thus passes the check
• GRUB loads RHEL kernel signed by the corresponding certificate (e.g. ‘Red Hat Secure Boot Signing 501’ on x86_64 architecture, can also be found in ‘redhat-sb-certs’ package). This certificate is signed by Red Hat Secure Boot CA and thus passes the check

In case of RHEL UKI:

• Shim loads RHEL UKI (‘kernel-uki-virt’ package) directly. The UKI is signed by the corresponding certificate (e.g. ‘Red Hat Secure Boot Signing 504’ on x86_64 architecture, can also be found in ‘redhat-sb-certs’ package). This certificate is signed by Red Hat Secure Boot CA and thus passes the check
• Optionally, if there are UKI command line extensions present, their signatures are checked against the Allowed Signature database (‘db’) and Machine Owner Key ('MOK') before the extension is used. This allows for having UKI extensions signed by the operating system vendor (Red Hat) as well as UKI extensions signed by the end user

When the RHEL kernel boots in Secure Boot mode, it enters ‘lockdown’ mode which implies that all kernel modules have to be signed by a trusted key. All the modules shipped with RHEL kernel (‘kernel-modules-core’, ‘kernel-modules’, ‘kernel-modules-extra’) are signed with an ephemeral key which is created every time the kernel is built. The private part of the key is then discarded. This process guarantees that no other module can be signed by this key. When RHEL kernel boots, it extends kernel keyring with this ephemeral key as well as a few other keys used for signing third party modules. The certificates from Secure Boot Allowed Signature database (‘db’) and Machine Owner Key ('MOK') are also trusted and can be used for signing kernel modules.

AWS

Currently, AWS supports Secure Boot for Nitro based instances. It is possible to use the feature both for Marketplace Amazon Machine Images (AMIs) and for custom RHEL AMIs.

Marketplace RHEL images

With 9.3 and 8.9 releases, RHEL Marketplace AMIs boot settings were updated to ‘UEFI Preferred’ making it possible to enable Secure Boot and Nitro vTPM features. Here is what RHEL 9.4 AMI in ‘us-east-2’ looks like:

$ aws ec2 describe-images --image-id ami-099f85fc24d27c2a7 --region us-east-2 | grep -E '"ImageId"|"Name"|"BootMode"'
 
 "ImageId": "ami-099f85fc24d27c2a7",
 "Name": "RHEL-9.4.0_HVM-20240423-x86_64-62-Hourly2-GP3",
 "BootMode": "uefi-preferred",

When an instance is created using RHEL Marketplace AMI, it boots into setup mode which allows for updating Secure Boot UEFI variables from within the instance:

# mokutil --sb-state
SecureBoot disabled
Platform is in Setup Mode

Let’s see how we can enable Secure Boot and add a custom signing certificate into ‘db’. As a general rule, it's better to not use the target instance for generating and keeping private keys. When Secure Boot secrets are kept on the same instance where they’re used, an intruder which gained access to the secrets will be able to escalate his privileges and defeat the sole purpose of the technology. First, we need to generate PK and KEK keys. These keys will be needed to update Secure Boot ‘db’ and ‘dbx’ variables in the future. We will be using ‘openssl’ and ‘python3-virt-firmware’ tools.

$ uuidgen --random > GUID.txt
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout PK.key -new -x509 -sha256 -days 3650 -subj "/CN=Platform key/" -outform DER -out PK.cer
$ virt-fw-sigdb --add-cert "$(< GUID.txt)" PK.cer -o PK.esl
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout KEK.key -new -x509 -sha256 -days 3650 -subj "/CN=Key Exchange Key/" -outform DER -out KEK.cer
$ virt-fw-sigdb --add-cert "$(< GUID.txt)" KEK.cer -o KEK.esl

Next, we need to create a list of allowed certificates (‘db’). Let’s generate a custom certificate and also add Microsoft’s “Microsoft Corporation UEFI CA 2011” certificate as it is used to sign ‘shim’ binary in RHEL.

$ openssl req -quiet -newkey rsa:4096 -nodes -keyout custom_db.key -new -x509 -sha256 -days 3650 -subj "/CN=Signature Database key/" --outform DER -out custom_db.cer
$ virt-fw-sigdb --add-cert "$(< GUID.txt)" custom_db.cer -o custom_db.esl
$ virt-fw-sigdb --add-cert 77fa9abd-0359-4d32-bd60-28f4e78f784b MicCorUEFCA2011_2011-06-27.crt -o ms_db.esl
$ cat custom_db.esl ms_db.esl > db.esl

The list of currently forbidden signatures ('dbx') can be downloaded from UEFI Forum.

Now, we can transfer PK.esl, KEK.esl, db.esl and x64_DBXUpdate.bin to the target instance and write the variables using ‘efivar’ package:

# efivar -w -n 8be4df61-93ca-11d2-aa0d-00e098032b8c-PK -f PK.esl
# efivar -w -n 8be4df61-93ca-11d2-aa0d-00e098032b8c-KEK -f KEK.esl
# efivar -w -n d719b2cb-3d3a-4596-a3bc-dad00e67656f-db -f db.esl
# efivar -w -n d719b2cb-3d3a-4596-a3bc-dad00e67656f-dbx -f x64_DBXUpdate.bin 

Now, we can reboot the instance and check that it has Secure Boot enabled:

# mokutil --sb-state
SecureBoot enabled

We can also check that our custom certificate got to the kernel keyring and can be used with ‘keyctl’ tool from ‘keyutils’ package:

# keyctl list %:.platform
4 keys in keyring:
136539426: ---lswrv     0     0 asymmetric: Signature Database key: f064979641c24e1b935e402bdbc3d5c4672a1acc
231557747: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 8: e1c6c580aa1e21d585aad9bf20f3929e5ec1f08b
939747660: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f
1040805986: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4

Custom RHEL images

When a custom RHEL AMI is registered, it is possible to have pre-filled UEFI Secure Boot variables. The advantage of this approach is that all instances launched from the AMI will have Secure Boot enabled with the desired variables even on the first boot. Similar to the previous example, let’s generate new PK/KEK keys:

$ uuidgen --random > GUID.txt
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout PK.key -new -x509 -sha256 -days 3650 -subj "/CN=Platform key/" -outform DER -out PK.cer
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout KEK.key -new -x509 -sha256 -days 3650 -subj "/CN=Key Exchange Key/" -outform DER -out KEK.cer

Let’s use Microsoft’s ‘Microsoft Corporation UEFI CA 2011’ certificate and a custom certificate to populate the Allowed Signature database ('db'):

$ openssl req -quiet -newkey rsa:4096 -nodes -keyout custom_db.key -new -x509 -sha256 -days 3650 -subj "/CN=Signature Database key/" --outform DER -out custom_db.cer

The list of currently forbidden signatures ('dbx') can be downloaded from UEFI Forum. Using ‘virt-fw-vars’ tool from ‘python3-virt-firmware’ package, it is possible to generate the required blob:

$ virt-fw-vars --output-aws aws_blob.bin --set-pk "$(< GUID.txt)" PK.cer --add-kek "$(< GUID.txt)" KEK.cer --add-db "$(< GUID.txt)" custom_db.cer --add-db 77fa9abd-0359-4d32-bd60-28f4e78f784b MicCorUEFCA2011_2011-06-27.crt --set-dbx x64_DBXUpdate.bin

Now, it is possible to register the AMI with the desired Secure Boot variable. Using ‘awscli2’ package, the AMI can be created from a disk snapshot:

$ aws ec2 register-image --name rhel9-test-ami --architecture x86_64 --virtualization-type hvm --root-device-name "/dev/sda1" --block-device-mappings "{\"DeviceName\": \"/dev/sda1\",\"Ebs\": {\"SnapshotId\": \"snap-fffffffffffffffff\"}}" --ena-support --boot-mode uefi --region eu-central-1 --uefi-data $(cat aws_blob.bin)

When an instance is created using this AMI, it boots with Secure Boot enabled and the expected Secure Boot certificates propagated to the kernel keyring:

# mokutil --sb-state
SecureBoot enabled
# keyctl list %:.platform
4 keys in keyring:
907254483: ---lswrv     0     0 asymmetric: Signature Database key: f064979641c24e1b935e402bdbc3d5c4672a1acc
203975646: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 8: e1c6c580aa1e21d585aad9bf20f3929e5ec1f08b
890038886: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f
990786053: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4

Google Cloud

Google Cloud supports Secure Boot with Shielded VMs. When creating an instance using the UI, ‘Secure Boot’ setting can be found under ‘Security’:

Publicly available RHEL images can be booted in Secure Boot enabled state. By default, this will populate the Allowed Signature database ('db') with Microsoft certificates:

# mokutil --sb-state
SecureBoot enabled
# keyctl list %:.platform
4 keys in keyring:
312311590: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4
700176698: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 8: e1c6c580aa1e21d585aad9bf20f3929e5ec1f08b
 65600894: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f
509209133: ---lswrv     0     0 asymmetric: Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53

It is also possible to specify custom SecureBoot variables when registering a new image. Similar to the AWS example, let’s generate new PK/KEK keys:

$ uuidgen --random > GUID.txt
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout PK.key -new -x509 -sha256 -days 3650 -subj "/CN=Platform key/" -outform DER -out PK.cer
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout KEK.key -new -x509 -sha256 -days 3650 -subj "/CN=Key Exchange Key/" -outform DER -out KEK.cer

Let’s use Microsoft’s ‘Microsoft Corporation UEFI CA 2011’ certificate and a custom certificate to populate the Allowed Signature database ('db'):

$ openssl req -quiet -newkey rsa:4096 -nodes -keyout custom_db.key -new -x509 -sha256 -days 3650 -subj "/CN=Signature Database key/" --outform DER -out custom_db.cer

The list of currently forbidden signatures ('dbx') can be downloaded from UEFI Forum.

Now, let’s register the new image by using ‘google-cloud-cli tool’. In this example, publicly available RHEL image is used as a template:

$ gcloud compute images create rhel94-custom-efi-image --source-image projects/rhel-cloud/global/images/rhel-9-v20240611 --platform-key-file=PK.cer --key-exchange-key-file=KEK.cer --signature-database-file=custom_db.cer,MicCorUEFCA2011_2011-06-27.crt --forbidden-database-file x64_DBXUpdate.bin --guest-os-features="UEFI_COMPATIBLE"

When a new instance is created from the newly registered image, it boots in Secure Boot enabled state and has the expected contents of the Allowed Signature database ('db'):

# mokutil --sb-state
SecureBoot enabled
# keyctl list %:.platform
4 keys in keyring:
757453569: ---lswrv     0     0 asymmetric: Signature Database key: f064979641c24e1b935e402bdbc3d5c4672a1acc
250442200: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 8: e1c6c580aa1e21d585aad9bf20f3929e5ec1f08b
 47707545: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f
489298485: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4

Microsoft Azure

Microsoft Azure supports Secure Boot technology with Trusted Launch VMs. In the UI, the setting can be found under ‘Configure security settings’:

Publicly available RHEL images can be booted in Secure Boot enabled state. By default, this will populate the Allowed Signature database ('db') with Microsoft certificates:

# mokutil --sb-state
SecureBoot enabled
# keyctl list %:.platform
5 keys in keyring:
447237609: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4
231070374: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 8: e1c6c580aa1e21d585aad9bf20f3929e5ec1f08b
121287780: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f
519351525: ---lswrv     0     0 asymmetric: Microsoft Corporation: Azure Services Linux Kmod PCA: 697cc4f41a2209a7abd0604c1f196f4d8c5735d9
356360234: ---lswrv     0     0 asymmetric: Microsoft UEFI CA 2023: 81aa6b3244c935bce0d6628af39827421e32497d

Microsoft Azure allows for adding custom certificates into UEFI Secure Boot variables when a new image version is registered in Azure Compute Gallery. To achieve this, let’s create a custom certificate with openssl and convert it into base64:

$ openssl req -quiet -newkey rsa:4096 -nodes -keyout custom_db.key -new -x509 -sha256 -days 3650 -subj "/CN=Signature Database key/" --outform DER -out custom_db.cer
$ echo `base64 -w0 custom_db.cer`
MIIFIjCCAwqgAwIBAgITNf23J4k0d8c0NR…..

Now, let’s create an Azure Resource Manager (ARM) JSON for registering a new Azure Compute Gallery version:

$ cat aztmpl.json

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "resources": [
        {
            "type": "Microsoft.Compute/galleries/images/versions",
            "apiVersion": "2023-07-03",
            "name": "your compute gallery/your image definition/version",
            "location": "location of the VHD",
            "properties": {
                "storageProfile": {
                    "osDiskImage": {
                        "source": {
                            "id": "your-storage-account-id",
                            "uri": "url-with-the-vhd"
                        },
                        "hostCaching": "ReadOnly"
                    }
                },
                "securityProfile": {
                    "uefiSettings": {
                        "signatureTemplateNames": [
                            "MicrosoftUefiCertificateAuthorityTemplate"
                        ],
                        "additionalSignatures": {
                            "db": [
                                {
                                    "type": "x509",
                                    "value": [
                                        "base64 of custom_db.cer"
                                    ]
                                }
                            ]
                        }
                    }
                }
            }
        }
    ]
}

The image version can then be registered using ‘azure-cli’ package:

$ az deployment group create --name custom-deployment-test1 --resource-group 
your-resource-group-name --template-file aztmpl.json

When a Trusted Launch or a Confidential VM instance is created using the newly registered image version, we can see that our custom ‘Signature Database key’ appears in the kernel keyring:

# mokutil --sb-state
SecureBoot enabled

# keyctl list %:.platform
6 keys in keyring:
997130443: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4
695633967: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 8: e1c6c580aa1e21d585aad9bf20f3929e5ec1f08b
239568845: ---lswrv     0     0 asymmetric: Red Hat Secure Boot CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f
586621657: ---lswrv     0     0 asymmetric: Signature Database key: f064979641c24e1b935e402bdbc3d5c4672a1acc
817181752: ---lswrv     0     0 asymmetric: Microsoft Corporation: Azure Services Linux Kmod PCA: 697cc4f41a2209a7abd0604c1f196f4d8c5735d9
140053514: ---lswrv     0     0 asymmetric: Microsoft UEFI CA 2023: 81aa6b3244c935bce0d6628af39827421e32497d

Note, instead of ‘MicrosoftUefiCertificateAuthorityTemplate’, it is possible to specify ‘NoSignatureTemplate’ and start with a clean slate with custom PK/KEK keys.

Wrap up

Secure Boot is a powerful tool for strengthening RHEL security which can be used on bare hardware, in virtual machines and on public cloud instances. Various cloud platforms support enabling and customizing Secure Boot settings for your RHEL instances. In particular, it is possible to add custom certificates to the Allowed Signature database ('db') making it possible to sign third party kernel modules and other objects like UKI command line extensions. This makes Secure Boot enabled instances not only more secure, but also very configurable.