Updating GPG keys for future Fedora and Red Hat Enterprise Linux releases

The SHA-1 hash function was originally published in 1995. Many researchers have published attacks since then, and the most recent estimate is that an attack cost 45,000 USD in 2020 with a predicted cost of 10,000 USD in 2025. The US National Institute of Standards and Technology NIST recommends that users switch to newer options as soon as possible and is planning to phase out SHA-1 completely by December 31, 2023.

GPG signature verification in RPM

Given this history, it is no surprise that Red Hat Enterprise Linux (RHEL) 9 deprecated SHA-1 for signatures in general and RPM package signatures specifically.

RPM packages use GPG keys for signing, and verification of GPG signatures is surprisingly complicated due to the structure of these keys. For example, the most common GPG key consists of a main RSA key used for certification and signatures and an RSA subkey for encryption. A self-signature packet created by the main key has metadata associated with it. A so-called "subkey binding signature" connects the subkey to the main key identity and includes similar metadata.

The key flags field of the self-signature will contain 0x3, indicating that the main key can be used for signatures. The same field in the subkey binding signature will be 0xC, which marks this key as usable for encryption of communication and storage only.

The following steps are therefore required to verify that an RPM package signature is correct:

• Verify that the signing key is in the keyring.
• Validate that the signing key's key flags field in its associated subkey binding signature or self-signature says that the key is a signature key.
• Confirm that an attacker has not modified the key flags field by verifying the subkey binding signature or self-signature against the main key.

RPM never implemented a full-featured OpenPGP parser, and there have been vulnerabilities in this code before: CVE-2021-3521, for example, is about omitting the last of these three steps.

When the gnupg2 maintainer tried to disable SHA-1 signatures in April in CentOS Stream 9, things quickly fell apart. Our servers were signing packages with SHA-256, but the signing GPG keys still contained subkey binding signatures that used SHA-1. Importing these keys into the keyring failed, which meant no updates for freshly installed systems. Additionally, pulling CentOS Stream or Universal Base Image containers with podman started failing. The maintainer reverted the change.

Better signature verification in RPM: rpm-sequoia

In 2021, the RPM maintainer Panu Matilainen wrote about the OpenPGP signature verification in RPM:

"It's not the most loved subsystem of rpm, exactly. Nobody ever stepped up to do the major rework (I would say redesign but that would imply a previous design…) it needs, so whatever "interface" there is, is pretty much all ad-hoc added for whatever the current need was."

Fortunately, Justus Winter did step up and contribute a better signature verification backend for RPM based on the Sequoia-PGP implementation written in Rust. A Fedora Change brought this backend into Fedora 38. During the beta phase, users noticed that many third-party RPMs are still signed by keys that use SHA-1 or even the obsolete DSA signature scheme. The Fedora Engineering Steering Committee decided to accept DSA and SHA-1 to sign RPM packages in Fedora 38 to give third-party vendors a grace period to update their keys to pass the stricter checks implemented in Sequoia.

Because RHEL 10 will fork from Fedora, its RPM package will likely also use the rpm-sequoia backend. Additionally, it will likely follow crypto-policies to decide whether to accept SHA-1. The DEFAULT crypto policy in RHEL 9 already denies signatures that use SHA-1. Now is a good time to verify that your RPM package signing keys do not use SHA-1 and pass the stricter verification checks in rpm-sequoia.

Testing your keys

The simplest method to test whether your public key needs a new signature is using the Sequoia-PGP command line tool sq in a Fedora container:

$> podman run --rm -it registry.fedoraproject.org/fedora:38 bash
$> dnf install -yq sequoia-sq
$> sq inspect yourkey.asc

Unlike sequoia-rpm, which still accepts SHA-1 signatures in Fedora 38, the sq command line tool marks SHA-1 invalid after February 2023. If your key needs updating, the output will contain a message similar to

Invalid: Policy rejected non-revocation signature (PositiveCertification)
        requiring second pre-image resistance
because: SHA1 is not considered secure since 2023-02-01T00:00:00Z

Re-signing your keys

To get rid of the SHA-1 self-signature or subkey binding signature that will fail to validate, you need to re-create these signatures. When done correctly, existing signed packages will continue to validate. Because of Sequoia's strict validation rules, this requires special attention to the timestamps in the GPG keys.

When creating signatures with GPG, the signer stores the current time in the signature packet. Sequoia always attempts to create a view of the key at signature time. A comment by one of the developers of Sequoia-PGP on GitHub explains the reason for this:

"Sequoia is trying to create a view of the certificate as it was when the signature was made at 2018-05-25T17:06:36Z, but that's not possible, because there is no valid binding signature as of 2018-05-25T17:06:36Z. The reason we don't accept a newer self signature is to avoid confusion-style attacks. A binding signature doesn't only contain expiration information, it includes other information that may change how a signature is interpreted (e.g., the key flags). As different interpretations can lead to different judgments, Sequoia conservatively rejects things that couldn't have been in effect at the time."

Sequoia checks whether the key usage flags did allow signatures when the package was signed. By default, new self-signatures will use the current timestamp and remove older signatures. After this operation, rpm-sequoia no longer knows what the key usage flags were before, and validation therefore fails. To avoid this, you can back-date the self-signature. How to do that depends on the tool used, though. The following sections offer instructions for GnuPG version 2.x and 1.x.

GnuPG 2.x

The simplest method to cause GnuPG to create new self-signatures is changing the expiry date of the key. To back-date the new signatures, first find the old signature creation times by using GnuPG's --with-colon output format and a bit of grep magic:

$> keyid=$(gpg --list-secret-keys --with-colons | \
        grep -E "^sec:" | \
        cut -d: -f5) # or specify manually
$> gpg \
    --list-keys \
    --with-colons \
    --fixed-list-mode "$keyid" | \
    grep -E '^[sp]ub:' | \
    cut -d: -f6

This process will print the key creation time as unix timestamps, one line per subkey. In most cases, these numbers will match, but if they do not, you need to edit each subkey separately or use the maximum of those values. GnuPG's --faked-system-time option allows setting a fixed timestamp to use as system time if the value ends with an exclamation mark. This is perfect to precisely control what timestamp to use, but a small piece of code will cause the signature operation to hang if you try to use the exact value:

    /* ... but we won't make a timestamp earlier than the existing
      one. */
    while(sig->timestamp<=orig_sig->timestamp)
      {
        gnupg_sleep (1);
        sig->timestamp=make_timestamp();
      }

This loop requires the new signature timestamp to be strictly larger than the earlier signature. To avoid triggering this path, add one second to the creation timestamp. This assumes that the key did not sign an RPM package in its first second of existence, which is probably safe.

Finally, to force the use of the newer SHA-256 digest algorithm, specify --cert-digest-algo SHA256.

With all required pieces in place, you can now edit the key to re-sign without SHA-1:

$> keyid=$(gpg --list-secret-keys --with-colons | \
        grep -E "^sec:" | \
        cut -d: -f5) # or specify manually
$> creation_time=$(gpg \
        --list-keys \
        --with-colons \
        --fixed-list-mode "$keyid" | \
        grep -E '^[sp]ub:' | \
        cut -d: -f6 | \
        sort -n | \
        tail -1)
$> gpg \
        --faked-system-time="$(( creation_time + 1 ))\!" \
        --cert-digest-algo SHA256 \
        --edit-key "$keyid"
gpg (GnuPG) 2.4.0; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: WARNING: running with faked system time: 2022-01-01 00:00:01
Secret key is available.

sec  rsa3072/344A807E6C877C1F
    created: 2022-01-01  expires: 2024-01-01  usage: SC
    trust: unknown       validity: unknown
ssb  rsa3072/BFD439D86A4DBE3C
    created: 2022-01-01  expires: 2024-01-01  usage: E
[ unknown] (1). Testy McTestface <test@example.com>

gpg> key 1

[…]

gpg> expire
Changing expiration time for a subkey.
Please specify how long the key should be valid.
        0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 4y
Key expires at Wed Dec 31 00:00:01 2025 UTC
Is this correct? (y/N) y

[…]

gpg> key 0

[…]

gpg> expire
Changing expiration time for the primary key.
Please specify how long the key should be valid.
        0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 4y
Key expires at Wed Dec 31 00:00:01 2025 UTC
Is this correct? (y/N) y

sec  rsa3072/344A807E6C877C1F
    created: 2022-01-01  expires: 2025-12-31  usage: SC
    trust: unknown       validity: unknown
ssb  rsa3072/BFD439D86A4DBE3C
    created: 2022-01-01  expires: 2025-12-31  usage: E
[ unknown] (1). Testy McTestface <test@example.com>

gpg> save

To verify whether the re-signed key does indeed not use SHA-1, pass it to sq inspect again and note the absence of any messages saying it is invalid:

$> gpg --export "$keyid" | sq inspect
-: OpenPGP Certificate.

    Fingerprint: 9F68A070F93C97E34606719E344A807E6C877C1F
Public-key algo: RSA
Public-key size: 3072 bits
  Creation time: 2022-01-01 00:00:00 UTC
Expiration time: 2025-12-31 00:00:01 UTC (creation time + P1460DT1S)
      Key flags: certification, signing

        Subkey: 57AB5C9B52D434543E1BF23EBFD439D86A4DBE3C
Public-key algo: RSA
Public-key size: 3072 bits
  Creation time: 2022-01-01 00:00:00 UTC
Expiration time: 2025-12-31 00:00:01 UTC (creation time + P1460DT1S)
      Key flags: transport encryption, data-at-rest encryption

        UserID: Testy McTestface <test@example.com>

GnuPG 1.x

Some older hardware security modules for OpenPGP keys use a patched GnuPG 1.x as the user interface. GnuPG 1 does not support the --faked-system-time option, so the approach outlined in the previous section does not work. Instead, you can use the libfaketime library to achieve the same result:

$> keyid=$(gpg --list-secret-keys --with-colons | \
        grep -E "^sec:" | \
        cut -d: -f5) # or specify manually
$> creation_time=$(gpg \
        --list-keys \
        --with-colons \
        --fixed-list-mode "$keyid" | \
        grep -E '^[sp]ub:' | \
        cut -d: -f6 | \
        sort -n | \
        tail -1)
$> faketime \
        -f "$(date --date="@$(( creation_time + 1 ))" '+%Y-%m-%d %H:%M:%S')" \
        gpg \
        --cert-digest-algo SHA256 \
        --edit-key "$keyid"
[…]

Publishing

As soon as you have the new public key, you can just replace your old public key. Systems that already have the old key in the RPM keyring will not re-import the SHA-1-free version automatically. Users that have issues can remove the old copy with rpm -e "gpg-pubkey-$keyid" as documented in the rpmkeys(8) manpage. The next install or update of a package signed with the key will prompt the user to trust the new version.

For more information about managing installed software with DNF, see the RHEL documentation. More information about the deprecation of SHA-1 is available in the customer portal and in an earlier post in this blog.