Cryptographic signatures for zip distributions

Red Hat's products are distributed through numerous methods, including RPMs, ISOs and zip files. Over the past several months, we have been working across the organization to design and implement a plan to provide signatures for all zip file types so that our customers have greater assurance that Red Hat actually creates the products they receive. This work is essential to our customers' trust in Red Hat and our products.

Released on October 4th, 2022, Red Hat Single Sign-On patch 7.5.3 is the first Red Hat zip distribution to include cryptographic signatures. More product releases will come with this vital security metadata in the coming months.

Background

Our products are signed with a cryptographic signature to provide evidence that the software has not been tampered with and to verify Red Hat as the software publisher. This provides a higher level of security and assurance for users of our products.

What are cryptographic signatures?

Cryptographic signatures are essential for verifying the authenticity and integrity of digital data. They work by verifying both the content creator’s identity and the content’s integrity. Signatures are created using a public and a private key—the public key verifies the signature, while the private key generates it.

SHA256 hashes

The Secure Hash Algorithm (SHA) is a cryptographic function to generate a number that identifies a file. It is known as a one-way function since it is easy to compute the hash given the input but very hard to reverse. This characteristic is often known as preimage resistance. SHA also provides "second preimage resistance," meaning that finding another input that would produce the same hash is difficult, even if you know the input. This assures that the file could not have been swapped out in transit or while at rest on our content distribution network (CDN) without a change being detected by the hash.

These characteristics assure our customers that the hash will be different if the file has been tampered with. We have chosen to use the 256-bit variant of the SHA algorithm, which is currently considered secure, and replaces SHA-1, which only provides a 160-bit digest and is known to be vulnerable to preimage attacks.

RSA signatures

Red Hat uses RSA signatures (known by the names of its creators, Ron Rivest, Adi Shamir, and Leonard Adleman) to sign lists of SHA256 hashes. RSA is a public-key encryption algorithm considered one of the first successful public-key cryptosystems. It is also considered highly secure when using modern key sizes. The algorithm is based on the fact that it is easy to multiply two large prime numbers together but very difficult to factorize the product. This makes RSA suitable for encryption and digital signatures.

We use a high-strength 4096-bit private key (release key 2), and our public keys are available on our website and the Massachusetts Institute of Technology (MIT) public key server.

How are we applying these technologies?

We are first using the security properties of both the SHA256 hash function and the RSA public key cryptosystem to hash all of the files produced for a release and then signing those hashes to confirm that Red Hat produced them. This provides a higher level of security for our releases and helps our customers to detect malicious or accidental corruption.

This process is integrated into our build and release process in two discrete steps: First, hashing the files as part of the build process, and second, packaging and signing the hashes as part of our QE and release verification procedures.

Hashing the zip file(s)

Our build system, Brew (powered by koji), produces our RPM and zip distributions and automatically hashes the archives it makes. These hashes are not created by hand; however, they are used to validate that the files have not changed before they are uploaded to our CDN and made available to customers. We have taken advantage of this aspect of our build process and extended it by combining all the hashes for a particular release and packaging them into a "SHA256SUM" file.

This file is in a standard format that lists the hash and the corresponding filename of the particular artifact. It is commonly used across the industry to provide integrity to binary files. However, it is not limited to that. The `sha256sum` command on RHEL, other Linux distributions and macOS natively supports this file format.

Signing the hashes

Once our software production team has completed their verification procedures, they sign off on the release from both a process and technical perspective. The "SHA256SUM" file they created in the previous step is signed by our latest release key, which produces a *.asc file. This file is an ASCII-Armor formatted detached signature file that proves the integrity and provenance of the "SHA256SUM" file (and, transitively, the zip artifacts themselves). The "gpg" command on RHEL, other Linux distributions and macOS supports the file format natively.

Due to the potential damage that a lost or stolen private key could cause, we have taken additional steps to add assurance to the signatures we produce. The primary technology behind this is our signing server.

Signing server

The signing server is a piece of infrastructure that provides secure access to our private keys to protect them from unauthorized access and use. We use a Hardware Security Module (HSM) to physically prevent the private keys from being exported (reducing the risk of theft). We then restrict the individuals and machines allowed to submit jobs to the signing server using Kerberos authentication and access control lists (ACLs). Our supply chain team carefully maintains and regularly reviews these ACLs to restrict the keys to authorized individuals on certain machines.

Beyond these controls, we also use the signing server to provide auditability and monitoring. Each submission to the server is logged for a minimum of one year within the system log and longer within Splunk. We collect the Kerberos username of the individual and the originating IP address to provide quick and effective incident response and forensics if a breach occurs.

How does this provide provenance and verification?

Verification

The SHA256SUM file's signature can be used to verify its integrity. The signature will no longer match if the file has been tampered with. Additionally, the hashes included in the SHA256SUM file can be used to verify the integrity of the downloaded files. These transitive cryptographic guarantees prove the integrity of our releases. Please see our knowledge base article for more details.

Provenance

The signature on the zip file can be used to verify the content creator’s identity. Since only specific and authorized individuals have access to use the keys (but not the ability to view or export them), the signature is a way to verify that Red Hat produced the content.

Summary

Red Hat now provides a new file for each zip release called "<product> Provenance and Verification." This file contains a SHA256SUM, SHA256SUM.asc, and README.md file, allowing our customers to validate that Red Hat created the release they have acquired. For more information, read Red Hat Release Provenance and Verification.