Kernel Live Patch Support Cadence Update

The Red Hat Kernel Live Patching Program has been in place for five years, at the time of writing.  Live patching has proven to be a more reliable way to help improve the security posture of running kernels, while at the same time minimizing application downtime. From the start, however, we've been trying to bridge a difficult gap: Support the running kernel for as long as possible while also keeping up with the many, many concurrent kernels customers are running.

We’re rolling out what we believe is an elegant solution that meets customer requirements. Going forward, live patching will become what amounts to its own patch stream, allowing customers to get live patches for a critical and important CVE for any supported kernel for a full year after the kernel is updated. This allows you to reboot on your own schedule. The tradeoff is that there are no longer live patches for every errata kernel, but only selected kernels (approximately quarterly) in the various errata streams.

This process is evolving based on feedback from key customers. For example, we're working to find the ideal balance between how many kernels we support and whether we need to support specific errata kernels for longer than one year (one year is the industry standard "no reboot" window.)

An example scenario: If you upgrade to RHEL 9.6 when it arrives, then you can get live patches for the initial 9.6 release kernel for up to a year (as compared to the previous 6 months). Red Hat believes it’s unwise to go longer than a year without rebooting and upgrading your kernel, but we want to move the decision of when to reboot from Red Hat to the customer.

Existing live patching release model

Before this change, Red Hat had been providing live patches for every errata kernel. Our kernel team released an errata kernel every six weeks for every actively supported minor and E4S version (currently 7.9, 8.2, 8.4, 8.6, 8.8, 8.10, 9.2, 9.4, and 9.3). That's almost 100 active kernels in Q2CY24, with a six-month support window for each kernel. It's a ton of kernels to evaluate for each patch.

In the existing model, no matter which errata kernel you were on, you could get live patches for about 6 months. After 6 months, you'd need to upgrade to a new errata kernel on your current minor version, or to a new minor version, to keep getting live patches.

Confusing, right? And on top of that, this cadence doesn't meet some customer SLAs (some vendors, for example, let you patch the same kernel for up to a year).

Example of the existing cadence

Under the current model, if you upgraded to RHEL 9.6 when it arrived, then you could get live patches for that kernel for six months. After six months, you'd need to either install the latest 9.6 errata kernel again to enable another 6 months of live patches, or upgrade to a later minor version (9.7 or 10.1). You'd have to do one or the other, and then reboot, every six months to stay covered.

A better idea, born of necessity

Recently, our kernel team decided to release errata kernels as often as weekly to better meet compliance requirements. This made the current "every errata" live patching model unsustainable, and forced us to rethink the problem.

Going forward, instead of trying to provide live patches for every kernel, we're asking customers using live patches to adopt a new patching pattern. Red Hat's live patching will patch fewer kernels, but provide live patches for those kernels for a longer stretch. This is a win-win because it removes any need to force a customer to reboot on a Red Hat schedule, and it doesn't require Red Hat engineering to meet a possibly unachievable demand.

Customers will be able to figure out which kernels have live patches by monitoring the errata stream.  We'll also provide an updated base kernel for each active errata stream approximately every three months.

Example of the new cadence

If you install the upcoming RHEL 9.6 on a machine you want to use with live patching, you'll install the newest available kernel that has live patching support. You'll then install the newest cumulative live patch set (if there is one). From that point forward, you'll get live patches on that kernel for a full year, with the ability to refresh the 9.6 kernel as often as every three months.

Customers get longer reboot windows AND fresh kernel quarterly

Red Hat still recommends you update your on-disk kernel as often as practical because:

• Live patches don't cover every available CVE and bug fix, so it's wise to update the underlying kernel periodically
• Customers probably need to reboot periodically for other reasons, which is a good opportunity to perform a kernel update
• If your organization has SLAs for CVEs not covered by live patching, you can meet those SLAs by refreshing the base kernel

That said, for as long as it works for your organization, you'll have strong coverage for critical and important CVEs without having to reboot as often just to get CVE-related patches.

For information on which kernels have kpatch support please see the Kernel Live Patch life cycles page.