RHEL: Behind the scenes of a simple backport

Bug fixes and security updates are essential components of the value of a Red Hat Enterprise Linux (RHEL) subscription. These ensure that you, as a RHEL subscriber, have access to the latest essential updates so your systems remain protected and refreshed. And it's all possible thanks to Red Hat's practice of backporting patches.

A "backport" is when a critical fix from a new version of software is applied to an old but supported version. Not all operating systems or distributions can manage that workload, but stability and security is important to Red Hat. When Red Hat backports a patch, you benefit from important security updates and bug fixes without the need of upgrading to, and testing, a new version of the software.

But what does "backport" really mean? How is it done? In this blog post, I'm going to give you a grand tour of the process from start to finish.

Software engineering a backport

One of my responsibilities as an associate software engineer has been to backport patches in RHEL for the binutils package. This process is, not coincidentally, similar to contributing to CentOS Stream (where open source community members develop, test, and contribute to a continuously delivered upstream for Red Hat Enterprise Linux). In practice, every update I've done has been slightly different, so there's no single complete step-by-step guide to fix all bugs. However, there are some common tasks and issues that come up, so this article describes a suitable all-purpose starting point.

A backport can be used to bring a fix for a flaw to an old version of a software. I've taken the fix for a bug in an upstream package, and applied it to an old version of that package distributed by Red Hat. That works well when a fix is available, but sometimes a bug is still present upstream. In those cases, I have to develop a fix and get it merged upstream first, so I can backport it to a Red Hat package.

The potential problem is that a newer version of software might have changes that the old version lacks. A backport isn't always just a matter of copying over a patch. A patch for code in one state might not fit with an old version of that code, or it might appear to work but then introduce a different issue. In that case, the first step is a test build.

To summarize, backporting a patch is an important part of maintaining functionality and security of old software. Developers can ensure that older systems continue to receive updates and bug fixes to meet client demand, while keeping that software fully operational.

How a backport is made

There are a number of things to keep in mind when doing a backport, and not all of them are based on source code.

• There are different internal targets to be met. This includes the time schedule, priority orders, the handoff to the next team, and more.
• A backport must be complete and confirmed functional locally before anything gets pushed.

Suppose I've found a bug in some software. I check the project's issue tracker and discover that the bug already has a fix in another version of the software. To get this fix I would need to backport it to all the affected versions of the software that are supported or where it's needed. In other words, if the bug is targeting version A, B, and C but the fix has only been applied to A, then I must backport the fix to B and C.

Here's how that's done.

1. Build a test environment

First, I have to build an environment where I can reproduce the bug for the targeted versions. This can be done by using a virtual machine, a container, or a physical machine with the correct version installed. I use a server belonging to Red Hat to ensure that I can match the specifications of a bug as closely as possible without having to change my own personal machine.

2. Reproduce the bug

Next, I try to reproduce the bug. Bugs aren’t always reproducible. They might be related to other software, or they might only show up sporadically due to factors beyond the developer’s control.

If the bug is reproducible: 

1. Test all supported versions of the software to identify which versions of the software can manifest the bug. This allows precise targeting of affected environments.
2. Understand the specific conditions that trigger the bug. This determines whether the bug is caused by the kernel, architecture, or library. This also reveals whether the bug is erratic with no predictable pattern, or deterministic. This allows me to use a systematic approach when devising for a solution.

3. Fix the bug

Once the bug has been rigorously examined, I must identify whether there's a fix for it yet. 

If there is a fix, which is typically the case during a backport process, I must determine whether it's possible to apply the fix to other versions of the software.

If it is, then the backporting can begin.

If not, then the patch might not be applicable. It may be necessary to make adjustments, modifications, or updates so that the fix bocemes compatible.

If a fix doesn't exist yet, then the next step is to create one. Of course, this requires programming and testing. It's not really part of the backporting process, though. Once a bug fix has been developed, then the backporting would begin back up at the start of the process.

4. Apply the patch

Finally, I apply the patch and test that it's working on each targeted version.

CentOS workflow

Once a suitable fix for a bug has been identified, I use CentOS Stream for the porting process. CentOS is upstream to RHEL, so establishing the fix in CentOS ensures that it's integrated into the RHEL dev cycle.

Of course, to minimize the risk of introducing new issues, it's crucial to verify that the backport is successfully applied and functioning locally before pushing any changes.

Here's how to create a backport using CentOS Stream.

1. Clone the repo

First, use the centpkg command to clone the source repository locally so you can test and work on the code.

When centpkg clones a repository, it performs two actions. It obtains a local copy of the code, and it  adds a new remote connection to a web-based version control platform used to store packages. At the time of writing, CentOS uses GitLab. Replace text between angle brackets (<>) with the actual URL and repository name: 

$ centpkg clone <URL of repo> && cd <repo>

2. Create a fork

CentOS uses the GitLab merge request workflow, which requires a fork.

$ centpkg fork

To test whether the fork was successful, use the command git remote to show aliases of the configured remotes. In a clean slate, the output is something like this:

$ git remote
user
Origin

A merge request workflow submits code for reviews and comments using the Merge Request function in GitLab. When a change is accepted, it's merged into the main repo.

3 Update the repo

When working remotely, it's vital to stay current with all potential updates. If your copy of the repo falls behind, there are likely to be merge conflicts when you try to get the code into the main repo.

To ensure your remote is up to date, run the command:

$ git fetch --all

The --all option checks all of the remotes for updates. 

4. Keep track of your branches

Now that the remote has been updated, make sure the work is being done in the right branch and that all the branches from all remotes are visible:

$ git branch --all

It can be easy to accidentally get on the wrong branch. Use the command git switch <branch> or centpkg switch <branch>.

Because a merge request workflow is used, it's important to work on a personal branch when making changes to source code. Do this using the following command:

$ git checkout -b <personal_branch> <remote_branch>

This does two things:

1. Creates <personal_branch> in your Git repository
2. Ensures that <personal_branch> is based on the <remote_branch>

5. Apply the patch

After creating your <personal_branch>, apply the patch. This is done by including the new patch, editing the RPM .spec file to include the patch, the version and release update, and a changelog entry to explain what you've done.

6. Build the package

To ensure that changes have been applied correctly, it's important to make a test build after applying the patch. You can do this with mock-build, local-build, or a scratch-build. For the purpose of  this example, I use scratch-build to target all supported architectures. Doing a scratch build requires special permissions, so I suggest that you perform a local build (use local-build instead of scratch-build in my example commands) or a mock build (use mock-build in the commands).

Generate a source RPM (SRPM) and to execute a scratch build:

$ centpkg srpm

$ centpkg scratch-build --srpm <srpm> --target 

The flag --target is used when trying to update a specific version of a software. It ensures that the target branch is the one being affected by the bug, not the branch you're currently working on (in this case, your <personal_branch>).

If your intent is to verify that a patch has been applied to the specific architecture that's being worked on, performing a scratch-build is unnecessary. Instead, use centpkg prep to verify the prep stage in the .spec file. This is a quick way to confirm that the patch is being applied without having to execute a scratch build. For example:

$ centpkg prep

7. Test the package

After a successful test build, copy the package to your test environment where you've reproduced the bug. Update the installed package with your patched one, and then verify that the error message revealing the bug is no longer triggered. 

Assuming that's the case, you can continue with the CentOS Stream workflow.

Before doing that, though, perform a mass rebuild on all of the dependencies for your newly updated package, or at the very least notify the maintainers of them, mainly to avoid introducing regressions due to your backport.

8. Commit

Create a commit with an explanation of your solution, and push it to your fork.

Once you've pushed the commit, open a merge request using the link returned by the push command. Give an explanation of why this is a valid merge request, and its intent. This gives the  maintainer information to help determine whether the merge request should be accepted.

Once approved and merged, the change is automatically synchronized  to the RHEL repository, meaning it isn't necessary to repeat the whole process to get the fix in RHEL.

Backports for stability

Backporting a patch plays a vital role in maintaining the functionality and security footprint of the software you rely on. By leveraging the advantages of a RHEL subscription, you gain access to timely updates and fixes. In addition to maintaining software functionality and enhancing system security, backporting patches for RHEL provides you with greater system stability, reduced downtime, and improved overall performance. It makes it possible for older systems to receive updates and bug fixes to meet client demands, and to keep mission-critical software operational.