Jumpstart GitOps with image mode

A year ago, I was introduced to image mode for Red Hat Enterprise Linux (RHEL). That introduction brought me back together with some folks I’d worked with in Project Atomic, and it proved that you could orchestrate the complete build and automation of an operating system using application pipelines. Finally, sysadmins can take advantage of the same build tools developers have.

Image mode for RHEL enables you to use container tools to assemble operating system artifacts. GitHub happens to have all the required tooling: source control, a built-in container registry and pipeline tools in the form of GitHub Actions.

By default, GitHub's infrastructure has no awareness of RHEL subscriptions. However, GitHub supports two kinds of execution hosts for their workflows: GitHub-hosted and self-hosted. Using a RHEL 9 host as a self-hosted runner solves the lack of subscription awareness.

A Containerfile for GitHub Actions

GitHub Actions are fairly straightforward: One or more jobs made up of a series of steps. One of the key benefits of the platform, especially for new users, is the diverse ecosystem of prebuilt actions available for use. You can take a manual process you already know, document it and pull in the appropriate actions that abstract the specific tasks needed to accomplish each step.

First, you need a GitHub repository you control. This process involves pushing code, and there are also settings that need to be adjusted, so you must have ownership of the repo. To build an image mode host, I wrote a Containerfile that creates a simple web server with the document root configured to a location that bootc manages:

FROM registry.redhat.io/rhel9/rhel-bootc
RUN echo "%wheel     ALL=(ALL)    NOPASSWD: ALL" > /etc/sudoers.d/wheel-sudo
RUN dnf -y install httpd && \
systemctl enable httpd && \
mv /var/www /usr/share/www && \
sed -ie 's,/var/www,/usr/share/www,' /etc/httpd/conf/httpd.conf
RUN echo "Welcome to a GHA built bootc-http instance!" > /usr/share/www/html/index.html

Any changes we want to make to our website to test the pipeline are controlled through the Containerfile. If you haven’t built an image mode for RHEL host previously, you can look at the quickstart guide or documentation to familiarize yourself with the process.

NOTE: If you fork the repository from the example above this article, be aware that it's only a starting point, so you'll most likely need to modify the Containerfile, image name, registry, tokens, and so on, for your own environment and needs.

Get a RHEL subscription

The first thing about automating RHEL builds with any tool is making sure you can access the repositories to install packages. If you don't have a subscription already, try a no-cost RHEL for developers subscription to get started.

Access your subscription during the build

To use subscription-manager in a pipeline or other automation, it’s best to use an activation key. If you’re a Red Hat Satellite user, you may already have activation keys in your workflows today.

If you aren't familiar with activation keys, here’s what the docs have to say:

An activation key is a preshared authentication token that enables authorized users to register and auto-configure systems. Running a registration command with an activation key and organization ID combination, instead of a username and password combination, increases security and facilitates automation.

In the Red Hat Console, create a key in RHEL > Inventory > System Configuration > Activation Keys. Make note of the Organization ID and the key name. These will be set up as repository secrets within your build workflow.

Getting access to the base image

Unlike UBI, the bootc base image does require a Red Hat account to access the image, because this is a full RHEL host. That means you must be logged in for the pull to work. To log in to the Red Hat Container Registry during a pipeline build or similar, create a registry service account in the customer portal. Head to the service account management page to find or create a new Service Account. The username and password from the Token Information page is required for use with the GitHub Action.

Repository secrets and variables

Now that you have your credentials, the next step is to go to the repository Settings in GitHub and find Actions under Secrets and variables. You must define 4 new secrets (I create these at the repository level, but read the documentation to make the best choice for your needs):

RHT_ORGID: Organization ID

RHT_ACT_KEY: Activation key name

RHT_REG_SVCUSER: Username including the single quotes and pipe character (from the Token Information page)

RHT_REG_SVCPASS: Password (from the Token Information page)

To use the GitHub Container Registry (ghcr.io) from within the workflow, permissions associated with the GITHUB_TOKEN secret must be updated. This secret is automatically created by GitHub when you enable Actions, and can be used for a variety of activities during a workflow. Permissions available to the GITHUB_TOKEN secret can be granted in a few different ways, so be sure to read the documentation to make the best choice for your needs. For my use case, I set my Workflow permissions to Read and write permissions in the Actions panel of the repository Settings. 

Step by step

With a Containerfile, Red Hat account credentials in place and a configured repository, I can start the interesting part, setting up a GitHub Actions workflow that will build a bootc image. All we need to start a workflow is to document the steps of a manual process. I’ve already talked about the prep work, such as obtaining the information for managing subscriptions, but let’s walk through the local build process we’re modeling from the start.

My local build host is RHEL 9.4 with an active subscription. I need to log into the Red Hat registry to be able to pull the bootc base image that we start from. I use podman to build and tag the bootc image as defined in the Containerfile from the Git repository. Then, I need to log into the target registry, usually quay.io, before I can push the final image there for distribution. Now, we just need to translate each of those steps to a GitHub workflow file.

A quick search of the GitHub marketplace shows me that there are a few different ways to build containers and push them to registries including options for both podman and buildah. However, nothing is currently available for handling RHEL subscriptions. Since the Linux runner provided by GitHub does not include subscription-manager support, that will be the only function that I’ll need to handle myself. GitHub Actions are defined in YAML format, so with all of this information in hand, I can get straight to creating the workflow.

The workflow file

Take a look at the workflow file if you want to see each of the steps in one location as we walk through the sections below. You need to sign into your GitHub account to view the logs of the individual steps in the Actions tab of the repository.

Basic controls

There’s nothing particularly interesting or creative in the setup of this project. The workflow is triggered by either a change to the main branch in the rhel directory, or when you manually invoke it in the workflow. The manual trigger is nice for debugging without cluttering your Git history with small changes just to trigger a workflow test.

name: Build RHEL bootc image with GHA
on:
 workflow_dispatch:
 push:
   paths:
     - 'rhel/**'
   branches:
     - main

Get our subscription details

Because the default Linux runner doesn’t know how to deal with Red Hat subscriptions, we need a stand-in for the physical RHEL host. UBI has everything we need for this particular workflow available in its repositories, and the default GitHub Linux runner includes the tools we need to run jobs in a container. In this case, I’m also running the steps inside UBI through containers with Podman.

To get the “Podman in Docker” build working, we do need to run this UBI container using the --privileged option. Otherwise, this is a pretty standard invocation of running a job in a container.

jobs:
 subs:
   name: Build RHEL bootc image with repo access
   runs-on: ubuntu-latest
   env:
    IMAGE_NAME: rhel9-cicd-bootc
    REGISTRY: ghcr.io/nzwulfin
   container:
     image: registry.access.redhat.com/ubi9/ubi
     options: --privileged

With UBI up and running, we can add the container tools we need and get access to our subscription. I pass the two secrets we created earlier for the activation key to the step, and set an environment variable that hints to subscription-manager that even though it's in a container, it shouldn’t look to the host for any information. Typically when dealing with subscriptions for UBI containers, Podman uses details passed up from the host. We want to make sure this container is independent of the host because we’re pretending that it's a full RHEL system. Otherwise, we're using an action that does nothing other than run a command directly. 

- name: Get container tools in UBI builder
 run: dnf -y install podman buildah skopeo
- name: Access a subscription via activation key
 env:
  SMDEV_CONTAINER_OFF: 1
  orgid: ${{ secrets.RHT_ORGID }}
  activation_key: ${{ secrets.RHT_ACT_KEY }}
 run: subscription-manager register --org=$orgid --activationkey=$activation_key

Off-the-shelf actions

With subscriptions pulled and repos available, I do want to take advantage of the automatic subscription handling in podman and buildah, so I use some actions from the marketplace that support use of those tools. These allow me to log into a registry, build the image and push the image to my target registry.

I’m not using any special arguments or anything non-standard. There’s one workaround for an open issue with the podman-login action, but once the issue is fixed, that step will disappear. Because GitHub has a registry that integrates directly with the repository packages, I use a Personal Access Token and the workflow actor (the username of the user triggering the workflow) in the last login as the target. This could be any other registry you have access to, like quay.io or an internal corporate registry.

Last but not least, I make sure not to hang onto any subscriptions we don't need once the image has been pushed to the GitHub registry:

 - name: Clean up the subscription
   env:
     SMDEV_CONTAINER_OFF: 1
   run: subscription-manager unregister

All that's left is to trigger a build, and watch the Action UI do its thing! If you're following along, the initial push with the Containerfile and workflow triggers a job. If the credentials aren't set up first, then the job fails. Once you've gotten everything configured, try committing a change to the Containerfile and you're off and running!

You’ve taken the first step to a new GitOps lifestyle for system management! My dreams have finally been realized, and yours can as well!

https://www.redhat.com/en/introduction-to-image-mode-for-red-hat-enterprise-linux-interactive-lab