Customize your Red Hat OpenShift nodes and keep them updated

Today we're excited to announce a new mechanism for admins to safely and easily customize an operating system deployment with highly refined needs while taking full advantage of the automation and power provided by Red Hat OpenShift. This means you don't have to second guess the need for special device drivers for uncommon hardware, system agents, or organizational demands that require more control over your host operating system.

Red Hat OpenShift is designed to run on a wide variety of hardware and operational contexts. OpenShift runs so well in a variety of environments that admins rarely need to intervene. But increasingly, we meet customers on the leading edge of innovation who want to modify default settings to provide the customized environment their business needs. For example, OpenShift clusters used for data science, artificial intelligence (AI), and machine learning (ML) workloads often require specialized hardware accelerators. Other workloads might be storage-bound or network-bound,  requiring specialized hardware such as storage or network interfaces for optimal performance.

Customizing a node with Machine Config Operator (MCO)

Included in each OpenShift cluster is the Machine Config Operator (MCO). The MCO modifies node operating system (OS) configurations, updates the node OS, and ensures that each node is in the desired configuration state. When the MCO rolls out a new configuration, it performs the following steps on each node until all nodes have received the updated configuration:

1. Cordons the node. This indicates that the node is not available for additional workloads.
2. Drains the node. This terminates all running workloads on the node, causing them to be rescheduled onto other nodes.
3. Applies the new configuration. It writes any new config files, enables systemd units, sets kernel arguments, or installs a new OS (assuming an OpenShift cluster is being upgraded to a newer OpenShift version).
4. Reboots the node.
5. Uncordons the node. This indicates that the node may have workloads scheduled on it again.

Customizing a node with On-Cluster Layering (OCL)

Cluster administrators can provide a Containerfile with an RPM image that layers customized content, such as device drivers or other specialized software, on top of the base OS image. Available in OpenShift 4.16 in tech preview, On Cluster Layering provides a final Open Container Initiative (OCI) image that is automatically:

• Built within the cluster
• Pushed to an OCI image registry
• Installed on each cluster node by default

Admins can customize the underlying cluster OS to take advantage of specialized hardware acceleration or enhanced security tooling without incurring the maintenance burden of keeping both the OS and the additional software up-to-date. Instead, cluster administrators can update the device drivers on all of their cluster nodes as soon as a new driver update is released with their custom image overlay. Those custom images can be kept up-to-date with each OpenShift upgrade.

For example, an admin of a cluster that requires additional tooling to enhance security and manage clusters effectively can now create a Containerfile that installs a logging or security agent onto the base image and configures the agent. On Cluster Layering takes the provided Containerfile, and builds and applies it to the same workflow that installs the OS on each cluster node.

Automatic or manual? You decide.

There are new opportunities for additional testing and automation to be added into this process. Any tooling that works with OCI container images, such as OpenShift Pipelines or security scans provided by Quay.io, can be used to verify the contents and functionality of these customized OS images before they're rolled out. Best of all, this process can be completely automated, as desired.

Stay up-to-date

OpenShift customers love how easy it is to upgrade to a new OpenShift release, and each new OpenShift release includes OS updates. On Cluster Layering works closely with the OpenShift upgrade mechanism to ensure that whenever an upgrade is performed, any customizations are applied to the new OpenShift release as well. A new customized OS image is automatically built before the upgrade process begins.

Optionally, cluster administrators can put this upgraded OS image through the same process to verify its contents and functionality before beginning the upgrade. Once built and tested, cluster administrators can start the upgrade process with the full confidence of knowing that not only do their updated OS images contain everything needed to get the best performance out of their hardware, but these images also have been tested and verified to work within their specific environment.

Image Mode for OpenShift

By now, you might have heard about Image mode for RHEL. As you might suspect, these changes are closely related. That's a big topic, but the most important things to understand are:

• Conceptually, everything is the same. You can now use all the same cloud-native tools, patterns, and infrastructure you know from application containers and use them with full operating system container images. Almost everything you learn from customizing CoreOS nodes carries over to Image mode for RHEL (and the other way round).
• While there are some current technical differences between the implementations in OpenShift and Image mode for RHEL, the approaches are actively converging.

The addition of On Cluster Layering to OpenShift provides a powerful mechanism to enable full control over OpenShift cluster nodes. It provides a self-contained solution for the creation and management of customized OS images within a cluster. This feature offers additional opportunities for testing and automation, providing full confidence in the contents and functionality of these custom OS images. By leveraging On Cluster Layering, cluster administrators can significantly reduce their maintenance burden while still enjoying the benefits of a fully automated and fully customized OpenShift environment.