Multi-arch workloads on hosted control planes on AWS

How would you like to save costs on operating your application services and workloads in Amazon Web Services (AWS)? Starting with Red Hat OpenShift 4.14, hosted control planes on AWS (available as a technology preview) can now schedule application workloads (also called NodePools) on two different CPU architectures, 64-bit Arm and 64-bit x86, from the same hosted control plane (you can't mix CPU architectures within the same NodePool, and each NodePool only contains nodes of one CPU architecture type).

This article examines the installation and benefits of a hosted control plane, and creating multi-architecture workloads on that Hosted Control Plane in AWS.

Hosted control planes for Red Hat OpenShift are based on the HyperShift project and make it possible to host OpenShift tenant clusters at scale in AWS. With each standalone OpenShift cluster, the control plane is scheduled across three machine nodes. However, with hosted control planes, the control plane is scheduled as pods on available worker nodes just like any other workload. This allows many control planes to be created and run on fewer nodes making clusters more affordable while still re-using OpenShift tenancy primitives such as projects, policies and so on.

Are your application services ready to run on Arm? Great! Save money by utilizing 64-bit Arm NodePools. Maybe some of your application services can still only run on x86? No problem! Run those services on 64-bit x86 NodePools. Manage any and all 64-bit Arm and 64-bit x86 NodePools from the same hosted control plane with ease.

Benefits

Utilizing the multi-architecture workload capability of hosted control planes on AWS offers these benefits:

• Run your application workloads on 64-bit Arm NodePools (an m6.large instance) can reduce your costs by approximately 25% compared to an equivalent 64-bit x86 (such as m5.large) NodePool.
• Run different parts of your application workloads on 64-bit Arm NodePools and 64-bit x86 NodePools while managing all your workloads from one centralized hosted control plane.
• Build native 64-bit arm and 64-bit x86 container images in a single cluster.

Cluster preparation

Here's what you need to prepare your management cluster to install HyperShift and create multi-architecture workloads:

• OpenShift 4.14.0 running as the underlying management cluster for either a 64-bit x86 or a 64-bit Arm OpenShift cluster hosted on AWS.
• Install the multicluster engine (MCE) operator, 2.4.0 or newer.
• Follow the steps to set up an AWS S3 bucket and S3 OIDC secret.
• Download the HCP CLI tool.

1. Configure environment variables

First, you must set up some necessary environment variables. These variables are used by the HCP CLI tool to create the hosted control plane, create the NodePools, create a kubeconfig, and destroy the hosted cluster and its corresponding NodePools.

You can obtain your image pull secret, referenced below as PULL_SECRET, from the Red Hat OpenShift Cluster Manager.

REGION=us-east-1
AWS_CREDS=~/.aws/credentials
BASE_DOMAIN=my.base.domain.com
PULL_SECRET=~/all-the-pull-secrets.json
BUCKET_NAME=my-s3-bucket
CLUSTER_NAME=my-hypershift-arm-cluster
AMD64_NODEPOOL_NAME=my-additional-x86-guest-cluster
ARM64_NODEPOOL_NAME=my-arm64-guest-cluster
NODEPOOL_REPLICAS=2

Set your KUBECONFIG environment variable to point at your OpenShift management cluster:

$ export KUBECONFIG=/Users/user/aws-oc-nightly/auth/kubeconfig

2. Create a hosted control plane and a 64-bit Arm NodePool

Now you need to create a hosted control plane with an initial 64-bit x86 NodePool. By overriding the standard release image with a multi-architecture release image, you allow the addition of an Arm64 NodePool later. It ensures that the correct bootstrapping configuration is used when NodePools with different CPU architectures are created.

This closely follows step 4, in 1.7.7.2. Deploying a hosted cluster on AWS.

$ hcp create cluster aws \
--name $CLUSTER_NAME \
--node-pool-replicas=$NODEPOOL_REPLICAS \
--base-domain $BASE_DOMAIN \
--pull-secret $PULL_SECRET \
--aws-creds $AWS_CREDS \
--region $REGION \
--release-image quay.io/openshift-release-dev/ocp-release:4.14.3-multi

Render a NodePool specification into a YAML file. Set the value for arch to arm64, and modify instanceType to an Arm Amazon Machine Image (AMI) instance type (in this example, m6g.large).

$ hcp create nodepool aws \
--cluster-name $CLUSTER_NAME \
--name $ARM64_NODEPOOL_NAME \
--node-count=$NODEPOOL_REPLICAS \
--render > arm_nodepool_spec.yml
NodePool my-arm64-guest-cluster was rendered to yaml output file
$ sed 's/arch: amd64/arch: arm64/g; s/instanceType: m5.large/instanceType: m6g.large/g' arm_nodepool_spec.yml > temp.yml && mv temp.yml arm_nodepool_spec.yml

Once you've modified the NodePool specification YAML file, apply it to your hosted cluster:

$ oc apply -f arm_nodepool_spec.yml
nodepool.hypershift.openshift.io/my-arm64-guest-cluster created

Verify that the NodePool creation was successful using the oc command: $ oc get nodepools -n clusters

$ oc get nodepools -n clusters
NAME                     CLUSTER...
My-arm64-guest-cluster   my-arm64-guest-cluster

Next, verify the successful completion of the hosted control plane. You may not see a completed progress until 10 to 15 minutes after applying the NodePool specification YAML file.

$ oc get --namespace clusters hostedclusters
NAME                       VERSION...
my-hypershift-arm-cluster  4.14.0-0.nightly...

Add an x86 NodePool

To add a 64-bit x86 NodePool to the hosted control plane, use the HCP CLI tool. NodePools default to an x86 CPU architecture, so there's no need to render the NodePool specification and modify it before applying it to the hosted cluster.

$ hcp create nodepool aws \
--cluster-name $CLUSTER_NAME \
--name $AMD64_NODEPOOL_NAME \
--node-count=$NODEPOOL_REPLICAS \
NodePool my-additional-x86-guest-cluster created

Use the oc command to verify that the new NodePool was initialized successfully:

$ oc get nodepools -n clusters
NAME                             CLUSTER
my-additional-x86-guest-cluster  my-hypershift-arm-cluster

Create a Kubeconfig for NodePools

Create a KUBECONFIG to query the nodes in the NodePools and verify that the AWS machines are ready.

$ hcp create kubeconfig > temp_kubeconfig
2023/08/23 16:12:55 selected 1 of 1 hostedclusters for the kubeconfig
2023/08/23 16:12:55 adding clusters/my-hypershift-arm-cluster to kubeconfig
2023/08/23 16:12:55 added clusters-my-hypershift-arm-cluster to kubeconfig
2023/08/23 16:12:55 created kubeconfig with 1 contexts

Verify the CPU architecture of the nodes. You should see a mix of arm64 and amd64 nodes.

Alternatively, you could run the following command to see the full specification of the nodes to see more fields like the name, the NodePool name, and CPU architecture for each node:

$ oc --kubeconfig=temp_kubeconfig get nodes \ -ojsonpath='{.items[*].status.nodeInfo.architecture}'
arm64 amd64 arm64 amd64 amd64 amd64

Deleting the hosted control plane and NodePools

Should you need to delete the hosted control plane and its corresponding NodePools at any time, use the hcp destroy command:

$ hcp destroy cluster aws \
--name $CLUSTER_NAME \
--aws-creds $AWS_CREDS

This closely follows step 1, in 1.7.7.8. Destroying a hosted cluster on AWS. Summary You now know the steps required to install and verify 64-bit Arm and 64-bit x86 workloads (that is, NodePools). You also know how to use the hcp command to create a hosted cluster, create a 64-bit Arm NodePool specification file, apply the specification to the hosted cluster, create a 64-bit x86 NodePool on the same hosted cluster, verify successful creation of the NodePools, and finally, delete the hosted control plane.

To further reduce costs, install your hosted cluster on a 64-bit Arm OpenShift cluster. Check out this previous blog post on details on how to do just that!