A Guide to Red Hat Hypershift on Bare Metal

April 20, 2022Benjamin Schmaus51-minute read

HyperShift is middleware for hosting OpenShift control planes at scale that solves for cost and time to provision, as well as portability cross cloud with strong separation of concerns between management and workloads. Clusters are fully compliant OpenShift Container Platform (OCP) clusters and are compatible with standard OCP and Kubernetes toolchains. In the following blog I would like to show how one can enable and provision a Hypershift based cluster in OCP.

This blog is a technical demonstration, and should not be used as an architecture for a production environment. Specifically, the use of ODF for host control plane etcd should only be used in a demonstration, given the etcd fast storage speed requirements. See this knowledge base article for details: https://access.redhat.com/articles/6970988

Before we begin lets start by describing the existing environment of the hub OCP cluster. This hub cluster (kni20) is where the Hypershift operator will run and also the control plane of the secondary cluster we deploy with Hypershift. The hub cluster is a OCP 4.10.3 compact 3 node cluster running on baremetal with OpenShift-SDN. The following operators have been installed for convience to provide the storage needed for Infrastructure Operator:

Along with installing and configuring the above operators we went ahead and set a default storageclass so that any requested persistent volume claims will be fulfilled automatically:

$ oc patch storageclass ocs-storagecluster-ceph-rbd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/ocs-storagecluster-ceph-rbd patched

At this point we can now start to configure this hub cluster for use with Hypershift. First we need to install one additional required operator: Infrastructure Operator for Red Hat OpenShift. We can install it from OperatorHub in the console.

We will be using the community operator here but in the future Red Hat Advanced Cluster Management for Kubernetes will actually integrate Hypershift and this will not be required as the Red Hat equivalents are pulled in when installing RHACM.

Once the Infrastructure Operator for Red Hat Openshift is installed we can proceed with configuring it. Here we need to create an agent service configuration custom resource that will tell the operator how much storage we need for the various components like database and filesystem and it will also define what OpenShift versions to maintain:

cat << EOF > ~/agentserviceconfig.yaml
apiVersion: agent-install.openshift.io/v1beta1
kind: AgentServiceConfig
metadata:
  name: agent
spec:
  databaseStorage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
  filesystemStorage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 100Gi
  imageStorage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
  osImages:
    - cpuArchitecture: x86_64
      openshiftVersion: '4.8'
      rootFSUrl: >-
        https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.8/4.8.14/rhcos-live-rootfs.x86_64.img
      url: >-
        https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.8/4.8.14/rhcos-4.8.14-x86_64-live.x86_64.iso
      version: 48.84.202109241901-0
    - cpuArchitecture: x86_64
      openshiftVersion: '4.9'
      rootFSUrl: >-
        https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.9/4.9.0/rhcos-live-rootfs.x86_64.img
      url: >-
        https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.9/4.9.0/rhcos-4.9.0-x86_64-live.x86_64.iso
      version: 49.84.202110081407-0
    - cpuArchitecture: arm64
      openshiftVersion: '4.9'
      rootFSUrl: >-
        https://mirror.openshift.com/pub/openshift-v4/aarch64/dependencies/rhcos/4.9/4.9.0/rhcos-4.9.0-aarch64-live-rootfs.aarch64.img
      url: >-
        https://mirror.openshift.com/pub/openshift-v4/aarch64/dependencies/rhcos/4.9/4.9.0/rhcos-4.9.0-aarch64-live.aarch64.iso
      version: 49.84.202110080947-0
    - cpuArchitecture: x86_64
      openshiftVersion: '4.10'
      rootFSUrl: >-
        https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.10/4.10.3/rhcos-4.10.3-x86_64-live-rootfs.x86_64.img
      url: >-
        https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.10/4.10.3/rhcos-4.10.3-x86_64-live.x86_64.iso
      version: 410.84.202201251210-0
EOF

Once the agent service configuration file is created apply it to the cluster:

[bschmaus@provisioning ~]$ oc create -f agent_service_config-kni20.yaml
agentserviceconfig.agent-install.openshift.io/agent created

After a few minutes validate that the pods for the Infrastructure operator have started:

$ oc get po -n assisted-installer
NAME                                       READY   STATUS    RESTARTS   AGE
agentinstalladmission-77bf8b8b8f-kcr8m     1/1     Running   0          27m
agentinstalladmission-77bf8b8b8f-ttxbg     1/1     Running   0          27m
assisted-image-service-0                   1/1     Running   0          27m
assisted-service-8876b7d45-9g2fb           2/2     Running   0          27m
infrastructure-operator-76d4b9c58f-ghvds   1/1     Running   0          32m

With our required operator installed we can move onto patching the provisioning configuration to watch all namespaces. This ensures that the bare metal hosts we are adding in a different namespace will get picked up:

$ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"watchAllNamespaces": true}}'
provisioning.metal3.io/provisioning-configuration patched

At this point we have completed all the pre-requisites required for Hypershift to be deployed so now we can proceed with deploying Hypershift on the hub cluster. To make deployment of Hypershift and our cluster easier lets go ahead and assign some basic variables that we can reuse during the rest of the installation process. These variables include definitions like: Hypershift namespace, Hypershift cluster name, infrastructure environment, base domain, sshkey, pull-secret, machine CIDR, OCP release and arhitecture, Hypershift image location and kubeconfig of hub cluster. Hypershift currently has to be installed via a container method because there is no operator in Operator hub to deploy Hypershift.

NAMESPACE=kni21ns
CLUSTERNAME=kni21
INFRAENV=kni21infra
BASEDOMAIN=schmaustech.com
SSHKEY=~/.ssh/id_rsa.pub
PULLSECRETNAME=${INFRAENV}-pullsecret
MACHINE_CIDR_CLUSTER=192.168.0.0/24
OCP_RELEASE_VERSION="4.10.3"
OCP_ARCH="x86_64"
HYPERSHIFT_IMAGE=quay.io/hypershift/hypershift-operator:latest
KUBECONFIG=kubeconfig-kni20
PULLSECRET=pull-secret.json

With our variables defined let move onto creating a temporary directory for our Hypershift install, copy in our kubeconfig and pull-secret:

$ mkdir /tmp/hypershift
$ cp ~/$KUBECONFIG /tmp/hypershift/
$ cp ~/$PULLSECRET /tmp/hypershift/
$ cd /tmp/hypershift/

Next lets make sure we are logged into Quay.io:

$ podman login quay.io
Username: bschmaus
Password: 
Login Succeeded!

Then lets pull the Hypershift image from Quay.io:

$ podman pull $HYPERSHIFT_IMAGE
Trying to pull quay.io/hypershift/hypershift-operator:latest...

Getting image source signatures
Copying blob de9bc33d7337 skipped: already exists  
Copying blob effc4ea612c8 skipped: already exists  
Copying blob 495726a4e351 done  
Copying blob ce753a481e9f done  
Copying blob 3b83e4259b50 done  
Copying blob 2de609a6fd40 done  
Copying config 12be255359 done  
Writing manifest to image destination
Storing signatures
12be2553593f133366e0b706c5bbcbab6e90c950b4bb1ea1feb1e25512791a9c

The Hypershift image that we pulled from Quay.io contains the Hypershift binary which we will use to install Hypershift operator into our hub cluster kni20. To do this we want to use a combination of an alias with the podman run command below. Note we are passing in the kubeconfig from kni20 so that the Hypershift binary is able to install on kni20.

$ alias hypershift="podman run --net host --rm --entrypoint /usr/bin/hypershift -e KUBECONFIG=/working_dir/$KUBECONFIG -v $HOME/.ssh:/root/.ssh -v /tmp/hypershift:/working_dir $HYPERSHIFT_IMAGE"

With the hypershift alias set we can proceed to run the hypershift install command. The output should look very similar to that below:

$ hypershift install --hypershift-image $HYPERSHIFT_IMAGE
created PriorityClass /hypershift-control-plane
created PriorityClass /hypershift-etcd
created PriorityClass /hypershift-api-critical
applied Namespace /hypershift
applied ServiceAccount hypershift/operator
applied ClusterRole /hypershift-operator
applied ClusterRoleBinding /hypershift-operator
applied Role hypershift/hypershift-operator
applied RoleBinding hypershift/hypershift-operator
applied Deployment hypershift/operator
applied Service hypershift/operator
applied Role hypershift/prometheus
applied RoleBinding hypershift/prometheus
applied ServiceMonitor hypershift/operator
applied PrometheusRule hypershift/metrics
applied CustomResourceDefinition /clusterresourcesetbindings.addons.cluster.x-k8s.io
applied CustomResourceDefinition /clusterresourcesets.addons.cluster.x-k8s.io
applied CustomResourceDefinition /clusterclasses.cluster.x-k8s.io
applied CustomResourceDefinition /clusters.cluster.x-k8s.io
applied CustomResourceDefinition /machinedeployments.cluster.x-k8s.io
applied CustomResourceDefinition /machinehealthchecks.cluster.x-k8s.io
applied CustomResourceDefinition /machinepools.cluster.x-k8s.io
applied CustomResourceDefinition /machines.cluster.x-k8s.io
applied CustomResourceDefinition /machinesets.cluster.x-k8s.io
applied CustomResourceDefinition /agentclusters.capi-provider.agent-install.openshift.io
applied CustomResourceDefinition /agentmachines.capi-provider.agent-install.openshift.io
applied CustomResourceDefinition /agentmachinetemplates.capi-provider.agent-install.openshift.io
applied CustomResourceDefinition /awsclustercontrolleridentities.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsclusterroleidentities.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsclusters.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsclusterstaticidentities.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsclustertemplates.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsfargateprofiles.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsmachinepools.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsmachines.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsmachinetemplates.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsmanagedmachinepools.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /azureclusteridentities.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /azureclusters.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /azuremachines.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /azuremachinetemplates.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /ibmpowervsclusters.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /ibmpowervsmachines.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /ibmpowervsmachinetemplates.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /ibmvpcclusters.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /ibmvpcmachines.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /ibmvpcmachinetemplates.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /kubevirtclusters.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /kubevirtmachines.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /kubevirtmachinetemplates.infrastructure.cluster.x-k8s.io
applied CustomResourceDefinition /awsendpointservices.hypershift.openshift.io
applied CustomResourceDefinition /hostedclusters.hypershift.openshift.io
applied CustomResourceDefinition /hostedcontrolplanes.hypershift.openshift.io
applied CustomResourceDefinition /nodepools.hypershift.openshift.io

Once the installation of the operator completes we can validate that we have a hypershift namespace an a running operator within it:

$ oc get po -n hypershift
NAME                       READY   STATUS    RESTARTS   AGE
operator-9c7f76468-2rzhl   1/1     Running   0          20m

At this point in the procedure we are now ready to congfigure the requirements needed to deploy our Hypershift cluster kni21. First lets create the namespace using the variable we defined above:

$ oc create namespace $NAMESPACE
namespace/kni21ns created

With the namespace created we can proceed in creating a pull-secret yaml file for that namespace:

cat << EOF > ~/$PULLSECRETNAME.yaml
apiVersion: v1
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
  name: $PULLSECRETNAME
  namespace: $NAMESPACE
data:
  .dockerconfigjson: $(cat /tmp/hypershift/pull-secret.json  | base64 -w0)
EOF

Once the pull-secret yaml has been created apply the custom resource:

oc create -f ~/$PULLSECRETNAME.yaml
secret/kni21infra-pullsecret created

Next lets create a infrastructure environment yaml file referencing the pull-secret we just created and also consuming our sshkey:

cat << EOF > ~/$INFRAENV.yaml
apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
  name: $INFRAENV
  namespace: $NAMESPACE
spec:
  pullSecretRef:
    name: $PULLSECRETNAME
  sshAuthorizedKey: $(cat $SSHKEY)
EOF

With the infrastructure environment yaml created proceed to create the custom resource:

$ oc create -f ~/$INFRAENV.yaml
infraenv.agent-install.openshift.io/kni21infra created

When the infrastructure environment is created it will proceed to create a discovery iso for that environment. To ensure that the discovery iso is created we need to check that the processes are completed and the iso exists. We can achieve this by first confirming that we are not seeing any pods in a state other then Completed or Running. If that looks good then we can further confirm by grabbing the iso download from the infrastructure environment and confirming the discovery iso is present.

$ oc get po -n $NAMESPACE|grep -vE 'Completed|Running'
NAMESPACE                                          NAME                                                              READY   STATUS      RESTARTS   AGE

$ oc get infraenv $INFRAENV -n $NAMESPACE -o json| jq -r .status.isoDownloadURL| xargs curl -kI
HTTP/1.1 200 OK
accept-ranges: bytes
content-disposition: attachment; filename=6440f9b6-44c9-452a-a820-906c582290fe-discovery.iso
content-length: 108984320
content-type: application/octet-stream
last-modified: Fri, 01 Apr 2022 14:09:52 GMT
date: Fri, 01 Apr 2022 14:09:52 GMT
set-cookie: 2d419d3e406946976a07970b1abc63e4=08f02ad56798ca18bf051a5067a87038; path=/; HttpOnly; Secure; SameSite=None
cache-control: private

With the discovery iso present we can now configure the two baremetal hosts that will become the physical worker nodes for our Hypershift cluster kni21. First lets again set some variables to make things easier. These variables will include the BMC username, BMC password, BMC address, worker name, boot MAC address of worker node, UUID of worker node and the RedFish URL. The UUID is not required in all RedFish enabled environments and depending on what type of hardware one is using (Dell/HP/SuperMicro/Etc) the RedFish URL could look slightly different as well so modify accordingly.

BMC_USERNAME=$(echo -n "admin" | base64 -w0)
BMC_PASSWORD=$(echo -n "password" | base64 -w0)
BMC_IP="192.168.0.10"
WORKER="worker-0"
BOOT_MAC_ADDRESS="52:54:00:ec:24:02"
UUID=1504e201-9385-4526-81e7-7d2c5f86791e
REDFISH="redfish-virtualmedia+https://${BMC_IP}:8001/redfish/v1/Systems/${UUID}"

With our variables defined lets go ahead and create the secret yaml for the first worker:

cat << EOF > ~/secret-$WORKER.yaml
apiVersion: v1
data:
  password: ${BMC_PASSWORD}
  username: ${BMC_USERNAME}
kind: Secret
metadata:
  name: ${WORKER}-bmc-secret
  namespace: ${NAMESPACE}
type: Opaque
EOF

Next create the secret for the first worker on the hub cluster kni20:

$ oc create -f ~/secret-$WORKER.yaml
secret/worker-0-bmc-secret created

Next create the baremetahost yaml file for the first worker:

cat << EOF > ~/$CLUSTERNAME-$WORKER.yaml
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: ${WORKER}
  namespace: ${NAMESPACE}
  labels:
    infraenvs.agent-install.openshift.io: ${INFRAENV}
  annotations:
    inspect.metal3.io: disabled
spec:
  automatedCleaningMode: disabled
  bmc:
    disableCertificateVerification: True
    address: ${REDFISH}
    credentialsName: ${WORKER}-bmc-secret
  bootMACAddress: ${BOOT_MAC_ADDRESS}
  online: true
EOF

And then create the baremetalhost for the first worker on the hub cluster kni20:

$ oc create -f ~/$CLUSTERNAME-$WORKER.yaml
baremetalhost.metal3.io/worker-0 created

After we have created the first worker baremetalhost we should watch the agent until the worker appears. We can do this with a until loop on oc get agent.

$ until oc get agent -n ${NAMESPACE} ${UUID} >/dev/null 2>&1 ; do sleep 1 ; done
$ echo $?
0

When the loop exits we can validate that the agent is listed by running the following command:

$ oc get agent -n ${NAMESPACE} ${UUID}
NAME                                   CLUSTER   APPROVED   ROLE          STAGE
1504e201-9385-4526-81e7-7d2c5f86791e             true       auto-assign

If the agent is indeed listed then we can move onto assign the agent name to the agent variable:

$ AGENT=$(oc get agent -n ${NAMESPACE} ${UUID} -o name)

With the agent variable populated lets go ahead and patch the agent with the installation disk, hostname and its role in the cluster:

$ oc patch ${AGENT} -n ${NAMESPACE} -p '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"'"$WORKER.$CLUSTERNAME.$BASEDOMAIN"'","role":"worker"}}' --type merge
agent.agent-install.openshift.io/1504e201-9385-4526-81e7-7d2c5f86791e patched

Now lets repeat the same steps for the second worker node by first assiging our variables:

BMC_USERNAME=$(echo -n "admin" | base64 -w0)
BMC_PASSWORD=$(echo -n "password" | base64 -w0)
BMC_IP="192.168.0.10"
WORKER="worker-1"
BOOT_MAC_ADDRESS="52:54:00:f3:c4:62"
UUID=c6828a00-169a-4578-a3ab-e62583b449be
REDFISH="redfish-virtualmedia+https://${BMC_IP}:8001/redfish/v1/Systems/${UUID}"

Next create the secret yam file for the second worker node:

cat << EOF > ~/secret-$WORKER.yaml
apiVersion: v1
data:
  password: ${BMC_PASSWORD}
  username: ${BMC_USERNAME}
kind: Secret
metadata:
  name: ${WORKER}-bmc-secret
  namespace: ${NAMESPACE}
type: Opaque
EOF

And then create the baremetal host yaml file for the second worker node:

cat << EOF > ~/$CLUSTERNAME-$WORKER.yaml
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: ${WORKER}
  namespace: ${NAMESPACE}
  labels:
    infraenvs.agent-install.openshift.io: ${INFRAENV}
  annotations:
    inspect.metal3.io: disabled
spec:
  automatedCleaningMode: disabled
  bmc:
    disableCertificateVerification: True
    address: ${REDFISH}
    credentialsName: ${WORKER}-bmc-secret
  bootMACAddress: ${BOOT_MAC_ADDRESS}
  online: true
EOF

And finally go ahead and create the custom resources for the second workers secret and baremetal host on the kni20 hub cluster:

$ oc create -f ~/secret-$WORKER.yaml
secret/worker-1-bmc-secret created

$ oc create -f ~/$CLUSTERNAME-$WORKER.yaml
baremetalhost.metal3.io/worker-1 created

Again we will run our until loop to wait for the agent to come online and then once it does assign the agent name to a variable and then patch the agent with the installation disk, hosname and role again:

$ until oc get agent -n ${NAMESPACE} ${UUID} >/dev/null 2>&1 ; do sleep 1 ; done
$ echo $?
0

$ AGENT=$(oc get agent -n ${NAMESPACE} ${UUID} -o name)

$ oc patch ${AGENT} -n ${NAMESPACE} -p '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"'"$WORKER.$CLUSTERNAME.$BASEDOMAIN"'","role":"worker"}}' --type merge
agent.agent-install.openshift.io/c6828a00-169a-4578-a3ab-e62583b449be patched

With the two worker baremetal hosts ready to be consumed we can move onto deploying the hypershift cluster. To do this we will again refer to the hypershift alias we setup earlier in this blog and leverage the hypershift pod image. To create a cluster we need to execute the following syntax and pass in our clustername, basedomain, namespace and agent namespace:

$ hypershift create cluster agent --name $CLUSTERNAME --base-domain $BASEDOMAIN --pull-secret /working_dir/pull-secret.json  --ssh-key /working_dir/id_rsa.pub --agent-namespace $NAMESPACE --namespace $NAMESPACE
2022-04-01T16:51:19Z	INFO	detected "192.168.0.213" from node "worker-0.kni20.schmaustech.com" as external-api-server-address
2022-04-01T16:51:19Z	INFO	Applied Kube resource	{"kind": "Namespace", "namespace": "", "name": "kni21ns"}
2022-04-01T16:51:19Z	INFO	Applied Kube resource	{"kind": "Secret", "namespace": "kni21ns", "name": "kni21-pull-secret"}
2022-04-01T16:51:19Z	INFO	Applied Kube resource	{"kind": "", "namespace": "kni21ns", "name": "kni21"}
2022-04-01T16:51:19Z	INFO	Applied Kube resource	{"kind": "Role", "namespace": "kni21ns", "name": "capi-provider-role"}
2022-04-01T16:51:19Z	INFO	Applied Kube resource	{"kind": "Secret", "namespace": "kni21ns", "name": "kni21-etcd-encryption-key"}
2022-04-01T16:51:19Z	INFO	Applied Kube resource	{"kind": "Secret", "namespace": "kni21ns", "name": "kni21-ssh-key"}
2022-04-01T16:51:19Z	INFO	Applied Kube resource	{"kind": "NodePool", "namespace": "kni21ns", "name": "kni21"}

This begins the cluster creation process but before we look at the pods coming up lets first observe the nodepool that was created. We should see that it has 0 desired nodes listed:

$ oc get nodepool ${CLUSTERNAME} -n ${NAMESPACE}
NAME    CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
kni21   kni21     0                               False         False        4.10.7

We actually want that nodepool to consume the two workers we configured so go ahead and scale the nodepool to have 2 nodes:

oc patch nodepool/${CLUSTERNAME} -n ${NAMESPACE} -p '{"spec":{"nodeCount": 2}}' --type merge

Once we scale the nodepool we can observe nothing has changed however the scaling has started processes in the backhground that will consume the two workers that we added above. We will check back on this in a bit.

$ oc get nodepool ${CLUSTERNAME} -n ${NAMESPACE}
NAME    CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
kni21   kni21     2                               False

Now lets focus back on the cluster itself. If we look in the following namespace made up of both our original namespace and clustername variables we can see that the control plane of a new cluster is running:

$ oc get pods -n ${NAMESPACE}-${CLUSTERNAME}
NAME                                              READY   STATUS    RESTARTS   AGE
capi-provider-7cfd7fd785-n4qzp                    1/1     Running   0          6m18s
catalog-operator-666b598b7b-4w6f8                 2/2     Running   0          5m41s
certified-operators-catalog-5869b59955-m2l5l      1/1     Running   0          5m43s
cluster-api-7c879944f6-fpn5r                      1/1     Running   0          6m18s
cluster-autoscaler-6fbc7b465c-kt6rl               1/1     Running   0          5m39s
cluster-policy-controller-8467d64db5-p4vsq        1/1     Running   0          5m46s
cluster-version-operator-5757c466c7-trwq8         1/1     Running   0          5m45s
community-operators-catalog-d4646d4cf-79q27       1/1     Running   0          5m43s
control-plane-operator-689d4f4b44-gtw7n           1/1     Running   0          6m17s
etcd-0                                            1/1     Running   0          5m47s
hosted-cluster-config-operator-59466cc47d-6qvs9   1/1     Running   0          5m44s
ignition-server-7566c666bf-tmw9v                  1/1     Running   0          6m16s
ingress-operator-695b4b78f-nf79g                  2/2     Running   0          5m45s
konnectivity-agent-5794855958-cvs85               1/1     Running   0          5m47s
konnectivity-server-5b86b5c944-s7bxq              1/1     Running   0          5m47s
kube-apiserver-69b77cd898-pltps                   2/2     Running   0          5m47s
kube-controller-manager-587879679c-lxpmq          1/1     Running   0          5m47s
kube-scheduler-74f5cf9bd8-r6bqc                   1/1     Running   0          5m47s
machine-approver-789674bd68-9zcmj                 1/1     Running   0          5m39s
oauth-openshift-9b78fcf8b-rh5lb                   1/1     Running   0          3m56s
olm-operator-6768fc984f-9w2r7                     2/2     Running   0          5m41s
openshift-apiserver-5f57ccc8cd-k7ss2              2/2     Running   0          5m47s
openshift-controller-manager-859644688f-fvhgn     1/1     Running   0          5m46s
openshift-oauth-apiserver-57bb6484c7-qqjlv        1/1     Running   0          5m46s
packageserver-68f897ffc9-k9v2j                    2/2     Running   0          5m41s
redhat-marketplace-catalog-6b4bf87d95-skp55       1/1     Running   0          5m43s
redhat-operators-catalog-f58ffd56-8g7lv           1/1     Running   0          5m43s

Seeing that the control plane is running we can go ahead and extract the kubeconfig for that cluster:

$ oc extract -n ${NAMESPACE} secret/${CLUSTERNAME}-admin-kubeconfig --to=- > ${CLUSTERNAME}-kubeconfig
# kubeconfig

With the kubeconfig extracted we can use it to see the status of the cluster operators of our new Hypershift cluster. At this point though we should be seeing only partial operators being available. That is the cluster is still in progress of installing.

$  oc get co --kubeconfig=kni21-kubeconfig
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console                                                                                           
csi-snapshot-controller                                                                           
dns                                                                                               
image-registry                                                                                    
ingress                                              False       True          True       6m17s   The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.)
kube-apiserver                             4.10.9    True        False         False      6m46s   
kube-controller-manager                    4.10.9    True        False         False      6m46s   
kube-scheduler                             4.10.9    True        False         False      6m46s   
kube-storage-version-migrator                                                                     
monitoring                                                                                        
network                                                                                           
node-tuning                                                                                       
openshift-apiserver                        4.10.9    True        False         False      6m46s   
openshift-controller-manager               4.10.9    True        False         False      6m46s   
openshift-samples                                                                                 
operator-lifecycle-manager                 4.10.9    True        False         False      6m27s   
operator-lifecycle-manager-catalog         4.10.9    True        False         False      6m36s   
operator-lifecycle-manager-packageserver   4.10.9    True        False         False      6m46s   
service-ca                                                                                        
storage

If we check back on our nodepool we can see that the status has changed and the updating version and config are set to true now:

$ oc get nodepool ${CLUSTERNAME} -n ${NAMESPACE}
NAME    CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
kni21   kni21     2                               False         False                  True              True             Minimum availability requires 2 replicas, current 0 available

If we wait a little bit longer for the cluster to instantiate and then look at all the running pods across all cluster namespaces but using the kubeconfig we extracted we can now see that everything is up and operational:

$ oc get pods -A --kubeconfig=kni21-kubeconfig
NAMESPACE                                          NAME                                                      READY   STATUS    RESTARTS        AGE
kube-system                                        konnectivity-agent-dpj5n                                  1/1     Running   0               7m53s
kube-system                                        konnectivity-agent-n4kf6                                  1/1     Running   0               8m4s
kube-system                                        kube-apiserver-proxy-worker-0.kni21.schmaustech.com       1/1     Running   0               9m46s
kube-system                                        kube-apiserver-proxy-worker-1.kni21.schmaustech.com       1/1     Running   0               9m25s
openshift-cluster-node-tuning-operator             cluster-node-tuning-operator-65b86d9b9b-77zn2             1/1     Running   0               26m
openshift-cluster-node-tuning-operator             tuned-vbpvx                                               1/1     Running   0               6m39s
openshift-cluster-node-tuning-operator             tuned-x4nmm                                               1/1     Running   0               6m39s
openshift-cluster-samples-operator                 cluster-samples-operator-948b7d55f-bnrcl                  2/2     Running   0               26m
openshift-cluster-storage-operator                 cluster-storage-operator-86d98bdc5d-75lhh                 1/1     Running   1 (7m11s ago)   26m
openshift-cluster-storage-operator                 csi-snapshot-controller-8544d7964b-2nwk5                  1/1     Running   0               6m39s
openshift-cluster-storage-operator                 csi-snapshot-controller-8544d7964b-xxng9                  1/1     Running   0               6m39s
openshift-cluster-storage-operator                 csi-snapshot-controller-operator-fcc99c578-7gtkg          1/1     Running   0               26m
openshift-cluster-storage-operator                 csi-snapshot-webhook-c76dfc54d-jqqhj                      1/1     Running   0               7m23s
openshift-cluster-storage-operator                 csi-snapshot-webhook-c76dfc54d-wp2vg                      1/1     Running   0               7m23s
openshift-console-operator                         console-operator-787c499759-qqf2r                         1/1     Running   1 (6m ago)      26m
openshift-console                                  console-797cd4885c-znb5k                                  1/1     Running   0               6m40s
openshift-console                                  console-797cd4885c-zpdv6                                  1/1     Running   0               6m39s
openshift-console                                  downloads-56bb8656db-mvgfg                                1/1     Running   0               7m28s
openshift-console                                  downloads-56bb8656db-w4xpq                                1/1     Running   0               7m28s
openshift-dns-operator                             dns-operator-9fbd86494-fqcdc                              2/2     Running   0               26m
openshift-dns                                      dns-default-498l5                                         2/2     Running   0               6m40s
openshift-dns                                      dns-default-vxpgr                                         2/2     Running   0               6m39s
openshift-dns                                      node-resolver-frxhm                                       1/1     Running   0               6m40s
openshift-dns                                      node-resolver-x2wwv                                       1/1     Running   0               6m39s
openshift-image-registry                           cluster-image-registry-operator-59d49c88cf-678bf          1/1     Running   0               26m
openshift-image-registry                           image-registry-6b55fbbd4-2pvl5                            1/1     Running   0               6m21s
openshift-image-registry                           node-ca-485vv                                             1/1     Running   0               6m23s
openshift-image-registry                           node-ca-w767j                                             1/1     Running   0               6m23s
openshift-ingress-canary                           ingress-canary-8wgk6                                      1/1     Running   0               7m53s
openshift-ingress-canary                           ingress-canary-vt5hf                                      1/1     Running   0               8m4s
openshift-ingress                                  router-default-596896b59c-nlnkg                           1/1     Running   0               26m
openshift-ingress                                  router-default-596896b59c-qj6mg                           1/1     Running   0               26m
openshift-kube-storage-version-migrator-operator   kube-storage-version-migrator-operator-79d98c76bb-bg5dh   1/1     Running   1 (6m16s ago)   25m
openshift-kube-storage-version-migrator            migrator-664f6b4cc7-v97vt                                 1/1     Running   0               7m29s
openshift-monitoring                               alertmanager-main-0                                       6/6     Running   0               6m
openshift-monitoring                               alertmanager-main-1                                       6/6     Running   0               6m
openshift-monitoring                               cluster-monitoring-operator-5486d59b75-kc7ng              2/2     Running   0               26m
openshift-monitoring                               grafana-ff8ff8857-pv74p                                   3/3     Running   0               5m59s
openshift-monitoring                               kube-state-metrics-5dffbb6d67-b8cfg                       3/3     Running   0               6m6s
openshift-monitoring                               node-exporter-k2cjj                                       2/2     Running   0               6m6s
openshift-monitoring                               node-exporter-zc78q                                       2/2     Running   0               6m6s
openshift-monitoring                               openshift-state-metrics-69b67595cb-gg6g4                  3/3     Running   0               6m6s
openshift-monitoring                               prometheus-adapter-564b459fd4-8lj7h                       1/1     Running   0               4m44s
openshift-monitoring                               prometheus-adapter-564b459fd4-wv6qd                       1/1     Running   0               4m44s
openshift-monitoring                               prometheus-k8s-0                                          6/6     Running   0               5m56s
openshift-monitoring                               prometheus-k8s-1                                          6/6     Running   0               5m56s
openshift-monitoring                               prometheus-operator-5b57d65c9b-lb6pb                      2/2     Running   0               6m40s
openshift-monitoring                               telemeter-client-c7b7cdb75-vb6c8                          3/3     Running   0               6m
openshift-monitoring                               thanos-querier-6878d74c84-4prng                           6/6     Running   0               5m58s
openshift-monitoring                               thanos-querier-6878d74c84-k2lb2                           6/6     Running   0               5m58s
openshift-multus                                   multus-additional-cni-plugins-ftbdr                       1/1     Running   0               9m26s
openshift-multus                                   multus-additional-cni-plugins-fv5h6                       1/1     Running   0               9m26s
openshift-multus                                   multus-admission-controller-5zr6r                         2/2     Running   0               7m53s
openshift-multus                                   multus-admission-controller-ppj9h                         2/2     Running   0               8m4s
openshift-multus                                   multus-dvl2v                                              1/1     Running   0               9m26s
openshift-multus                                   multus-h77hw                                              1/1     Running   0               9m26s
openshift-multus                                   network-metrics-daemon-2tdfc                              2/2     Running   0               9m25s
openshift-multus                                   network-metrics-daemon-pwg92                              2/2     Running   0               9m25s
openshift-network-diagnostics                      network-check-source-65cbf8c547-c7zqt                     1/1     Running   0               9m12s
openshift-network-diagnostics                      network-check-target-bl8sw                                1/1     Running   0               9m11s
openshift-network-diagnostics                      network-check-target-rc9wk                                1/1     Running   0               9m11s
openshift-network-operator                         network-operator-779fd555b-9xnq7                          1/1     Running   0               26m
openshift-sdn                                      sdn-4fl62                                                 2/2     Running   0               9m12s
openshift-sdn                                      sdn-controller-9wktp                                      2/2     Running   0               9m12s
openshift-sdn                                      sdn-controller-g2fgg                                      2/2     Running   0               9m12s
openshift-sdn                                      sdn-lqbb6                                                 2/2     Running   0               9m12s
openshift-service-ca-operator                      service-ca-operator-5687654cb9-s9p2c                      1/1     Running   1 (5m58s ago)   26m
openshift-service-ca                               service-ca-6694d7d4fd-9f8rm                               1/1     Running   0               7m22s

We can also validate that the worker nodes have joined the cluster and are in a ready state:

$ oc get nodes -A --kubeconfig=kni21-kubeconfig
NAME                             STATUS   ROLES    AGE   VERSION
worker-0.kni21.schmaustech.com   Ready    worker   15m   v1.23.5+9ce5071
worker-1.kni21.schmaustech.com   Ready    worker   14m   v1.23.5+9ce5071

And if we check back on our nodepool we can see we have indeed 2 nodes in the pool:

$ oc get nodepool ${CLUSTERNAME} -n ${NAMESPACE}
NAME    CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
kni21   kni21     2               2               False         False        4.10.9

And finally if we check our cluster operators again on the Hypershift cluster we can finally see that they too are all online and available:

$ oc get co --kubeconfig=kni21-kubeconfig
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console                                    4.10.9    True        False         False      4m26s   
csi-snapshot-controller                    4.10.9    True        False         False      3m10s   
dns                                        4.10.9    True        False         False      2m4s    
image-registry                             4.10.9    True        False         False      88s     
ingress                                    4.10.9    True        False         False      118s    
kube-apiserver                             4.10.9    True        False         False      23m     
kube-controller-manager                    4.10.9    True        False         False      23m     
kube-scheduler                             4.10.9    True        False         False      23m     
kube-storage-version-migrator              4.10.9    True        False         False      4m2s    
monitoring                                 4.10.9    True        False         False      40s     
network                                    4.10.9    True        False         False      3m22s   
node-tuning                                4.10.9    True        False         False      2m1s    
openshift-apiserver                        4.10.9    True        False         False      23m     
openshift-controller-manager               4.10.9    True        False         False      23m     
openshift-samples                          4.10.9    True        False         False      96s     
operator-lifecycle-manager                 4.10.9    True        False         False      22m     
operator-lifecycle-manager-catalog         4.10.9    True        False         False      23m     
operator-lifecycle-manager-packageserver   4.10.9    True        False         False      23m     
service-ca                                 4.10.9    True        False         False      3m59s   
storage                                    4.10.9    True        False         False      3m49s

Hopefully this gives a preview of how Hypershift can be used to deploy a cluster where the control plane is inside of an existing OpenShift cluster and then the workers are on real baremetal. As I mentioned earlier in the blog Hypershift will eventually be wrapped into Red Hat Advanced Cluster Management to make things easier for trying Hypershift out.

About the author

Benjamin Schmaus

Principle Product Manager - Edge Solutions

Benjamin Schmaus is a Red Hat Principle Product Manager for Edge Solutions. He has been involved with Linux since 1998 and has supported business environments in a variety of industries: retail, defense, software development, pharmacy, financial, higher education and K-12. His experience in those industries along with various positions within Red Hat have enabled him to have a broad and deep understanding of customer challenges. Most recently, he has been focused on enabling our customers at the edge by driving the Red Hat portfolio to deliver edge solutions that meet their ever diverse needs as they journey through modernization.

Read full bio

Browse by channel

Explore all channels

Platform products

Try & buy

Featured

By industry

Featured

Topics

Articles

More to explore

For customers

For partners

About us

Open source

Company details

Recommendations

Select a language

Select a language

A Guide to Red Hat Hypershift on Bare Metal

About the author

Benjamin Schmaus

More like this

Browse by channel

Products

Tools

Try, buy, & sell

Communicate

About Red Hat

Select a language

Red Hat legal and privacy links

Red Hat legal and privacy links