Deploying bare metal at the edge with Red Hat Advanced Cluster Management and zero touch provisioning

Zero touch provisioning (ZTP) is a technique that deploys new edge sites with declarative configurations of bare-metal equipment. Template or overlay configurations install Red Hat OpenShift features required for edge workloads. All configurations are declarative and managed with Red Hat Advanced Cluster Management for Kubernetes.

ZTP leverages Red Hat Advanced Cluster Management and GitOps approaches to manage edge sites remotely. The service is optimized for single-node OpenShift (SNO) and three-node OpenShift. I will deploy a SNO scenario for this example.

Configure channels in Red Hat Advanced Cluster Management that contain URLs and secrets to connect to Git and subscriptions that have Git paths, branches or tags. The configuration will be generated using kustomize, which will merge the base templates with the proper overlay for a specific site.

The base directory is used as a template and contains all global information. Inside the overlay folder is a subfolder for each SNO containing the specific configuration for the given site.

Use the files in my GitHub repository as examples and templates.

SNO installation steps

The following steps are executed automatically via an assisted installer and Red Hat Advanced Cluster Management. The following list is only for explanation, and no manual actions are required for the SNO installation:

1. The assisted installer generates an ISO image used for discovery.
2. The discovery ISO is mounted using the Redfish API.
3. Boot the host with the discovery ISO. DNS services resolve the URL exposed by the assisted installer (assisted-image-service route). The rootfs image is downloaded.
4. By default, the discovery network is configured via DHCP.
5. The agent service on the host starts the discovery and registers with the Red Hat Advanced Cluster Management Hub. The AgentClusterInstall and ClusterDeployment CRs define install configurations like cluster version, name, networking, encryption, SSH and proxies.
6. The cluster installation begins. When complete, the ManagedCluster and KlusterletAddonConfig CRs trigger the registration of the cluster to Red Hat Advanced Cluster Management.

Prerequisites

1. Enable the Redfish API at the remote location for hardware.
2. At least one cluster hub for management must exist (Red Hat Advanced Cluster Management as a single pane of glass).
3. Configure the network infrastructure (DNS, DHCP and NTP) and network firewall rules for connectivity.
4. One or more Git repos containing manifests for the following:
   1. Day 1 ZTP deployment (site config custom resources (CRs)).
   2. Day 2 infrastructure configuration (governance policies).
   3. Application-related manifests (using Red Hat Advanced Cluster Management application subscriptions or Red Hat GitOps).
5. A vault for keeping external secret operator secrets (only if you want to manage secrets with the external secret operator).

Git structure

The Git structure for SNO provisioning is divided into base and overlays. This uses the best features of the kustomize tool, avoiding duplication of common information. The following is an example:

Installing Red Hat Advanced Cluster Management

1. Install the operator:

---
apiVersion: v1
kind: Namespace
metadata:
 name: open-cluster-management
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
 name: open-cluster-management
 namespace: open-cluster-management
spec:
 targetNamespaces:
 - open-cluster-management 
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: advanced-cluster-management
 namespace: open-cluster-management
spec:
 channel: release-2.7
 installPlanApproval: Automatic
 name: advanced-cluster-management
 source: redhat-operators
 sourceNamespace: openshift-marketplace
---
apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
 name: multiclusterhub
 namespace: open-cluster-management
spec: {}

2. Define the AgentServiceConfig resource and the Metal3 provisioning server using the code below:

---
apiVersion: agent-install.openshift.io/v1beta1
kind: AgentServiceConfig
metadata:
 name: agent
spec:
 databaseStorage:
   accessModes:
   - ReadWriteOnce
   resources:
     requests:
       storage: <db_volume_size> # For Example: 10Gi
 filesystemStorage:
   accessModes:
   - ReadWriteOnce
   resources:
     requests:
       storage: <fs_volume_size> # For Example: 100Gi
 imageStorage:
   accessModes:
   - ReadWriteOnce
   resources:
     requests:
       storage: <image_volume_size>  # For Example: 50Gi

3. Proceed with the configuration of the Metal3 provisioning server resource:

apiVersion: metal3.io/v1alpha1
kind: Provisioning
metadata:
 name: metal3-provisioning
spec:
 provisioningNetwork: Disabled # [ Managed | Unmanaged | Disabled ]
 virtualMediaViaExternalNetwork: true
 watchAllNamespaces: true

4. Set the Red Hat Advanced Cluster Management role-based access control (RBAC) configuration:

oc adm policy add-cluster-role-to-user --rolebinding-name=open-cluster-management:subscription-admin open-cluster-management:subscription-admin kube:admin
oc adm policy add-cluster-role-to-user --rolebinding-name=open-cluster-management:subscription-admin open-cluster-management:subscription-admin system:admin

Example use case

ztp/
├── README.md
├── environments
│   ├── base
│   │   ├── 00-namespace.yaml
│   │   ├── 01-bmh.yaml
│   │   ├── 02-pull-secret.yaml
│   │   ├── 03-agentclusterinstall.yaml
│   │   ├── 04-clusterdeployment.yaml
│   │   ├── 05-klusterletaddonconfig.yaml
│   │   ├── 06-managedcluster.yaml
│   │   ├── 07-infraenv.yaml
│   │   └── kustomization.yaml
│   └── overlays
│       ├── sno1
│       │   ├── 00-namespace.yaml
│       │   ├── 01-bmh.yaml
│       │   ├── 02-pull-secret.yaml
│       │   ├── 03-agentclusterinstall.yaml
│       │   ├── 04-clusterdeployment.yaml
│       │   ├── 05-klusterletaddonconfig.yaml
│       │   ├── 06-managedcluster.yaml
│       │   ├── 07-infraenv.yaml
│       │   └── kustomization.yaml
│       └── sno2
│           ├── 00-namespace.yaml
│           ├── 01-bmh.yaml
│           ├── 02-pull-secret.yaml
│           ├── 03-agentclusterinstall.yaml
│           ├── 04-clusterdeployment.yaml
│           ├── 05-klusterletaddonconfig.yaml
│           ├── 06-managedcluster.yaml
│           ├── 07-infraenv.yaml
│           └── kustomization.yaml
└── policies
   ├── policy-htpasswd.yaml
   ├── policy-lvms.yaml
   └── policy-registry.yaml

Using OpenShift GitOps and SiteConfig custom resource definitions (CRDs)

siteconfig
├── environments
|   ├── site1-sno-du.yaml
|   ├── site2-standard-du.yaml
|   └── extra-manifest
|       └── 01-example-machine-config.yaml
└── policies
   ├── policy-htpasswd.yaml
   ├── policy-lvms.yaml
   └── policy-registry.yaml    

Manifest

To speed up the provisioning of the SNO on a single system, I did not install OpenShift GitOps. Therefore, I could not use a single SiteConfig manifest, as shown below:

apiVersion: ran.openshift.io/v1
kind: SiteConfig
metadata:
 name: "<site_name>"
 namespace: "<site_name>"
spec:
 baseDomain: "example.com"
 pullSecretRef:
   name: "assisted-deployment-pull-secret" 
 clusterImageSetNameRef: "openshift-4.10" 
 sshPublicKey: "ssh-rsa AAAA..." 
 clusters:
 - clusterName: "<site_name>"
   networkType: "OVNKubernetes"
   clusterLabels: 
     common: true
     group-du-sno: ""
     sites : "<site_name>"
   clusterNetwork:
     - cidr: 1001:1::/48
       hostPrefix: 64
   machineNetwork:
     - cidr: 1111:2222:3333:4444::/64
   serviceNetwork:
     - 1001:2::/112
   additionalNTPSources:
     - 1111:2222:3333:4444::2
   #crTemplates:
   #  KlusterletAddonConfig: "KlusterletAddonConfigOverride.yaml" 
   nodes:
     - hostName: "example-node.example.com" 
       role: "master"
       #biosConfigRef:
       #  filePath: "example-hw.profile" 
       bmcAddress: idrac-virtualmedia://<out_of_band_ip>/<system_id>/ 
       bmcCredentialsName:
         name: "bmh-secret" 
       bootMACAddress: "AA:BB:CC:DD:EE:11"
       bootMode: "UEFI" 
       rootDeviceHints:
         wwn: "0x11111000000asd123"
       cpuset: "0-1,52-53"
       nodeNetwork: 
         interfaces:
           - name: eno1
             macAddress: "AA:BB:CC:DD:EE:11"
         config:
           interfaces:
             - name: eno1
               type: ethernet
               state: up
               ipv4:
                 enabled: false
               ipv6: 
                 enabled: true
                 address:
                 - ip: 1111:2222:3333:4444::aaaa:1
                   prefix-length: 64
           dns-resolver:
             config:
               search:
               - example.com
               server:
               - 1111:2222:3333:4444::2
           routes:
             config:
             - destination: ::/0
               next-hop-interface: eno1
               next-hop-address: 1111:2222:3333:4444::1
               table-id: 254

The SiteConfig CRD simplifies cluster deployment by generating the following CRs based on a SiteConfig CR instance:

• AgentClusterInstall
• ClusterDeployment
• NMStateConfig
• KlusterletAddonConfig
• ManagedCluster
• InfraEnv
• BareMetalHost
• HostFirmwareSettings
• ConfigMap for extra-manifest configurations

I created these files in the Git repo, as described below:

---
apiVersion: v1
kind: Namespace
metadata:
 annotations:
   apps.open-cluster-management.io/do-not-delete: 'true'  
 labels:
   kubernetes.io/metadata.name: sno1
 name: sno1
---
apiVersion: v1
data:
 password: XXXXXXXXXXXXX
 username: XXXXXXXXXXXXX
kind: Secret
metadata:
 name: bmc-credentials
 namespace: sno1
 annotations:
   apps.open-cluster-management.io/do-not-delete: 'true' 
type: Opaque
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
 annotations:
   bmac.agent-install.openshift.io/hostname: sno1
   inspect.metal3.io: disabled
   apps.open-cluster-management.io/do-not-delete: 'true'
 annotations:
   apps.open-cluster-management.io/do-not-delete: 'true'  
 labels:
   infraenvs.agent-install.openshift.io: sno1
   cluster-name: sno1
 name: sno1
 namespace: sno1
spec:
 automatedCleaningMode: disabled
 online: true
 bmc:
   disableCertificateVerification: true
   address: redfish-virtualmedia+https://example.com/redfish/v1/Systems/1/
   credentialsName: bmc-credentials
 bootMACAddress: 5c:ed:8c:9e:23:1c
 #rootDeviceHints:
 #  hctl: 0:0:0:0
---
apiVersion: v1
data:
 .dockerconfigjson: XXXXXXXXXXX
kind: Secret
metadata:
 annotations:
   apps.open-cluster-management.io/do-not-delete: 'true'  
 name: pull-secret
 namespace: sno1
type: kubernetes.io/dockerconfigjson
---
apiVersion: extensions.hive.openshift.io/v1beta1
kind: AgentClusterInstall
metadata:
 name: sno1
 namespace: sno1
 annotations:
   apps.open-cluster-management.io/do-not-delete: 'true'  
   agent-install.openshift.io/install-config-overrides: '{"networking":{"networkType":"OVNKubernetes"}}'   
spec:
 clusterDeploymentRef:
   name: sno1
 imageSetRef:
   name: img4.12.14-x86-64-appsub
 networking:
   clusterNetwork:
   - cidr: 100.64.0.0/15
     hostPrefix: 23
   serviceNetwork:
   - 100.66.0.0/16
   userManagedNetworking: true  
   machineNetwork:
   - cidr: 10.17.0.0/28
 provisionRequirements:
   controlPlaneAgents: 1    
 sshPublicKey: ssh-rsa XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
---
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
 name: sno1
 namespace: sno1
 annotations:
   apps.open-cluster-management.io/do-not-delete: 'true' 
spec:
 baseDomain: example.com
 clusterInstallRef:
   group: extensions.hive.openshift.io
   kind: AgentClusterInstall
   version: v1beta1
   name: sno1
 controlPlaneConfig:
   servingCertificates: {}    
 clusterName: sno1
 platform:
   agentBareMetal:
     agentSelector:
       matchLabels:
         cluster-name: sno1
 pullSecretRef:
   name: pull-secret
---
apiVersion: agent.open-cluster-management.io/v1
kind: KlusterletAddonConfig
metadata:
 annotations:
   apps.open-cluster-management.io/do-not-delete: 'true'
 name: sno1
 namespace: sno1
spec:
 clusterName: 
 clusterNamespace: 
 clusterLabels:
   cloud: hybrid
 applicationManager:
   enabled: true
 policyController:
   enabled: true
 searchCollector:
   enabled: true
 certPolicyController:
   enabled: true
 iamPolicyController:
   enabled: true
---
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
 annotations:
   apps.open-cluster-management.io/do-not-delete: 'true'
   apps.open-cluster-management.io/reconcile-option: replace
 labels:
   name: sno1
   cluster-name: sno1
   base-domain: example.com
 name: sno1
spec:
 hubAcceptsClient: true
 leaseDurationSeconds: 120 
---
apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
 annotations:
   apps.open-cluster-management.io/do-not-delete: 'true'  
 labels:
   agentclusterinstalls.extensions.hive.openshift.io/location: sno1
   networkType: dhcp
 name: sno1
 namespace: sno1
spec:
 clusterRef:
   name: sno1
   namespace: sno1
 agentLabels:
   agentclusterinstalls.extensions.hive.openshift.io/location: sno1
 cpuArchitecture: x86_64
 ipxeScriptType: DiscoveryImageAlways
 pullSecretRef:
   name: pull-secret      
 sshAuthorizedKey: ssh-rsa XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

If you need to enter custom configurations at Day 1, you can provide them as a ConfigMap to the AgentClusterInstall. Below is an example:

---
apiVersion: extensions.hive.openshift.io/v1beta1
kind: AgentClusterInstall
metadata:
 name: sno1
 namespace: sno1
spec:
 manifestsConfigMapRefs:
   - name: sno1-customconfig
---
apiVersion: v1
kind: ConfigMap
metadata:
 name: sno1-customconfig
 namespace: sno1
data:
 htpasswd-authentication.yaml: |
   apiVersion: config.openshift.io/v1
   kind: OAuth
   metadata:
   name: cluster
   spec:
   identityProviders:
       - htpasswd:
           fileData:
           name: htpass-secret
       mappingMethod: claim
       name: HTPasswd
       type: HTPasswd

Wrap up

Zero touch provisioning simplifies bare-metal deployments at edge sites by combining the power of Red Hat OpenShift, Red Hat Advanced Cluster Management and GitOps. This demonstration used a single system to deploy a small SNO cluster as a proof of concept.

Build on this design in your own environment to learn more about deploying these features in your own network.