In this article, we describe an approach that OpenShift administrators can use to apply and manage Day 2 configuration for any OpenShift cluster using features available in Red Hat Advanced Cluster Management (RHACM). By Day 2 configuration, the following aspects of OpenShift cluster configuration should be complete by the cluster administration for production:

• Deleting Kubernetes administrator (kubeadmin) account
• Enhancing ETCD configuration (encryption, disk placement)
• Enabling cluster storage configuration (configure storage provisioner, storage classes, etc..)
• Complete configuration of various cluster components:
  • Ingress Controller
  • OpenShift monitoring and alerting, including adding custom alerts configuration
  • MachinePools configuration
  • Cluster-wide proxy configuration
  • Network policies configuration

The approach we propose allows administrators to define cluster configuration as a set of YAML manifests describing the desired state of Kubernetes cluster resources. These configuration resources are continuously synced from a Git repository and applied to the cluster. These YAML manifests can be simple Kubernetes YAML manifests or can take advantage of the powerful templating features of RHACM, providing a way for administrators to define consistent (yet unique if required) configurations to many combinations of OpenShift clusters.

This approach is implemented using the following Governance features, available in RHACM:

• RHACM Configuration policies - to apply configuration on a managed cluster.
• RHACM Policy Generator – to generate RHACM Configuration policies based on a set of Kubernetes YAML manifests.
• RHACM Policy templating – to customize RHACM Configuration policies and Kubernetes YAML manifests for an OpenShift cluster.

RHACM Configuration policies are policies can be used to configure any Kubernetes resource across all managed clusters. RHACM Configuration policies not only enforce the configuration, but also can inform administrators and users if there is a drift in the configuration without changing it. This might be useful for some cases where there is a need to avoid touching resources on many clusters in an automated way. For example, if there is a mistake while modifying storage configuration, it might ruin the data.

To create a RHACM Configuration policy the following Kubernetes YAML manifests need to be created for each policy:

• Policy manifest containing Kubernetes resources to apply to a managed cluster, and how the policy should be applied (i.e., inform or enforce mode)
• PlacementBinding Kubernetes manifest to bind a policy to a placement
• PlacementRule Kubernetes manifest – to define cluster labels

This requirement might seem complicated when multiple policies and labels need to be combined to cover all aspects for configuration of the OpenShift cluster. However, the Policy Generator in RHACM can be used to generate all manifests described in the previous list (Configuration policy, PlacementBinding and PlacementRule) on the fly in RHACM. As an input for Configuration policy you only need the path with YAML manifests that you want to apply to a managed cluster.

RHACM Policy Generator is a tool embedded in RHACM since version 2.4. It builds RHACM Configuration policies from Kubernetes YAML files and works as a Kustomize generator plugin – but does not add any Kustomize complication to the solution. To generate Configuration policies, a user needs to create a simple YAML manifest (kind: PolicyGenerator) where they can describe how their policy or multiple policies should look, and provide a list with paths to the Git repository where Kubernetes manifests are located. After creating the manifest, RHACM Application and Subscription resource can be used to generate all policies by pointing to the Git repository with this YAML file.

However, in a real environment, cluster administrators might need to modify a Kubernetes manifest before applying it to a cluster, to adjust the configuration for that particular cluster. View the following configuration examples:

• Use the cluster hostname or domain in a manifest – for example, you can configure certificates on OpenShift API and Ingress endpoints.
• Set the number of ingress replicas in Ingress Controller configuration based on cluster type. For example, set replicas to 5 for any cluster in a cloud and set replicas to 3 if a cluster is on-premise (OnPrem), or even set it per cluster.
• Precisely define Red Hat OpenShift Data Foundation (storage) configuration.
• Set the operator installation approval to Manual or Automatic based on the cluster type (for example, automatically deploy an operator if a cluster is for development, but require approval to install it for a production cluster), and many more.

To achieve this, Kubernetes tools like Helm or Kustomize are often used, but using these tools can sometimes make the solution too complicated and require cluster administrators to generate or support additional manifests (such as creating Helm charts or supporting multiple Kustomize overlays).

The Policy Templating feature (available in RHACM since version 2.3) enables users to define configuration policies dynamically. With this feature, users can create adaptable configuration policies that can be applied to multiple clusters at scale as a single customized policy; rather than having to create multiple policies for each managed cluster that requires slightly different behavior. Users can utilize Golang template syntax (similar to Helm) to define values dynamically in configuration policies or Kubernetes YAML manifests used by the Policy Generator.

To define values in Kubernetes YAML manifests before they are applied to a managed cluster using various template features that are supported, such as getting values from ConfigMaps, Secrets, labels, other resources, etc. Detailed descriptions of these functions can be found in the RHACM Template function documentation. By default, templates are customized to the target managed cluster. In addition, RHACM supports hub cluster templates that can retrieve values from resources on the RHACM hub cluster to generate Kubernetes manifest on a managed cluster, but any referencing Kubernetes objects must be in the same namespaces as Configuration policies on the hub cluster). The hub cluster templates also support a special variable .ManagedClusterName, which resolves to the name of the target managed cluster and can be used to customize manifests that should be unique to a particular managed cluster.

In summary, to configure OpenShift clusters managed by RHACM the following items are required:

• Kubernetes YAML manifests that describe the desired state of Kubernetes resources. These manifests can be used to set the desired state of the resource or to patch it.
• A Git repository that contains all these YAML manifests.
• Some of these manifests are templated so they can be customized per OpenShift cluster or group of OpenShift clusters.
• At least one YAML file describing how to apply these manifests.

One more RHACM feature can be used to customize the configuration of OpenShift clusters – cluster labeling. RHACM supports administrators adding labels to managed clusters and these labels (by utilizing PlacementRule and PlacementBindings resources) determine what configuration policies are applied to the managed cluster. For example, if a cluster needs specific configuration for a cloud environment, administrators can set the label platform-config=cloud on the cluster - and RHACM then applies the manifests relevant only for clusters running in a cloud. Similarly, if the label is set to platform-config=onprem, RHACM applies the manifests relevant only to clusters running on-premise.

When an administrator adds a label to a managed cluster, the RHACM agent creates a corresponding Kubernetes resource on the managed cluster (cluster-scoped), clusterClaim. This resource can be used in templated manifests to generate configuration settings based on the labels assigned to the cluster. After a cluster has been provisioned or discovered and added as a managed cluster to RHACM, it applies default labels containing some useful information about the cluster, such as the OpenShift version or the cloud platform infrastructure it is running on. Detailed information about these labels or clusterClaims can be found in RHACM documentation, Cluster claims. These labels also can be used to generate configuration settings properly for a managed cluster. View the following screen capture:

Conclusion

Red Hat Advanced Cluster Management for Kubernetes (RHACM) allows cluster administrators to implement the GitOps approach for configuring any managed cluster whether provisioned by RHACM or imported manually, with minimal effort. In the this article, we briefly described RHACM features and embedded tools that cluster administrators can leverage to configure OpenShift clusters. For more detailed description of how to implement various use cases of how cluster administrators can use YAML manifests, cluster labels, and templated policies to manage OpenShift cluster configuration. Check out some examples in the vadimzharov/ocp-gitops-acm repository.