Leveraging the Red Hat Advanced Cluster Management Observability to Support OpenShift 3 to OpenShift 4 Migration

Since its release back in 2015, OpenShift Container Platform 3 gained a lot of traction in the market, with more than 1,000 organizations across industries and around the world using it. With OpenShift Container Platform 3, administrators individually deployed Red Hat Enterprise Linux (RHEL) hosts, and then installed OpenShift Container Platform within these hosts to form a Kubernetes cluster. Administrators were responsible for properly configuring these hosts and performing updates.

Main point: Cluster management was a challenge and configuration drift was a constant headache.

During Red Hat Summit 2019, OpenShift Container Platform 4 was introduced and presented a significant change in the way the OpenShift Container Platform clusters were deployed and managed. OpenShift Container Platform 4 included new technologies and functionality, such as Operators, machine sets, and the immutable operating system, Red Hat Enterprise Linux CoreOS (RHCOS), which are core to the operation of the cluster. This technology shift enabled clusters to self-manage - even self-remediate - some functions which previously were only performed by administrators. Key takeaways were platform stability and consistency, and simplified installation and scaling.

Despite the ease of setup and day-2 configurations from OpenShift 4, upgrading from 3.x to 4.x is not available and users must start a new installation of OpenShift Container Platform 4. Hence, migrating workloads from OpenShift 3 to OpenShift 4 requires some planning and possibly additional tooling, depending on the situation. Ideally, organizations can redeploy applications to the new environment making use of their CI/CD pipeline and subsequently copy any persistent volume data. If the latter is not an option, Red Hat also provides a solution called Migration Toolkit for Containers, which is described later in this blog.

Regardless, most organizations follow a similar migration journey, as depicted in the following diagram:

Migration considerations

When it comes to migration concerns, cluster administrators have to analyze storage, networking, authentication and a few other components throughout all the steps of the previously mentioned journey, in order to successfully migrate. In the next section, I explain how RHACM can help. I’ll focus on the Setup and Certification and make sure there is a concise environment setup to certify the migration process.

Environment Specifications

As mentioned previously, let's make use of an OpenShift cluster with RHACM installed, and have the source and target clusters as managed clusters. The environment used in this example have the following specifications:

• Red Hat OpenShift Container Platform 4.11 (or later) installed
  • 3 Control Plane Nodes (vCPU 4, RAM 16GB)
  • 3 Compute Nodes (vCPU 4, RAM 16GB)
  • At least one StorageClass defined
• Red Hat Advanced Cluster Management 2.5.1 (or later) installed
• S3-API compatible Object Storage

Installing the multicluster observability add-on

When installing the RHACM operator, you access some observability features right away - you can use the Search capability from the Overview page that shows some gauges and details about the managed clusters. This includes an add-on (search-collector) that runs on the managed cluster, which collects Kubernetes resources (CRDs, secrets, kinds, pods, storage, etc, just all the 'raw' resources that are in a cluster) and allows the user to search through these things across all the clusters. This is enabled by default, out-of-the-box.

Multicluster observability, which in this case, is specifically the collection and visualization of OpenShift with Kubernetes platform metrics and alerts, disabled by default on the RHACM hub cluster. Users must configure an objectStorage and have that ready for Thanos on the hub cluster to store all cluster platform metrics.

Observability is included in the RHACM installation, but must be enabled to use it. Follow along to enable observability:

Create the open-cluster-management-observability project on your RHACM hub cluster with the following command:

$ oc create namespace open-cluster-management-observability

After successfully creating the namespace, you need to retrieve your pull-secret. You can get it from your account in cloud.redhat.com or extract it from the openshift-config namespace. Run the following to extract your pull-secret from the openshift-config:

DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-`

Copy the pull-secret from the openshift-config namespace into the open-cluster-management-observability namespace. Run the following command:

oc create secret generic multiclusterhub-operator-pull-secret -n open-cluster-management-observability --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" --type=kubernetes.io/dockerconfigjson 

After adding your pull-secret, your observability components in this namespace have credentials that are used to access images in the registry. Now it’s time to configure a connection for our S3-compatible object store. In this example, like stated before, let's use AWS S3. From the RHACM web console, click on the add icon (+) in the console header to add your AWS secret, which Thanos retrieves as an S3 resource:

apiVersion: v1
kind: Secret
metadata:
  name: thanos-object-storage
  namespace: open-cluster-management-observability
type: Opaque
stringData:
  thanos.yaml: |
    type: s3
    config:
      bucket: YOUR_S3_BUCKET
      endpoint: s3.amazonaws.com
      insecure: false
      access_key: YOUR_ACCESS_KEY
      secret_key: YOUR_SECRET_KEY

You can now enable the multicluster observability add-on. From the OpenShift Container Platform console navigation menu, select Installed Operators and select the RHACM operator (open-cluster-management project). Navigate to MultiClusterObservability and click Create instance:

If everything goes smoothly, a Grafana link is displayed in the RHACM Overview page:

Import Source and Target Clusters

One of the greatest values that RHACM brings into this migration process is the ability of managing both source and target clusters in various ways. One is providing access to a central inventory of clusters and being able to display all of them in the Clusters view, and verifying they are connected to the hub cluster (with the ready status, ). In the following diagram, the target cluster is an OpenShift v4.10 cluster called aws-management-1 and the source cluster is an OpenShift v3.11 cluster named ocp3-11.

Note: When importing a cluster that is not created by OpenShift Container Platform, you need a multiclusterhub.spec.imagePullSecret defined. Check your MultiClusterHub CR from your RHACM installation to check if this field is set or follow the product documentation, Custom image pull-secret for steps on how to define the pull-secret.

Assuming you already have your OpenShift 3 clusters installed and active, import the source cluster from the Clusters page by clicking Import cluster. You are prompted to give it a name, a cluster set (optional for RBAC scenarios, and empty for this example), and select the Import mode: to choose to run commands manually through the oc cli for your OpenShift 3. See the following screen capture:

Select what’s most convenient for you and in a few minutes your cluster is fully imported to the Clusters view. You can follow these same steps to import source and target clusters.

In the case that you don’t have a fully setup OpenShift 4 cluster installed and functioning, RHACM can also help spin up a new one to migrate your workloads from OpenShift 3. Try selecting Create cluster from the RHACM Clusters page to view options that guide you to create your own OpenShift 4 cluster interactively and very easily. Keep in mind that credentials to the infrastructure provider are required for this automated installation. You can set up these credentials from the Credentials page. For more information, see Creating a cluster section for a list of supported infrastructure providers for this capability.

Visualize high-level indicators of migration health with built-in Grafana dashboards

As soon as you get your clusters and the Grafana instance installed, RHACM provides immediate value for the OpenShift Container Platform 3 to OpenShift Container Platform 4 migration path. If you access the Grafana link from the Overview page, navigate to the Grafana menu and select Search Dashboards icon (the magnifying glass). Notice there's a folder specifically for OpenShift 3 metrics, where the observability component delivers out-of-the-box dashboards that the customer can immediately start to observe, as illustrated in the following GIF:

This Grafana dashboard serves as an entry point for using customized dashboards for better visualization, but the sky is the limit for adjusting these visualizations to better serve your needs. Check RHACM docs with the possibility of creating custom dashboards, custom metrics, predefined rules and times of data collection for better resource consumption and configurable search. More than just providing monitoring for K8s resources, RHACM uses this Thanos, Prometheus, and Grafana setup to give you a way to be alerted of misused and/or failed components, and help you with root cause analysis, if needed.

Summary and links

Regardless if you’re using CI/CD to move your application artifacts and your workloads are just a click away from being placed in the target cluster or if you need a bunch of the other tooling to successfully migrate, the goal here is to introduce RHACM and its flexible observability component. The observability component can definitely help you certify that your environments are compatible with your migration strategy.

Check MTC Docs and our Migration webpage for more insights and requirements regarding networking, storage and other computing resources for a successful migration. Red Hat Training also provides a course to help prepare your teams for migration from OpenShift 3 to OpenShift 4 - DO328.

Get started today! Ask your Red Hat account team for more details on the OpenShift 3 to 4 upgrade offering for, or fill out this form and submit to get RHACM subscriptions at no cost to help you with your migration and get the most of RHACM and OpenShift 4.