Deploying multicluster Kubernetes applications with GitOps: What architects need to know

Greater adoption of Kubernetes naturally leads to more multicluster Kubernetes applications, and architects must know how to address the challenges that come with these environments.

One of these is how to standardize and centralize essential functions across all environments, including:

• Application lifecycle management
• Governance
• Observability
• Multicluster lifecycle management

This article demonstrates how Red Hat Advanced Cluster Management (ACM) for Kubernetes addresses this challenge by showing an example application deployment. It also shows the two possible ways (at the time of writing) to deploy GitOps-based applications using ACM:

1. Managing apps with Git repositories: Deploying resources from Git repositories using the ACM web console's built-in mechanism
2. Configuring managed clusters for OpenShift GitOps operator and Argo CD: Registering a set of one or more ACM-managed clusters to an instance of Argo CD before deploying the application to those clusters

The solution relies on ACM for Kubernetes 2.5 and OpenShift GitOps 1.5.4 (or Argo CD).

In addition, you need OpenShift CLI version 4.3.0 or greater for Kustomize.

Finally, you can take advantage of the Web Terminal Operator if you prefer to use only a web browser for the entire deployment.

[ Learn how to build a flexible foundation for your organization. Download An architect's guide to multicloud infrastructure. ]

Prerequisites

The following prerequisites must be in place to follow along with the steps in this article:

• Install Argo CD or the Red Hat OpenShift GitOps operator on your Red Hat ACM for Kubernetes platform. I've included a policy to install the OpenShift Pipelines operator in my repo for this article. 
• Import one or more managed clusters.
• You'll need the ACM subscription administrator role to leverage all the automation components from the policies. (You need it even if you hold the cluster-admin role.) 

Implementing GitOps

The following image shows how ACM implements GitOps with the application subscription model.

Image

The GitOps example in this tutorial is highlighted in the diagram above.

[ Download The path to GitOps to put the right tools, workflows, and structures in place to enable a complete GitOps workflow.  ] 

Subscriptions (subscription.apps.open-cluster-management.io) allow clusters to subscribe to the following source repositories (channels):

• Git repository
• Helm release registry
• Object storage repository

Image

Subscriptions use channels to identify new or updated resource templates. The subscription operator can download templates directly from the storage location and deploy them to targeted managed clusters without checking the hub cluster first. With a subscription, the subscription operator can monitor the channel for new or updated resources instead of using the cluster with the ACM Hub.

Deploy an application using the ACM web console

The administrator doing the deployment must have an access level that can create applications. You can only perform actions that are assigned to your role. The cluster-admin role can deploy the application, but if you need more granular access, review ACM's role-based access control documentation.

1. From the ACM web console, select Applications → Create application → Subscription.

Image

Select YAML: On to view the YAML in the console as you create your application (or you can import the YAML directly instead of using the interactive web form).

Image

2. The application in this example is called pacman. Enter the following values in the specified fields:

• Name: pacman-app
• Namespace: pacman

Select a namespace from the list. If you have the correct access role, you can also create a namespace using a valid Kubernetes name.

3. Choose Git from the list of repositories.

Image

Image

If you want to learn more about the different options and how they work, refer to the Managing apps with Git repositories documentation.

[ For more on OpenShift, Argo CD, and Tekton, download the complimentary eBook Getting GitOps. ]

4. Click Save, and ACM redirects you to the app dashboard, where you can see all the objects that compose the app and if they are correctly deployed.

Image

Image

Deploy Argo CD ApplicationSets in ACM

An ApplicationSet is an Argo CD subproject that adds multicluster support for Argo CD applications. You can create an ApplicationSet from the product console editor, as shown below.

[ A complimentary guide from Red Hat: The automation architect's handbook. ] 

Configure managed clusters for Argo CD

There are two approaches to configuring managed clusters for Argo CD:

1. Follow the step-by-step procedure in the official Configuring managed clusters for OpenShift GitOps operator and Argo CD documentation.
2. Install the policies in my repo for this article.

NOTE: Remember to grant your user the subscription administrator role before installing the policies.

If you prefer to continue with the automated approach, once the local-cluster complies with the policies, add the clusters you want to the all-openshift-clusters ClusterSet created by the policy-openshift-gitops-acm-integration policy.

You can choose any name for the ClusterSet, as ACM supports integration with non-OpenShift clusters. I used all-openshift-clusters in this case because the ClusterSet consists of OpenShift clusters alone. This is for simplicity, since the labelSelector of the Placement object, in charge of integrating the clusters in ArgoCD, is based on the label with the key/value vendor=OpenShift, which is the default label with any OpenShift cluster.

Image

1. Select Clusters → Cluster sets → Manage resource assignments.

Image

2. In the next screen, select the clusters, click Review, and then Save.

Image

You can confirm the assignment in two ways. The first method is by using the command interface:

$ argocd cluster list

[ Learn more about cloud-native development in the eBook Kubernetes Patterns: Reusable elements for designing cloud-native applications. ]

The other approach is by accessing the Argo CD web console at Settings → Clusters:

Image

The two managed clusters (highlighted in the screenshot) have been added to the list of clusters managed by Argo CD.

Deploy Argo CD ApplicationSets from ACM web console

Once you have enabled integration between OpenShift GitOps and ACM, you can deploy ApplicationSets with the ACM web console.

Start by deploying the ApplicationSet:

1. From the ACM web console, choose Applications → Create application → Argo CD ApplicationSet.

2. Navigate through the different steps on the web console's interactive form and fill in the relevant data. Here is an example:

Image

Note that the destination namespace could be any value. It's not necessary for the ApplicationSet itself (but mandatory for the ACM form). It is better to leave the value bgdk to avoid misinterpretation because bgdk is the namespace that will generate the app deployment itself.

3. Clicking Create, and you are presented with the application. To check it, go to the Applications menu:

Image

Drill down to see its evolution and whether all the elements are correctly deployed in both clusters by clicking on its name.

Image

Image

In this case, select the Route object and check that it is deployed in both clusters and that you have direct access to the URL or can edit the object's YAML.

You can do the same with the Argo CD web console:

1. Check the general status.

Image

2. Drill down by clicking on one of the instances (one per cluster).

Image

Deploy with a YAML file

You can do the same as in the previous section using either the web console or the command line.

Create a YAML file with these contents:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: acm-appsets
  namespace: openshift-gitops
spec:
  generators:
    - clusterDecisionResource:
        configMapRef: acm-placement
        labelSelector:
          matchLabels:
            cluster.open-cluster-management.io/placement: acm-appset-placement
        requeueAfterSeconds: 180
  template:
    metadata:
      name: 'acm-appset-'
    spec:
      destination:
        namespace: bgdk
        server: ''
      project: default
      source:
        path: apps/bgd/overlays/bgdk
        repoURL: 'https://github.com/RedHat-EMEA-SSA-Team/ns-apps/'
        targetRevision: single-app
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Here is the command-line example:

$ oc apply -f rhacm/deploy/acm-appset.yaml

Other methods include using the OpenShift web console (+Add → Import YAML), Argo CD web console (+NEW APP → EDIT AS YAML), or ACM web console (with the Create application menu).

Wrap up

At the beginning of this article, I outlined the challenge of deploying multicluster applications (whether in private, public cloud, or hybrid environments) while standardizing and centralizing critical functions across all environments. This includes:

• Application lifecycle management
• Governance
• Observability
• Multicluster lifecycle management

I have given more attention to the application lifecycle and governance part (thanks to ACM policies) in this article. For insight on multicluster management and observability, please consult the official documentation on these topics:

• Multicluster lifecycle management
• Observability (Grafana, Prometheus + AlertManager)