GitOps quality of life tips

As a person who spends time playing computer role playing games (cRPGs), I see a lot of discussion of the quality of life (QoL) aspects of these games. Does the game have an overly complicated inventory system, a burdensome user interface or require you to manually buff your characters before each fight? Improvements in these areas enhance the gamer's quality of life.

Similarly, there are a variety of QoL tips that can improve the experience of developers and operators when interacting with Red Hat OpenShift GitOps. This blog looks at a few of my favorites.

Use annotation tracking

Argo CD, the upstream project of OpenShift GitOps, needs to track the resources in the cluster it has deployed when deploying applications. By default, both Argo CD and OpenShift GitOps use labels for the tracking method. While label tracking does the job, it only includes a minimal amount of information due to the Kubernetes 63-character limit.

This limitation manifests itself when Argo CD deploys resources that, in turn, generate additional resources. This is most commonly experienced deploying custom resources for operators. The operator will often copy all of the labels and annotations from the custom resource to the additional resources it creates, including the Argo CD tracking label.

While this behavior is generally desirable, copying the tracking label causes Argo CD to treat the application as out of sync. This occurs because Argo CD sees the tracking label and thinks it deployed the resource, but it cannot find the resource in Git since it was created by something else.

You can work around this behavior by adding the IgnoreExtraneous annotation to custom resources where this situation occurs, but it results in additional work and maintenance.

A better solution is to change the tracking method annotations. With annotation tracking, Argo CD can include additional information since annotations do not have the same character limit as labels. Argo CD uses this additional information to automatically weed out false positives.

To change the tracking method in OpenShift GitOps, simply add the following to your Argo CD custom resource:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
 name: openshift-gitops
 namespace: openshift-gitops
spec:
 resourceTrackingMethod: annotation
 ...

Note that you will likely need to re-sync all applications after making this change. Fortunately, you can use the Argo CD user interface to quickly sync everything in one go.

There is an issue upstream with changing the default tracking method from labels to annotations (or annotations+labels). However, caution is warranted to avoid disruption in existing installations (that's why it isn't the default).

Override automatic sync for App of Apps

App of Apps is a common pattern in OpenShift GitOps where you have an Argo CD application that deploys several other applications. This can be useful when you want to simplify a bootstrap process or deploy applications in a specific order via sync waves.

When troubleshooting one of the dependent applications, however, you cannot temporarily disable automatic sync on the application. If you attempt to do so, the parent application will simply reset it back to its state in Git. The workaround is a two-step process of disabling auto-sync on the parent application and disabling it in the child application.

Fortunately, you can use the ignoreDifferences feature in the parent application to simplify this process. To do so, add ignoreDifferences to your parent App of App as per the example below, taken from my cluster bootstrap application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: cluster-config-bootstrap
 namespace: openshift-gitops
spec:
 ...
 ignoreDifferences:
   - group: argoproj.io
     kind: Application
     managedFieldsManagers:
       - argocd-server
     jsonPointers:
       - /spec/syncPolicy/automated

This configuration tells Argo CD to ignore any changes made by the argocd-server service account to the spec.syncPolicy.automated field. The argocd-server service account is used by the pod providing the Argo CD user interface. Any changes made to an application from the UI will use this service account.

Thus, this change enables you to use the Argo CD UI to override the automated field as needed since Argo CD will now ignore this difference.

I cannot take credit for this tip. I stumbled across it somewhere and have since lost where I got it. Kudos to the original contributor, whomever they may be!

Easily test custom health checks

I firmly believe in writing custom health checks to support the custom resources that various operators make available. These custom health checks enable Argo CD to understand the state of the resource (Healthy, Progressing, Degraded, etc.) and are required if you want to use sync waves with the resource.

The custom health checks are written in the Lua programming language. Testing complex checks in Argo CD can be challenging and time-consuming. You can use the argocd command line interface to quickly and easily test these health checks independent of an Argo CD deployment on a cluster.

To test the resource, you need a YAML file copy of the argocd-cm configmap and a copy of the resource you want to check. In OpenShift GitOps, the argocd-cm configmap is automatically generated in the namespace where your Argo CD instance is located. For example, to get the configmap for the default openshift-gitops instance, you can do the following to retrieve it and save it as a file:

oc get configmap argocd-cm -n openshift-gitops -o yaml > argocd-cm.yaml

Use a similar command when you want to retrieve the resource. Once you have these two files, you can test the health check with the following command:

argocd admin settings resource-overrides health <resource-yaml> --argocd-cm-path <argocd-cm-yaml>

While debugging a problematic health check for a subscription, I ran the command with the following results:

$ argocd admin settings resource-overrides health ./bad-sub.yaml --argocd-cm-path ./argocd-cm.yaml
INFO[0000] Starting configmap/secret informers
INFO[0000] Configmap/secret informer synced
STATUS: Healthy
MESSAGE: 1: CatalogSourcesUnhealthy | False

You can see the health check was successful and returned a Healthy status. Also, note the MESSAGE block. This is the message portion of the health check. It is useful to include additional information that could help teams to better understand why a specific status was returned.

Use kubectl-neat to export clean resources

It's a common GitOps practice to deploy things into a cluster initially using a GUI like the OpenShift console or tools like oc new-app and then export the resources into a Git repository.

Resources in a Kubernetes cluster tend to have a lot of cruft in them (managed fields, status, etc.) that is painful to remove from the exported resources manually. A great tool for handling this automatically is the kubectl-neat plugin.

This tool is a plugin to kubectl and oc. To install it, get the binary from the GitHub repository releases and put it somewhere on your path. Then, when fetching a resource from the cluster, you can pipe the results to neat. Here's an example:

oc get service my-service -o yaml | oc neat > my-service.yaml

Use Global projects

In multi-tenant GitOps, the recommended practice is to use Argo CD Projects (AppProject) to manage access privileges and restrictions for the tenants. The AppProject enables the Argo CD administrator to restrict access to specific resources via allow- and denylisting, define team roles for role-based access control (RBAC), sync windows and more.

It's not uncommon in larger multi-tenant Argo CD instances to have many different projects for the various teams. Certain configuration items are often common across all projects or specific subsets of projects.

Rather than repeat this configuration over and over, you can define it once in a Global project instead. The other benefit is that changes to the global project will automatically be applied to the dependent projects, reducing the overall maintenance burden.

Here is an example of a Global project I use with my tenants in a cluster-scoped OpenShift GitOps instance. In this example, I denylisted all cluster-scoped resources along with specific namespace-scoped resources. The namespace-scoped resources remove some common ones, like ResourceQuota and LimitRange. It also removes some resources the OpenShift GitOps operator provides by default in cluster-scoped instances. It looks like this:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
 name: global
spec:
 description: Global Configuration
 clusterResourceBlacklist:
 - group: '*'
   kind: '*'
 namespaceResourceBlacklist:
 - group: ''
   kind: Namespace
 - group: ''
   kind: ResourceQuota
 - group: ''
   kind: LimitRange
 - group: operators.coreos.com
   kind: '*'
 - group: operator.openshift.io
   kind: '*'
 - group: storage.k8s.io
   kind: '*'
 - group: machine.openshift.io
   kind: '*'
 - group: machineconfiguration.openshift.io
   kind: '*'
 - group: compliance.openshift.io
   kind: '*'

The OpenShift GitOps operator does not have direct support for configuring the match expression needed for other projects to reference it. However, you can add the configuration using the extraConfig capability in the operator, as shown in this snippet:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
 name: argocd
spec:
 extraConfig:
   globalProjects: |-
     - labelSelector:
         matchExpressions:
           - key: argocd.argoproj.io/project-inherit
             operator: In
             values:
               - global
       projectName: global
 ...

This Global project can then be referenced in other Projects by adding a label to it:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
 name: product-catalog
 labels:
   argocd.argoproj.io/project-inherit: global
spec:
 ...

Wrap up

This blog went over several ways to improve your overall quality of life when working with OpenShift GitOps and Argo CD.