As a person who spends time playing computer role playing games (cRPGs), I see a lot of discussion of the quality of life (QoL) aspects of these games. Does the game have an overly complicated inventory system, a burdensome user interface or require you to manually buff your characters before each fight? Improvements in these areas enhance the gamer's quality of life.
Similarly, there are a variety of QoL tips that can improve the experience of developers and operators when interacting with Red Hat OpenShift GitOps. This blog looks at a few of my favorites.
Use annotation tracking
Argo CD, the upstream project of OpenShift GitOps, needs to track the resources in the cluster it has deployed when deploying applications. By default, both Argo CD and OpenShift GitOps use labels for the tracking method. While label tracking does the job, it only includes a minimal amount of information due to the Kubernetes 63-character limit.
This limitation manifests itself when Argo CD deploys resources that, in turn, generate additional resources. This is most commonly experienced deploying custom resources for operators. The operator will often copy all of the labels and annotations from the custom resource to the additional resources it creates, including the Argo CD tracking label.
While this behavior is generally desirable, copying the tracking label causes Argo CD to treat the application as out of sync. This occurs because Argo CD sees the tracking label and thinks it deployed the resource, but it cannot find the resource in Git since it was created by something else.
You can work around this behavior by adding the IgnoreExtraneous annotation to custom resources where this situation occurs, but it results in additional work and maintenance.
A better solution is to change the tracking method annotations. With annotation tracking, Argo CD can include additional information since annotations do not have the same character limit as labels. Argo CD uses this additional information to automatically weed out false positives.
To change the tracking method in OpenShift GitOps, simply add the following to your Argo CD custom resource:
apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: openshift-gitops
namespace: openshift-gitops
spec:
resourceTrackingMethod: annotation
...
Note that you will likely need to re-sync all applications after making this change. Fortunately, you can use the Argo CD user interface to quickly sync everything in one go.
![Screenshot of the "Sync Apps" button in the Argo CD user interface](/rhdc/managed-files/styles/wysiwyg_full_width/private/gitops-tips-img1.png.webp?itok=GB2Vk4_A)
There is an issue upstream with changing the default tracking method from labels to annotations (or annotations+labels). However, caution is warranted to avoid disruption in existing installations (that's why it isn't the default).
Override automatic sync for App of Apps
App of Apps is a common pattern in OpenShift GitOps where you have an Argo CD application that deploys several other applications. This can be useful when you want to simplify a bootstrap process or deploy applications in a specific order via sync waves.
When troubleshooting one of the dependent applications, however, you cannot temporarily disable automatic sync on the application. If you attempt to do so, the parent application will simply reset it back to its state in Git. The workaround is a two-step process of disabling auto-sync on the parent application and disabling it in the child application.
Fortunately, you can use the ignoreDifferences
feature in the parent application to simplify this process. To do so, add ignoreDifferences
to your parent App of App as per the example below, taken from my cluster bootstrap application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cluster-config-bootstrap
namespace: openshift-gitops
spec:
...
ignoreDifferences:
- group: argoproj.io
kind: Application
managedFieldsManagers:
- argocd-server
jsonPointers:
- /spec/syncPolicy/automated
This configuration tells Argo CD to ignore any changes made by the argocd-server
service account to the spec.syncPolicy.automated
field. The argocd-server
service account is used by the pod providing the Argo CD user interface. Any changes made to an application from the UI will use this service account.
Thus, this change enables you to use the Argo CD UI to override the automated field as needed since Argo CD will now ignore this difference.
I cannot take credit for this tip. I stumbled across it somewhere and have since lost where I got it. Kudos to the original contributor, whomever they may be!
Easily test custom health checks
I firmly believe in writing custom health checks to support the custom resources that various operators make available. These custom health checks enable Argo CD to understand the state of the resource (Healthy, Progressing, Degraded, etc.) and are required if you want to use sync waves with the resource.
The custom health checks are written in the Lua programming language. Testing complex checks in Argo CD can be challenging and time-consuming. You can use the argocd
command line interface to quickly and easily test these health checks independent of an Argo CD deployment on a cluster.
To test the resource, you need a YAML file copy of the argocd-cm
configmap and a copy of the resource you want to check. In OpenShift GitOps, the argocd-cm
configmap is automatically generated in the namespace where your Argo CD instance is located. For example, to get the configmap for the default openshift-gitops
instance, you can do the following to retrieve it and save it as a file:
oc get configmap argocd-cm -n openshift-gitops -o yaml > argocd-cm.yaml
Use a similar command when you want to retrieve the resource. Once you have these two files, you can test the health check with the following command:
argocd admin settings resource-overrides health <resource-yaml> --argocd-cm-path <argocd-cm-yaml>
While debugging a problematic health check for a subscription, I ran the command with the following results:
$ argocd admin settings resource-overrides health ./bad-sub.yaml --argocd-cm-path ./argocd-cm.yaml
INFO[0000] Starting configmap/secret informers
INFO[0000] Configmap/secret informer synced
STATUS: Healthy
MESSAGE: 1: CatalogSourcesUnhealthy | False
You can see the health check was successful and returned a Healthy
status. Also, note the MESSAGE
block. This is the message portion of the health check. It is useful to include additional information that could help teams to better understand why a specific status was returned.
Use kubectl-neat to export clean resources
It's a common GitOps practice to deploy things into a cluster initially using a GUI like the OpenShift console or tools like oc new-app
and then export the resources into a Git repository.
Resources in a Kubernetes cluster tend to have a lot of cruft in them (managed fields, status, etc.) that is painful to remove from the exported resources manually. A great tool for handling this automatically is the kubectl-neat plugin.
This tool is a plugin to kubectl and oc. To install it, get the binary from the GitHub repository releases and put it somewhere on your path. Then, when fetching a resource from the cluster, you can pipe the results to neat. Here's an example:
oc get service my-service -o yaml | oc neat > my-service.yaml
Use Global projects
In multi-tenant GitOps, the recommended practice is to use Argo CD Projects (AppProject) to manage access privileges and restrictions for the tenants. The AppProject enables the Argo CD administrator to restrict access to specific resources via allow- and denylisting, define team roles for role-based access control (RBAC), sync windows and more.
It's not uncommon in larger multi-tenant Argo CD instances to have many different projects for the various teams. Certain configuration items are often common across all projects or specific subsets of projects.
Rather than repeat this configuration over and over, you can define it once in a Global project instead. The other benefit is that changes to the global project will automatically be applied to the dependent projects, reducing the overall maintenance burden.
Here is an example of a Global project I use with my tenants in a cluster-scoped OpenShift GitOps instance. In this example, I denylisted all cluster-scoped resources along with specific namespace-scoped resources. The namespace-scoped resources remove some common ones, like ResourceQuota
and LimitRange
. It also removes some resources the OpenShift GitOps operator provides by default in cluster-scoped instances. It looks like this:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: global
spec:
description: Global Configuration
clusterResourceBlacklist:
- group: '*'
kind: '*'
namespaceResourceBlacklist:
- group: ''
kind: Namespace
- group: ''
kind: ResourceQuota
- group: ''
kind: LimitRange
- group: operators.coreos.com
kind: '*'
- group: operator.openshift.io
kind: '*'
- group: storage.k8s.io
kind: '*'
- group: machine.openshift.io
kind: '*'
- group: machineconfiguration.openshift.io
kind: '*'
- group: compliance.openshift.io
kind: '*'
The OpenShift GitOps operator does not have direct support for configuring the match expression needed for other projects to reference it. However, you can add the configuration using the extraConfig
capability in the operator, as shown in this snippet:
apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: argocd
spec:
extraConfig:
globalProjects: |-
- labelSelector:
matchExpressions:
- key: argocd.argoproj.io/project-inherit
operator: In
values:
- global
projectName: global
...
This Global project can then be referenced in other Projects by adding a label to it:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: product-catalog
labels:
argocd.argoproj.io/project-inherit: global
spec:
...
Wrap up
This blog went over several ways to improve your overall quality of life when working with OpenShift GitOps and Argo CD.
저자 소개
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.