External Secrets Operator and Doppler with OpenShift GitOps

Managing secrets when using GitOps to manage Kubernetes clusters and applications is challenging with many potential solutions. These solutions tend to fall into one of two following patterns:

1. Encrypt secrets in Git. SealedSecrets and Mozilla SOP are examples of this.
2. Externalize secrets. Examples include solutions like Hashicorp's Vault or a cloud provider's key management solutions coupled with the External Secrets Operator or the Secrets CSI driver.

I recently acquired a second server in my home lab. While I was relatively satisfied with SealedSecrets as a simple way to manage secrets, the second server increased my desire to examine externalizing my secrets for the benefit of centralization.

Since this is for a home lab scenario, minimizing costs is important. As a result, I settled on using the External Secrets Operator (ESO) along with the Doppler provider as the back-end for a completely free solution. The Secrets CSI driver is a fine solution, but at the time of this writing, the only free provider it supported was Community Vault, which requires significant effort to set up.

This blog looks at setting up ESO in a cluster to access secrets in Doppler. While I use Doppler, much of what I show applies to other providers. A complete list of providers that ESO supports is available here, and the specific documentation on using Doppler as a provider is available here.

External Secrets Operator Installation

Since I am doing GitOps, it is only natural to install the External Secrets Operator in OpenShift using OpenShift GitOps. Fortunately, this exercise is made trivial thanks to the efforts of the Red Hat GitOps Community of Practice team, which maintains a catalog of community-supported manifests for installing commonly used software. This catalog already contains manifests for the External Secrets Operator, found in GitHub.

The catalog splits the installation into the operator and an ESO instance. However, I prefer to use a single Argo CD Application to install both. This is easy to accomplish by using a customization to aggregate both items together as follows:

    kind: Kustomization
    apiVersion: kustomize.config.k8s.io/v1beta1

    commonAnnotations:
        argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

    resources:
        - github.com/redhat-cop/gitops-catalog/external-secrets-operator/operator/overlays/stable
        - github.com/redhat-cop/gitops-catalog/external-secrets-operator/instance/overlays/default

Note that I use the SkipDryRunOnMissingResource annotation to prevent the application from failing due to the required Custom Resource Definitions not being installed ahead of the instance.

Next, create an Argo CD application pointing to your repository's above customization file. Here is an example:

    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
        name: external-secrets
        namespace: openshift-gitops
    spec:
        destination:
            namespace: openshift-operators
            server: https://kubernetes.default.svc
        project: cluster-config
        source:
            path: components/apps/eso/overlays/aggregate
            repoURL: https://github.com/gnunn-gitops/cluster-config.git
            targetRevision: HEAD
        syncPolicy:
            automated:
                selfHeal: true

Sign up for Doppler

Before proceeding further with ESO, you will need to create an account with Doppler (or the provider of your choice). A Developer level account is available for free with Doppler, and I have found it sufficient to meet the needs for home labs or learning. As a small plug for Doppler, I like the fact that some of the more advanced features, like branching, remain available in the free tier.

To sign up for an account, visit doppler.com and follow their registration procedure. It is straightforward, and I shan't elaborate on it here.

Once you can access your account, click Projects, as this is where you set up secrets. When you first log in to Doppler, you will notice it has a default project. I deleted it and created a new one called "homelab" as per the screenshot below. However, feel free to use whatever name suits your needs. Note also that you can have multiple projects. You can have one project for cluster configuration and one or more application projects. Again, how you organize things is up to you.

Within the confines of a Project, Doppler assumes you will have environments like configs following dev, stage, and prod. In my case, I do not need these, so I opted to have a single base config called "cluster" that will contain the common secrets used across all clusters. For cluster-specific secrets, I opted to leverage Doppler's branch feature, where each config inherits from the root config but allows me to override or add additional secrets to the branch config.

I have two clusters in my home lab—home and hub—represented by the two branches shown below. Note I'm still relatively new to Doppler, so thoughts would be welcome on whether using environments or branches to model different clusters is better.

I can now add the secrets into Doppler as needed. I'll leave this as an exercise for you. However, I recommend reading the next section first to understand how ESO handles multi-key secrets, such as certificates, to minimize maintenance.

Finally, I need to configure an access token for ESO to pull secrets from Doppler. In Doppler, the access tokens are associated with the config. I created individual access tokens for the two branch configs, cluster_home and cluster_hub, to use with my home and hub clusters, respectively.

Configure an ESO SecretStore

For ESO to access a secrets provider, I need to define a secrets store. ESO supports a ClusterSecretStore and SecretStore, which are cluster and namespace scoped, respectively. For simplicity in my home lab, I used the ClusterSecretStore to provide cluster-wide access to secrets.

For folks operating in a more real environment, I recommend checking out the ESO multi-tenancy documentation. The TL;DR is that you will likely need to use the namespace-scoped SecretStore to operate safely in a multi-tenant OpenShift environment.

Here is an example of the ClusterSecretStore I am using:

    apiVersion: external-secrets.io/v1beta1
    kind: ClusterSecretStore
    metadata:
    name: doppler-cluster
    spec:
    provider:
        doppler:
        auth:
            secretRef:
            dopplerToken:
                name: eso-token-cluster
                key: dopplerToken
                namespace: external-secrets

Notice that this is referencing a secret. This secret contains the cluster-specific access token and thus varies with each cluster. An example of this secret is as follows (minus the real token, of course):

    apiVersion: v1
    kind: Secret
    metadata:
        name: eso-token-cluster-home
        namespace: acm-policies
    data:
        dopplerToken: XXXXXXX

While not in the scope of this blog, I'm using the RHACM policygenerator to distribute the ClusterSecretStore and secret to the managed clusters along with my GitOps installation. Interested readers can see it here.

Create an ExternalSecret

Here is the fun stage—creating ExternalSecret objects to reference the secrets in Doppler. The ESO guides do an excellent job of covering the common patterns used to create secrets, so I want to highlight a couple of the more common patterns I've been using.

- All keys, one secret. It is common to have a secret that has multiple keys in it. For example, certificates typically need keys for the private certificate, a public certificate, and a CA. While you can store each of these as separate key-pairs in your secret provider, it can be painful. ESO allows you to keep these key-value pairs in Doppler as a single JSON document and will automatically decompose it into multiple key-pairs in the secret.

Also, note that the Import Secrets feature in Doppler can reduce the effort involved in using this pattern.

- Common Secret Types. Kubernetes supports various secret types besides generic, including basic-auth, tls, docker, etc. In ESO, you can use templating to create secrets of the required type. For example, here is a secret for a dockerconfigjson type:

    apiVersion: external-secrets.io/v1beta1
    kind: ExternalSecret
    metadata:
        name: dest-docker-config
        namespace: product-catalog-cicd
    spec:
        refreshInterval: 1h
        secretStoreRef:
            name: doppler-cluster
            kind: ClusterSecretStore
        target:
            template:
            type: kubernetes.io/dockerconfigjson
            data:
                .dockerconfigjson: "{{ .docker_secret | toString }}"
            name: dest-docker-config
            creationPolicy: Owner
        data:
        - secretKey: docker_secret
            remoteRef:
            key: DOCKER_CONFIG_JSON

Wrap up

ESO provides an easy way to access secrets from an external centralized provider.