Introducing the external secrets management console plug-in

Red Hat recently released the initial version of the external secrets management console plug-in. It’s an extension of the Red Hat OpenShift web console, which lets you inspect any of the resources deployed by Red Hat’s portfolio of secrets management operators: cert-manager operator for Red Hat OpenShift, external secrets operator for Red Hat OpenShift, and secrets store CSI driver operator. This technology preview is available as a Day 2 operator for Red Hat OpenShift 4.22, installable through the Red Hat Ecosystem Catalog.

Secrets management in Red Hat OpenShift

Last year, Red Hat made the external secrets operator generally available (GA), rounding out the trio of operators for managing credentials on OpenShift.

• [GA] Cert-manager operator for Red Hat OpenShift for automated lifecycle management of x.509 certificates
• [GA] External secrets operator for Red Hat OpenShift for automated management of credentials using Kubernetes secrets
• [GA] Secrets store CSI driver operator for automated management of credentials in highly secure environments without using Kubernetes Secrets

Each of these 3 operators offers a unique solution to automating management of credentials that workloads on OpenShift use. They also help cluster administrators adopt a stronger security posture for their clusters. Why to use secrets management tools isn’t the core topic of this blog. However, at a high level, using secrets management tools helps admins reduce the risk of leaked credentials by ensuring a single source of truth for all credentials on a cluster, minimizing the blast radius should a credential be leaked and eliminating the use of manually configured Kubernetes Secrets.

Referencing Figure 1, blast radius is generally measured on 2 axes: exposure and severity. Exposure represents the amount of time a credential is valid on a cluster. Assuming a bad actor gains access to the cluster, a Kubernetes Secret represents the worst possible exposure, as there are no controls to automatically rotate or revoke a credential. Severity is the measure of how permissive a leaked credential is. It’s disastrous when cluster admin permissions are leaked, whereas the service account for a single workload in an isolated namespace might be less critical. OpenShift’s secrets management operators enable automation to periodically rotate credentials living on the cluster, reducing exposure. They also allow admins to manage larger scales of credentials, decreasing the need for generalized, more permissive secrets to exist in the 1st place.

Figure 1: This diagram illustrates the potential cost of a leaked credential based on the amount of time a credential is exposed and the permissions level of the leaked credential.

To learn more about how Red Hat views secrets management in OpenShift, check out this learning path or the documentation. 

A user interface for secrets management

So, why do we care about a user interface (UI) for tools that automate management of credentials? Administrators should just be able to sit back and allow the operators to run, right? For the most part, yes. However, adopting secrets management at scale shifts failure points away from the basic Kubernetes Secret resource, changing the dependency graphs of workloads using these credentials. As a result, administrators have fewer resources to keep track of, but the health of these resources becomes much more critical. Having a unified UI visualizing these resources can be extremely useful. Consider the following scenarios:

Cert-manager operator for Red Hat OpenShift

The cert-manager operator helps admins automatically issue and rotate Transport Layer Security (TLS) certificates for hundreds or even thousands of applications on a cluster, increasing security and reducing overhead. Previously, admins needed to monitor every single certificate in a cluster to make sure none were expiring and manually issue new certificates for certificates reaching their expiration date. The cert-manager operator now automates this process thanks to the introduction of new Custom Resource Definitions (CRDs)—Issuers and Certificates. 

Issuers define how the cert-manager operator communicates with an external Public Key Infrastructure (PKI) to obtain and renew TLS certificates, and Certificates define what these TLS certificates should look like. The cert-manager operator abstracts away the task of manually issuing and renewing certificates at scale. However, admins now have the critical task of ensuring that the cert-manager Issuer is configured correctly and healthy. In many cases, multiple Certificates will rely on a single Issuer or ClusterIssuer, the cluster-scoped version of an Issuer. This layer of abstraction shifts the point of failure from the TLS certificate itself to the Issuer Custom Resource (CR). If part of the configuration for the Issuer changes or if the endpoint the Issuer expects to communicate over goes down, multiple certificates and their respective workloads could be affected. 

Figure 2 shows a screenshot of the new external secrets management console plug-in, where you can see an ideal scenario. The cert-manager operator is working as expected, and all deployed resources are “Ready.”

Figure 2: This screenshot shows the ideal scenario of deployed cert-manager operator resources. All objects are successfully deployed and in a “Ready” state.

However, what happens when an accidental change misconfigures an Issuer or a bug causes problems for the Issuer when attempting to connect to the external PKI? In Figure 3, you can see that an Issuer has entered a “Not Ready” state, and a number of certificates are set to expire soon. Previously, troubleshooting this issue would require a series of `oc` commands in the command line, but with the UI, you can resolve the issue faster. First, you can see that the Certificates with the yellow warning states are likely false positives, as their Issuer is in a “Ready” status. Therefore, this may not warrant immediate action. However, you’ll quickly see that the Certificate with the red “critical” state is about to expire within the next 24 hours and is attached to an Issuer in a “Not Ready” state. This requires immediate action. 

Figure 3: This screenshot depicts how an administrator might begin triaging a Certificate resource that’s about to expire.

Conveniently, you can also inspect the Issuer and Certificate in order to triage the issue further. Clicking on the kebab icon to the right of each resource will redirect you to another page with detailed information regarding the resource you’re inspecting. In the image below we can see the events log of the expiring Certificate. 

Figure 4: This screenshot shows the ideal flow of certificate requests, followed by certificate issuance (green box), followed by a failed flow. The certificate renewal request is not followed by a successful issuance event (red box).

The green box shows a successful issuance of a new certificate. By comparison, the red box shows the cert-manager operator subsequently attempted to renew the expiring certificate. However, no successful issuance event followed the request. This indicates a potential problem with the Issuer the Certificate is using to renew/obtain TLS certificates. You can use a similar approach to diagnose the Issuer now that it’s been identified as the problem. Compared to the traditional method of issuing and managing certificates, the point of failure has shifted, and this UI lets admins quickly assess the state of managed TLS certificates and their Issuers with just a few clicks. 

External secrets operator for Red Hat OpenShift

The external secrets operator has a similar basic deployment architecture as the cert-manager operator. SecretStores, like Issuers, define how the operator communicates with an external secrets management system. ExternalSecrets, like Certificates, define what credentials to obtain and how to store them on OpenShift. However, the external secrets operator differs from the cert-manager operator in that it’s capable of managing almost any credential type, whereas the cert-manager operator is designed specifically for managing TLS certificates. At a high level, the triage of a problem with a credential managed by the external secrets operator is similar to the cert-manager operator. 

But the external secrets operator showcases another example of how things change when adopting external secrets management: the dependency graph shifts. The external secrets operator introduces several new CRs that make it a flexible tool, including:

• SecretStores
• ExternalSecrets
• PushSecrets
• Generators

These new resources bring added flexibility and control to how you manage credentials living on OpenShift. However, compared to managing large sprawls of Kubernetes Secrets, admins now can introduce more complex topologies of automatically managed credentials. In Figure 5, we can see a more advanced deployment example. 

Figure 5: This diagram shows a more complex deployment topology of the external secrets operator resources.

If you follow the solid and dotted arrows, you can see a flow to and from the external secrets management database. An external secrets operator Generator can create a credential that’s populated into a Kubernetes Secret, after which a PushSecret uploads the generated credential to the external database. Next, you can configure a SecretStore to connect to the external database to obtain these credentials. Finally, you can create an ExternalSecret for each individual application workload. These define which credentials to obtain from the external database and how the credentials should be stored within a Kubernetes Secret. This might be a little overwhelming to tease apart from a command line, but as you can see in Figure 6, the UI is handy for understanding the connections between the different external secrets operator objects. 

Figure 6: This screenshot shows how an administrator can trace the dependencies between different external secrets operator resources. 

Future work

Red Hat is invested in providing a complete solution for management of secrets and credentials on OpenShift, and that includes providing the UI components admins need to interact with these products. The Technology Preview of the console plug-in is limited to administrator users only. The GA release will include a number of planned improvements including non-admin views and improved usability features. Additionally, our longer term roadmap includes topology graphs, deeper integration with observability features like alerts, and more.

Try the external secrets management console plug-in

The operator is available in the Red Hat Ecosystem Catalog and can be installed on any OpenShift 4.22 cluster by following this documentation.