Introducing the Secret Store CSI Driver in OpenShift

The phrase “Kubernetes secrets are not secrets” is commonly heard, but what are these secrets and why are they important? 

Secrets refer to sensitive data such as passwords, certificates, bearer tokens, and more, which are essentially the core assets for an application owner. Compromising these secrets risks not only the application but potentially the entire infrastructure, underscoring the need for secure storage and management. 

Kubernetes provides Secret objects that can store this sensitive information independently of  the application, ensuring a clear separation from application code. The Secret objects are stored in the etcd datastore using base64 encoding. The intent behind base64 encoding is not to safeguard the secrets, but rather to provide binary-to-text encoding for ease of storage and transfer. Etcd provides the option to encrypt these secrets within the database, adding an additional layer of protection against unauthorized access. Etcd encryption primarily provides safeguards to prevent anyone with unauthorized access from reading data directly from etcd. 

However, if role-based access control (RBAC) is not set up properly on a Kubernetes cluster, anyone with API access can retrieve or modify these secrets. Additionally, anyone who is authorized to create a pod in a namespace can use that access to read any secret in that namespace. Therefore, in production environments it is essential for customers to establish robust security measures for storing and managing secrets and sensitive data effectively. 

To address these challenges, RedHat OpenShift is introducing Secret Store CSI Operator as TechPreview. Secret Store CSI (SSCSI) driver allows OpenShift customers to mount secrets from external secret management systems like AWS Secrets Manager or Azure Key Vault via a provider plugin. While the secrets can be used by applications, they do not persist on the system once the application pod is destroyed. This gives the customers complete control over storing and managing secrets using centralized secret storage that is external to the cluster. These external secret management solutions have far more stringent access controls and encryption options to secure and manage secrets.  

Some of the key features include: 

• Mounts secrets/keys/certs to pod using a CSI Inline volume
• Supports mounting multiple secrets store objects as a single volume
• Supports multiple secret stores as providers. Multiple providers can run in the same cluster simultaneously
• Supports pod portability with the SecretProviderClass CRD
• Supports Linux and Windows containers
• Supports sync with Kubernetes Secrets

How the Secret Store CSI Driver Works

Image source: https://secrets-store-csi-driver.sigs.k8s.io/concepts 

Here is how the secrets workflow works with the SSCSI driver, the SSCSI provider, and any external secret source to which the provider is connected: 

1. A pod is created and scheduled on a node  

1. Kubelet issues a grpc request to CSI driver to mount the volume
2. Secret Store CSI driver starts the process for volume mount to the pod as tmpfs  
3. SSCSI driver then sends a grpc request to the SSCSI provider 
4. SSCSI provider will talk to the Secret Store using the identity of the pod (this can be short-lived identity tokens or service account tokens for pods)
5. Once it fetches the secrets, it sends content back to the SSCSI driver as a gRPC response
6. Driver writes the secret to the filesystem 
7. Volume is successfully mounted as tmpfs in the pod
8. Volume is removed if the pod is deleted as part of cleanup

User Scenarios  

There are many use cases for using SSCSI driver to enhance security and streamline compliance and operations on the OpenShift platform, including when:

• Compliance managers would like to have all application secrets stored in a centralized secret management system that meets the organization’s regulatory requirements. This helps to limit the impact of potential threats and remediate issues more quickly.
• Cluster administrators want to allow applications to fetch secrets from the external provider and leverage existing secret management procedures with minimal overhead.
• Application developers want their application to fetch secrets from supported secret stores automatically, so that secrets can be updated without manual intervention.

How to install the Secret Store CSI driver operator

The Secret Store CSI Driver Operator is managed by OLM and can be installed from the Operator Hub. While installing, ensure that “All namespaces on the cluster” is selected and the install namespace is set to “openshift-cluster-csi-drivers”. For a cluster instance of the driver, add the following yaml on the instances tab: 

apiVersion: operator.openshift.io/v1
kind: ClusterCSIDriver
metadata:
    name: secrets-store.csi.k8s.io
spec:
  managementState: Managed

Refer to our documentation for additional details. As previously noted, The SSCSI Driver mounts secrets only into ephemeral volumes, so there is no need to do any volume provisioning because the volume will be deleted when pod is deleted. 

CRDs with Secret Store CSI Driver

The Secrets Store CSI Driver Operator creates a SecretProviderClassPodStatus resource in the same namespace as the pod. You can review this resource to see detailed information, including versions, about the secrets in the pod volume mount.

$ oc get secretproviderclasspodstatus <pod_name>-<namespace>-<secret_provider_class_name> 

SecretProviderClass custom resource is used to provide driver configurations and provider-specific parameters to the CSI driver. An example with Azure Key Vault is below:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-azure-provider                 
  namespace: my-namespace                 
spec:
  provider: azure                         
  parameters:                             
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    userAssignedIdentityID: ""
    keyvaultName: "kvname"
    objects: |
      array:
        - |
          objectName: tls-cert
          objectType: secret
    tenantId: "tid"

Features with Secret Store CSI Driver Operator

There are two optional features that are enabled by the operator by default:

Secret Auto Rotation 

The Secrets Store CSI Driver periodically rotates the content in the mounted volume with the content from the external secrets store. If a secret is updated in the external secrets store, the secret will be updated in the mounted volume. The Secrets Store CSI Driver Operator polls for updates every two minutes. If sync secrets are enabled, the Kubernetes secrets are also rotated. Applications consuming the secret data must watch for updates to the secrets. 

Note that if using subPath volume mount, the secret updates are not propagated on rotation. This is a known limitation (see details here). 

Sync as Kubernetes Secret 

Sync secrets allows you to create Kubernetes secrets from the content on the mounted volume. An example where you might enable synchronization is when you want to use an environment variable in your deployment to reference the Kubernetes secret. 

Here’s an example of how you would add the Kubernetes secret information to the SecretProviderClass CRD:

  secretObjects:                          
    - secretName: ingress-tls-csi         
      type: kubernetes.io/tls             
      data:
        - objectName: tls-cert            
          key: tls.key                    
        - objectName: tls-cert
          key: tls.crt

Looking Ahead

We try to keep your secrets safe! With the Secret Store CSI Driver solution, RedHat OpenShift  aims to provide a vendor-neutral solution that can integrate with secret storage systems such as Vault, AWS Secrets store, AWS Parameter store, and Azure Key Vault that are used by customers to securely manage application secrets. The end-to-end solution relies on working with external secrets providers and we encourage you to share  which providers you are using and how we can incorporate support for them in future. We hope to evolve the feature sets on this solution closely with upstream and work with our ecosystem of partners to provide integrations for an end-end solution.