Accelerate STIG compliance with Red Hat OpenShift’s built-in security features: From 40 CAT I items to 7 (Part II)

Part I of this series compared the CAT I STIG items for Kubernetes on Red Hat Enterprise Linux 8 vs Red Hat OpenShift and how OpenShift's built-in security reduces the number of STIG items needed to achieve compliance. This article reviews the seven CAT I STIG items in OpenShift and puts you on the path to STIG compliance.

Keep in mind that this portion of the blog discusses the technical implementation of the STIG items and assumes familiarity with Kubernetes and OpenShift on a technical level. Refer to the product documentation for further details regarding topics that come up during checks and remediations. Please note that you must log in with a user with the cluster-admin role to perform the commands to check and remediate STIG items.

Before diving into the OpenShift STIG items, I want to use two Kubernetes STIG items on an OpenShift cluster to demonstrate why there are fewer items to harden. These items concern control plane hardening in the /etc/kubernetes/manifests directory, which contains static pod manifests for control plane components.

1. Vul ID: V-242381     Rule ID: SV-242381r879522_rule     STIG ID: CNTR-K8-000220

Rule Title: The Kubernetes Controller Manager must create unique service accounts for each work payload.

The first item states to check for unique service accounts by using the grep command. If the output contains use-service-account-credentials=false or if it's not configured, it's considered a finding because the default setting is to use the same service account with the same credentials.

In OpenShift, we discourage using SSH to access the nodes and make direct changes, but you can still check for configurations in the manifest files. Direct configuration changes may be reverted when the node reboots because the state of the operating system is defined in the MachineConfig objects. 

The recommended method to check these files is to look inside the etcd pods in the openshift-etcd namespace.

After listing the pods, choose one and use the oc exec command to get a shell into the container and perform the check.  

After changing into the /etc/kubernetes/manifests directory, you can now use the grep -i use-service-account-credentials * command per the Kubernetes STIG instructions.

The grep output is --use-service-account-credentials=true, which is not a finding. This result means unique service accounts exist for each controller instance.

2. Vul ID: V-245542     Rule ID: SV-245542r918141_rule     STIG ID: CNTR-K8-002620

Rule Title: Kubernetes API Server must disable basic authentication to protect information in transit.

Another quick check is basic authentication which uses usernames and passwords with no encryption. We can check this with the grep command for basic-auth-file in the /etc/kubernetes/manifests directory. Since you're already in that directory from the previous check, use the grep -i basic-auth-file * command for this STIG item.

The grep command output shows nothing returned, indicating there is no basic-auth-file to remove and, therefore, no findings. You can also verify that the grep search returned nothing with the echo $? command, which shows the return code of 1.

You've confirmed that OpenShift control plane components comply with some of the Kubernetes STIG. Next is achieving CAT I compliance in OpenShift. It is important to note that there are two main ways to remediate items in OpenShift. 

The first is through the creation and deletion of Kubernetes API objects using the OpenShift console or command line, and the second is by applying MachineConfigs, which make changes to files and services on the operating system (RHCOS). The STIG may ask to create/edit/remove a Kubernetes object such as a secret, config map, ingress controller, or RBAC which does not involve changes to the OS.  To see the kinds of API objects included with OpenShift, use the oc api-resources command.

I have listed the seven CAT I items with their respective IDs and Rule Titles. For a full description of each item, download the STIG here. Also, due to the length of the MachineConfig object definitions, the YAML for these is not shown here. Please refer to the STIG to see the MachineConfig YAML definitions.

1.  Vul ID: V-257513 Rule ID: SV-257513r921482_rule STIG ID: CNTR-OS-000090

Rule Title: OpenShift RBAC access controls must be enforced.

This item requires that cluster admins review user and group permissions to make sure they have the proper access to cluster and project resources.

Use the following commands to check these permissions:

oc describe clusterrolebinding.rbac

oc describe rolebinding.rbac

Remove improper permissions with the following commands:

oc adm policy remove-cluster-role-from-user <role> <username>
oc adm policy remove-cluster-role-from-group <role> <groupname>
oc adm policy remove-role-from-user <role> <username>
oc adm policy remove-role-from-group <role> <groupname>

* If your organization will have many users and groups, it would be better to automate this task using a tool such as Ansible automation, which you can run repeatedly and only execute the tasks that change. If a list of users and groups with their respective RBAC permissions changes, you can re-run the Ansible playbook, and only the new roles and rolebindings, clusterroles, and clusterrolebindings are affected.  

* If you use an LDAP provider such as Active Directory, you can integrate it with OpenShift, and groups can be synchronized to make this task easier and minimize operator error.

2.  Vul ID: V-257519 Rule ID: SV-257519r921500_rule STIG ID: CNTR-OS-000170

Rule Title: Red Hat Enterprise Linux CoreOS (RHCOS) must initiate session audits at system startup.

This check monitors user activities and system events from system startup. Check it with the following command:

for node in $(oc get node -oname); do oc debug $node -- chroot /host /bin/bash -c 'echo -n "$HOSTNAME ";  grep audit /boot/loader/entries/*.conf || echo "not found"' 2>/dev/null; done

Correct the finding by creating a MachineConfig object that sets kernel parameters of audit=1 and audit_backlog_limit=8192.  

* As with other STIG items that require OS configuration changes, a MachineConfig object must be created for the changes to be applied and persist. It is not necessary to edit the configuration files on each node.

3. Vul ID: V-257540 Rule ID: SV-257540r921563_rule STIG ID: CNTR-OS-000400

Rule Title: OpenShift must disable root and terminate network connections.

By default, OpenShift is not configured for root login to the nodes, but you can check this setting with the following command:

for node in $(oc get node -oname); do oc debug $node -- chroot /host /bin/bash -c 'echo -n "$HOSTNAME "; grep -i PermitRootLogin /etc/ssh/sshd_config' 2>/dev/null; done

Run the following command to check network connection termination settings:

for node in $(oc get node -oname); do oc debug $node -- chroot /host /bin/bash -c 'echo -n "$HOSTNAME "; grep -i clientalive /etc/ssh/sshd_config
' 2>/dev/null; done

In this case, the setting ClientAliveCountMax is commented out and is a finding. Uncommenting this setting and changing its value to 0 would remediate this.

4. Vul ID: V-257543 Rule ID: SV-257543r921572_rule STIG ID: CNTR-OS-000460

Rule Title: OpenShift must use FIPS validated LDAP or OpenIDConnect.

Check the cluster's identity provider by running the following command:

oc get oauth cluster -o
jsonpath="{.spec.identityProviders[*].type}{'\n'}"

* Remediate this finding by setting up LDAP or OpenID Connect. Ansible Automation can configure LDAP and sync LDAP groups.

5. Vul ID: V-257546 Rule ID: SV-257546r921581_rule STIG ID: CNTR-OS-000510  

Rule Title: OpenShift must protect authenticity of communications sessions with the use of FIPS-validated 140-2 or 140-3 validated cryptography.

To check whether a cluster is FIPS enabled, run the following command:

for node in $(oc get node -oname); do oc debug $node -- chroot /host /bin/bash -c 'echo -n "$HOSTNAME "; sysctl crypto.fips_enabled' 2>/dev/null; done

* If the cluster is not FIPS enabled, OpenShift must be reinstalled with fips: true stated in the install-config.yaml file. FIPS must be enabled before the operating system used by the cluster boots for the first time so it can not be enabled after installation. 

The product documentation for Red Hat Enterprise Linux 8 states, “Only enabling FIPS mode during the RHEL installation ensures that the system generates all keys with FIPS-approved algorithms and continuous monitoring tests in place.”  This is the same for RHCOS so it is necessary to enable FIPS at installation.

To ensure that all key material is generated with FIPS compliant cryptography, run the openshift-install binary from a server that is FIPS enabled. On a RHEL 8 server, we can check the FIPS status with the fips-mode-setup --check command.  

This screenshot shows that the RHEL 8 bastion host is not FIPS enabled and it would need to be enabled before installing OpenShift with FIPS. To ensure FIPS compliance on RHEL, follow the guidance in the security hardening documentation.

For more information on FIPS with OpenShift, click here.

6. Vul ID: V-257557 Rule ID: SV-257557r921614_rule STIG ID: CNTR-OS-000660  

Rule Title: Container images instantiated by OpenShift must execute using least privileges.

Run the following command to check for SecurityContextConstraints that allow privileged containers or containers access to the host network or filesystem resources:

oc get scc -ojson | jq '.items[]|select(.allowHostIPC or .allowHostPID or .allowHostPorts or .allowHostNetwork or .allowHostDirVolumePlugin or .allowPrivilegedContainer or .runAsUser.type != "MustRunAsRange" )|.metadata.name,{"Group:":.groups},{"User":.users}'

For each SCC listed, if any of those users or groups are anything other than the following, this is a finding:

  * system:cluster-admins
  * system:nodes
  * system:masters
  * system:admin
  * system:serviceaccount:openshift-infra:build-controller
  * system:serviceaccount:openshift-infra:pv-recycler-controller
  * system:serviceaccount:openshift-machine-api:machine-api-termination-handler

Check roles and clusterroles to see if they use non-permitted SCCs with the following commands, which will print the clusterrole/role's name and namespace, followed by a list of resource names and if that resource is an SCC:

oc get role.rbac --all-namespaces -ojson | jq -r '.items[]|select(.rules[]?|select( (.apiGroups[]? == ("security.openshift.io")) and (.resources[]? == ("securitycontextconstraints")) and (.verbs[]? == ("use"))))|.metadata.name,{"scc":(.rules[]?|select((.resources[]? == ("securitycontextconstraints"))).resourceNames[]?)}'

oc get clusterrole.rbac -ojson | jq -r '.items[]|select(.rules[]?|select( (.apiGroups[]? == ("security.openshift.io")) and (.resources[]? == ("securitycontextconstraints")) and (.verbs[]? == ("use"))))|.metadata.name,{"scc":(.rules[]?|select((.resources[]? == ("securitycontextconstraints"))).resourceNames[]?)}'

After checking SCCs, clusterroles, roles, and service accounts, use the following commands to edit or remove SCCs or clusterrolebindings/rolebindings:

oc edit scc <SCC>

The following instructions will remove the user or group from the cluster role binding for the SCC policy.

Remove a user from the SCC policy binding by executing the following command:

oc adm policy remove-scc-from-user <SCC> <USER>

Remove a group from the SCC policy binding by executing the following command:

oc adm policy remove-scc-from-group <SCC> <GROUP>

Remove a service account from the SCC policy binding by executing the following command:

oc project <SERVICE_ACC_PROJECT>
oc adm policy remove-scc-from-user <SCC> -z <SERVICE_ACC>

Remove any roles that allow the use of non-permitted SCC policies (excluding platform-defined roles) by executing the following command:

oc delete clusterrole.rbac <ROLE>

or

oc delete role.rbac <ROLE> -n <NAMESPACE>

* To thoroughly check all SCCs, roles, clusterroles, rolebindings, clusterrolebindings, and service accounts would take a lot of time and some things could get overlooked. Automating this task through Ansible automation would speed up the process and minimize the risk of missing non-permitted permissions.  

7. Vul ID: V-257583 Rule ID: SV-257583r921692_rule STIG ID: CNTR-OS-001010

Rule Title: Red Hat Enterprise Linux CoreOS (RHCOS) must disable SSHD service.

Managing OpenShift and RHCOS nodes should be done through the OpenShift API using either the command line or the OpenShift console and not through direct SSH connections to the nodes.

Run the following command to check if SSH is enabled on the RHCOS nodes:

for node in $(oc get node -oname); do oc debug $node -- chroot /host /bin/bash -c 'echo -n "$HOSTNAME "; systemctl is-enabled sshd.service; systemctl is-active sshd.service' 2>/dev/null; done

* Only disable SSH access after the successful installation and configuration of OpenShift. Remember, OpenShift allows SSH access with the core user by default.

If there are any installation failures or issues, it may be necessary to SSH into the bootstrap node or another node to check logs or services such as bootkube, kubelet, or cri-o. It is possible to gather logs without SSH access by using the oc adm node-logs command.  Examples on gathering logs can be found here.

Wrap up

This article provides explanations and instructions for mitigating seven CAT 1 STIG items. By following these steps you gain an understanding of the issues and have the instructions for meeting compliance with the DISA OCP STIG.

Be sure to review the first article in the series for information on built-in security features of OpenShift and how Red Hat provides a comprehensive container application platform with a lower number of STIG items for compliance. Also, stay tuned for the final article in the series which will discuss scanning and remediating OpenShift STIG items with the Compliance Operator.