Red Hat Enterprise Linux CoreOS Integration with LDAP and SSSD

Introduction

If you are using OpenShift 4.x then you are no doubt becoming more and more familiar with Red Hat Enterprise Linux CoreOS (RHCOS) and how it is managed differently than a traditional Linux distribution. By default there is only one user on an RHCOS machine known as the "core" user and this account is used for all traditional remote access via ssh and a private key. But what if I told you there is a way to make additional user accounts available using LDAP.
If you have a corporate requirement for individual named users to log into the system, or corporate requirements to leverage a centralized Identity Management System such as Windows Active Directory, or LDAP, well then this post is for you.

RHCOS supports the use of LDAP for centralized administration through the use of the System Security Services Daemon (or SSSD for short) and OpenShift MachineConfigs to properly configure the RHCOS nodes to leverage a centralized IDM solution. SSSD is available as part of the standard RHCOS image and just needs to be enabled and configured in order to take advantage of this support.

In this post, we will see how to configure SSSD on Red Hat Core OS nodes through the use of MachineConfigs in OpenShift to allow for centralized management of user accounts in an LDAP directory.

Warning!

RHCOS nodes are designed to be managed through the use of MachineConfigs and MachineConfigPools. Editing files directly on the node over SSH is not supported and will lead to issues with maintainability and supportability of the cluster. The use of SSH for remote access to the nodes should be used as a last resort for debugging/diagnostic issues only.

Prerequisites

This blog post will assume that you have the following available to you:

• Working OpenShift Cluster
• Working LDAP Server
• Connection information for your LDAP server
• Cluster Admin role in your OpenShift Cluster
• openssl binary
• base64 tool
• git

This post will leverage FreeIPA as our LDAP server, but with some modification of the configuration files, you can use any Identity Management server that has LDAP support. See the SSSD documentation for how to properly configure SSSD for your particular LDAP implementation.

LDAP Requirements

You will need the following information in order to proceed with this tutorial. If you do not have this information you will need to work with your LDAP administrator:

• ldap_uri - URI of your LDAP instance eg. ldaps://ldap.xphyrlab.net
• ldap_default_bind_dn - Username to connect with eg. uid=sssd_ldap,cn=users,cn=accounts,dc=example,dc=com
• ldap_schema - Schema used by your LDAP implementation eg. ipa
• ldap_default_authtok - Password to use for connection eg. password
• ldap_user_ssh_public_key - object attribute type for ssh Public Key (if supported) eg. ipaSshPubKey
• ldap_user_search_base - user search base in LDAP where your users are defined eg. cn=users,cn=accounts,dc=example,dc=com
• ldap_group_search_base - group search base in LDAP where your groups are defined eg. cn=groups,cn=accounts,dc=example,dc=com
• ldap_sudo_search_base - sudo search base (if supported) eg. ou=sudoers,dc=example,dc=com
• ldap_access_filter- filter to limit access to RHCOS boxes eg. memberOf=cn=rhcosadmins,cn=groups,cn=accounts,dc=example,dc=com

Clone the Template Repo

We will use a a set of template files that have been created to help make configuration of the LDAP integration easier. Start by cloning the git repository https://github.com/rh-telco-tigers/idm_rhcos and then change into the templates directory of the repository:

$ git clone https://github.com/rh-telco-tigers/idm_rhcos
$ cd idm_rhcos/templates

We will use these files through the remainder of this post.

Retrieve SSL Certificate

The first order of business is to retrieve the SSL certificate for your LDAP server. This certificate is used to validate our connection to your LDAP server and secure our connection to the server. This certificate can be obtained using the following command, making sure to update the <ldap server name> with your specific server:

$ openssl s_client -showcerts -connect <ldap server name>:636 </dev/null 2>/dev/null|openssl x509 -outform PEM > ldap_ca.crt

This command will create a file called “ldap_ca.crt” which we will use shortly as part of our MachineConfiguration.

Updating Template Configuration Files

Now that we have our LDAP server certificate, we need to update our template files. The template files can be found in the git repository we cloned at the beginning of this post.

There are four configuration files in the templates directory however, we only need to edit two of them.

• sssd.conf - our main configuration file
• sshd_config - enable password auth and/or ssh_keys stored in LDAP if supported

Updating the sssd.conf file

Start by editing the template/sssd.conf file and updating the required configuration items. There are nine items that need to be updated. You should have previously gathered this information in the LDAP Requirements section.

• ldap_uri
• ldap_default_bind_dn
• ldap_schema
• ldap_default_authtok
• ldap_user_ssh_public_key - if you do not wish to use SSH keys stored in your LDAP directory remove this line
• ldap_user_search_base
• ldap_group_search_base
• ldap_sudo_search_base - if you do not wish to use sudo rules stored in your LDAP directory remove this line
• ldap_access_filter- this can be used to define a LDAP group that users must be a member of in order to successfully log into the node.

Your particular implementation of LDAP may require additional configuration options. Be sure to see the documentation on Configuring SSSD to use LDAP for additional information.

$ vi sssd.conf

SSHD Configuration

As part of the decision to enable the use of an LDAP directory for login to RHCOS machines, you will need to make a decision on how you would like authentication to work. You have two main options:

1. Enable tunneled clear text passwords
2. Use ssh keys stored in your LDAP directory

To enable password based authentication, edit the sshd_config file and remove the “#” from the beginning of the following line:

#PasswordAuthentication yes

To enable the use of ssh public keys stored in your LDAP directory, you will need to uncomment the following two lines:

#AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys

#AuthorizedKeysCommandUser nobody

If you choose to use ssh public keys stored in your LDAP directory you must also ensure that you properly configured SSSD in the Updating the sssd.conf file section above.

You must make one or both of these changes, or you will not be able to access the RHCOS nodes with users stored in LDAP.

Generate base64 encodings

We will be using OpenShift MachineConfigs to manage these changes on our RHCOS nodes. OpenShift MachineConfigs leverage base64 encoded files for distribution. For each file in the template directory, we will create a base64 encoding file. We can quickly convert all our files using the following bash script:

for i in nsswitch.conf  sshd_config sssd.conf system-auth ldap_ca.crt
do
  base64 -w0 $i > $i.base64
done

Validate that you have five new files ending in “.base64” in your directory by running the _ls _command

$ ls *.base64
nsswitch.conf.base64  sshd_config.base64  sssd.conf.base64 system-auth.base64 ldap_ca.crt.base64

Creating a MachineConfig to apply our files

The GitHub repo contains template files that we will use to build our MachineConfig that will be applied to our cluster. The GitHub repo comes with two MachineConfig template files:

• sssd_control_mc.yaml
• sssd_worker_mc.yaml

If you inspect the files you will see that there is one main difference.

• metadata.labels.machineconfiguration.opepassword-based

This label allows us to target the specific type of machines that we will be targeting with this change. The “sssd_worker_mc.yaml” file will only apply to Kubernetes Worker nodes, and the “sssd_control_mc.yaml” file only applies to the Kubernetes Control Plane (master) nodes. You will also notice that there are placeholders for the base64 encoded files we previously created. We will use these placeholders in the forthcoming sections.

Enabling sudo Rules from LDAP

It is possible to deliver sudo rules from an LDAP directory. If your system is set up to deliver sudo rules, and you configured SSSD in the Updating the sssd.conf file to use sudo rules, you will need to uncomment the “nsswitch.conf” section in both the sssd_control_mc.yaml and sssd_worker_mc.yaml files before proceeding.

Updating MachineConfig Files

Use the following command to replace the base64 placeholders in our MachineConfig files with the contents of our base64 strings.

for i in nsswitch.conf  sshd_config sssd.conf system-auth ldap_ca.crt
do
  sed -i "s/<$i.base64>/$(cat $i.base64 )/" sssd_control_mc.yaml
  sed -i "s/<$i.base64>/$(cat $i.base64 )/" sssd_worker_mc.yaml
done

Review the updated sssd_control_mc.yaml file and sssd_worker_mc.yaml to ensure that the placeholders have been updated with the base64 strings before continuing.

Applying the sssd machine config

With our MachineConfig files created, we will now apply them to our cluster. If you wish to apply this change to only the worker nodes, only apply the “sssd_worker_mc.yaml” file to your cluster. If you want to apply this change to all nodes, apply both the sssd_control_mc.yaml and the sssd_worker_mc.yaml files. Ensure that you are logged into your cluster with “Cluster Admin” privileges before attempting to apply these changes:

$ oc login
$ oc create -f templates/sssd_control_mc.yaml
machineconfig.machineconfiguration.openshift.io/55-sssd-control-ldap created
$ oc create -f templates/sssd_worker_mc.yaml
machineconfig.machineconfiguration.openshift.io/55-sssd-worker-ldap created

The MachineConfig will now be applied to all control-plane and worker nodes. You can watch the progress as the changes are applied machine by machine by running the following command:

$ oc get mcp --watch
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-7342437503dd945f89093e6853496e54   False     True       False      3              0                   0                     0                      42d
worker   rendered-worker-e8bbbd02a1037009686b488d3d770e21   True      False      False      5              5                   5                     0                      42d

Wait for the section titled “READYMACHINECOUNT” to match “MACHINECOUNT” to know that the changes have been applied.

NOTE: This change will reboot all the affected nodes, in a controlled manner.

Testing our Changes

With the MachineConfig applied to all our OpenShift nodes, you should be able to connect to them via SSH using any account that exists in the group you identified as a filter. For example, we will log into a worker node with the user “markd” which comes from the LDAP directory:

$ ssh markd@172.16.25.123
Red Hat Enterprise Linux CoreOS 47.83.202106032343-0
  Part of OpenShift 4.7, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.7/architecture/architecture-rhcos.html

---
[markd@ocp47-jzgl9-worker-jt5ng ~]$ sudo su -

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for markd: 
[root@ocp47-jzgl9-worker-jt5ng ~]# 

Troubleshooting LDAP Connections

Due to the complexity of LDAP and every enterprise implementation differs, you may run into issues with getting LDAP integration to work properly on your first try. There are a few places to look to determine what might be going wrong.

Check the SSSD Service

The first place to start is with the sssd service itself. Check to ensure that it is running and is not showing any logs. Remotely connect to a test node using the “core” user and check the status of the sssd service using the systemctl status sssd command:

$ ssh core@<ip address>
$ sudo su -
# systemctl status sssd

You can also use the command journalctl -u sssd to see the output of the service as it starts up to see if there are any errors.

Check the SSH Service

If the sssd service is running, check to ensure that the ssh service is also running and not throwing any errors:

$ ssh core@<ip address>
$ sudo su -
# systemctl status sshd
# journalctl -u sshd 

Check the logs

If both the sssd and sshd services are running and not showing any errors, then it is time to look at the detailed logs for clues. Review the logs in the following two locations for clues as to why the authentication is not working:

• /var/log/audit/audit.log
• /var/log/sssd

There are multiple logs in the /var/log/sssd directory, you may need to consult one or all of them in order to find where the issue may be.

Conclusion

The configuration changes we have made today are the same changes that one would need to make for any Red Hat Enterprise Linux system to integrate SSSD with LDAP. We leveraged OpenShift MachineConfigs in order to ensure that the files were properly copied to all our control plane and worker nodes. While the options used in this post reflect the use of FreeIPA, with some small tweaks to things such as schema, and the object references to your ssh keys and sudo rules, you can make this work with any LDAP service.