Locking down sshd

SSH, or the Secure SHell, replaced Telnet somewhere in the 1990s as the remote access protocol of choice, and for good reason. SSH allows administrators—or users—to access a remote shell through a secure tunnel, by connecting their SSH client to an SSH server. SSH can also handle file transfers, which should replace FTP, though there are a surprising number of situations which still rely on good old clear text FTP.

Why? Ugh.

For the reasons above, modern Linux systems are managed through SSH. Most experienced sysadmins love the direct access and power they get from securely connecting to their system's shell with relative ease. In this article, I’ll talk specifically about the OpenSSH server daemon, sshd. We’ll cover some of the security problems you can run into, and how to mitigate—or outright solve—them.

Why we need SSH

As I mentioned, SSH is a powerful tool. Through an SSH session, you can connect a virtual terminal directly to your target system. In the wrong hands, this ability can be problematic, so why do we leave such power accessible? Well, the power of SSH involves a delicate balance. Within minutes, an administrator can have a secure session open to their server to respond to an incident. The simplicity is elegant.

SSH is also at the core of automation tools like Ansible. Ansible can apply changes to a system through SSH without the need of an agent. All changes are carried out, instead, using SSH and Python.

SSH can also be used, as I mentioned, for secure file transfers. For example, rsync tunnels through SSH for secure data synchronization. SSH can also be used to tunnel through one system to get to another, which is great for troubleshooting, but also great for an attacker.

If the preceding paragraphs haven’t convinced you, I’ll say this again: SSH is a powerful tool, and like any powerful tool, it can be abused. If an attacker can gain a secure shell into your system, it’s game over. And attackers know this, so they’re constantly looking for systems with SSH open to the world so they can attack.

Lock SSH down

The single most common attack against SSH is brute force. On my systems, I personally have never had a brute-force attack succeed against SSH. However, before I started following simple lock-down rules, I can tell you they certainly tried.

A quick search on Shodan—a search engine that targets Internet-connected devices—shows 20,984,090 systems with SSH open to the world. Do they need to be set up this way? I can almost guarantee you that each and every one of those systems receives several SSH brute force attacks per second. While I was writing this article, I started a droplet on Digital Ocean, left SSH open to the world, and tailed /var/log/secure. Within around 20 minutes, I saw the first SSH probe. Within an hour, the flood of brute force attacks started.

Jun 23 01:35:33 centos-s-1vcpu-1gb-sgp1-01 sshd[10926]: Did not
receive identification string from 81.22.45.137 port 61000
Jun 23 01:37:47 centos-s-1vcpu-1gb-sgp1-01 sshd[10928]: Invalid
user fake from 104.248.132.25 port 36050
Jun 23 01:37:47 centos-s-1vcpu-1gb-sgp1-01 sshd[10928]:
input_userauth_request: invalid user fake [preauth]
Jun 23 01:37:47 centos-s-1vcpu-1gb-sgp1-01 sshd[10928]: Received
disconnect from 104.248.132.25 port 36050:11: Bye Bye [preauth]

In short, a bad password on a default sshd installation could be all that stands between the bad guys and your system. While the default configuration for OpenSSH is decently secure, it can stand to be hardened. Luckily for you, I have suggestions.

Restrict access to port 22

Start by checking to see if the default SSH port (port 22) is open to the world. You can do this by running Nmap, which will probe your network according to your specifications:

Starting Nmap 7.70 ( https://nmap.org ) at 2019-06-22 20:56 EDT
Nmap scan report for 167.71.200.117
Host is up (0.26s latency).
Not shown: 995 closed ports
PORT    STATE    SERVICE
22/tcp  open     ssh

Narrowing down who can access port 22 is the most basic thing you can do to secure your system. I realize that this tactic may not work for every scenario. Maybe you’re running a public use system, or maybe the CIO travels a lot and might access systems from many different places (we’ll talk about VPNs in a bit). My point is that you may have a valid reason that SSH absolutely needs to be open to the general public. Think very hard about avoiding that configuration, though. I’ve done it, but mostly out of laziness. There’s usually a better way.

If you’re using a cloud provider, configure a security group to only accept SSH traffic from the network you're working from. This task should be simple if you’re accessing this provider from an enterprise environment. You probably have your own IP address range, which means that you can whitelist it and block everything else. If you’re accessing the cloud provider from home, though, you may have to perform some tricks to whitelist your ISP’s dynamically-allocated IP block. Using a security group should let you change the IP address range through your cloud provider’s self-service portal if needed.

If you’re hosting a system that's not behind a firewall, use the host-based firewall to block port 22 from anywhere but your IP address range, using the same methods I just described. The problem there, of course, is that if you're locked out because your IP address changed, you’ll have a harder time making the necessary changes.

And don't forget that there are benefits to an enterprise environment. If your server is on an enterprise network, there’s a good chance there’s a network firewall you can use to lock down SSH access, so use that fact to your advantage. Defense in depth folks, it’s a thing!

Require a VPN for remote access

Ok, so let’s talk about that traveling CIO who absolutely needs access to ssh from a hotel room. The solution seems obvious to those of us who have been around the block, but if you’re considering opening SSH access to the entire world, maybe you need to hear this: A VPN, or Virtual Private Network, lets you build an encrypted tunnel from an endpoint—like your CIO’s laptop at that hotel in Maui—back to your enterprise network in Seattle. This connection is protected in its own way, and encrypted.

However, your dozens of servers that aren’t accessible to the world could become accessible through this single point. Yes, this practice shifts the attack target to the VPN, but that means there's one more hurdle for the bad guys to get through.

Deny root access

Now we’re getting down to actual sshd configuration. The OpenSSH daemon has an option called PermitRootLogin. By default, this option is set to Yes, since when you install some systems, only root exists at first. When this situation happens, you need the root account in order to access the system and perform the initial configuration.

I recommend adding a user during the install process, or as part of your golden image, and turning root logins off from the get-go. There’s no reason to log in as root, and there’s certainly no reason to log in as root over SSH. So, change the configuration line to: 

PermitRootLogin no

Implement key-based authentication

Initially, I used key-based authentication as a convenience. I generated a private key, added the public key to my authorized_keys file on the servers I managed, and could then securely connect to my systems without a password. Doing this saved time, and made automation possible. WIN! It wasn’t until later that I found out that SSH key-based authentication can be leveraged to eliminate password brute force attempts.

To generate a key, run ssh-keygen on a Mac or Linux box. Maybe you Windows folks can do this natively now, but on Windows machines, I’ve always used PuTTYGen (and then used the key with the PuTTY client). You’ll be asked for a type of key, and a length. I’ve been making 4096 bit RSA keys lately. The more bits, the harder the key is to crack. So why not?

[gangrif@meliodas ~]$ ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/gangrif/.ssh/id_rsa): ./test-key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ./test-key.
Your public key has been saved in ./test-key.pub.
The key fingerprint is:
SHA256:xfHOf4t3V3CCkJ+3FbEnQusAjBwOjBXfOa9WdGXBMqc gangrif@meliodas.undrground.org
The key's randomart image is:
+---[RSA 4096]----+
| ++o.+. ....+o.|
| . .+o..+o+o+o..|
| o + =o=B..o|
| = *E.+.+|
| S o +. * |
| o .. .|
| o . o|
| . .o+|
| ...o|
+----[SHA256]-----+

What you then end up with is a new file called test-key (in this case), and test-key.pub. The key in test-key is my private key, and test-key.pub contains the public key. Protect the private key with your life. You can leave the public key laying around wherever you’d like, though, as it’s useless without the private key.

[gangrif@meliodas ~]$ cat test-key.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDAQxmjoHO6i4Kkq8vp49KtqAsMcw+ptgvd+PGjxVYkDPGdzRZpJhq0c3BDstFs86ENlojs61zZaP3MihLR0BlDdIyM3jh9TmVvmOPiylhu4X9QliAca/ODxyVp76OpTKm+QcgBi/7i/JNiJdEgcSy8izB0Oil7LZjIhS6wCs3mQFVbkPXPfe1lmHQIcuMDOKO0RuyecY/lrKodcr/YbEE4GsM/P11QdDPZ78lq83G3H5vqLqMrxVKkLw7Z4//+PZPFFxi/N1EBwBp/uPEo4MfD1IwSmnKromRlkYTdOYlbiCNosdvEobtMARySdqjv7RDn0Iwb3JxioUW6jQdAXfl8d01LvX/0Yb1tq5hF44ahwvI6TyoB4z3DRs+3cFiMPWQR7LK8/qQOvHk5I2NMBAV9KuSN2ML0d7xeB8tglw38PYjhZsXl/t2rAWT4n8PU0o6+Hrrli+WEfvk8QWpLesmBBmzG0sLjH0mXBU/xN4UBLLJNlrsUQ/TzEsdknntMhh7leOzechlU3PiTNuUitweT9NqLJwCP+CHbeSJn2M5qzb090CPnCGpnr2GOfm2dL+RfHQi2R1Cf04nBDq3WuhAPJY5SoEw2llfSgA2GlOdhcY9N9Bl9rLUBPgQ6ttBd7qZJ6E3BkiPlHnU+kSSpYr8Gkw1oDGbtd2UUjExZqvz4rQ== gangrif@meliodas.undrground.org

On machines you want to connect to without using a password, copy the public key into .ssh/authorized_keys (or have the user do it, depending on the situation). Now, I say without a password, but you still need to unlock the private key with the password you set during generation. You did set a password, right?

You can also remotely copy this key to a server using ssh-copy-id.

Disable password-based authentication

Remember when I said that key-based authentication opens you to the possibility of locking down SSH brute force attempts completely? Well here’s how. With your private and public keys in place, SSH isn't prompting you for a password anymore, right? So why leave that avenue open as a vector of attack?

On every Linux server I run, I add my public key as part of our base install, and I turn off SSH password authentication before the system even hits the network. This is a setting right in sshd’s configuration file. Again, the default setting allows password authentication, because that feature has to work for configuration. Once you have your keys in place, though, turn that off.

PasswordAuthentication no

Congratulations, you’re now immune to password brute force attacks.

SSH user jail, with chroot

The chroot—usually pronounced “chi-root”, or “ch-root”—command is a neat tool. It lets you change the root directory seen by a process and its children, hence the name. It’s great for troubleshooting a system where you can access the disk, but it won’t boot. You just mount its disk and chroot to /mnt/whatever.

This is also a neat security tool for something like a shell server. You can chroot a user at login, so they literally can’t see the rest of the filesystem. I don’t do this often, but it is a very viable jail for users. I found what looks to be a decent write-up here.

Rotation

It’s worth considering password and ssh key rotation. I’ll try to outline the good and bad here.

Password Policy and Rotation

I’ll lump password policy in with password rotation because they’re related. Password policies that enforce complexity are a great way to make your users pick a “strong” password. However, this has some caveats, especially when coupled with arbitrary (i.e. timed) password expiration. I could write a whole article on how I feel about password policy and rotation, but I’ll try to keep it brief here.

Last year, NIST did something of an about-face on their password guidelines, changing a few of their suggestions that had been ingrained in most of our minds for most of our careers. The Infosec community (which I dabble in) has been suggesting for years, an 8 character random passwords, or 8 character minimum length with strict complexity rules, were doing more to HARM password hygiene than help it. Combine strict complexity rules with a 3-6 month password rotation policy, and you’re setting your users up for frustration. That frustration leads to password re-use and other bad habits. So think hard about whether you want to impose this on your users. The new recommendations lean toward expiry only when a breach or compromise is suspected, and policy which leads your users into strong pass phrases rather than passwords. “This 1s my p@ssw0rd 2019.” is a lot harder to guess, and crack, than “W1nt3r2019”. Which, by the way, is a common go-to for both password setters, and red teamers. Bear in mind however, if you’re asking smart admins to rotate their passwords, you may not have this problem (because they understand WHY they’re doing it, and how important it is). But for your users at large, arbitrary expiration and complex “pass words” are becoming a thing of the past.

SSH Private Key Rotation

Like any identifier that can be generated, it might be good to consider rotating your private key periodically. Your private key isn’t quite as easy to steal or guess as a password might be, but it is possible that someones lifted your key without your knowledge. This gets into the “how paranoid would you like to be” territory. Changing the password on your private key isn’t good enough however. That password unlocks that key, if I swipe your key today, and you change your password tomorrow, the copy if your key that I have still has your old password. If you’re going to do this right, you need to generate a whole new key. Now’s a good time to make the key length longer if the standard has moved up since you last generated a key. Maybe you generate a new key every time your employer issues you a new laptop, maybe you do it once a year on your work anniversary. I haven’t found a solution that will manage this for you though.

Bear in mind, that generating a new key means all of the public keys you have out there in the world are now invalid. Or at least they’re not valid with your new key. You’ll likely need to use your old key, to place your new public key.

MFA

Multi factor authentication is becoming the norm. Some people have tried to claim that SSH keys are a form of MFA, they kind of aren’t. Especially if you’ve disabled password auth. You can however integrate mfa like TOTP, or YubiKey into your systems PAM config. Central auth solutions like FreeIPA make this easy.

Conclusion

So there you have it, I hope you take this information and apply it to your systems to improve your security. Don’t be one of those 21 million systems with SSH open to the world. There’s just no reason for it.

Thanks for reading!

This article was originally published at Undrground.org. Republished with permission.