Five best practices for administering remote systems

For some of us, administering systems that are fully remote and in a different facility is a daily way of life. For others, recent work-from-home restrictions might be one of the first times that you have administered a system that you have no physical access to if something goes wrong. Fear not! In this article, I'll take you through some simple suggestions to set you up for success when administering your systems from your home or from wherever you might be.

Tip 1: Lockdown remote access protocols

The first step to managing a remote system is... well, being able to manage it! Remote access protocols, such as SSH (and in rare cases, remote desktop), enable you to configure and manage your systems from anywhere. But you don't want your remote management strategy to turn into a security risk inadvertently. You should always take the time to understand the remote management protocol or system that you are using and secure it accordingly.

At a minimum, you should follow two standard best-practices for SSH security.

1. Disable password authentication and use SSH keys
2. Disable root login over SSH

SSH is highly configurable, and you can implement plenty of security enhancements, such as disabling weaker ciphers or limiting the use of particular SSH keys to specific IP addresses. Be sure to check out the sshd_config man page for more information. If you're feeling overwhelmed and don't know where to start, Enable Sysadmin has you covered with an article about locking down SSH.

Tip 2: Limit ingress points for remote management traffic

If you're suddenly in the position of managing remote systems and need to get access as soon as possible, it might be tempting to start loosening security safeguards to get you up and running. Punching holes in your firewall to permit SSH connections and loosening security ACLs permits remote management protocols through your edge network.

The problem with such "temporary" workarounds is that they inevitably become permanent. Set yourself up for success by only allowing remote access protocols, such as SSH or remote desktop, through a secure VPN with appropriate authentication and authorization controls. While setting up a VPN environment might seem like a bit of initial up-front work, the security benefits and reduction of administrative burden pays dividends. You won't have to worry about home or remote IP addresses changing since everyone goes through the VPN endpoint. And your systems are more secure because they can only be reached through an official VPN setup.

Tip 3: Know your network fundamentals

When administering a remote system, you don't have the luxury of walking over to a rack and tracing a network cable to figure out what the server is plugged in to. Additionally, making a network configuration mistake can be a costly accident: you might suddenly lose remote access entirely. Knowing your network fundamentals is absolutely essential for successfully managing remote systems. Using command-line tools, such as ip, lldpctl, and tcpdump, can help you build a mental picture of your network topology. This mental image enables you to make safe, sensible network changes even if you're hundreds of miles away from your beloved servers.

Knowing network fundamentals can also save you if you lose access to a remote system. For example: knowing that hosts on the same Layer 2 broadcast domain can reach each other can allow you to fix a routing change gone awry: simply use one server as a jump box to the other server, and you're in!

Tip 4: Have access to a remote console

There's nothing quite like rolling the classic keyboard, video, mouse (KVM) "crash cart" over to a completely unresponsive server and working to diagnose it while basking in the warmth of the data center hot aisle. But what will you do if one of your servers suddenly dies and you can't drive into the office to get it back online? You want to be confident in the remote access technology available to you for your servers when you need to administer them remotely.

For physical servers, this is the remote management facility provided by the hardware vendor (such as IPMI). For virtual machines, a console is provided by the hypervisor, such as a virsh console in KVM. Either way, you want to be comfortable with the tools available to you if something goes wrong.

It's also a best practice to confirm remote access to a server before you make any potentially dangerous network changes, such as changing the IP address. If something goes wrong and you lose your ability to SSH to the machine, you want to be able to log in and revert your changes. Confirming access includes verifying that the root password you have on file works. The worst feeling is staring at a login prompt that keeps failing authentication because your root password doesn't work.

Tip 5: Use SSH agent forwarding to your advantage

If you've read our previous articles about using SSH keys, then you're familiar with the idea of using ssh-agent and ssh-add to cache your key locally. Key-based authentication avoids the need to continually enter the passphrase for your private key when connecting to remote systems. But there's another useful feature in the SSH protocol that you can use to your advantage when administering remote systems: ssh-agent forwarding.

Using ssh with the -A flag forwards your ssh-agent session to the remote host. This allows your private key to be "available" on that remote host. You can then "jump" from the first system to the second system, using the same SSH key authentication that you would use from your PC.

When is this useful? Consider the following common remote administration scenario: You have a very large file on host1, and you want to transfer it to host2. Both systems trust your SSH key. You could rsync the file to your local machine from host1, and then rsync it up to host2, but that would be inefficient. It would be much better if you could just copy it directly from host1 to host2. Agent forwarding can help with this:

# Add my SSH key to the agent
anthony@laptop$ ssh-add
Enter passphrase for /home/anthony/.ssh/id_rsa:
Identity added: /home/anthony/.ssh/id_rsa (/home/anthony/.ssh/id_rsa)

# Connect to host1 with a forwarded SSH agent
anthony@laptop$ ssh -A ansible@host1

# Rsync a file to host2. This uses the forwarded agent
ansible@host1:~$ rsync some-file.txt ansible@host2:~/

# SSH to host2 from host1. This also uses the forwarded agent
ansible@host1:~$ ssh ansible@host2

# Looks like the file made it over OK!
ansible@host2:~$ ls
some-file.txt

Wrapping up

This article covered some basic best practices for administering remote systems. For some sysadmins, this is nothing new: managing remote server farms is a daily activity. For others, the current world climate might be forcing you to be farther from your servers than you'd really like. Either way, this article reviewed some useful tips and tricks for keeping the servers humming, even when you're not in the office.

[ Free online course: Red Hat Enterprise Linux technical overview. ]