How to hire the right sysadmin for the job

I have been in IT for 45 years and been on both sides of hiring interviews for Sysadmins and other technical positions. The techniques I have experienced are many and varied, but most were rooted in layers of bureaucracy, inertia, and institutional bias. Some of these layers were intended to meet specific legal requirements, but many of them came about because those places had no idea how to find and hire good Sysadmins.

In this article, I will present my ideas for how to attract and hire the best candidates for Sysadmin positions, then interview them. In the interest of full disclosure, I am not an attorney, nor am I an HR expert. I will discuss this topic strictly from a technical standpoint and how to best discover a candidate’s qualifications. We will also look at how intrinsic bias affects hiring. Some parts of this article are based on excerpts from my book, The Linux Philosophy for SysAdmins, Apress, 2018.

Specifications

As a Sysadmin, when I start a project I like to have a clear set of specifications. You should also have one when hiring a Sysadmin. The worst thing I encountered when seeking a job was a set of job requirements that read like a kid’s Christmas wish list.

So the organization wants someone who is an expert with at least a Ph.D., multiple certifications, 10 years of experience in 14 distros of Linux, three versions of Unix, and all versions of Windows going back to Window 95 because they still haven’t upgraded. Plus, the right candidate also must be able to program in 8 different languages and interpret a TCP/IP dump on the fly. In addition, they want someone who can understand and rack and stack hardware of all kinds, fix it, and build a highly secure, fault-tolerant data center with a remote data recovery plan. The hiring manager also wants someone to be available 24x365 and—well you get the idea. On top of all of this, most of the time these organizations are only offering to pay entry-level wages with minimal vacation and benefits.

Although this list is somewhat exaggerated, it is not by much. I skipped over those ads. If you advertise unrealistic job requirements you will never attract good people. So here is my personal set of requirements for a mid-level Linux Sysadmin, modified just a bit to be more realistic. The ideal candidate will meet about 80% of these and can learn the rest.

• Sysadmin with three to five years of experience with at least two common Linux distributions including the distribution in use at the hiring organization.

• Installs and configures Linux for specific roles such as workstation, router/firewall, and servers of various types.

• Creates scripts using bash or another appropriate scripting language to automate recurring and one-time tasks.

• Uses package management tools to update, remove, and install software.

• Performs Linux upgrades to new releases.

• Configures workstation network connectivity.

• Manages system services and can monitor, start, and stop them.

• Monitors system resources and responds accordingly when problems occur.

• Performs Linux and hardware problem determination and then resolves these problems.

• Uses virtual consoles, tools like screen, and terminal multiplexers to leverage command line efficiency.

• Configures and uses SSH and public/private key pairs (PPKP) to perform all tasks on any remote Linux host.

• Installs various hardware devices such as hard drives, SSDs, graphics adapters, all types of PCI pluggable adapters, printers, replacement motherboards, various types of memory, and performs any required configuration from the command line.

• Implements and monitors workstation security using commonly available tools such as firewalls, file ownership and permissions, and reactive tools such as Fail2Ban.

Yes, this is a long list and I could add more, but by now you get the point. Every item relates directly to the skills needed by a mid-level Linux Sysadmin. There are no superfluous lists of acronyms that have no relation to the job. You could add optional requirements as well, giving extra points for these but considering none to be a deal breaker. Frankly, the best will meet most of these optional requirements:

• Has at least one Linux certification of any type, but preferably from Red Hat or Cisco.

• Has a home lab to experiment with even if it only contains a few Raspberry Pi single board computers (SBCs).

• Builds their own personal computers from scratch from parts purchased online or at a local computer store.

For an upper-level Sysadmin, I would also add server requirements.

Notice that Windows is nowhere to be found. Anyone who is or wants to be an expert Linux Sysadmin will not have time to deal with Windows in an effective manner. And the same is true for really good Windows admins; they will have little or no time for Linux.

Salary and so much more

Your job advertisement should also include the perks, such as salary range, vacation—yes, Sysadmins need a vacation—and other paid time off, whether working from home at least part of the time is an option, and opportunities for advancement.

One of the most important perks for many Sysadmins is how much outside technical training they can get each year, paid for by the organization. This training should be in the form of classes or attending one or more of the many Linux- and open source-related technical conferences.

The Interview

There are many methods for hiring the right people, but none are foolproof. I have, however, found that the right interviewers and the right interview strategies go a long way toward making that happen. Many people who interview for Sysadmin positions are not ready, because they have no idea how to solve problems. Sometimes you cannot tell until you hire the person, and it may be difficult to unhire them at that point.

The interview structure I recommend here can help ensure that the people you hire can use critical thinking to identify and solve problems. Here is the key: you are not looking for the person who can best regurgitate memorized answers. You need the person who has the best critical thinking skills.

The best interviews I have participated in—only a couple of times—consisted of three parts, which I will address below. The best interview I ever had as an applicant was at Cisco. They used two of these three parts, but only because I had worked extensively with the lab manager at another organization so he already knew my skill set. Even so, that interview took more than four hours.

Part 1: Meet the manager

The first fifteen minutes of my best interviews were almost always meeting with the hiring manager, getting acquainted, and discussing the job’s requirements and parameters. For the applicant, this part is about discovering whether the job and the manager are worth continuing the interview. As an applicant, I have walked out of interviews at this stage because I knew the position would not work for me.

For the manager, this part is about assessing the applicant to see whether they are a good fit for the job. I have also been told at this stage that I would not be a good fit, so it works both ways. In one case I was told this because they wanted a superstar Linux Sysadmin who would also be the Windows Sysadmin. Personally, I have no time for Windows. There are always new things to learn about Linux.

After fifteen minutes of this part, the Cisco hiring manager withdrew and turned me over to their team.

Part 2: Meet the team

This is where department subject matter experts meet with the applicant and ask non-stop technical questions for a couple of hours. It is ok—really it is—if the applicant says they don’t know something. The idea here is to define the limits of the applicant’s knowledge and skills. The “I don’t know” answers help to delineate that.

Questions should start with the obvious, things that the applicant should know, and then progress to the more complex topics that the applicant may or may not know. In my Cisco interview, I was asked questions that were completely out of my area of expertise, but the objective, in that case, was to assess my critical thinking skills more than anything else. This is a good tactic and should be introduced to the applicant as such.

Part 3: The skills demonstration

This is the last and probably should be the hardest section of the interview. Most organizations don’t do this because they don’t know how. There also can be legal issues with testing but, if the test is truly representative of the type of work the applicants will be doing, and all of the applicants are required to take the same test, then (check with your lawyers) it should be fine to test.

One place I worked used a hands-on test. Our test was simple. The organization set up a Linux host with three specific but fairly simple problems that the applicant had to fix within a specified amount of time.

Although this test was about finding and fixing problems under the duress of time pressure, our primary concern was the manner in which the applicant approached the task. Those who panicked or who proceeded more or less randomly, thrashing about with little direction were quickly eliminated from our consideration. Even if they did not resolve all of the problems, those who proceeded with some sense of purpose, with a well-developed problem-solving algorithm, and who exhibited well-developed critical thinking skills, were the ones we considered most likely to be successful. We could easily teach technology. We could not easily teach problem-solving techniques, critical thinking, or Zen.

Other tests you might consider are to have the applicant install the Linux distribution used most commonly in your organization and to configure the resulting host in a manner typical to those you normally use. Just give the applicants a set of specifications and leave them alone for a specified period of time.

Skills testing like this should be done on a single host using virtual machines (VMs) on a virtual test network that can be rebooted from snapshots to prepare for the next applicant. The host used for testing can be completely air-gapped from the organization’s physical network to provide the maximum amount of security.

About certifications

Although I have listed a certification as an optional requirement, let me dispel the myth of certification.

As a Sysadmin, part of my responsibility in many of my jobs has been to assist with hiring new employees. I participated in many technical interviews of people who passed many Microsoft certifications and had fine resumés. I also participated in many interviews in which we were looking for Linux skills, but very few of those applicants had certifications. This was at a time when Microsoft certifications were the big thing but during the early days of Linux in the data center when few applicants were certified.

We usually started these interviews with questions designed to determine the limits of the applicant’s knowledge. Then we would get into the more interesting questions, ones that would test their ability to reason through a problem to find a solution. I noticed some very interesting results. Few of the Windows certificate owners could reason their way through the scenarios we presented while a large percentage of the Linux applicants could.

I think that result was due in part to the fact that obtaining the Windows certificates relied upon memorization rather than actual hands-on experience, and the fact that Windows is a closed system which prevents Sysadmins from truly understanding how it works. I think that the Linux applicants did so much better because Linux is open on multiple levels, and logic and reason can be used to identify and resolve any problem. Any Sysadmin who has used Linux for some time has had to learn about the architecture and has a decent amount of experience applying knowledge, logic, and reason to solve problems.

So if you use certification as an optional requirement, I recommend Cisco or Red Hat. The main reason I recommend these two specifically is that both, at least in part, are task-based. I have worked for both companies, myself, and taken the tests for each. In both cases I was unable to pass the first couple of tries because the tests were not about regurgitating memorized facts, they were required actually performing specific tasks. These tests are not easy nor should they be. That said, none of these tests are perfect and the tests I took for Red Hat and Cisco certification had issues. Both have changed over the years, so I have no idea how they stand now.

Just as task-oriented tests such as Red Hat’s and Cisco’s should be valued over multiple choice, so should a hands-on job applicant test be preferred as the primary arbiter of who can best do the job. I would personally rather hire someone who can follow the scientific method of problem-solving that I outline in my book, even if they do not finish in the allotted time, than someone who can spout rapid-fire answers to questions.

Understanding your intrinsic biases

Intrinsic biases are tricky and—completely aside from legal issues—can cause hiring organizations to overlook some of the best-qualified candidates. See Examples of Unconscious Bias in Job Descriptions for more information on this topic.

In one study—Why Women Don’t Apply for Jobs Unless They’re 100% Qualified—women are likely to apply for a position only if they think they meet 100% of the requirements listed, while men apply if they think they meet 60%. Another article—Tech industry job ads: Older workers need not apply—shows that ageism is rampant in tech job listings, and phrases like “recent grad” is blatant code for, “don’t apply unless you are really young.” Way back when I was in my sixties and looking for a job, most of the job descriptions used “recent grad” as a requirement. I applied for some of these anyway and got zero calls back for months. I finally found a job description for a company that wanted a Linux “superstar.” That job was so horrible I quit after a year and started my own consulting business.

Consider what happens when you hire for “culture fit.” Other terms used here might be how well the applicant will “mesh,” “fit in,” or “integrate” with the team. How does any person of color fit in with the culture of your organization if everyone else is white? The article Is hiring for culture fit another form of unconscious bias? discusses three factors that lead to cultural bias. These include using “intuition” which is really a way of saying that this person is enough like me that I think they will be a good hire. Using subjective criteria for evaluating the new hire after a few weeks or months in the job is based on the hiring manager’s need to justify hiring this person regardless of the actual outcome. Finally, if almost everyone in the organization is white males of similar age, there is the group-think tendency to find that applicants of similar characteristics would be “the best fit” for the job.

Another result of intrinsic bias is the bigotry built into devices produced by technical companies that have few or even zero black employees or other people of color working for them. In one case, discussed in Bigotry Encoded: Racial Bias in Technology, a soap dispenser would not recognize the hand of a black person because none of the white engineers thought to test it on the hands of anyone but white people.

Other egregious examples of racism include terminology used to describe inter-server relationships such as master/slave in technologies like BIND, databases, hard drives on a PATA bus, and many more. Some companies and software projects are trying to correct these problems but much more needs to be done. In any event, using racially charged and offensive terms can cause candidates of color to skip your job no matter how much you describe your organization as inclusive and diverse.

There is also ability bias. In one instance, I participated in interviews for several candidates and, based on phone interviews, invited three to continue in person. I could tell as soon as we met the young man with sight problems that the manager would never hire him, no matter that he looked like the best candidate to me and most of the other interviewers. The blatant eye rolls gave it away: no worries about this person seeing that, right? Nothing was said overtly but there was plenty of code coming from management that a disabled person would never be hired for that position, regardless of their qualifications.

Final thoughts

This article covers some of the technical aspects of hiring the right people for your Sysadmin jobs. Accurately representing the job requirements is probably the most important factor in attracting the right people. We also looked briefly at a few of the typical biases that are encountered in job ads and interviews, and how they might affect your ability to hire the best people as well as the effect that might have on your products.

It is my assertion that using the procedure I outlined here will result in hiring the best person for your position—so long as you understand and deal with your institutional biases.