The 5 things I wish I'd known before becoming a sysadmin

For those sysadmins with a few years of experience behind you, do you ever feel like there were some things you should have known before flipping the, "I want to be a sysadmin" switch? I'm sure there are. I don't know that it would have changed my mind or the course of my career but there are a handful of things I wish I'd known before taking the job. The five things I list in this article are the ones that I find to be the most important. You might have different ones.

Users - Our reason for existing

Whether they're named users in your organization that you have some support responsibility for or all those random, anonymous users on the Internet-at-large, users are our reason for existing. My sysadmin colleagues used to complain about users, their weird requests, and incidents involving security, neglect, or other issues. I always took a different approach to end-user support. I had the idea that without users, they wouldn't need me to do a job. In other words, I exist because they exist.

[ You might also like: How to manage Linux permissions for users, groups, and others ]

This approach helped me to avoid the angry and irritable reactions that many users received when presenting their issues to a sysadmin or through a ticketing system. I felt bad for some of my users because I know that they could have solved most of their own problems if we'd granted them administrative privileges. Those who couldn't solve their own issues were at the mercy of my groups' perennial bad attitudes and dismissive demeanor. Not all of my coworkers were like this. We had some very friendly support folks who really enjoyed customer service and making our end users happy and productive. I fell somewhere in-between. I neither held users in contempt or in particularly high regard.

I once had an experienced developer bring me her keyboard complaining that it didn't work and that she required a new one. I noticed the wholesome aroma of chicken soup emanating from the keyboard when she handed it to me. Before I spun around in my cubicle to give her a new keyboard, I asked, "Was it homemade chicken soup?" She paused for a beat and then responded, "Yes, it was." I gave her a new keyboard and said, "Totally forgivable then." She walked away smiling. It could have gone much differently had I wanted to give her a hard time about rendering a perfectly good keyboard useless with chicken soup. Her productivity was worth far more than a $40 USD keyboard. Plus, I gave her a better impression of the support staff in my group. I call it a win.

You might not know this, but as sysadmins, we serve the end-user. I think it's better to do so with a smile.

Security - The constant battle

Am I wrong? Security is a constant battle on all fronts. User security, system security, application security, network security, service account security, host-based firewalls, keeping up with the constant flow of common vulnerabilities and exposures (CVEs), and maintaining patches accounts for most of the gray hair on many a sysadmin's heads.

In my years as a sysadmin, I've seen security discussions go from, "Security is an important aspect of our jobs" to "Security is the most important job we have." Everyone has to think about security. As one colleague put it, malicious actors are always on the job, and we have to be as well.

I don't know if you've ever watched your system's logs or your corporate firewall's logs but it's very telling indeed. Multiple times per day, or even per hour, your network gets probed. Malicious actors continuously run scans, probes, vulnerability tests, and reconnaissance against your network. It's shocking to see how many times and ways they attempt to get in. And it isn't just over-the-network attempts, is it? It's phishing, it's ransomware, social engineering, social media, card readers at your bank or at the gas station. It's everywhere, and it gets exhausting. It makes me tired just thinking about it.

What keeps us coming back for more? Every day is a new challenge.

Late nights and all-nighters - Dealing with maintenance windows

So, you're expected to roll into an office every day somewhere between 8 AM and 9 AM, head home from 5 PM to 6 PM, and then connect into the corporate VPN again at 11 PM for maintenance windows that might last anywhere from one to six hours or more. And, then be back to work the next day at the usual time. Great, where do I sign up for that? Can you also pay me in cryptocurrency?

I hope your hours are more flexible than mine were when I worked as a full-time sysadmin. We were told that "It doesn't matter if you stayed up all night. You're expected to be at your desk every day." This was the corporate standard. To be fair, most managers allowed us to come in after lunch to compensate for the lost sleep and lack of family time. But we were still expected to have our phones on, charged, and by our beds even if we took "comp" time just in case our managers needed to contact us.

Although many standard racked and cloud-based systems have enough redundancy built into them that maintenance could be performed any time of day, it generally is not. Late nights are still the industry fashion and probably will be for some time to come.

Prepare yourself for late-night maintenance windows with a step-by-step list of things you need to do. Have a good back-out plan too.

On-call rotation - When things go wrong

If you're a sysadmin, then you know all too well what on-call rotation is. On-call rotation, for you non-sysadmins, is the periodic handoff of all system administration notifications, escalations, and meetings to each sysadmin in a support cluster. For example, if there are five sysadmins in a support cluster for company X, then every five weeks, you are the primary on-call sysadmin, and then likely the next week, you are secondary.

This all means that for one week, all evening and weekend issues come directly to you for resolution. For the week that you're secondary, you might not have to do anything, depending on how good and available your primary is. I've had both good and bad primary sysadmins in my rotations. A good one might ask you to assist if something big comes up. A bad one won't answer the text messages, and so the Help Desk will escalate to you, the secondary.

On-call rotation is a necessary evil for sysadmins, but it doesn't have to be something to face with dread. Be sure that your coworkers are all prepared to fully support your clients. Mentoring and screen-sharing can help you teach junior-level or new sysadmins while working through on-call issues.

Hardware - The ever-evolving hardware landscape

Have you created any "golden" images to install on all new systems as they arrive? How many golden images have you created? Have you entirely given up on creating golden images for deployment? When you work in a large environment where you support hundreds or even thousands of server systems, you need to be able to streamline operating system installation, software deployment, standard security settings, and so on. It's hard to do with the ever-evolving hardware landscape, isn't it?

This is one great point in favor of using cloud-based systems or virtual machines. You can create templates and deploy them at will without concern for any underlying hardware inconsistencies.

The workaround for hardware changes is to create a custom ISO image set up with all of your chosen features. It's not as good as a fully automated solution. However, I've also found that even with fully automated solutions, you still have to customize your system with hostname, IP address, certain system-specific configurations, updates, and more.

Don't fight the changing hardware scene. Install new bare metal systems with a custom ISO or even from a standard downloadable vendor image. Create templates for your virtual systems for faster deployment.

[ Free online course: Red Hat Enterprise Linux technical overview. ]

Wrap up

There you have the five things I wish I'd known before becoming a sysadmin. Again, I'm not sure it would have changed my mind to know them but being informed might have made my sysadmin life a little happier overall. Knowledge is power. My best advice is to learn all you can about everything you can. Specializing is not a great thing for your long-term career. Embrace end-user support because users aren't going away. Ever. The other things you can deal with as they come along. Being a sysadmin isn't a bad job but you have to remember that it's just a job.