How to harness the power of incremental updates

How many times have you put off patching and updating because something of higher priority comes along, only to finally perform a "big bang" update that almost always never goes as planned?

How many times have users complained that systems are old, slow, and out of date?

How many times has a security bug bitten you that a recent system patch might have fixed?

How many systems do you outright fear patching because it's been so long, and you're not sure what will happen if you do?

Although keeping systems patched and up to date is a core task of IT operations teams, let's be honest: Nobody really likes doing it. Patching systems is like flossing teeth: It's necessary but really not enjoyed.

What if, instead, you could reach a state where keeping systems patched and up to date—from the smallest virtual machine to the largest systems—could be done reliably, repeatedly, and in an automated fashion?

Is it possible to actually enjoy keeping systems patched and updated?

I think that it is—but only with evergreen IT.

Enter evergreen

Fundamentally, evergreen IT is about having IT services continually kept up to date. The concept is not particularly new. The earliest reference I can find to "evergreen IT" comes from PriceWaterhouseCoopers in 2009. Since then, we have had an explosion of cloud and automation technologies to help us achieve a state of evergreen.

Generally speaking, you can categorize evergreen environments by the following:

• Small, frequent, incremental updates. Infrequent, terrifying "big bang" updates do not occur.
• Heavy use of automation and a tightly managed IT infrastructure. You cannot do this manually.
• A culture that's committed to continuous improvement and willing to continually iterate its processes.

Microsoft's decision to change Windows to a "rolling-release" model (since Windows 10) is a prime example of shifting to an evergreen operating system. Frequent, small, incremental updates are easier to deploy, test, and keep everything up to date.

[ Download the checklist to learn 6 ways to promote organization-wide IT automation. ]

Here are some other benefits:

• Reduced disruption: Small, frequent changes are less likely to cause outages, particularly when they result from repeated and reliable automation.
• Better user experience: Users get access to newer software faster.
• Happier business-as-usual (BAU) teams: Teams can tackle higher-value tasks.
• Reduced incidence of shadow IT: As evergreen increases confidence in corporate IT delivery, there's less shadow IT going on.

Sounds great, but it doesn't drive revenue

I consider keeping systems patched and updated a "necessary, non-value-add activity." It doesn't really generate any extra revenue for a business or improve service delivery. Nobody's going to pay you for patching your enterprise resource planning system. But still, someone must do it.

You must do it to comply with policy or legislation in your industry, avoid exposure to ever-evolving security threats, and give your users access to the latest tools available to them.

Keeping systems patched is necessary, but it's not the "pointy end" of the business. So if you've got to do it (which you do), it makes sense to make patching and updating as low-cost and low-friction as possible, right?

That's where evergreen IT comes in.

How do we make it happen?

Make no mistake: Achieving a state of evergreen IT is an activity in transforming IT delivery. It takes a concerted effort and a dedicated focus on making it happen. You cannot buy it off the shelf, and you have to do it from within the organization.

[ Explore considerations for a variety of modern cloud architectures in the eBook An architect's guide to multicloud infrastructure. ]

To achieve it, you need:

• Automation everywhere: The goal is never to patch a system manually. If you have to log into a terminal to issue an update command, that's not evergreen, and it won't scale. All systems under evergreen control must have automation that enables them to get updated without IT operations intervention. Pick an automation technology with a low entry barrier, so newcomers can contribute too.
• Pre-production environments: You should never apply untested updates directly into production systems. Always have a pre-production environment that is as representative of production as possible. Your automation needs to test these changes in this environment first, and only when you are satisfied should the rollout to production occur.
• Automated testing: How do you know if a system is working as expected? You need a battery of automated tests for each system that give you confidence that the system is operating as expected. These tests should come from the domain experts who designed and deployed the system. Testing is a core part of evergreen, and it needs to be automated; again, manual testing just won't scale.
• Reliable content streams: You know never to open attachments in emails you don't recognize; applying patches to systems should be the same. You need a reliable, verifiable stream of content. This content must come from a trusted source, be verified as free of malware, and ideally come with digital signatures to verify authenticity and integrity. The lifecycle of the content should be at least as long as the anticipated lifecycle of your system. Otherwise, you'll be left holding a system that you can't patch, yet is still in active use. That's a risk.
• To shift your mindset: Evergreen is not just about technology. It is also about people and processes (sound familiar?). You need to bring an agile approach to the problem and be willing to test and adjust your approach continually. It requires an investment of time and energy but pays itself back quickly.
• To start small: Don't tackle your most complex system first: It will probably fail and demoralize the team. Pick a small system (it could be a single host!) to bring under evergreen control, and use the lessons from that experience to inform the next. Maintain focus and discipline, and more and more systems will fall under evergreen control.
• To keep an eye on things: Maintain visibility of all systems, evergreen and not. When were they last patched? What's the status of any known but unpatched vulnerabilities? Are all functionality tests passing? You can't maintain an evergreen state unless you have wide-ranging visibility over your infrastructure and software.

Last, please don't underestimate the power of open source and open standards in helping you achieve evergreen IT. Open source and open standards help keep your options open long term. They free you from needing to keep an old, aging, unsupported system running because it's the last version of the software that supports closed source or proprietary APIs or data formats.

Apply the Pareto principle

Evergreen IT is an excellent state to achieve, but you don't have to achieve 100% evergreen. It's a laudable goal, but it's probably out of immediate reach for most organizations. Follow the Pareto principle instead. Focus on the 20% of systems that take 80% of your time and bring those under evergreen control.

That will be enough to see significant service maintenance and delivery benefits in your IT organization. Better services, reduced downtime, happier BAU teams, and less effort spent on the necessary non-value add activities. Once that 20% gets dealt with, you'll have far more time available to tackle the remaining 80%.

Lastly, one final reminder to start small. Get some quick wins on the board, and over time, the big wins will take care of themselves. Plant the seed, keep it watered, and it will grow. We might not enjoy flossing, but perhaps it's still possible to enjoy patching.

This article is adapted from the author's LinkedIn post and is republished with permission.