Transforming Your Secrets Management

00:02 — Jamie Parker
In this season of Code Comments, we're covering the well-explored paths to digital transformation. You might think there are a few secrets left to uncover. As it turns out, when your IT modernization project scales into the stratosphere, there are more secrets to know than ever. But these aren't process or development secrets to uncover and unlock the potential of your teams and infrastructure. No, these are secrets you want to keep hidden. These secrets unlock the doors to your application's proverbial kingdom. In this episode, developer advocate, Rosemary Wang and product manager Steve Almy, both at HashiCorp, explain why keeping a company's secrets safe means being a detective, a teacher and a bouncer.

00:49 — Rosemary Wang
There's a joke that secrets are not fun. And I think that that is very much true as a company, when you face the secrets sprawl challenge.

01:02 — Jamie Parker
For the unaware, that quote Rosemary is referring to is, "Secret secrets are no fun. Secret secrets hurt someone." Managing an organization's secrets is a monumental task. While the original quote implied keeping secrets was harmful, in our case, it's sharing them that can hurt people. So first off, what is the secret? Rosemary defined them as, "Anything that gives you access to another part of the system." Think login credentials or certificates or API keys or tokens, et cetera. They're called secrets for a reason. You don't want to share them with anyone who doesn't legitimately need them. A distributed system, a cloud system, a modern system if you will, has secrets in droves. With the proliferation of components and applications needed to run a system, comes an equal proliferation of secrets to gate and grant access. And unless you've been diligent about tracking them from the start, you're going to have some sleuthing to do.

02:03 — Rosemary Wang
Because at this point we're now reaching a point where our systems are so complex we can't really keep track of what's going on.

02:11 — Jamie Parker
There's a qualification. We can't really keep track of what's going on without automated assistance. With dozens to thousands of human users and who knows how many more interconnected application components to wrangle, secrets are all over the place and they're frequently only handled by the people who set them up. Left to their own devices, people naturally do things in their own way.

02:36 — Rosemary Wang
A lot of organizations have gone to cloud and experienced this organic growth of where they put their secrets. If you're using different systems, virtual machines across multiple clouds, maybe you store it in the individual Secrets Manager that is specific to that cloud provider. Other times you might choose to use Kubernetes Secrets on some other cluster somewhere. And other times you might decide to encrypt them and put them in version control and distribute them using some kind of GitOps tooling. And so the result is that you might have thousands of developers who each have a different way of doing secrets and no consolidated way to do it.

03:16 — Jamie Parker
While each method may keep individual secrets well guarded, such a scattered approach can cause problems. Steve recounts a situation when a company lost access to some very important infrastructure.

03:28 — Steve Almy
I might add, even for smaller organizations, it's about de-risking the whole product space. A former employer of mine, a very small organization, had a reduction in force and reduced the person who held all of the AWS credentials. And not that they leaked, not that this individual posted them on the internet, but they didn't know how to restart systems or log into them and check their billing the next day.

03:56 — Jamie Parker
In one day, they lost the ability to log into core systems. They couldn't reset them, pay for them, change configurations. What does this tell us? Proper secrets management isn't just about keeping the bad guys out. It's also about making sure the right people have access when they need it. This example with only one person having access to important infrastructure happens a lot and it happens multiple times across a single organization. Someone sets up their application components complete with their own secrets. Often they're the only ones who need access, so they don't share the secrets. But other parts of the system still depend on those components and when they go down, there's a cascading effect. If you're not on top of it, your whole system can go down and leave you scrambling trying to figure out how to get it up and running again. Let's look at another scenario. Many teams needing access to the same component, except nobody kept track of who.

04:58 — Rosemary Wang
And by that point we've cascaded and propagated all of these configurations and templates across tens if not twenties and thirties and hundreds of teams. And so by the time you start trying to identify who's using a secret and tracking down where it's located, you have to go, it's almost like a little hunt for where it started in the first place.

05:20 — Jamie Parker
Who's using the secret? Are you using it? Did you let your friends borrow it too? How long ago was that? Rosemary has had to play detective to help customers before. She's had some unsolved mysteries land on her desk with little to go on.

05:35 — Rosemary Wang
Part of the challenge is that whenever I was working on a system, the first time I heard that something was wrong about rotating a secret after I did, it was a user complaint logged to the help desk. And it was fine, but I wasn't part of the help desk.

05:55 — Jamie Parker
Rotating a secret is a common security practice. You update your password, you renew a certificate and so on. Standard, expected stuff. Where things get difficult is when someone needed access and automated the login, then forgot about it and the secret changed without their knowledge. Now they're getting an error message and they have no clue what's going on. That's what happened with that user complaint log. A customer service representative was handling a ticket from a customer who couldn't get into their system and only got a vague error about why.

06:29 — Rosemary Wang
And it took two weeks for that ticket to get circulated to my platform team. And it wasn't because it was inherently a problem, it wasn't like we didn't have systems in place to monitor it. We just didn't realize that basically three layers of backend systems had failed because one system in the very backend couldn't log in. And we had not assessed this system as being impacted by this rotation.

06:59 — Jamie Parker
Without a record of who needed access to the system, some users were left out of the loop when the password was reset. And the error message wasn't clear about why they could no longer access their system. Updating secrets is a fact of digital life, but just because it's routine, it doesn't mean you should be blase about it.

07:18 — Rosemary Wang
Rotating a password is really helpful and you can say, "I'll rotate it and see what it impacts," and then go back and remediate. But oftentimes this will affect customers and users for weeks, if not months, and you won't find these problems until much later. It's almost something waiting to happen.

07:34 — Jamie Parker
Rotating a secret, like changing a password, without telling everyone using it about the change is going to lead to some inconvenience at a minimum and major problems at worst. Relying on someone to request restored access assumes people will know what their sudden issue is. Troubleshooting could take a long time. Let's recap what we've heard.

08:01 — Jamie Parker
Unmanaged secret sprawl is an issue. It creates risk because not everyone safeguards their secrets in the same way. It also creates a problem if someone is the only person with access and then is no longer part of the organization for whichever of the myriad reasons that happen every day. And not knowing who's using a secret creates problems when they're not told when that secret is rotated. These are all problems that assume secrets are being handled appropriately. As Steve and Rosemary remind us, when you don't have standard procedures to set up secrets, there are ample opportunities for them to leak.

08:42 — Rosemary Wang
In any number of ways and usually by accident, I have exposed it in a couple different ways before, so I'll talk about at least my ways. So for me, sometimes it's accidentally printing it out in a CI pipeline or continuous integration pipeline. The output runs and there's the password or there's the username.

09:00 — Jamie Parker
Automation is a great time saver. We covered a lot of this in season two of this show. But sometimes a little bit of carelessness can lead to an automated security leak. Think forgetting to secure a secret in a pipeline or forgetting to secure it in another space.

09:19 — Rosemary Wang
Other times it could be that you forgot to encrypt that secret before you pushed it to version control.

09:25 — Jamie Parker
So how does this kind of thing happen? Rosemary shares how working a little too quickly and without revision can lead to these unfortunate situations.

09:36 — Rosemary Wang
Sometimes you just copy, paste it, put in some configuration somewhere, and it's not redacted, it's not marked as sensitive. It's very easy just to put it in configuration and forget that you set it there. Part of this is that we have a tendency to distribute the secret without necessarily communicating that we distributed it.

09:55 — Jamie Parker
You do the setup to finish your task and don't consider the implications of how you set it up. And sometimes they're not even secrets you set up. So if the owners of that application rotate the secret, they have no way of knowing that they need to let you know. A lot of InfoSec advice around secrets is to keep them secret, keep them safe. Don't tell anyone your password. And that's still pretty much built into the process of creating credentials.

10:22 — Rosemary Wang
These are things that we individually configure because secrets are so essential to how they start. And so because they're manually created in the first place, there's not much incentive to automate that kind of bootstrap.

10:34 — Jamie Parker
With modern IT, cloud deployments, distributed computing, all of that changes how we need to handle secrets. It's a radical change in infrastructure and consequently needs an equivalent change in how we manage access to the system. And the new way of doing things means breaking some of the habits we've built up over the years. Some, but not all. After the break, we'll cover some of the strategies you could use to better manage your secrets.

11:08 — Jamie Parker
Imagine you're trying to get past an organization's defenses. What would you call a repository of that organization's passwords, credentials and secrets? The word jackpot may come to mind. Putting all your secrets in one place seems like it might increase your risk of something terrible happening. But properly implemented, that's exactly what you'd want to do.

11:33 — Rosemary Wang
It seems counterintuitive to decide you're going to centralize your secrets. Because now you're saying anybody who compromises that central component that houses all of those secrets and manages them now has access to everything. But the reality is that centralizing your secrets allows you as an organization, especially as you scale across thousands of developers, to identify who's using a secret, when it's being used, you're able to audit how long it's been issued. It allows you to rotate those credentials on a life cycle.

12:04 — Jamie Parker
See, if you make everyone record, store and pull their secrets from the same place, a lot of the problems we've been talking about go away. You can keep a log of who's using the secrets. Everyone gets notified when those secrets change. And eventually you can set up automations to pull those secrets from that database rather than coding them in, minimizing the chances of leaving them unencrypted in a pipeline or configuration file. While getting there isn't always easy though, it takes a lot of education and changing habits. It takes being a good teacher.

12:38 — Rosemary Wang
And you do have to invest in education as well because it changes the way developers are retrieving secrets or anybody in the organization is retrieving secrets. And so a lot of folks have to invest in developer enablement, but after that initial developer enablement, usually that pattern starts to get adopted and more and more people understand it.

12:58 — Jamie Parker
Getting people on board and into the right habits is only part of the battle. You then have to do the work of setting up a central repository. And that can turn out to be the more difficult part.

13:09 — Steve Almy
The challenge, if you will, is once those secrets get out of control, rating them back in, pulling them back into the central place. Once you get there, much closer to utopia in terms of having control, having awareness, first of all, being able to know how to change or how to update or rotate a very specific secret to deal with a very specific unintended release of that secret.

13:36 — Jamie Parker
You have to do all of that detective work we were talking about earlier. Figure out what all the secrets are, when they were created and who's using them. It's likely too much to tackle all at once. Steve advises that organizations take a staggered approach to building a centralized secrets management system.

13:55 — Steve Almy
What we found is that that really starts with having a good awareness of what secrets are being used and which services. We find customers have early successes that they can then build on over time by pulling in a couple of services at a time into a central system such as Vault. Being able to work around some of those issues with updating those secrets, proving that systems do not go down, takes a little bit of time to make sure that those rollouts happen cleanly. And then just over time, building on that success, being able to prove that some rotations are safe now to do, that services can be restarted on a very short lead time, if necessary.

14:36 — Jamie Parker
You build your proof of concept, your bouncer at the speakeasy, asking for the password, teach a team how to use it and help them figure out how to get the most out of the new system. Once they've got it down and show that it works and remains secure, you can start moving on to the next team. The sprawl of secrets doesn't exactly stop. Your system keeps growing, as do the number of teams who need access to them, but now that sprawl becomes controlled growth and the benefits multiply.

15:07 — Steve Almy
There is some payback for that initial investment on the engineering development side that all of a sudden those phones don't ring, the pagers don't go off when it comes time to rotate a secret. Because it's just totally managed by the security team, the application restarts by the DevOps team, production team, and the developer can have the weekend free to enjoy dinner or whatever they're doing outside of work.

15:34 — Jamie Parker
It sounds like a fairy tale ending. That's how it would be if your entire system is running on modern infrastructure. But as is often the case, especially for companies early or part way through their digital transformation, some elements are still based in traditional infrastructure. That means there's a few more considerations to keep in mind.

15:55 — Rosemary Wang
It's also worth noting that it's a challenge because it's a paradigm shift in the way we think about how configuration should be treated. Most applications, they don't change much configuration once it's been deployed. You could redeploy the application because you're deploying new features or you're changing the code itself, but you're not often changing a configuration. And centralized secrets management means that there's another dependency for that application to rely on, which means that application needs to be able to reconnect to a central Secrets Manager if something happens to the network.

16:33 — Jamie Parker
Another shift in how you could handle your secrets. At this point, it's one of many. But it's manageable with proper planning. That's the mindset Rosemary suggests keeping in mind when tackling the secrets management side of digital transformation.

16:47 — Rosemary Wang
Embrace the fact that everything changes quickly. I think ephemerality, when you talk about cloud native or you talk about digital transformation, the point is that change is inevitable. Your systems must evolve with it. And you must support evolution, you're no longer supporting systems, you're supporting evolution of those systems. And change has become, I think, the primary factor in adoption of architecture. Can it change with the times?

17:22 — Jamie Parker
Tech has had a lot of systems evolution over the years. Digital transformation is the latest in a long history of change. But it's different in that modern systems are now constantly changing, instead of static deployments like in the days of yore. All the support systems need to be flexible in order to keep up with the new normal and secrets management is an important part of that support system. Centralizing your secrets and implementing automations to help your team access them when needed can do a lot for your organization. Without those options, secrets can sprawl, they could be unintentionally exposed and you'll likely get a slew of help desk tickets, potentially over the course of months, whenever you need to rotate those secrets. Whatever gains you might be getting from modernizing your infrastructure could be undermined by these mounting issues. As Rosemary and Steve explained, a centralized secret system can help with those issues. It takes dedicated work to implement properly, but doing so helps you in the present and sets you up to better adapt to the unknown challenges that could come your way.

18:32 — Jamie Parker
You can learn more at redhat.com/codecommentspodcast or visit redhat.com to find our guides to digital transformation. Many thanks to Rosemary Wang and Steve Almy for being our guests. Thank you for joining us. This episode was produced by Johan Philippine, Kim Huang, Caroline Creaghead and Brent Simoneaux. Our audio engineer is Mark Angley. The audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oates, Rachel Ertel, Carrie da Silva, Mira Cyril, Ocean Matthews, Paige Stroud, Alex Traboulsi, Boo Boo Howse and Victoria Lawton. I'm Jamie Parker and this has been Code Comments, an original podcast from Red Hat.