You Can’t Automate Collaboration

00:00 — Jamie Parker
There's an implicit assumption in the automation pitch that you're going to want to automate as much of your IT stack as possible. That reliability, consistency, and ability to scale only really kicks in when a significant proportion of your repeated processes aren't done by hand anymore.

00:21 — Jamie Parker
There's a small hitch with that. There probably isn't a single person or even a single team who has the breadth of knowledge—let alone the time—to successfully automate everything across your organization. It takes the whole organization to pitch in, but they probably don't know how.

00:37 — Jamie Parker
We've covered how individual teams pioneered automation within their organizations and how changing culture is key to supporting adoption. In this episode, lead infrastructure automation engineer Sudheer Kumar Donthineni explains how his team at Ulta Beauty helps other internal teams build automation to their processes.

00:58 — Jamie Parker
It's like what they say about teaching someone how to fish—except we're going to fish at scale and then there's going to be a lot of fish scales—and I think my simile broke down somewhere around here.

01:10 — Sudheer Kumar Donthineni
Ulta Beauty is well on its way in its automation journey, with I would say approximately 50% of infrastructure components including cloud services are already automated. And when we joined in 2017, it was hardly 1%.

01:22 — Jamie Parker
Sudheer joined Ulta Beauty in a small team of automation experts. The three of them were assigned the gargantuan task of automating the company's infrastructure. In 2023, that team looks back on a string of successful projects that automates about half of the company's IT infrastructure, and the size of that team today? It's still three people. How did they get that done? Well, they didn't do it alone. In fact, they're not even the ones who did most of the work. They guided other teams through the process of automating.

01:54 — Sudheer Kumar Donthineni
You have an issue. You don't know where to go. You come to us, a center of excellence team, which will help you to drive your automation within yourself. We will provide you the best practices.

02:05 — Jamie Parker
There was no other way for them to accomplish that level of automation without helping those other teams do a lot of that work because those teams are the experts of their domains.

02:15 — Sudheer Kumar Donthineni
We wanted to leverage, extend that skillset across the team, so they can focus on manual tasks, and they can also focus on the automation tasks because they are the ones who are performing the actual heavy lift. So they exactly know what time, what effort it's taking for a task to be completed. That means they can write their story very well.

02:38 — Jamie Parker
It's the age-old question. If you need to save the world from an approaching asteroid, do you train oil drillers to be astronauts, or do you train astronauts to be drillers? For Sudheer and his team, learning the ins and outs of every system that needed automation wasn't an option. They had to convince leadership it would be more effective for them to act as an example and a resource for the rest of the company.

03:02 — Sudheer Kumar Donthineni
Our strategy around changing the company's culture, where we started as a team, and then where we are today, is by going to the leadership and getting leadership buy-in. Our leaders demonstrated support for automation initiatives and highlighted their importance, and this actually sent a clear message to the rest of the organization.

03:23 — Jamie Parker
Ulta Beauty chose to empower each of their teams to write their own stories. They decide what to automate based on their own needs and processes. To do so, they need to learn the tools, principles, and best practices of automation, and it was up to Sudheer and his team to provide that knowledge. The best way for them to prepare was to undertake a project themselves.

03:46 — Sudheer Kumar Donthineni
The general roadmap on how automation principles can be scaled up, involved larger teams who are performing day-to-day activities in the organization as a whole. The first thing that we have done is we have documented the process. We started to document the success achieved within the small team, and we highlighted the benefits, the efficiency gains, the improvements in the operational processes—resulting in the automation.

04:13 — Jamie Parker
That document served as proof positive that adopting automation can help the infrastructure teams reach their goals, but it also became the roadmap for everyone to follow, and follow they did.

04:24 — Sudheer Kumar Donthineni
Whatever the success we have achieved through automation actually built more awareness and enthusiasm among all of our employees over the following years.

04:34 — Jamie Parker
Those projects became a reinforcing mechanism that spread.

04:38 — Sudheer Kumar Donthineni
So we have created a success story and benefits of automation with other team members and departments. We presented that information to this team members in the team meetings, town halls, and even internal communications, one-on-one communications to generate interest and curiosity, and then that's when we started the education and training.

04:59 — Jamie Parker
That education and training became available for a large chunk of the company. Even employees without technical skills could learn about the principles and best practices of automation.

05:09 — Sudheer Kumar Donthineni
We have organized cross-functional workshops, webinars, and forums where teams can share their automation experiences, challenges, and successes. This encouraged collaboration and facilitated the exchange of the best practices within the organization itself—with their own learnings, with their own issues, with their own best practices.

05:29 — Jamie Parker
Anyone could become a key player in the automation effort. They could help each other set up, troubleshoot, and maintain automation systems because remember—Sudheer's team remains quite small.

05:41 — Sudheer Kumar Donthineni
This actually helped to bridge the knowledge gap and actually empowered all the employees to adopt automation practices. And then what we have done is we have identified champions. We have identified individuals or teams, sometimes even we did the teams, where different departments who are very enthusiastic about automation are willing to champion its adoption.

06:05 — Jamie Parker
While there's plenty to celebrate about the power of automation being implemented to create efficiency, there are potential pitfalls along the way.

06:14 — Jamie Parker
When we come back, Sudheer and his team light the way, so others know what to look out for and when to ask for help.

06:27 — Jamie Parker
One pitfall to look out for is giving in to the pressure to introduce automation quickly.

06:33 — Sudheer Kumar Donthineni
A rush to write code, especially in the context of automation and software development, which can lead to a variety of issues that affect the quality, reliability and even the maintained—we cannot maintain that. Why they do it is because of the project deadlines.

06:48 — Jamie Parker
Deadlines. Can't stand them, can't get anything done without them. It's a tough balancing act. Sudheer has witnessed what happens when they're set too aggressively.

06:58 — Sudheer Kumar Donthineni
Rushing through the code development can lead us to overlooking bugs, logic errors, and might cause unexpected behavioral system failures. Poor design, skipping proper design and architecture considerations can result in a code base that was difficult to maintain, extend, and refactor.

07:15 — Jamie Parker
And that's not all.

07:16 — Sudheer Kumar Donthineni
It will lead into a lot of other issues. Poor design, like I said, inefficient code, limited testing, no documentation, maintaining challenges. There is a risk of security as well, and we cannot reuse that code anymore. The main purpose of automation is reusability, and if you're writing an automation for that single achievable goal that is not reusable, then no.

07:40 — Jamie Parker
Obviously poor design, technical debt, and security risks are not the goal. And if you're sinking a lot of time into an automation system, it makes sense to make it reusable to save time on future similar projects. So take Sudheer's advice. Don't try to rush through your automation projects. Build in enough time to get it right because there's other things to worry about.

08:04 — Sudheer Kumar Donthineni
Maybe the engineers have done it manually, and maybe they have not been documented properly. And when we run the automation, automation is only going to pick up what is in there. It doesn't pick up anything which isn't there. So data complexity is one of the issues that we have seen, and legacy systems. Integration with older legacy systems might prove more challenging than anticipated due to outdated protocols, or limited documentation, or compatibility issues. We have dependencies that we came up through. There were unidentified dependencies between obligations and databases and components that led to disruptions during migration.

08:44 — Jamie Parker
Data complexity, unidentified dependencies, compatibility issues. Automation processes have a lot of moving parts. While there's room for reusability, each project is unique, and that underscores the ever-present need to get things right.

08:59 — Sudheer Kumar Donthineni
Security concerns. There was a security vulnerability that was exposed during migration, and we had to follow additional security measures and modifications for rewriting those.

09:10 — Jamie Parker
It's all a lot to wrap your head around, and it's hard to take a step back, take time, and take a scaled out view of what you're building when you're hurrying to get it done. It's especially useful if—in that time of reflection—you can identify where else the automation could be used.

09:26 — Sudheer Kumar Donthineni
A lot of teams, when they automate their stuff, they write their custom code. When they write their custom code, if they're encountering issues which require immediate attention, they are going to write some bad code in it just to fix the current issue. That customization actually creates problems in the longer run. So we always recommend, that's why there should be an individual or a team who specifically checks all this in order before the merge of the code happens.

09:55 — Jamie Parker
There's a lot to learn and to keep in mind as time goes on. And it's worth the effort for teams to learn all of these best practices because as we mentioned at the top of the show, there's an assumption that automation will be applied to a large percentage of the IT stack.

10:10 — Sudheer Kumar Donthineni
I would say automation is not a one-time project, but an ongoing journey. Regular evaluation, adaptation, and learning are crucial to harness its full potential. In a world where technology advancements accelerate and demands become more dynamic, automation is just not a tool, but a mindset that embraces efficiency, innovation, and progress. And I would say by leveraging automation strategically, organizations can shape a future that is more efficient, responsive, and rewarding for both teams and even customers alike.

10:50 — Jamie Parker
It's a huge help to have a dedicated team to teach the fundamentals and help everyone else get started, but Sudheer and his team do much more than that. When a project gets complicated, they can step in to help out too.

11:06 — Jamie Parker
Sometimes, automation projects follow common workflows. Sometimes not so much.

11:12 — Sudheer Kumar Donthineni
If it is straightforward, "Okay, here is a simple task. This is A to B, B to C, C to D," they just do it. But if they wanted A to C, C to D, B to A, this kind of a different ratio, that's when it gets complicated, and they reach out to our team to seek help.

11:30 — Jamie Parker
But what does that help look like? Sudheer and his team are experts in automation workflows, but likely don't know the intricacies of the complex system the team needs to help with. Their advice and process may sound familiar, but it works.

11:45 — Sudheer Kumar Donthineni
So most of the design pieces are done by our team, but the actual implementation of coding itself is done by the respective team members. We started small. We encouraged other teams to start small with their automation initiatives. Identify simple—just start with a simple automation, repetitive tasks that can be easily automated, and have an immediate impact. Starting with small wins builds confidence and demonstrates the value of that automation. We worked with individual teams to identify automation opportunities that align with their specific needs and processes.

12:19 — Jamie Parker
Starting small, breaking the project up into chunks, finding the next opportunity to automate, zooming in to the individual components that can be automated before undertaking the rest of the infrastructure, helps overcome the overwhelming complexity that can stop a team in its tracks. Sudheer doesn't necessarily know what those first small components are going to be, but he can work with the team to find out and then advise them on the technical approach to getting those first automations running.

12:50 — Sudheer Kumar Donthineni
I may think I have the subject matter expertise in a specific area, but I may be missing a point somewhere where other team members can actually find it, which is why I always encourage team members, before you write an automation, always check with your peers. Have an architectural diagram. Though it may be a simple task or a major task, present it to the team. Take all the inputs, all the pros and cons, then move onto writing your code, rather than just jumping onto the code.

13:17 — Jamie Parker
It's no secret that we are big fans of collaboration here at Red Hat, so we're going to shout about it from the rooftops every opportunity we get. When they step in to help out, Sudheer and his team aren't there to take over the project. They advise, help plan, and help with some of the technical aspects, but the original team is still in charge.

13:36 — Sudheer Kumar Donthineni
We never ask them to jump into a critical task and then give a deadline to them. We have them do it in their own pace. At the same point, we encourage them to do it continually. Customizing solutions to address each team's unique challenges increases the likelihood of successful adoption. That's how we saw it.

13:54 — Jamie Parker
They also make the effort to carve out the time needed for individuals to dive in and learn these new tools, processes, and principles. And it's not just technical docs or webinars. Ulta Beauty encourages their employees to learn by doing.

14:10 — Sudheer Kumar Donthineni
We have as Red Hat to actually come in and do hackathon sessions with all these individuals, give them the technology, and we have given certification methods to have them go learn. We have created specific dedicated hours to specifically state, "Hey, these are the dedicated hours for you to learn your automation journey." So we have used that kind of process. It's not only for the organizational growth, but it's for their individual professional growth as well. It often requires team members to learn new tools and skills. This provides opportunities for continuous learning and professional growth, making the organization an attractive place for those who value that development within the company and also for the future talent that we are actually seeking to get into the company.

14:55 — Jamie Parker
Though it may not seem like it, the last step of their process is just as important as any other.

15:01 — Sudheer Kumar Donthineni
And then we celebrated the successes as well because recognition is a key thing, recognizing and celebrating automation success across the organization. We are highlighting teams that have achieved significant efficiency gains or process improvements through automation. This reinforced the past impact of automation. That's how the automation journey has been so robust within Ulta.

15:25 — Jamie Parker
When you're relying on teams to learn the skills to automate their own projects, recognition can add to the feeling that those efforts were worth it, and showcasing successful implementations plays a vital role in raising the visibility of the worth of automation. In the years he's been at Ulta Beauty, Sudheer and his team have gone from 1% to 50% of the infrastructure automated. Where are they headed next?

15:50 — Sudheer Kumar Donthineni
I would like to say 100%, but strategically speaking, that wouldn't be possible. So there is always some kind of a human touch that is needed, some kind of an involvement that is needed.

16:04 — Jamie Parker
Next time on Code Comments, we hear from a security expert at Microsoft, who's learned a lot about the need to adjust security automation signal-to-noise ratio in order to clear the way for productivity.

16:17 — Jamie Parker
You can read more at redhat.com/codecommentspodcast, or visit redhat.com to find out more about our automation solutions.

16:25 — Jamie Parker
Many thanks to Sudheer Kumar Donthineni for being our guest, and thank you for joining us.

16:31 — Jamie Parker
This episode was produced by Johan Philippine, Kim Huang, Caroline Creaghead, and Brent Simoneaux. Our audio engineer is Elisabeth Hart. The audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oates, Rachel Ertel, Carrie da Silva, Mira Cyril, Ocean Matthews, Paige Stroud, Alex Traboulsi, and Victoria Lawton.

16:56 — Jamie Parker
I'm Jamie Parker, and this has been Code Comments, an original podcast from Red Hat.