Aligning With Open Source Principles

00:05 — Burr Sutter
I'm sure many of you know Red Hat is very well known as an open organization with things like Linux and Kubernetes application servers, all our intellectual property, our code out there as open source. But what does it mean to be an open organization that is working with its partners? Well, in this episode, we'll be talking to David Duncan, a former Red Hatter, and now Senior Manager Partner Solutions Architect fo r Linux at Amazon Web Services.

00:33 — David Duncan
I came to work at Amazon to work with some of my favorite open source developers, not to work against them.

00:43 — Burr Sutter
I'm Burr Sutter and this is Code Comments, an original podcast from Red Hat. Well, David it's absolutely fantastic to have you today. I know you're formally of Red Hat, we knew you from a Red Hat standpoint, and now you're over at Amazon. Can you tell us about that? And I'm particularly interested in how you feel that open source principles and open source mindset blends and benefits within the concept of the Amazon Leadership Principles.

01:11 — David Duncan
Well, first off, I want to say thank you for having me today, Burr. And yes, I did come from Red Hat. I came over to Amazon to work with some people that I had worked with at Red Hat formally. The concepts here at Amazon are very much built in open source software, and so we have a lot of people who understand open source and understand how those principles work. And Amazon puts a lot of emphasis around the Leadership Principles. And in those leadership principles, we have a lot of directives that help to hone the practices and help to guide us in the way that we are effective.

01:47 — Burr Sutter
The Leadership Principles actually start with the concept of customer obsession. I see that at the top of the list and I am really curious to know how that meshes with your open source ideals. I feel like some of these things are related to each other.

01:59 — David Duncan
Yeah, so definitely customer obsession is top of the list for anything that's related to Amazon and Amazon Web Services and how you get there is really important. Listening to what they have to say, looking at those requests and how they make those requests and then helping to solve the real problem. A lot of people ask for things that they think they want, but then they really want something else. And a great example of that in the Amazon world is lower cost. You always see people saying, "Oh, well we want lower cost, lower cost, that's why we're moving to the cloud." But really a lot of times what they want is they want agility. They want to figure out a way... And this is the same in open source... They really want to be able to use software in a way that really meshes with their business directives, their goals and their skills, the skill levels that they have. They want to be able to implement things in a way that is consistent with their line of business. And so finding a way to help them means presenting information in a way that is easier to absorb. That's something you have to do in the open source world as well. You can't just go off and start working on projects and just come to a decision over like, today, this hundred lines of codes is going to solve the problem. We have to know what direction we're going, create a roadmap for ourselves, find a way to express exactly what the design goals are and then make those design goals real by putting in the time and the effort and breaking them down into component parts that we can really understand.

03:38 — Burr Sutter
Yeah, I think that's such an important point there that you mentioned the concept of really, let's say, asking the five why's, getting behind the initial request, getting into some depth there. It's not just about cost, it's about agility. And that agility comes from greater knowledge, greater flexibility, and I think that does apply to open source because you have access to the source, you have access to the community, you have access to all the knowledge base, practically at this point, around the entire globe. There's everything on Google and Stack Overflow at this point. So there's so many forms where you can get help and assistance, and I think that's what makes open source so powerful. And I do see some of those same aspects that you mentioned there when it comes to the Amazon Leadership Principles. But maybe there's another leadership principle that is of interest to you and something of that nature that you really feel applies and helps us think about the linkage between your experience at Red Hat and now your experience at Amazon.

04:32 — David Duncan
Well, invent and simplify comes to mind when I think of things that was really important, both at Red Hat and at Amazon. Looking at software and design or the requirements customers have, you really want to do more with less and creating that solution that gives you exactly that is really, really an important part of our process. So finding ways to deliver - I'll say it in the Leadership Principle terms - to deliver results by simplifying the requirements and delivering on that in a fast way is something that I think is really important. And actually, there's another principle that I think is really important, and this is also extremely important in open source design, is the idea of the two-way door. No one-way doors is not a leadership principle, but it's definitely an important part of our process, which is that if we're moving forward at a fast pace and we're trying to go in the right direction, you know we're going to make mistakes. We're working with as many of the details as we can get together and then we're trying to make sure that we get everything together that we can and go in a direction that we think is right, but we want to make sure that if it's not right, that we can quickly recover and change direction. So it's a very important part of our process to eliminate all of the one-way doors. And I think this works really well with... It takes me back to system wide changes versus confined changes in the Fedora world where a system wide change has to have a rollback requirement before it's considered. All of those component parts have to be examined. And if it doesn't, someone in the council, there's obviously going to be a council discussion around that from the Fedora side, and if there's no way to roll back, it's a big decision. A lot of people get involved. There's usually a fairly large community momentum, and as an open source community, you have that big discussion. Same thing happens with anything that would happen here at Amazon or Red Hat, same deal.

06:49 — Burr Sutter
Well, I like that point a lot because one of the things that open source is incredibly well known for, at least the great communities around open source projects that are being very successful, and that is the principle of transparency. And I think that's so powerful and such a great enabler for the open source communities and certainly open source projects. Would you say that really applies within the concept of Amazon? From an outsider looking in, for those of us looking at Amazon, it's unclear if there's as much transparency there and I'm curious to what you think about that.

07:16 — David Duncan
Well, I mean, I point to a lot of the open correction of errors posts that we've made around issues that were related to service availability. And I think that in a lot of cases, all of the information that we're collecting together is shared with customers in the sense that we have these field requests, we're constantly looking for new ways, new technologies that are going to make it possible for them to resolve their issues. Our architectural principles are all out there, so all of the things that we use with decision records and more complex components of the architecture are out there on the aws.amazon.com/architecture website. So you can go in, look at some of the hard problems that we've solved and see what the component parts were and the decision process on how those solutions came together.

08:16 — Burr Sutter
David just told us about aspects of Amazon that are more open and transparent. Next, we're going to dig deeper into how David's experience in Red Hat and open source has influenced his current role and his current work at Amazon. I know you're deeply involved with alliances, so your overall day job is managing a team of people working with AWS partners. And I'm curious to know what advice you have for those teams at your partners as they build out their ecosystems, how does your open source background, open source mindset, and even what we just discussed here, how does this help you advise them? And what kinds of things do you do there?

08:53 — David Duncan
Okay, so this is interesting because this does come back to my Red Hat experience as well. One of the things that I have focused on, and as someone who works in the alliances team, I spend a lot of time with operating system partners, Red Hat being one of those. And we spend a lot of time talking about finding ways to create good timing. Cadence is something I come back to quite a bit. Looking at how we can align with the timing for releases and new features is something that we try to do all of the time. Internal to AWS, I'm always pushing on this. We know when the release schedule is for this partner. We want to create a good cadence, we want to make sure that we have all of these new features, these new instance type supported. And all of that means making sure that we're months in advance really in lockstep. So looking at roadmaps, taking those into account, looking at how we handle our discussions, that part of the alliances experience is something that I love working on. Making sure that we have more open and aligned practices is a really big part of my day job. And one of my favorite examples of this comes from working on the hibernation, and this will give you a little bit of an idea, the EC2 hibernate is it's a technology that from a server perspective, you don't think about a lot, but it turns out that it's a really beneficial thing to have in the context of cloud native architecture because we have this ability to take advantage of opportunistic compute. And so hibernation is one of those things that finds its way into your process, in your practice. So when I was first approached by our hibernate team, they said, "Hey, we've got a hibernation practice and we want to introduce this to our operating system partners." There was no precedent for that. There were two things that they wanted to do. One was to have support for server, was hibernation on server. And then the second one was to support a swap file. The swap file was something that I knew we couldn't support right out of the box. We had to find ways to introduce this gently into the process. And that is, to me, a breakdown of the cadence. The request was much bigger, but then once we started to break this down into component parts, it was things that we could resolve. And there were some problems with the way that the hibernation functionality was presented and I had a young software engineer here at Amazon and he worked on the hibernation team and just put his heart and soul into making some modifications to the Linux kernel code to make sure that it did exactly what we needed to do. And I'm trying to paraphrase here a little bit. The result was that he pulled together his first patch for the Linux kernel, and this was super exciting. We had a great collaboration, a very open collaboration on the request and the structure. And the result was that Dave actually mentored this new engineer, this fresh engineer, and helped him through that first commit, really literally helped him through his first commit to the Linux kernal that changed the way we could support hibernation in the XFS file system on Red Hat server and brought that really to the community at large, which I thought was super exciting.

12:28 — Burr Sutter
I'd have to say that is super exciting because that was the whole point of open source to begin with and having that open culture to basically allow everybody to contribute. And I think that is incredibly powerful. Well, you talked about cadence there and I'm very interested in that concept. I would love to double click on that a little bit more because that concept of pace interval, a known release schedule kind of thing, is that what you mean by cadence or can you explore that further for us?

12:54 — David Duncan
Yeah, so there are a lot of things involved in that. So we all have roadmaps, we all are very excited about making sure that we can get our product out as fast as possible. And I think that that's great. I would definitely not say that you have to wait on someone else, but sometimes, and this was a perfect example of that, the goal can look very daunting in the context of other roadmaps. So finding ways to interact with one group versus another group is a very important part of our process. And something that I very much focus on in my daily experience is that the hibernation team came with a simple goal. Their goal was to have their software supported on Rel as it was in their development environment, which was Amazon Linux. And the answer back was we weren't on the right schedule. We didn't have the same goals. So just simply, the answer was no. No, we don't have the ability to make this claim. But when you start to break it down and create a series of smaller steps, small changes, and to develop a cadence, a lock step with the processes that are associated with the two different products, then we started to see a path forward. Now we can see that today we don't have what we need, but tomorrow we will and we can create solid goals and solid planning that all of our product managers and all of our developers can really work to. And the collaboration, even that collaboration literally led to finding a bug in the NVMe code for the Linux kernel that hadn't been discovered for more than three years.

14:50 — Burr Sutter
Oh wow. But that, again, is the power of open source. So I love that concept where you basically are saying we had some challenges, we had some wins, but it's all about the learning. I think that's really the core of it here, learning and collaboration and that constant iteration means that we eventually get to a better outcome.

15:09 — David Duncan
Yeah, indeed. Yeah, I know one of my favorite Fedora tenants is friends; that one of the things that we assume walking into all of our development experience and all of, really, our roadmaps is that we're friends and that we have common goals.

15:28 — Burr Sutter
Absolutely. Absolutely. I do love that principle. Now when it comes to your Cloud API collection that's out on GitHub though: are you guys truly open to the entirety of the community? You want everybody to come show up there and become familiar with code base?

15:43 — David Duncan
Most definitely. So to review, to look at how the schema is arranged to take advantage of it and then to make a steady comment, I think that everyone in the Cloud Control API team, and really CloudFormation in general, just wants to make it better. They want to make sure that the experience is absolutely the best they can make it. When we started talking to the Ansible team way back in probably 2015, one of the things that we constantly look for was ways in which we could celebrate and maintain a relationship in between the CloudFormation team and the Ansible team that was more open and gave more of an opportunity for everyone to provide feedback directly. And that's really where the Cloud Control API came from, is that having that common goal, that common experience.

16:38 — Burr Sutter
And I can totally see that where you really want the best user experience possible. You guys opening up that API allowed other tooling creators, if you will, like the Ansible community to basically come in and participate with that.

16:52 — David Duncan
Yeah, so let's talk a little bit about that. So the Cloud Control API is an API with access point, endpoint, that allows you to make some calls for creating and updating, deleting, and even listing component parts of the AWS products and services that are used by customers on AWS today. And part of, I mean, what got me excited about this was that this was a perfect fit for Ansible. And looking at where we have been going in the Ansible community for, I say we because it's been so dear to my heart and I've spent so much time with people, people who have been super influential in Ansible for a good long while and their work, and this was an opportunity for us to really bring together a lot of the things that we had been hoping to do, which was to provide a supported collection for Amazon. And this API was out there, we were talking to the PMs about it, and I said to the PM, Rahul, this is a perfect opportunity for working with the Ansible community. And he was in violent agreement. He thought this was fantastic. So we had some preliminary planning meetings and then just got right down into it, created some communication strategies to get things off the ground, and then had a couple of the Ansible folks just working directly on this. But all that work was done out in the open and there were pull requests and changes that came from many different developers in the resource community. And this wasn't just a one person project, this was a lot of people getting involved in creating a structure that made it possible for us to have a truly supported collection.

18:44 — Burr Sutter
Well, David, and it sounds like the concept, of course, the Cloud Control API that then scores the API, which is for AWS, but what this means is there's a, I don't want to call it an SDK, but it's like that. It's now an open source set of Ansible technology that people can go in and look at right now and you can collaborate with people out in the open.

19:03 — David Duncan
Really a schema. And then beyond that, this schema now is something that is easily consumable. So instead of creating something that was spoon fed one piece at a time, this became something that we could use to auto generate the responses. And then the feedback loop is the more important part of that, I think, is that now we have this charged feedback loop in the open. There's a very public discussion around this collection in the Ansible collections repository, and it leads to changes in the Cloud Control team for how they're going to build and develop their next generation API controls. For example, I guess one of my favorites is presence, the statefulness of an object, an instance, an S3 bucket, anything like that, in Ansible, you have to go and get that and there has to be some handle for making that request or ensuring that when you list those details, you get back the item potent object that you were looking for. Let me give you an example, like a security group. If you're defining a security group, you would look at the name, the ID, the region that it was in, and all of that would align with a single specific security group. So you have to be able to find that. And when the Cloud Control API came out originally, that was not part of the process or part of the schema. And now that's being worked on directly with the help of the open source community around the AWS Ansible experience.

20:47 — Burr Sutter
I think you just tapped on something though that's incredibly important. And that is the fact that you guys defaulted to open when it came to this API, putting this out on GitHub. And that allowed you to work with partners and get the feedback from people who really wanted to use the API. Now you have a situation where that feedback helps you build a better product.

21:05 — David Duncan
Yeah, and it's directing changes in the way that the CloudFormation team is building out objects and making it possible for new advances in partner technologies and incorporating those partner technologies onto the AWS experience.

21:22 — Burr Sutter
So are you guys totally open to pull requests?

21:24 — David Duncan
Well, not on the API itself, but from the experience side, we are taking all customer feedback and pulling that in. From the perspective of the Cloud API or the Ansible collection generator, we rely on our friends at Red Hat and in the Ansible community to help us make that better and then to provide feedback that changes that Cloud Control API. Obviously, that API is part of the grander API for Amazon, but there are several libraries there that are associated from the that perspective like our common runtime libraries. Those definitely have an open first policy.

22:06 — Burr Sutter
How many different client run times or tools have been integrated with the API? Do you track a lot of language-based SDKs or tools like Ansible in this context?

22:16 — David Duncan
Yeah, sure. I mean, HashiCorp has tools there like Terraform. Pulumi has integrated their strategy with the Cloud Control API. So there's a lot of open source technologies out there that have created providers around this or methodologies for leveraging the Cloud Control API. And that feedback, similarly, comes from them. There's some unique features to Ansible, I think, like the concept of state that makes it an interesting fit and has always made it one of my first go-to tools.

22:54 — Burr Sutter
Well, and I'm curious too when it comes to the Cloud Control API, how much coverage would you say is it of the Amazon ecosystem, the AWS ecosystem? Can I do most things through the API that I would do through the user interface or through the CLI?

23:07 — David Duncan
I mean, ultimately, yes, you will be able to, but today, it's a smaller subset of the overall tools. There's a focus on specific ones like S3, where we have more defined state. For example, EC2 instances are still on the roadmap.

23:26 — Burr Sutter
So as someone who understands the power and potential of open source and that background, you have an open source, what are you excited about now?

23:33 — David Duncan
I'm excited about the concept of the open source software as a service. So Amazon has a team called the SaaS factory team, which has always been one of my favorite teams because it helps customers and partners to build out software as a service, service delivery, on AWS. And I work a lot with Red Hat, OpenShift on AWS. I worked in the early stages of development and it was the time of my life, really, just a super amount of fun. Almost all of that work is structurally available for review. And the SREs that I worked with, we spent a lot of time working in GitHub and working back and forth on different problems.

24:14 — Burr Sutter
Well, I know that's a great example, by the way, when it comes to the Red Hat OpenShift service on AWS. I know when we're going through that process, I was even one of the little internal testers on our side, but there were bugs found on for both parties. We worked together collaboratively to basically make it bigger, richer. Do you have any specific examples of where that partnership really worked out nicely?

24:36 — David Duncan
Well, I mean, I think the ROSA CLI is a really great experience on that side. All of that development was done in the open and it's written in go and lots and lots of things that we did there in the early days were experimental. Some of the practices that I love are YouTube videos, sharing YouTube videos where we would just create a small feature or add some augmentation and then demonstrate that to the rest of the group. That crossed all barriers. We had a link, that link went out, we had a conversation, either through email or discussion in the commentary for the issue or the pull request, and then that would take us in one direction or another.

25:22 — Burr Sutter
I think you just tapped on another key open source principle and that is doing things asynchronously. In other words, we're not going to all sit around on video calls talking to each other all day, but we will document, describe things, put comments in, pull request, video recording. I use video recording a ton, even with my work within Red Hat. So I really like that idea. That is another great open source principle.

25:45 — David Duncan
And it feeds right into something else that I think is really important and also related to this concept of software as a service as an open source experience, which is the Operate First group; Karsten Wade, who is responsible for a lot of the work that's being done on the Operate First now has been instrumental in other practices like The Open Source Way. And Karsten and I talked about this extensively around how do we create this cloud experience that transcends all borders and then gives us this opportunity to take advantage of exactly what it is that we need to get the job done. And part of his process has been creating and defining architectural decision records. And I think this is just an amazing practice. And something that I love about doing this in an open way is that we can share not only a catalog of services and technical options, but we can create for ourselves from a building block, a way of expressing our decision to people who come after us or people who want to be a part of the project long after decisions have been made.

27:02 — Burr Sutter
Yeah, absolutely. Those recorded decisions so that people can understand, hopefully, the logic behind it and why the decision was made, that is incredibly important when it comes to working on complex software in the open source world. And with that asynchronous team with people in different time zones around the globe, but different languages, in many cases, English is not their first spoken language. Well, now they can participate in a more effective way.

27:26 — David Duncan
At Amazon, they have what we call the Well-Architected Framework, and that's built around these five pillars, security and operational excellence, those kinds of things. The idea of the Well-Architected program is very similar to that, that you have a decision and that there is a decision tree that you can refer back to and you can even look at how that works. And some of the things that Karsten and I have talked about is building out a structure for identifying where in that relationship do we fit and how do we determine, in an open source way, that we think we're going in the right direction.

28:00 — Burr Sutter
Well, I'm glad you mentioned, by the way, the Operate First Cloud. I want to come circle back to that because I don't know if everyone here listening to the podcast has really understood what that is. It's not well-advertised, let's say. The Operate First Cloud is an incredibly powerful idea because in the history of open source, we've often had all these open source bits out there, let's say on GitHub and of course in binary form at various download sites, and you could basically pull it down to your laptop and execute it. But that doesn't necessarily really test the software in an operational sense. And that's really when you get into production, you're trying to host it, you're trying to create software as a service out of it. Now, you're really learning a lot more and exploring a lot more about the software. And of course, that leads to greater innovation. And I'm curious to know, how do you think about that and what do you think about what Karsten is doing over there with the Operate First Cloud and how you know might be involved?

28:52 — David Duncan
Well, I mean in UMass Lowell, I mean, he's doing some great work with some people who are working on this concept of a public private cloud, where they have a cloud that's built inside of the university and then open to users. And that's part of where their project started, but that wasn't necessarily where it was intended to land. That's just the tip of the iceberg. So he has a place where people who are experimenting with GitOps and delivering open source software as a service can get their start, but then that can just extend out in every direction. And that's that truly hybrid experience. And I feel like that's exactly what he's focused on, is creating this truly hybrid experience. So that's where we obviously have some interlock there because I'm part of that. And the excitement for me was that Karsten was looking at how he could create this structure that as we build out better and better models, that we can do that in a truly open way. And that as a service design, it could be something that we can share and we can all use.

30:07 — Burr Sutter
Well, Dave, I've really had a great time talking to you today. I think it's absolutely fantastic the experience you've had both at the Red Hat and with the open source ecosystem. And I love how you've brought some of those same principles and shared ideas into the AWS ecosystem inside of Amazon. And of course, we certainly enjoy working with you as a partner. So thank you so much for your time today.

30:25 — David Duncan
Burr, it's my pleasure to spend some time with you.

30:32 — Burr Sutter
It is so great to hear these stories of open source and open innovation happening inside of Red Hat's partner community. You can read more about Red Hat's partnership with AWS at redhat.com/codecommentspodcast. Many thanks to David Duncan for being our guest, and thanks to all of you for joining us today. This episode was produced by Brent Simoneaux and Caroline Creaghead. Our sound designer is Christian Prohom. Our audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Johan Philippine, Kim Huang, Nick Burns, Aaron Williamson, Karen King, Jared Oates, Rachel Ertel, Devin Pope, Matias Faundez, Mike Compton, Ocean Matthews, Alex Traboulsi, and Victoria Lawton. I'm Burr Sutter and this is Code Comments, an original podcast from Red Hat.