Ready to Launch

Launching a new product should be cause for celebration. All too often in the tech industry, launch day is a source of worry, and in the worst cases, scrambling panic. But it doesn’t have to be that way.

What can we learn from the gaming industry’s frequent launches? Why is it so difficult to get launch day right? And why aren’t companies better prepared for the problems they know are coming?

00:04 — Johan Philippine
We talk a lot about building software and infrastructure and the people who make it all work. Today we're going to cover an oddly persistent issue in the internet age, the oft-dreaded launch day. Whether it's the rollout of a new major healthcare platform for the whole country to use or when a superstar's concert tickets go on sale, there's a team of engineers with their fingers crossed that everything's going to go smoothly and everyone will be able to at least access the system properly.

00:33 — Johan Philippine
The players and game developers and the audience are keenly aware of just how well or how terribly things can go wrong. With several launches a year that run the gamut, we turn to the games industry to help us find out: why is launch day so difficult to get right, and why aren't companies better prepared to handle the problems they know are coming?

01:01 — Brent Simoneaux
This is Compiler, an original podcast from Red Hat.

01:05 — Angela Andrews
We're your hosts.

01:06 — Brent Simoneaux
I'm Brent Simoneaux.

01:08 — Angela Andrews
And I'm Angela Andrews.

01:10 — Brent Simoneaux
We go beyond the buzzwords and the jargon to simplify tech topics.

01:15 — Angela Andrews
Today we hear about why it's so hard to launch multi-user software.

01:25 — Brent Simoneaux
Producers Johan Philippine and Kim Huang are here to share what's at the end of the ethernet road.

01:33 — Johan Philippine
Our first guest is Stephen Gordon. He's a strategic accounts manager for public sector clients here at Red Hat. More specifically, he's helping the armed forces stand up defense training simulations. His experience working with 343 Industries, the studio behind the Halo franchise, was a big help. He clued us into some of the challenges multiplayer games face.

01:58 — Stephen Gordon
Typically when you launch multiplayer services games, there's a couple of different problems that you're trying to solve for, regardless of the size of your studio. You're trying to reach players globally, so reach. Spikes are a problem.

02:13 — Johan Philippine
So we've got two things to start with. You've got reach and spikes.

02:18 — Brent Simoneaux
Reach and spikes.

02:20 — Johan Philippine
Reach is all about being able to have players to connect from all over the world to your game. Depending on where your servers are, depending on how many you have, that can affect who can connect and how much latency they experience. Spikes refer to large numbers of players trying to connect at the same time. As fast as computers are, there are limits to traffic and loads.

02:45 — Stephen Gordon
Traditionally, without the cloud or without environments digitally that can handle seasonality, additional players coming onto your game, like say Minecraft. So all of a sudden 10,000 new users show up. I've got to handle the spikes, handle the fast-growing number of players.

03:03 — Johan Philippine
We'll get into the technical aspects of that a little bit later in the episode, but I wanted to introduce it here because it's a concept that all multiplayer games face: seasonality.

03:13 — Johan Philippine
You're not always going to have the same number of users connecting to your games. Excitement and novelty often lead to these spikes. The most common of which being on launch day.

03:24 — Brent Simoneaux
That makes sense.

03:25 — Johan Philippine
Everyone wants to try out the game when it first comes out, and as time goes on and the novelty wears off, so does the population trying to connect. There's also a day/night cycle to consider when traffic is highest, and then dips probably a little bit too long after a reasonable bedtime, and then picks up the next day. Then when a big update or a content drop or some expansion hits, you get that spike of users again who want to try out the new content. Now on top of connecting to and running the actual game, there's a host of other secondary services that go along with these games, and they present their own challenges.

04:01 — Stephen Gordon
Then there's the data element. There's scale stored around player data globally. There's leaderboards. Now there's more monetization of games and sponsored game players.

04:14 — Brent Simoneaux
Help me out here, Johan.

04:15 — Johan Philippine
Yeah.

04:15 — Brent Simoneaux
What does the infrastructure look like for what he's talking about?

04:21 — Johan Philippine
Well, it's pretty complex. There are three basic parts to running a multiplayer game. The first thing that players are going to encounter is the matchmaker. Think of it as a waiting room where you first connect to the online service, and then that matchmaker determines, okay, who am I going to connect into a game together? Based on skill, who's going to be playing against each other and then what server are they going to go to?

04:51 — Johan Philippine
You've got the game server itself, which is, okay, now that we've got our players, we're going to spin up a server so that they can play this game. Then that's going to run all the different elements of the game.

05:02 — Johan Philippine
Then you've got the net code, which is the infrastructure that connects the players and the matchmaker to the game servers itself. That is there to make sure that everyone is having a smooth experience, that latency is low, and that everyone is able to stay connected to the game.

05:19 — Angela Andrews
Okay.

05:19 — Johan Philippine
Stephen gave us a little bit more details on what that could look like in terms of the actual hardware running games these days.

05:28 — Stephen Gordon
If I'm hosting a game on a local server, I'm going to have to continue to rack servers and add capacity, scale out, scale up, maybe add memory. By the way, the conventional CPUs are not ideal for rendering what's in modern gaming. So now you're talking about GPUs.

05:51 — Stephen Gordon
GPUs on a server in the cloud are not cheap. So I've got to find a way that I can persist and continue to add that capability. But it does solve a scale problem. So prices come down. I have availability to play on cloud-hosted architecture with everything behind it, including large-scale databases, distributed databases, different regions for availability, failover. So you have much more data loss, you have a lot more opportunity for low-latency games. Some games require a lot more latency or low latency.

06:31 — Brent Simoneaux
Angela, can you help me out here?

06:33 — Angela Andrews
Sure.

06:34 — Brent Simoneaux
I'm not sure I quite understood what Stephen said.

06:38 — Angela Andrews
Well, he's talking about making sure the infrastructure is robust enough to handle these types of games. You need to be able to scale up immediately, if you're scaling up.

06:53 — Brent Simoneaux
Those spikes, right?

06:54 — Angela Andrews
Right. But we're talking cloud infrastructure, so that usually means scaling out. So instead of having a server, that one server may scale up to three or five. They may scale up in the United States region or a European region, depending on where the users are.

07:12 — Brent Simoneaux
Okay.

07:12 — Angela Andrews
Now he's talking about GPUs, because we're talking about graphics processing, and that is very expensive, expensive on hardware, expensive money-wise, but if you want the best experience for your players, you want to make sure that you have the best infrastructure behind it.

07:30 — Angela Andrews
So he was talking about a lot of data. So you need databases to be able to scale. They have to be distributed because your customers are distributed. They need to be able to keep track of their scores and where they are in the game and their positioning. It's a lot of moving pieces.

07:48 — Johan Philippine
Yeah.

07:48 — Angela Andrews
So what he's saying is you can have much more data loss because, depending on where you are, if there isn't a really big connection, you may be dropping packets somewhere from where you are to where the servers are. So there's a lot of moving parts.

08:05 — Angela Andrews
It comes down to how fast your connection is to the internet. We all know know that can vary. Then what does the infrastructure look like behind these games and how is it supported?

08:18 — Angela Andrews
So, like you said, launch day is so important, and you want to make sure you have to set aside enough infrastructure to make sure that you have the capacity to give your customers what they want. But how do you know? How do you know how many people are going to join you?

08:35 — Johan Philippine
Yeah.

08:36 — Angela Andrews
That's where the magic comes. How do you decide how much to put up in advance of launch day? So I guess we'll find out more about that.

08:44 — Johan Philippine
Mm-hmm.

08:45 — Angela Andrews
Okay.

08:46 — Johan Philippine
Absolutely. Yeah. Just to build on that, because that was all excellent and on point, you really need powerful systems that can handle high volumes, right?

08:56 — Brent Simoneaux
Yeah.

08:56 — Johan Philippine
Not only that, but then you need to keep latency or lag or ... Again, the amount of time between a command is sent from a player's computer to the server, and then it being recognized and sent back, you need to keep that very, very low. These are measured in milliseconds.

09:13 — Brent Simoneaux
Yeah.

09:13 — Johan Philippine
When any of the elements that we've been talking about aren't right, the latency, the inability to connect, if there's too many people trying to connect to a server, and then you get things like long queues to be able to get into the game, if any of those aren't right, it makes for a bad experience, and those players are likely to go elsewhere, right? They're-

09:31 — Brent Simoneaux
The stakes are high.

09:32 — Johan Philippine
The stakes are really high, especially for launch day, right?

09:34 — Brent Simoneaux
Yeah.

09:35 — Johan Philippine
Because if you make a big splash and people are able to connect and have a good time, then that'll build, and you'll get more and more players and have a successful game. But if it goes the other way, if it's a big colossal failure, I mean that's going to be losses in the long term, right?

09:50 — Angela Andrews
Indeed. People have long memories.

09:52 — Johan Philippine
Yeah.

09:52 — Brent Simoneaux
Yeah.

09:54 — Johan Philippine
Now Stephen shared just a few of the technologies that are helping with these issues. Our next guest is going to give us some of the technical considerations that can help companies avoid or at least better prepare for the problems that could arise.

10:14 — Kim Huang
Okay. So I spoke with someone who has a little bit of knowledge about the technology needed to scale operations in gaming.

10:25 — Aaron Moon
My name's Aaron Moon. I am a senior partner engineer with Unity Gaming Services, specifically with multiplay. Big, long word that basically means I scale video games.

10:36 — Brent Simoneaux
Ooh, he's at Unity.

10:37 — Johan Philippine
Mm-hmm.

10:38 — Kim Huang
Yes, Unity. Unity is an engine that a lot of independent game developers use to build their games. Aaron talks a lot about the cultural divide between enterprise IT, so people like us, and the video game industry in general.

10:55 — Aaron Moon
Most of the games industry is programming, and they're not really into infrastructure. They digest hosting as a commodity. And so, these things were just not a focus of theirs. So that was surprising. So I had to bring some of that with me. Shell scripting, a lot of those fun things, working on game servers. Those were skills that I brought that were born and bred the hard way. Former RHCSA, RHCSE. So those skills actually did come to bear, and it was very useful.

11:23 — Brent Simoneaux
RHC-

11:25 — Kim Huang
Yes, RHCSA. That stands for Red Hat Certified Systems Administrator, and RHCE, which stands for Certified Engineer, respectively. Those are the two certifications that Aaron cites as being in his background, and that was part of the experience that he brought to Unity Gaming Services.

11:42 — Brent Simoneaux
Got it.

11:45 — Kim Huang
So you could say that Aaron brings that kind of experience, that enterprise IT mindset, to work that largely doesn't fit that mold in video game development. But why is that? Aaron says, for smaller development studios, the money just isn't there.

12:03 — Aaron Moon
From an indie perspective, you've got an ROI per concurrent user that you're trying to preserve. You have a budget which you can spend in infrastructure. So a lot of those solutions that would pass muster in the enterprise realm, they just ... Financially, they're not feasible here yet.

12:20 — Angela Andrews
That is so true. You want high availability, you want redundancy, you want fault tolerance. You want to be in multiple availability zones. Every word I just said costs more money.

12:31 — Brent Simoneaux
Yeah.

12:33 — Kim Huang
Yeah.

12:33 — Brent Simoneaux
Okay.

12:34 — Angela Andrews
If you're talking about spending money and making sure that the concurrent user can match up to your infrastructure, sometimes you're not going to get the infrastructure that you may think your game deserves, because all of this is super expensive.

12:49 — Kim Huang
Yes. It also requires people to maintain, if I'm not mistaken.

12:54 — Angela Andrews
Indeed. Indeed.

12:55 — Kim Huang
So we have a lot of game developer teams that are only three, four, or five people, that they're trying to scale up and they're trying to have that high availability, but they don't have the money or the people in order to support that, at first at least. But studios who have to, or want to rather, satisfy large game publishers and player demand, they want to expand and be more successful and be larger, have that money to hire more people and build out their operations.

13:28 — Kim Huang
They have to enter a world where it looks a lot like where we work every day, the enterprise level. There are so many things to consider. How is the game going to make money? That means in-game purchases or a marketplace where people can buy, for example, cosmetics for their characters. Will there be expansions or other downloadable content? All of that involves programming, data, servers, product design, stacks, platform integration, and a whole lot of planning.

14:00 — Johan Philippine
It's a lot.

14:01 — Angela Andrews
Yes. It is.

14:04 — Kim Huang
Yeah. As games are launched, it sounds like some developers suffer from what I like to call the curse of success. I'll let Aaron explain.

14:13 — Aaron Moon
The infrastructure is really also indicative of the scale. We speak with studios out there, small ones that have blown up. I've seen situations where they have their own small scaling infrastructure, and then they blast up into the millions, kind of that lottery ticket scenario. And so, they actually have to change in step, like while their game is launching and blowing up.

14:34 — Aaron Moon
So I would say if you're doing the session-based, a lot of the infrastructure you want to set up is to be able to quickly scale in multiple regions and location no notice. You're going to rely on the cloud in so many ways.

14:50 — Aaron Moon
Now if you can mitigate the cost and have bare-metal infrastructure staged there, maybe you have marketing data that tells you you've got this many people signed up for pre-launch, so it guides your hand, that'll help you set up a server infrastructure.

15:07 — Kim Huang
Now, Angela, earlier in the episode, you said something along these lines. How do you accurately predict how many people are going to be there on day one? How do you know many users you're going to have on a launch?

15:19 — Kim Huang
One of the metrics that people use in the gaming industry is pre-orders or pre-launch signups. There are a number of other types of metrics and things that they use, beta. A lot of games will open for beta to see what the interest is and measure how many people sign into sessions, how many people were able to get into sessions.

15:40 — Kim Huang
It serves two purposes. It's kind of like a stress test for the game that you've built. Then on top of that, you also can gauge interest to see how many people will sign in on that day one.

15:50 — Angela Andrews
So that's interesting because you want to see, well, how have we been marketing and who's been interested so far? So at the very least, you have to account for that, and you want to make sure that your infrastructure, in the beginning, before any real scaling happens, is already in place to match what you think your capacity will be.

16:12 — Angela Andrews
But then right after that, you have to make sure that if there's this lottery ticket scenario that Aaron's talking about, this could blow up. How do you make sure you have the right infrastructure and scalability and automation and availability in place? If this game goes global, how do you make your customer in Philadelphia, Pennsylvania have an amazing experience and how do you have your customer in Paris, how do you make sure everyone's getting what they need? You're right, the curse of success, if it blows up past what you've allotted, someone might be upset.

16:51 — Kim Huang
Exactly.

16:52 — Angela Andrews
Someone's going to be upset.

16:54 — Brent Simoneaux
What does that actually ... Because I've never experienced this before. What actually happens when your game blows up?

17:03 — Kim Huang
Let's say the indie game developer and they've built a new game. Maybe their intention is to have this game and publish this game on their own and continue on as an indie developer, or maybe they want to get noticed by a larger gaming publisher, which you have a larger ... Like Microsoft is a big game publisher. If you're aware of the market, you know that they have a lot of different studios that they work with to make the games that they release. But in order to get their attention, you have to have a pretty successful launch, right?

17:36 — Johan Philippine
Yeah.

17:37 — Kim Huang
So let's say this new game launches and the server infrastructure is built and tested ... And we'll get to that later, testing. But it's built and tested for about 50,000 concurrent players, and the sessions are very small. Let's say each session is three to four players, or something like that, but the sessions go very quick. So maybe a battle royale-style game.

18:01 — Kim Huang
But for whatever reason, the game goes viral and, overnight, you have hundreds of thousands of players trying to access the game, but they can't get into a session because there's too many people in the lobby. So they go to social media and let the devs know, let everybody who will listen, they let them know, that there's a problem with the game. They can't get in, they can't play. It's day one. They want to play their game. They spent their money.

18:26 — Kim Huang
Media picks up the story, and now you have a situation where a developer is ... They have a bad reputation or a bad game launch. Now the developer has to work hard to scale up their game. That typically involves a hybrid cloud infrastructure nowadays, which costs money and time, like we talked about earlier.

18:48 — Johan Philippine
Yeah.

18:48 — Kim Huang
Basically they have to hope that people come back to improve that long tail player number. So it's post-launch, you're going into the weeks and months following, and you're trying to entice people to come back to your game. Maybe you're doing some more marketing. Maybe you're doing a lot of community management. That's where community managers come in and they go, "Hey, the player community, we want you to come back. We want you to try this new expansion pack or these new downloadables. We want you to come in and explore what we did to fix all the issues we had before."

19:20 — Kim Huang
The positive outcome is that players come back, they enjoy the experience, and sometimes that can lead to more projects being greenlit, especially if they have an agreement with a larger publisher, and people get to work on new games.

19:33 — Brent Simoneaux
I imagine that's not always the case, though, right?

19:37 — Kim Huang
The negative outcome is that players, well, don't come back and those long-tail numbers don't go up. The game pretty much flops, and game publishers lose faith in that and new projects don't get greenlit. If new projects don't get approved, the studio has to go out of business.

19:56 — Brent Simoneaux
What I'm thinking about is that there's probably some kind of playbook or some kind of-

20:02 — Angela Andrews
That's what I was just thinking. What's the formula? Is there a formula-

20:06 — Brent Simoneaux
Yeah, some kind of formula-

20:07 — Angela Andrews
... that we could follow?

20:08 — Brent Simoneaux
... tried and tested-

20:10 — Angela Andrews
Yeah.

20:10 — Brent Simoneaux
... that we can follow. At the same time, though, I'm thinking about how each one of these cases is going to be unique. So there's a playbook, but then you're probably going to have to adapt and be really nimble and ... You know?

20:24 — Kim Huang
It's a catch-22. To me, the thing that makes a game enticing is how it sets itself apart. A lot of that has to do with the actual play experience. That's why you have so many different types of technologies that stack, and the engine and the middleware, the net code, like Johan was talking about earlier. Those things come together and create a unique player experience. That's why, according to Aaron at least, no two launches are the same.

20:53 — Johan Philippine
Yeah.

20:54 — Kim Huang
This is why testing well and testing often is very vital.

21:00 — Aaron Moon
So the most common ones I see are the failure to adequately test their backend. The way that you approach that is frequent testing. So in the industry, we like to simulate launch failures. Think of it like the Chaos Monkey scenario. We're going to throw everything at it and watch it catch fire, analyze the ashes, and then we come back and we get ready for launch.

21:23 — Brent Simoneaux
Wait, that's real?

21:23 — Johan Philippine
That's a real thing.

21:27 — Kim Huang
Yes, that's a real thing. You deliberately crash your servers. You deliberately try to break what you've built just to see if it can stand the fire, if it can handle the load.

21:41 — Johan Philippine
We have more on this in season two of Command Line Heroes-

21:45 — Brent Simoneaux
Oh.

21:45 — Johan Philippine
... if you want to hear more about the Chaos Monkey and the-

21:47 — Brent Simoneaux
And Chaos Monkey is the official name?

21:51 — Johan Philippine
Yeah.

21:52 — Kim Huang
Yeah.

21:52 — Johan Philippine
It's a real tool.

21:53 — Kim Huang
Yes. But what you were talking about earlier, Brent, Angela, about the formula, there is no real formula here. It just depends on trying to predict, and sometimes you can't predict popularity. You can't predict success in this case, and therefore you can't predict that your servers will be able to handle everything.

22:19 — Brent Simoneaux
That is not the answer I want, Kim.

22:21 — Angela Andrews
Me either.

22:22 — Kim Huang
No, but it's true.

22:26 — Brent Simoneaux
But I guess that's the reality we have to live with.

22:28 — Angela Andrews
Yeah.

22:28 — Kim Huang
That's right.

22:30 — Angela Andrews
I mean it feels like if you've done all the things properly, you've done your due diligence, you've done your QA testing, you've done your Chaos Monkey testing, and you discover things that are broken. You fix them, you built in that resiliency, and the game comes, the players come, and it's still not enough? What do you do next? You've done everything you could possibly do and it still has the risk of not being successful. That's a lot of pressure.

23:02 — Kim Huang
It is. But if you think about it, it's not very different than what happens in the enterprise. What do you do? You go to your users and keep open a clear line of communication with them and be honest.

23:13 — Brent Simoneaux
Yeah.

23:13 — Angela Andrews
That's true.

23:13 — Kim Huang
You say, "We're working to resolve the issues on the back end. We're working and doing the best that we can with the resources that we have, and we hope this experience will not put you off on coming back or using our platforms and using our products, because we're actively trying to adjust and improve and adapt."

23:33 — Brent Simoneaux
Is this just because people are unpredictable? Like you don't really know until you get lots of people, like real people, in there playing.

23:43 — Kim Huang
Yes, that's exactly it. Aaron can speak a lot better to this than I can.

23:48 — Aaron Moon
You can test all you want, but until you put real players at mass in your game, you don't really understand how well they're going to interact. You don't understand what kind of load profile that is.

23:59 — Johan Philippine
This is a persistent problem even for game studios who have a lot of experience launching their games, right?

24:08 — Kim Huang
Absolutely.

24:08 — Johan Philippine
We've seen some AAA games that, on launch day, even though they've had big games launched in the past, because each game is new and different, it requires a slightly different set of infrastructure. Because of that, you're going to have some interactions that you're not prepared for.

24:28 — Kim Huang
Right. Maybe this time around you're launching a very large cosmetics-based aspect of your game, and that introduces a whole new level of physics and things with your characters interacting, that maybe on a previous launch you didn't have to worry about, but now you do. You're introducing new features. When you introduce new features, you have to figure out how that's going to affect the way that all the rest of your infrastructure works together.

25:00 — Brent Simoneaux
I want to come back to something we were talking about earlier, which is the cost of all of this.

25:06 — Kim Huang
Yeah.

25:07 — Brent Simoneaux
I was just thinking about how it's probably just as important to be able to scale down-

25:12 — Angela Andrews
Yes.

25:12 — Brent Simoneaux
... than it is to be able to scale up.

25:17 — Kim Huang
Yes.

25:17 — Angela Andrews
Scalability in and of itself is a powerful feature of the cloud. That's what makes it amazing. The ability to, at the click of a button, you're literally able to scale that infrastructure and everything underneath it as well, servers, firewalls, regions, load balancers. These things grow and grow, but when the traffic goes down, you cannot-

25:44 — Brent Simoneaux
No.

25:44 — Angela Andrews
... keep all this stuff up and running. So your app has to be smart enough and say, okay, our load has gone down. It has been down for whatever threshold you've set. Then it starts to orderly scale things down. That's what you want. You want that built-in scalability to make sure that one, yes, you can hit those marks when you need to, but definitely don't leave stuff running-

26:10 — Kim Huang
Right.

26:10 — Angela Andrews
... when you don't ... Don't do that.

26:12 — Kim Huang
Yes.

26:12 — Johan Philippine
It's burning money over there.

26:15 — Kim Huang
Yup. Burning money. That's exactly ... Actually, I'll bring Aaron back because he says it ... Again, he says this in a way that I feel is pretty powerful.

26:24 — Aaron Moon
In hybrid, if I buy a machine for a month and I can host X amount of sessions on it, and it's always full, I'm winning, because that's probably going to be a lot cheaper than cloud. But if I am doing the same thing in cloud, three times the expense. It's so much more expensive. So unless that thing is only on when it needs to be used and turned off as soon as it doesn't, again, I'm just bleeding out spend.

26:47 — Aaron Moon
So scaling down is definitely something that needs to happen the right way, because when you're running a live service game, you can't just push the stop button. There's no empty button to kick people out of servers.

26:57 — Angela Andrews
That's interesting. When you're talking about cloud, you're paying for that data, that ingress and egress. When people are using that game, every click, all of that costs money. Like you said, if it's always running, if it's just your system and you don't have to worry about bandwidth or things like that. When you're in the cloud, everything costs, and having that data get back to your customer and then putting data up there, sometimes there's no ingress costs, but definitely egress costs. Man, you're burning money.

27:32 — Kim Huang
That's right.

27:33 — Angela Andrews
So Aaron has a point here. You have to be very frugal and smart about that, because you're going to be losing money.

27:44 — Johan Philippine
One of the things he was telling us about that inexperienced game studios face is they've scaled up and they've got their cloud instances running. Then when they try to scale down, they've got ... Maybe one of the cloud servers has a game running and they don't want to just cut people off because that's a bad experience.

28:02 — Brent Simoneaux
Yeah.

28:02 — Johan Philippine
But in the time that it takes for that one game to finish, it may launch another single session on that server. And so, unless you've got the proper load balancing to really offload any games from appearing on that cloud server again, you're coming up with a problem where that one server is being used for a very low amount of population. So it's something that you really need to take into account when trying to figure out how you're going to scale down that cloud footprint.

28:34 — Kim Huang
I want to switch gears and talk a little bit about patching. So if you talk to players, they will always look at you whenever you say day one patch with a mixture of happiness and fear. A day one patch happens when there are issues or other desired changes ... Maybe not, but related to bugs ... that need to be done after a release. It's not a sign of failure necessarily. It's meant to make the playing experience better. But Aaron says proper planning for scale when patching a game is key because it can make or break a studio.

29:12 — Aaron Moon
It also has to be thought of when you're patching, because how do you bleed them from one version to the other? So it's a very long, multifaceted issue, but scaling down is where people really lose and where we see games, when they start to go into long tail numbers, they just simply go out of business if this is not mastered.

29:31 — Angela Andrews
So there's multiple issues here that I hear Aaron mentioning. He's talking about the issue of not being able to scale down because all of your players may be spread apart all these different servers, and you can't remove those resources. Well, that's a problem.

29:48 — Angela Andrews
But when you're talking about this day one patch where, well, how do you intelligently decide which players are going to get patched and how? How do you roll these patches out so at some point every player has the newest version of the software?

30:06 — Angela Andrews
Again, this is another planning exercise. You have to make sure it's done smart because, again, if a customer has a bad experience anywhere in between, they're going to feel it and you're going to hear it.

30:20 — Brent Simoneaux
So, Kim, Johan, Angela, we have been talking a lot today about games. But launch days are a reality for a lot of people outside of gaming. So I'm curious what we can take away from this example and pour it over into other scenarios or other facets of our lives.

30:46 — Angela Andrews
I'll start.

30:46 — Brent Simoneaux
Yeah.

30:46 — Kim Huang
Yeah.

30:46 — Angela Andrews
Planning is key.

30:48 — Brent Simoneaux
Okay.

30:48 — Angela Andrews
That's what we're hearing. Planning is so important. Testing is important. Listening-

30:55 — Kim Huang
Important.

30:56 — Angela Andrews
... is important. These are all skills that are totally transferable, even if you're talking about game launches or software launches or concert ticket, first ticket days. Those are things that are synonymous across all of those different use cases. When things go wrong ... And apparently they will, because it sounds as if you really can't gauge it. Sometimes things go wrong. But an effective postmortem, what did you do exceptionally well? Well, why'd you do it so well? How? Then if something goes wrong, well, what went wrong? What needs work? Well, why? Then you figure those things out.

31:36 — Angela Andrews
I think having open communication, it doesn't matter the vertical or what you're launching, but those are the things that I think we can take away, that this is how you'd be remotely successful and this is how you bounce back.

31:53 — Brent Simoneaux
How about you, Johan?

31:54 — Johan Philippine
There's a couple of things that came out for me, and I'm going to use the game industry as, again, the metaphor here. But the biggest thing is that no two game launches are alike. To best prepare your launch and to best build the infrastructure to support that launch, you really need to keep the type of game that you're running in mind. Is it going to be a lot of short sessions that need servers to go up and down in quick succession, or is it longer sessions like something like a massively multiplayer game where the server stays on for a very long time? Those two launches have very different needs.

32:35 — Kim Huang
Correct.

32:36 — Johan Philippine
So you keep that type of launch in mind when you plan for that infrastructure. What that leads to is that, this is what Angela was saying, success isn't necessarily in the right kind of tools. It's more about the process of launching the game, right?

32:49 — Brent Simoneaux
Yeah.

32:50 — Johan Philippine
You want to make sure that, because each launch is different, you are asking the right questions, that you are planning the right way, and that you are testing everything over and over and over again until you get to that launch day.

33:07 — Brent Simoneaux
And, Kim, how about you?

33:09 — Kim Huang
Well, I definitely came away from this episode with a better understanding of why software launches are so complicated. There's the extremes of failure and then too much success, when, honestly, if a development team is working with a small budget and the resources are finite, the ideal might be somewhere in between those two things. But there's no way to gauge or predict success or trends or user demand.

33:38 — Kim Huang
I think that there's a lot of mind share and a lot of collaboration that could happen between people who work at enterprise IT and people who work in video game development. There's different things that they could learn from each other. I think that development teams can do a lot of research and planning and exploration of open culture and other kind of use cases and all of the knowledge that people have shared about the process to try to get through a product launch the best way they can.

34:12 — Kim Huang
Also, I want to talk a little bit about the Red Hat Community of Practice, because they did help us make this episode a little bit. So for those of you who don't know, Red Hat has a community of practice that is a community of Red Hatters who use open source software to create games. If you stick around for the credits, we can tell you exactly how to find out more and get involved.

34:34 — Angela Andrews
So that was the launch day episode. We learned so much about game launch days, but we want to hear about you. What are some of your favorite game launch days and what did you learn from them? Would you mind tweeting us at Red Hat? We would love to hear about it. Don't forget to use the hashtag, #CompilerPodcast. We'd love to hear it.

34:54 — Kim Huang
I have a couple of favorites, but I'm not going to name names. But please feel free to tweet us and name names.

34:58 — Angela Andrews
Yes, name names. We would love to know what some of your favorite game launches are, so we can keep an eye out for those, or, on the flip side, if you've experienced horrible game day launch days, we want to hear about those, too. Come on, share the dirt. Share the dirt with us.

35:13 — Kim Huang
Yup.

35:20 — Brent Simoneaux
And that does it for this episode of Compiler.

35:24 — Angela Andrews
Today's episode was produced by Johan Philippine, Kim Huang, and Caroline Creaghead.

35:29 — Brent Simoneaux
A big, big thank you to our guests, Stephen Gordon and Aaron Moon.

35:35 — Angela Andrews
Victoria Lawton makes sure our distribution infrastructure can handle the load.

35:41 — Brent Simoneaux
So Red Hat has its very own gaming community of practice, and they are currently looking for contributors. If you want to learn more or get involved with a launch of your own, visit arcade.redhat.com. That's arcade.redhat.com.

36:01 — Angela Andrews
Our audio engineer is Kristie Chan. Special thanks to Shawn Cole. Our theme song was composed by Mary Ancheta.

36:10 — Brent Simoneaux
Our audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oats, Rachel Ertel, Devin Pope, Matias Foundez, Mike Compton, Ocean Matthews, Paige Johnson, and Alex Traboulsi.

36:27 — Angela Andrews
If you like today's episode, please follow the show, rate the show, leave a review, share it with someone you know. It really helps, because sharing is caring.

36:38 — Brent Simoneaux
All right, we will see you next time.

36:41 — Angela Andrews
Later, guys and gals.

36:42 — Kim Huang
Bye.

36:44 — Angela Andrews
Later, folks.

36:47 — Brent Simoneaux
All right, bye.

36:48 — Johan Philippine
And, scene.