Online multiplayer games are a staple of modern entertainment. But internet connectivity is an unpredictable factor in the experience. That’s why netcode exists. But what is it? And how does it really affect players?

00:02 — Kim Huang
Here's a riddle for you. What's big, full of revolutionary software and emerging technology, and makes more than 300 billion US dollars a year? Give up? The answer is the video game industry. And with that kind of money being spent, it's important all the tech works and players have good experiences. Something that plays a key role in that is netcode. But what is netcode? How does it work? And if a developer wants to build a game for players to enjoy, what are some of the things they need to know?

00:43 — Angela Andrews
This is Compiler, an original podcast from Red Hat. I'm Angela Andrews.

00:50 — Kim Huang
And I'm Kim Huang.

00:51 — Angela Andrews
We go beyond the buzzwords and jargon and simplify tech topics.

00:56 — Kim Huang
Today's episode, the truth about netcode. Angela, do you play video games?

01:07 — Angela Andrews
No, I don't.

01:09 — Kim Huang
Okay. Well, I do, and this is one of the situations where I have a lot of experience. I feel like it's very rare that I walk into a situation where I'm the expert.

01:21 — Angela Andrews
You are the expert.

01:23 — Kim Huang
Yes. But today's episode we're going to talk about video game development and in particular netcode, which we'll talk about what it is in a second. But first I want to introduce someone I talked to with a lot of knowledge about video games.

01:40 — Angela Andrews
My name's Hunter Kepley. I'm an associate software engineer at Red Hat. I've done game development for about 10, 15 years on my own just as a hobby thing, as well as I've released a couple of things in the past. My older brother got me into Linux and he got me into development, and then I started with Pygame and it all went off from there.

02:00 — Kim Huang
All right. So I spoke to Hunter because he's kind of a resident expert at Red Hat on interconnectivity between games or online gaming and netcode. And I can try to kind of explain netcode, my own definition in 30 seconds or less.

02:18 — Angela Andrews
Oh, you got this.

02:19 — Kim Huang
You sure?

02:19 — Angela Andrews
Yeah.

02:20 — Kim Huang
Okay. All right. I'm going to try. Netcode to me is kind of like the glue that connects different experiences in gaming to each other. So if you're in California or you're in the United States and you want to play a game online with someone in Croatia or Romania or Thailand, netcode is there to kind of make sure that the experience goes as smoothly as possible. It handles interconnectivity, it handles lag and latency issues. Think about the episode that we did on edge computing and video gaming. A lot of that discussion is going to come back in this episode. We're talking about interconnectivity in gaming in online multiplayer situations. So netcode is the software that is used to mitigate data being sent back and forth between players and sometimes between the player and the server. Does that make sense?

03:15 — Angela Andrews
A little bit.

03:16 — Kim Huang
Hunter can probably do a better job at this, so I'm going to let him do it.

03:19 — Angela Andrews
Okay.

03:19 — Hunter Kepley
So netcode is pretty much the networking for video games. At its core level, the only difference between that and let's say the networking between your computer and Google is that it's going to use unreliable networking rather than reliable. So it's much different because you're not always expected to receive the data from the game, but you have to have a good way of handling it.

03:44 — Kim Huang
What do you think, Angela? Is that better?

03:46 — Angela Andrews
So as soon as I heard networking, my ears perked up because I know a little bit about networking. He didn't say it's not networking, but it's more dealing with unreliable networks.

03:57 — Kim Huang
Yes.

03:58 — Angela Andrews
So is it networks or is it the code that helps with these unreliable networks and the gaming experience? I have a lot of questions.

04:10 — Kim Huang
Yeah, absolutely. It's the latter. Think of it as software solutions that are in place to handle or compensate for unreliable connectivity. So if one person has a really good connection and the other person does not, they can still have an enjoyable experience because of this kind of layer of software that's sitting as middleware, essentially middleware that is connecting the data to the application itself. Interesting. All right. Tell me more. I asked Hunter how this all looks to a player.

04:46 — Hunter Kepley
The way that all works is that you are in the future and everybody that you're playing with is in the past on your screen. And that goes for everybody. And that is all because of lag compensation. That will never go away in those games.

04:58 — Angela Andrews
Okay.

04:58 — Kim Huang
Yes. Exactly.

04:59 — Angela Andrews
The way that the players are presented is what matters.

05:04 — Kim Huang
Yes.

05:05 — Angela Andrews
If I'm in the future and other players are in the past, they're getting past me, which means I'm moving forward. And because this is compensating for said lag or disrupted networks, their experience won't be hindered by any of that because I'm in the future.

05:26 — Kim Huang
Yes. And what the netcode does is kind of anticipate the inputs from the player that's in the future and make sure that they are correct and that they're syncing up with what the person's actually doing.

05:37 — Angela Andrews
Okay. Light bulb moment.

05:39 — Kim Huang
Yep. Exactly.

05:39 — Angela Andrews
Got it.

05:40 — Kim Huang
It's pretty genius if you think about it.

05:41 — Angela Andrews
It is.

05:42 — Kim Huang
Yep. As long as people in Los Angeles want to play games with people in say, Berlin, internet connectivity will always be an unpredictable factor in the experience. Data moves back and forth between the client or the player's machine and the server where much of the compute lives in this case. Hunter gives more detail into what that looks like behind the scenes.

06:07 — Hunter Kepley
Along the way, several packets have gone missing. They just didn't get to you. You don't have that data. Suddenly, you're going to see players start teleporting around. So really the biggest thing is to deal with that kind of stuff. And in that case it would be interpolations. They're not going to just jump around, they're going to stop for a second and then slowly keep moving. And if you speed it up to a certain amount and you tweak it looks pretty smooth and it looks like people are just moving around normal.

06:32 — Kim Huang
That's kind of what the netcode is doing there.

06:34 — Angela Andrews
Yeah. This is definitely a light bulb moment and that explanation kind of put the head on it. Okay. So again, we are trying to make sure that the experience is normal. It looks like what's going on is actually what's going on, but it's making concessions and it's anticipating.

06:55 — Kim Huang
Yes.

06:56 — Angela Andrews
So this is how they do it. Okay.

06:58 — Kim Huang
This is how it's made. Obviously there's a lot more to talk about.

07:03 — Angela Andrews
Oh, I'm sure.

07:04 — Kim Huang
But I wanted to stop and ask, maybe it's a little bit evident, especially if you play video games and you listen to our podcast, by the way thank you. But why is netcode important?

07:15 — Hunter Kepley
It comes down to the experience. Certain actions need to be pretty reliable. If you're going to pick up an object off the ground, you want to make sure that that gets picked up. Every single player knows that's picked up. That has to go straight to the server.

07:28 — Kim Huang
Back up a little bit. He does talk about server, and I talked a little bit about client client side and server side. So I'm going to break this down a little bit. The client side is like the player. If you think about what goes into the player side, you have data from your character. If you're playing an RPG or a role-playing game, maybe your character has blue hair or they have pants instead of a dress or they have a sword, or they have a spear, that's personalized. That's your side, that's coming from the client side. And inputs also from your keyboard, or if you have a controller, a gaming console, that's your controller, those are coming from you. That's the client side.

08:13 — Kim Huang
The server side is the game world or the environment. It's also where matchmaking takes place if you're playing a competitive kind of online think fighting game or something. CPUs or AI controlled characters and NPCs or non-playable characters also live on the server side. It's helpful in this case, if a metaphor is needed, to think of the server as the canvas, the players as the artist, the client is the brush and the inputs are kind of like the brush strokes and the paint.

08:46 — Angela Andrews
I like that visual.

08:47 — Kim Huang
Did I do good?

08:48 — Angela Andrews
Yes.

08:48 — Kim Huang
Yay. I did good.

08:50 — Angela Andrews
I'm a visual person. Thank you for laying that out for me.

08:53 — Kim Huang
No worries.

08:54 — Angela Andrews
So I got it. I got it.

09:00 — Kim Huang
But there are different types of video games so that it automatically means that there's different types of netcode. What are they?

09:07 — Hunter Kepley
There's quite a few. Mostly what's used nowadays is going to be, at least for FPS games, some form of lag compensation which pretty much just does server rollback. There's also delay-based, which is the most popular for turn-based games, for card games, for anything where you don't need that competitive advantage or anything like that.

09:27 — Kim Huang
So I'll break down kind of what he's saying.

09:29 — Angela Andrews
Okay.

09:29 — Kim Huang
For FPS, so first-person shooters, think of most popular one, Call of Duty.

09:35 — Angela Andrews
Got it.

09:35 — Kim Huang
Think of these kind of big maps, these big server game worlds where different players, and it's a lot of them, it could be dozens of them, are all moving around at the same time and trying to compete with each other. Lag compensation or rollback is kind of that time travel that we were talking about earlier, Angela. It's you're in future, they're in the past. They're dealing with kind of past you and it's all kind of syncing together. And what the server and what the game is trying to do is make sure that the inputs that you're using or the commands that you're putting into the controller or the keyboard are matching up with... They're making predictions of what you're going to do or what your inputs are going to be, and then they're syncing it or checking it with the actual inputs you're making. And that's kind of how rollback happens.

10:23 — Angela Andrews
Understood.

10:26 — Kim Huang
So there's a lot of discussion online about good netcode and bad netcode. I wanted to know what makes good netcode. According to Hunter, that's often a loaded question.

10:43 — Hunter Kepley
The average gamer, they really don't know what the good netcode is, the good anything is in terms of an engine, anything.

10:51 — Kim Huang
And that's because most players don't really know what netcode is and what it isn't. What I feel most people mean when they talk about netcode, they're talking about the particular experience they're having in the context of the game. And remember, netcode is a solution or a software that is supposed to mitigate connectivity issues.

11:16 — Kim Huang
So when these connectivity issues are being encountered, it's not necessarily that the netcode is not working. It just could be such an extreme kind of difference in connectivity, or it could even be an issue on the other client side or the other player side, or it could even be a server issue itself because again, this is all a lot of different points of data and different applications all kind of stacking and playing together and melding together to make this experience. Any one thing that kind of goes wrong or if there's a bug, it could appear like an issue with the netcode. And that's why I feel like a lot of players don't really understand. When they're looking at something happening, some people can look at it and diagnose, "Oh, this is like a netcode issue." But for the most part it could be anything going on really and it's not necessarily connected to the netcode itself.

12:11 — Angela Andrews
So the issue could lie just about anywhere. But it appears when you're having this discussion about netcode it gets blamed when it could be a myriad of things.

12:22 — Kim Huang
Yes. Yes.

12:22 — Angela Andrews
Kind of like DNS is always the issue. You know that joke that we say?

12:27 — Kim Huang
Exactly.

12:27 — Angela Andrews
Okay.

12:28 — Kim Huang
I don't really think that there's such a thing as bad netcode in my experience because it's usually a conflict between what the solution does, what the game is supposed to do and what the players want to do. And that's whether or not the netcode is open source or proprietary. So up next we'll go over the most well-known types of software around netcode and we'll discuss how developers choose to handle it or sometimes choose not to handle it.

13:07 — Kim Huang
Okay. Let's pick up where we left off. Players rely on netcode to mitigate distance and connectivity issues in order to have a close-to-real-time game experience. Netcode solutions are considered middleware since it connects data, the packets we talked about earlier, and applications. So what are the most well-known types of netcode?

13:30 — Hunter Kepley
The most popular for fighting games now is rollback. They used to use delay-based, and then people got kind of sick of it. So if I'm making a fighting game, rollback is the standard now, GGPO, which is the standard. Everybody uses that.

13:43 — Kim Huang
Some solutions around rollback netcode are proprietary. GGPO, it stands for "Good Game, Peace Out" is famous because it's synonymous with fighting games, big ones like Street Fighter

13:55 — Hunter Kepley
With rollback, the worst thing that will happen is that someone's going to do a move and then suddenly teleport back and not do that move. And that's because you both are experiencing the same rollback. So if something doesn't go through and there's rollback frames, then it won't go through for both of you and it will be immediately corrected.

14:10 — Angela Andrews
So no one gets an advantage.

14:12 — Kim Huang
Exactly. Yes.

14:13 — Angela Andrews
Oh, very good. Okay.

14:15 — Kim Huang
That's usually in fighting games where the inputs are kind of instant. You have to be able to kind of counter moves. In turn-based games, it's less important. Kind of what Hunter was saying earlier, you have a delay-based netcode for online games that are kind of turn-based RPGs, real-time strategy games. They don't need that kind of instant gratification or there's, like you said, an instant advantage over one person over the other. They don't need that. So I think of rollback as for everyone else that has instant kind of input, instant stimulus, instant reaction times that are needed. And then for all the other online games that are not so time sensitive, delay-based is fine.

14:56 — Angela Andrews
Gotcha.

14:58 — Kim Huang
GGPO is also famous for being open-source since 2019, allowing for both major and smaller independent developers to use the software for their games. But there are a lot of other options for netcode on the market. Sometimes developers don't think about online play until after the game is built or even after it's been released.

15:20 — Angela Andrews
What?

15:21 — Kim Huang
I know. Yeah. Why?

15:24 — Angela Andrews
Okay. Now, really, that's an afterthought?

15:27 — Kim Huang
Yeah.

15:28 — Angela Andrews
You have to tell me why, because that my friend did not make sense. But okay.

15:33 — Kim Huang
It doesn't, but it does. And I'll tell you why. There's a few reasons. Here's some really big ones. One: money. Building online multiplayer is expensive. Two: time. It can delay the time a game goes to market. So if you're a small development studio and you want to just get a game out there to make some money or satisfy a contract that you've made with a publisher, you might just want to build a single-player game. And then once it gets to a certain level of popularity, you can go back and add online multiplayer as a feature or as an update. And the third reason is culture. Some developers just don't think it's that important that players will just kind of grin and bear it in order to play their game.

16:23 — Angela Andrews
Really. So do you find that users or gamers are just, "Oh, we're just going to deal with how terrible this game is," as opposed to doing the opposite and, "Oh, I'm not playing this game." What are people doing?

16:40 — Kim Huang
It's funny you say that because those attitudes are changing swiftly, especially with the amount of competition out there.

16:50 — Angela Andrews
Explain. For the better I hope, right?

16:53 — Kim Huang
Yes, for the better.

16:55 — Angela Andrews
Okay.

16:55 — Kim Huang
I'll use an example. I'm a really big fan of Metroidvanias. A Metroidvania is a subgenre of game. It's like an action-adventure type of game. It's a portmanteau of Metroid as in the Metroid series and Castlevania as in the Castlevania series. A long time ago, if you wanted to play that type of game, you had two options, Metroid or Castlevania. But now there are so many different games in the market and with online marketplaces like Steam and Epic Game Store and the PlayStation network, there's so many different choices for players to make as far as what they want to play. That means if you don't prioritize an online experience, people can take their money and go somewhere else.

17:42 — Angela Andrews
Exactly.

17:43 — Kim Huang
Exactly.

17:44 — Angela Andrews
So that last one where people will grin and bear it, I don't think that's going to ever be the case.

17:50 — Kim Huang
It was for a while

17:52 — Angela Andrews
It was, but now people have games for the taking. They're so prevalent nowadays. Users have choices.

18:00 — Kim Huang
Yeah. I do want to kind of acknowledge that a development studio that does not have a lot of money can use a single player release to kind of gauge interest.

18:10 — Angela Andrews
Okay.

18:10 — Kim Huang
Do people even want to play a particular game online with their friends? They're putting the game out there to explore that question. And if the answer is yes, then they can spend the time and the resources building the functionality. They could even maybe get a deal with a larger publisher and get that extra money, that extra cash infusion to build out that experience so that the game will be bigger and better than it was before.

18:34 — Angela Andrews
Okay. Kind of like a startup.

18:35 — Kim Huang
Exactly.

18:36 — Angela Andrews
I was just going to use that analogy, the old startup analogy, and get that big infusion of cash and now you can really do something with your product.

18:43 — Kim Huang
Yes, yes.

18:44 — Angela Andrews
Same thing in the gaming world.

18:46 — Kim Huang
Same thing in the gaming world. That's right.

18:47 — Angela Andrews
Okay. But I will say pushing online multiplayer down the line isn't always the best decision. Here's Hunter again.

18:58 — Hunter Kepley
It really first of all depends on the type of game that you want, as well as it depends on, let's say you create a game that is single player, everyone loves it. But it's in a world where everyone wants to play multiplayer, they want to play together. Well, for the longest time they said that it is impossible because, "The engine's done, the game's done. We cannot just throw in netcode," which makes perfect sense. You really can't do that. It takes double the amount of time it would've in the first place, at least.

19:25 — Kim Huang
Hunter is saying adding in netcode as an afterthought is difficult and time-consuming. And bottom line, if the online experience in a game is bad, players will take their time and their money somewhere else. Hunter explains players' responses to a game with bad netcode.

19:43 — Hunter Kepley
There's so many cases where especially nowadays, games will just get review bombed and go from perfect reviews to 40% overall on Steam, for example, all because a server goes down or there are some issues with matchmaking or anything like that.

19:59 — Kim Huang
It has little to do with the game per se. It will be, "This game is good, but I can't play it because the experience is so bad." Or, "I want to play your game. But because you put the netcode, you just kind of slapped it in there on top after the fact as an afterthought in the development, it doesn't work correctly or it's not the best experience I could have. And if there's another game out there that did the right things and thought about online multiplayer earlier on the development cycle, I'm going to go play that game instead even if I really want to play your game."

20:36 — Angela Andrews
So is there a such thing as gamer loyalty with this proliferation of games that are available and if something doesn't float your boat, you can immediately switch to something else? Does that exist?

20:47 — Kim Huang
I think it does. There's definitely developers that have developed a good brand and they are very responsive to player feedback and they kind of anticipate correctly what players want, and they're very successful and they are not plagued with these types of problems. Thinking about netcode culturally in a different way across the board is really important for these companies to have continued success because the expectation moving forward is that more connectivity is needed, not less. We're not going to put the genie back in the lamp. It's out.

21:20 — Angela Andrews
No, we're not.

21:21 — Kim Huang
Yeah.

21:21 — Angela Andrews
I think the way that games, even the simplest game, the games that you download on your phone and you play, and I'm not talking about multiplayer games or anything like that. I'm like the mind wasting games where you can say, "You know what? Let me play X and I can play against someone else." Right? That's almost like the culture. That's in your brain. Sometimes you just do want to play by yourself, but then that little pop-up comes up like, "Would you like to challenge?" Why not? So it sounds like that is becoming the norm.

21:51 — Kim Huang
Yeah. People want to play with their friends. People want to play against each other. They want to play competitively. You have an entire corner of the market that plays video games competitively, and there's a lot of money in that. So all of these things coming together, it's a cultural change around this kind of technology, this kind of layer, really necessary layer of technology that video game developers have to understand. Well, I wanted to talk to the developers in the audience because with all of that in mind, how can a person building their own video game best figure out their needs around netcode?

22:28 — Hunter Kepley
There's so many videos out there where people explain every little thing on how to do netcode for these types of games. There are articles that are wonderful and there are even dedicated blogs that people have made for netcode. So you can read exactly how it works.

22:44 — Kim Huang
Yes. GGPO, that's one of the open source netcode solutions that's out there, there is a proprietary version of it, of course, that some major publishers and major developers use. But it's out there and all the documentation, the technical documentation is out there for free. You can go and look it up. I don't want to leave out Godot, which is an open source engine. There's a netcode that you can use for Godot. There's all kinds of different options out there, and there's tons of information out there. You don't have to have a full boutique studio with a budget to understand how netcode works and try to put it in your own little project.

23:24 — Angela Andrews
Got to love open source.

23:26 — Kim Huang
Yeah. Building a game isn't easy, whether you're doing it alone with a group of friends or with a big studio. But Hunter says, for him, it's worth the effort.

23:42 — Hunter Kepley
The reason that I love game development is that at its core, it is one of the most rewarding things because you spend weeks to months to even years on this one little project, and it could become something that is fully deviated from your original idea. But it is so satisfying to have someone download your game and play it. And even more satisfying is be able to see that other person on your screen and everything goes well.

24:07 — Kim Huang
He also wants aspiring game devs to not be intimidated by the intricacies of things like netcode. That way more people will get involved in building games for people all over the world to enjoy.

24:21 — Hunter Kepley
Don't be afraid of how difficult it may seem at first. It's just like any other thing you've done with programming in your life or even just anything at school or anything like that where it's hard to learn and wrap your head around. But once you get your head around it, it's just second nature.

24:36 — Angela Andrews
That's tech in general. Sometimes we bang our heads on problems and you have to go around and around and around, but once you get it, it's yours. So I like that little analogy right there. Thank you to Kim for bringing this to us. Thanks to Hunter for all of his insight. Let us know what you thought about this episode. Hit us up at Red Hat using the hashtag compiler podcast and tell us what you thought about it. What games are you playing? Have you noticed the netcode, is it good? Is it bad? Just tell us. We want you to share with us what you thought about this episode and, what games are you playing? We'd love to hear it.

25:22 — Kim Huang
Yeah,.definitely tell me what games you're playing right now. It's time for me to get some more in my library.

25:27 — Angela Andrews
She's looking for ideas.

25:29 — Kim Huang
Yes, yes. Hit me up on social media and tell me what you're playing. And that does it for this episode of Compiler.

25:39 — Angela Andrews
Today's episode was produced by Kim Huang with help from Johan Philippine and Caroline Creaghead. A big thank you to our guest, Hunter Kepley.

25:49 — Kim Huang
Hunter is a part of the Red Hat gaming community of practice. If you want to learn more about what they do, check out Arcade.redhat.com. Victoria Lawton is the main character, definitely not an NPC.

26:02 — Angela Andrews
Special thanks to Britt Duggan. Our theme song was composed by Mary Ancheta. Our

26:08 — Kim Huang
Audio team includes Brent Simoneaux, Leigh Day, Stephanie Wonderlick, Mike Essar, Nick Burns, Aaron Williamson, Karen King, Jared Oates, Rachel Ertel, Devin Pope, Mike Compton, Ocean Matthews, Paige Johnson, Alex Traboulsi, and the magnanimous Mira Cyril. If you like today's episode, please follow the show, rate the show, and leave a review on your platform of choice or even share it with someone. It really helps us out.

26:38 — Angela Andrews
Take care everybody. Thank you for listening. Keep on gaming.

26:42 — Kim Huang
Bye everyone.

26:43 — Angela Andrews
See you.