
Is it just me or is networking really hard? - rinesh
http://gafferongames.com/2015/09/12/is-it-just-me-or-is-networking-really-hard/
======
chetanahuja
_" You can’t even imagine a way that reliability could be implemented on top
of UDP that beats TCP? What total bullshit"_

This. This "TCP already does it best in every situation" is a common trope
usually spouted by people who know nothing about what TCP does and why. The
easiest way to get over this misconception is to ask yourself, can you think
of a protocol that not only works _well_ over all sorts of networks, all the
way from two boxes connected by a 10Gig switch (with 10us ping latency) vs two
hosts talking to each other via a satellite link (with perhaps, 2 _second_
ping latency) but is the _best_ option across all those situations?

The truth is that TCP is the common denominator that's available in pretty
much every device you would want to use. But in almost every _individual_ use
case, you could come up with a far better protocol if you took the time and
effort to treat the problem seriously.

~~~
nubb
UDP is used when it's designed by network engineers. That's why we have VOIP
and video streaming - all are served UDP.

dont get me started on the lack of multicast out there. imagine game servers
running multicast. one day.

~~~
perlgeek
> UDP is used when it's designed by network engineers.

UDP is used when low latency and low jitter is more important than complete
information.

For telefony it's better to lose a few packets than to wait a second or two
for a retransmit. Some for game state, I believe.

And I do believe that TCP was also designed by network engineers :-)

~~~
Retric
I really depends on the game, if your writing a chess game then TCP is a much
better choice than UDP. Though for anything really complex your probably going
to want to use both.

------
babuskov
I'm thinking to add online multiplayer to my game:
[http://store.steampowered.com/app/363670](http://store.steampowered.com/app/363670)

...so this was very interesting read. I investigated a little bit, and found
that there are more protocols than just pure UDP, and now I can't decide which
one to try. I narrowed it down to these three:

* PCC [http://modong.github.io/pcc-page/](http://modong.github.io/pcc-page/)

* NORM [http://www.nrl.navy.mil/itd/ncs/products/norm](http://www.nrl.navy.mil/itd/ncs/products/norm)

* ENet over UDP [http://enet.bespin.org/](http://enet.bespin.org/)

What do you think?

~~~
Mithaldu
Since your players don't seem to be affecting each other a lot, you might even
be able to go with simple TCP. UDP is needed when latency matters a LOT, which
is mainly when people fight one another.

~~~
TazeTSchnitzel
I believe World of Warcraft and some other MMORPGs use TCP, since latency
isn't too big of a deal. Note that they do some things to compensate for this,
however, like making players not collide with each other.

------
OliverJones
I sometimes ask the question "how does the internet work?" in interviews. The
vague and wrong answers I hear are astonishing. I'm looking for a basic
understanding of the layering of TCP, or RDP, or ICMP (or anything actually)
on top of IP. I don't get that level of insight much. The people who do have
that insight usually can go deeper -- slow-start, BGP, etc.

I wonder if we have a modern version of C.P. Snow's mid-twentieth-century two
cultures lament.
[https://en.wikipedia.org/wiki/The_Two_Cultures](https://en.wikipedia.org/wiki/The_Two_Cultures).
That's the complaint that educated humanities folks don't know the basics of
science.

Are we miseducating our so-called "full-stack" engineers by not offering them
a basic understanding of telecom? Or are they not listening?

~~~
akerl_
This goes on the list of interview questions that's great at letting the
interviewer claim pretty much any candidate they want to discard is unworthy
of the position. It's the tech community's variant of a voting literacy test.

If you ask me "how does the internet work" and what you really want is details
on the underlying protocols, at around the range of TCP/IP, why aren't you
actually asking for that? Your question is so broad that it's not shocking in
the slightest that most people pick a different focal point for their answer,
especially given the already hostile interview environment and the added
agitation provided by such an unclear question.

If your goal is to elicit and understanding of somebody's background in TCP/IP
and the surrounding protocols, ask them a question about that. If that's not
relevant for the position (it's almost certainly not for most "full-stack"
engineers), switch to asking questions that are relevant to the work they'll
actually be doing.

~~~
mikehollinger
this.

I'll chime in as someone with a skill set that is completely not special
(linux application development, and hardware / firmware / software
integration), but in certain circles, is unique since it doesn't involve
JavaScript (yet).

I think the question is too vague, but I've asked a similar question:

To kernel hackers (not application developers) - Tell me what you know about
the Linux boot process

To anyone with a 5th-gen language on their resume: how's a hash table
implemented? (use of a feature or function without understanding why and how
it works can lead to really interesting performance problems, even if the code
is functionally correct)

~~~
akerl_
I'd argue that the cases where a hash table's performance characteristics come
into play are few and far between for most developers, and when they do, it's
way more relevant that the person have a general idea how to profile perf in
their code and say "yup, this is the spot that I need to worry about
optimizing first".

By marking down people who don't know how to implement a hash table, you're
cutting out a large chunk of your candidate pool, and I'm willing to stake my
hiring choices on the fact that a decent chunk of them could easily by taught
that knowledge if it became relevant for their career on your team.

~~~
davnicwil
I did an interview once where they asked me questions around the
implementation of a hashmap. At the time I'd been working about a year out of
university, and had not in that job come across any situation where I'd needed
to know anything about how hashmaps worked under the hood (if anyone had a
first job where you did, I'm happy for you, yours was a lot more interesting
than mine). Any remnant of an explanation I'd got from my CS degree, most
likely in first year, was way back in the mists of time. In the pressured
situation, I simply admitted I didn't know.

At this point the interviewer's face soured, and he basically wrapped up the
interview straight away - it had been running about 5 minutes at that point,
with only one question before that, so it was pretty obviously being cut
short.

Immediately after the interview I went to Wikipedia to look up hashmaps,
instantly understood the basic mechanics, and 5 minutes later could have given
a stellar answer to that question.

If that interviewer was looking for someone with an encyclopaedic knowledge of
data structures, with flawless recall under pressure, then his question served
him well in filtering me out. I sincerely doubt that's what he was looking
for, though.

Since that experience I've been very careful, doing interviews myself, never
to ask straight-up knowledge questions like this - or at least if I veer into
a question like that, allow the candidate to explore the idea (even tell them
the answer, and just talk about it to see if they get it) if they initially
don't know. That will get you a lot closer to what you're actually trying to
assess them for.

It's very easy to fall into a trap of interviewing with a 'mirror' approach,
where you assume that the candidate must know what you know, or else be bad.
Kind of a paint by numbers, where the finished product looks a lot like you.
But everyone learns on a different path. You need to remember that there was a
time where each concept you use take for granted every day in your job was
once new and alien to you, too. You picked it up just fine, and so can they.
In fact, this is by far the most core skill you are looking for - that ability
to actually pick up new things and use them.

~~~
Gibbon1
The comment about mirror approach to interviewing reminds me of the first
place I worked. At one point we needed to hire someone to do something outside
the usual skill set. The CTO interviewed a bunch of people and hired the one
that knew slightly less than he did. What O got was the ones that knew what
they were doing, disagreed with the CTO on some issues, and so were out of the
running.

------
drxzcl
I know this attitude all too well. I've replied to a few questions on
gamedev.stackexchange with pointers to the usual articles on game state
synchronization and lag compensation, and I get lukewarm reactions of "too
complicated" type. Meanwhile mr rpc-over-tcp get tens of up votes and accepted
answer.

Their loss I suppose.

~~~
zamalek
> Their loss I suppose.

Our loss, really: less good quality games for everyone.

I threw in the towel at GameDev for reasons very similar to yours ("premature
optimization" also comes up far too frequently) - from, what can only be,
people who have never even worked on specific components of games that they
are claiming knowledge on.

The StackOverflow system somewhat fails on GameDev, because there aren't
enough knowledgeable people on the site to issue corrective votes (likely
because these people are scarce in general).

Incidentally the bad advice is bad on many levels: even if you somehow ignore
UDP it's _still_ bad advice to do "RPC over TCP". TCP works best (fastest,
least latency) when the send buffers are full at both ends - which RPC, by
definition, does not do. This means that these people simply couldn't have
researched the fundamentals of the protocol that they assume to be "good
enough," how could they therefore be aware of how it is better or worse than
alternatives?

------
Arnt
Networking has local time, like the relativity theory. "Those two packets are
at the same time at point a, but not point b." "A sends b and C sends d, but
since time is local you cannot" etc.

So IMO it makes sense that networking is really hard. I like it though.
Networking problems are clear when you think hard about them, unlike so many
of today's programming problems, where the key is to know some half-documented
aspect of a large framework/library/whatnot.

------
hebdo
Well, technically speaking any multiplayer game is a distributed database,
except that more inconsistency is tolerated ("lag"). But all of the classic
results, such as CAP or FLP, apply here as well. And given how hard even the
simple distributed protocols are (eg. Paxos) it should not be a surprise that
writing a good networking stack for a multiplayer game is also hard.

------
staunch
> _We have the Tribes Networking Model, The Unreal Networking Model, Valve
> Latency Compensation article._

Which are all derivatives of the Quake model.

~~~
Mithaldu
Can you point out where the Quake model is described in sufficient detail that
one can reimplement it for one's own game? Or is it open source?

~~~
tinco
Perhaps if you're going for simplicity or something then that might be an ok
idea, but he mentions those three because they are more modern or important
additions to the Quake model. He is of course very aware of their Quake
heritage, at the end of this article he credits John Carmack (in an indirect
way).

I recommend reading the rest of Gaffer's articles before doing anything, it
will give you a basis that's needed to understand why Quake does what it does.

------
jokoon
TCP is suitable for "flat" static content, which you only retrieve at punctual
intervals, and is relatively large. A good example is a file which won't
change for the next 5 min, and you know that you want that exact file BUT you
are ready to wait until it loads perfectly.

UDP is for real time, continuous data, when you can't wait for your hardware
to check for packet consistency.

A good example is paper mail versus a telephone call. The paper mail will
land, but you don't care if your phone line cuts for 1 second. You just wait
for the other guy to check if he can hear you

UDP is really much better when you want to have performance, especially when
it comes to responsiveness and latency, but can afford to discard packets if
they're not reliable.

I think it boils down to people not realizing how hard it is to transmit data
over long distances, and that the protocol don't consider that data
transmission is 100% good, because it never is. You can have a lot of
interference in your network, but TCP will always manage to land those
packets.

------
toolslive
I think the topics should be split: you have 'networking' which is about
connecting computers typically using TCP, UDP, RDMA, ... and 'distributed
systems' which is about how to use connected computers to achieve something.
And yes, both are difficult, but for other reasons:

\- networking is hard because the APIs are old, platform specific and probably
don't behave very intuitively. So trying to build messaging systems on top of
sockets is not simple, (but you will only find out after you've moved beyond
the prototype stage).

\- distributed systems is hard because the odds of successfully exchanging
messages aren't in your favour.

------
Animats
It's not that networking is really hard, it's that development of synchronized
games that run over unreliable links is really hard.

If you want a simple solution for a simple game, here's one. Use a fixed-
format UDP packet to pass current location and orientation of the player and
whatever else changes rapidly. This should be stateless; if you lose a packet,
the next one has a full state update. This is enough for a simple FPS game.

Anything else goes over one or more TCP connections. If you have to load
assets (level maps, textures, etc.) just use HTTP over TCP, which means you
get to use standard servers and client software for that stuff.

If your game is too complex for that model, you're probably going to have to
do some serious thinking about distributed synchronization.

~~~
jstimpfle
> This should be stateless; if you lose a packet, the next one has a full
> state update. This is enough for a simple FPS game.

Also enough considering packet reordering? And corruption?

~~~
callesgg
the next packet should come within a few milliseconds so yes.

but I see your point adding a simple counter might be somewhat useful ignore
any packet that has a Lower counter value.

coruption there is generaly enogh checksums in earlier layers so that it is
neglisheble.

------
ninjakeyboard
You know what I found that is actually quite amazing: Video encoders for
streams of linear/live video feed for broadcast go over UDP to the next device
in the chain. An encoder will often output a stream of video over UDP to
devices that will then split the video into multiple renditions.

------
zurn
The single problem with TCP this rightly points out is head-of-line blocking.
But overcoming that still leaves you with the problem of coping with out of
order or missing events at app level, with clients and servers seeing events
in differing sequences (fex packet sequence of move-duck-fire, now shuffle the
move around...). I witnessed one game engine project implement custom
networking on UDP and then disable unreliable & out of order messages because
of game logic headaches.

And the payoff is quite small, since packet loss is rare in healthy networks
and TCP handles the rare loss pretty efficiently based on ack clocking, not
timeouts (fast retransmit, basically same as his idea of "redundantly sending
un-acked data").

~~~
jschwartzi
How many networks are actually healthy, though? Networked gaming is less about
a hardwired LAN and more about reliably synchronizing state using your crap
wifi in a oversaturated band when Comcast hasn't repaired the infrastructure I
your neighborhood in years.

In other words, a healthy network is the edge case.

~~~
zurn
Sure there are people with rotten last mile conditions, but healthy networks
are the normal case. You get subpar download speeds with http and choppy
skype/facetime/ssh if your network drops packets frequently. It's bad for
business from the ISP POV. Using UDP will not work around the conditions you
describe, game will still be laggy and choppy.

~~~
Dylan16807
Downloads will work just fine if you're dropping .1% of packets. Skype is very
resilient to dropped packets. Console use isn't quite so fast that a half-
second hiccup is a big problem.

But a naive game implementation over TCP will stutter badly and obviously
every time.

~~~
zurn
Yep, and that's different from the hopelessly choppy last mile. The question
here is if there's a big difference between a good (not naive) implementation
over TCP vs various DIY UDP schemes, and whether that difference justifies the
big complexity increases in game design due to trying to cope with out of
order/lost events.

~~~
Dylan16807
You can only do so much over TCP. Unless you are willing to put in a
significant delay, dropped packets will cause problems. You can hide them, but
there will still be shots that miss, enemies that were never where they
displayed, etc.

------
ilurk
I know the article is language agnostic but I don't think anyone would develop
their networking modules (in the context of videogames or finance) in anything
but C++ (or Erlang/Elixir?).

So could the local experts give their opinion on C++ networking libraries? I
know there are a few [1] but how do they compare?

I've only used Qt networking and POSIX sockets (although for C), but the
application didn't have much of latency or throughput requirements.

[1] [http://stackoverflow.com/questions/118945/best-c-c-
network-l...](http://stackoverflow.com/questions/118945/best-c-c-network-
library)

~~~
Const-me
You shouldn’t just compare C++ libraries. First, define your requirements,
both functional (what should your networking do) and technical (latency,
throughput, available bandwidth, operating systems of both clients and
servers). Second, pick a protocols stack. Only then, look for those C++
libraries.

If on step #2 you’ll choose e.g. ProtoBuff + UDP, you’ll have totally
different set of available libraries than if you choose SOAP + HTTPS. Besides,
very few of those libraries work well on multiple platforms, e.g. libuv does,
while libevent does not.

In addition, C++ is not the only choice. I once developed networking module
for a game in C#. Have a few friends who are doing the same in finance
industry.

------
Const-me
Not every game is critical to network latency. Not every game is tolerant to
packets loss. If those 4-8 players in the game are playing 3-dimension virtual
reality poker, TCP or even HTTP will work just fine.

I have read the original thread, and the author says somewhere in the comment,
“it is likely to be similar to that you would see in something like Diablo II.
Players exist in a world, attack enemies and interact with NPCs (including
trading).”

Which makes the article irrelevant to the original question. Games like Diablo
2 are not real-time; they can usually use TCP just fine.

------
Schwolop
The best part of this article is the two comments at the end. The first is by
the person who posted the Reddit question that prompted the entire rant,
describing how asking them question made them rethink everything, and how they
ultimately settled on a UDP design just like the one promoted in this article.
The second is by the article's author: "Bravo and well done. I’m quite
impressed".

------
vans
What's the point of this insulting article full of rage ? nothing to do at
this ranking on HN.

------
nubb
As a network engineer, this was great. No body never bothers to learn
networking.

The article is annoying tho, basically says use UDP idiots. I mean, why do we
think VOIP uses UDP? Same concept.

If only more devs would implement multicast...

~~~
mahyarm
Isn't the problem with multicast is most networks filtering it out or not
supporting it? Thus it only stays useful for the LAN network scope? So I
cannot just multicast to 5000 random IPs on the internet.

~~~
yusyusyus
Not supporting it because why though? IP multicast routing requires end-to-end
state for each (source , group). Further, the boxes on the market for doing
multicast (not a little GPCPU router) have limitations concerning how they do
multicast within the box that makes scaling multicast a real pain (basically,
internally, they will send forwarding engines traffic they don't need and
choke up there because it runs out of internal ways to express the separate
streams).

Some of this can be shortcut by tunneling techniques, but it is just masking
the situation and adding another layer of network complexity (in an area where
humans still try to manage/troubleshoot mostly by hand).

------
zobzu
netcode is the most difficult part of fast multiplier games. then again one
could just read quake3 source - its actually, 16 years later, still one of the
best for that.

------
tormeh
>You have whole industries (gaming, video streaming, VoIP) that will laugh at
you if you send time critical data over TCP and rightly so

Skype uses TCP, no? Laugh, indeed.

~~~
Too
This might finally explain why the video quality is so bad then. If you tried
any commercial video call software it's quite hard to look back to skype.

~~~
tormeh
Google Hangouts isn't much better. I would assume it's more about the
constraints of free services than anything else.

If you're talking about systems like Cisco Telepresence, then those are
obviously not comparable because of (lots of) dedicated hardware, better
connections etc.

------
mmaunder
I guess that's one way to package up a fairly fundamental piece of networking
knowledge. Wrap the fucker in expletives to get attention.

------
known
[http://www.kegel.com/c10k.html](http://www.kegel.com/c10k.html)

------
ggambetta
I've written a series of articles about this topic:
[http://gabrielgambetta.com/fast_paced_multiplayer.html](http://gabrielgambetta.com/fast_paced_multiplayer.html)
It's usually well received, perhaps because it builds the ideas from the
ground up and does _not_ include the words _" What the fuck is wrong with
you?"_.

~~~
exDM69
To be fair, Gaffer's original articles are well written, thoughtful and
informative too.

This is him ranting about the unfounded, negative and incorrect feedback he's
been receiving about them from people who are willfully ignorant about
networking. He could or should have replied with less attitude and more facts,
though. I understood what and why he's saying what he is, but I didn't enjoy
the negativity either.

~~~
ggambetta
His articles are _excellent_ , and I've learned a lot from them myself. But I
cannot sympathise with lowering the level of a technical discussion to
personal insults.

~~~
angersock
I don't know if this is a general failing of people's critical thinking
abilities, a sloppiness in their reading, or something else, but I _have_
noticed over the past (feels like) few years a growing problem:

A lot of folks conflate personal insults ("you, ggambetta, are clearly a
dunderhead when it comes to reading comments") with general sentiments about a
group or abstract population ("Hacker News posters tend to be overly sensitive
to the point of being crybabies").

A great deal of the flavor and comedy in writing, especially pieces that are
self-identified rants, are _not_ from personal insults but instead from
hyperbolic and sometime vitriolic statements about an abstract other.

~~~
gafferongames
Spot on and this was firmly written with tongue in cheek! It's funny because
it's true though. I harbor no ill will to anyone. Also, a nice ending to this
whole thing, the original poster found the article, and halfway through
reading it, realized it was about them and posted a comment!

------
mahouse
Reading this one could think you would be a fool to use TCP, yet some games
(such as World of Warcraft) use TCP exclusively with no problems... What
gives?

~~~
exDM69
World of Warcraft is a pretty slow-paced game on "the inside", it just looks
like a fast real time game but it's almost turn based when you analyze the
gameplay. When you cast a spell, you'll instantly see the animation that your
character is casting the spell, but the actual effect comes only when the
animation is complete. This animation is used to hide the network latency
(other players might see the animation played back slightly faster to
compensate for the lag). Further, once you cast the spell, it must succeed and
you can't really do anything before it has completed. In other words,
reliability is essential and latency can be hidden.

Contrast this with a fast paced, (soft) real time multiplayer game. When a
player jumps, the jump has to start immediately. If you want to jump and
shoot, the shots have to be fired right away, you can't wait for the jump to
complete before shots are fired. With TCP, you'd be stuck on the ground until
the "jump" packet is re-transmitted (two or more network round trips, tens to
hundreds of milliseconds, very noticeable) and no shots would be fired until
the character is off the ground.

In a real time game, old packets are next to useless. Re-transmitting is
wasting bandwidth and causing lag by blocking on information that is no longer
useful. The networking model (like OP describes) is a constant stream of
packets containing redundant information, minimum latency is essential and
loss of packets is tolerated.

You should be aware that TCP vs. UDP becomes apparent only when network
conditions are bad. You could choose either and have satisfactory results 90%
of the time, but once packets get lost, TCP does the _wrong_ thing when it
comes to fast paced real time games.

If you do take the time to read Gaffer's original article, you should see that
it is different from TCP in many ways. Yes it does similar things
(reliability, flow control, etc) but in a completely different manner, tuned
for a completely different use case.

------
hmans
Someone needs a bit of anger management therapy.

