
Never trust the client - netinstructions
http://gafferongames.com/2016/04/25/never-trust-the-client/
======
xenadu02
"Never trust the client" is good advice for _every_ kind of application.
Clients are filthy liars.

Clients have bugs. Old clients don't upgrade. Packets arrive out of order.
ACKs never make it to clients, causing clients to repeat what the client
believes to be a failed operation. All of this is before even considering an
attacker actively trying to subvert you.

The server should always be the source of truth. It should always enforce all
the consistency rules. Database schemas can be a critical tool here as well.

Don't blindly slap whatever the client sends into a no-SQL database, then
vomit it back out for queries.

~~~
djsumdog
I saw a talk where a guy was showing how to cheat at some games. A lot of
games will detect and crash you out if they detect a debugger attached to
them.

Things like Punk Buster, et. al. are more like anti-virus software: they try
to find the signatures of known chat clients.

So it's best to write you own...and rename your debugging process to "Google
Chrome." He got pretty far with his auto-walker in some MMOs. He had limiters
so his runs wouldn't be too fast and would go up/downhill correctly...or so he
though. He forgot to multiple by -1 somewhere and the server banned him for
attempting to fly of into space.

~~~
drdaeman
More modern anti-cheat software seem to have a trend to move to kernel
space[1] and block user-mode access. So it's not just signature checking, but
some generic attack prevention as well. (Cheats also move to kernel space, of
course.)

Wonder how soon there will be a market of PCI-e cheat rigs for DMA attacks...

____

[1] Yep, gamers are literally accepting rootkits that allow arbitrary remote
code execution on their machines.

~~~
smelendez
> Yep, gamers are literally accepting rootkits that allow arbitrary remote
> code execution on their machines.

That's certainly risky and rather horrible, but what's the alternative?
Multiplayer games are a huge business and a huge culture, and all that goes
down the drain if people cheat.

Game companies have millions of people around the world competing for prestige
and money. They're running game software on their own personal computers--
machines that are also full of their financial records, naked pictures and
personal correspondence. If your game is full of cheaters, you're done. If
your game gets players hacked, you're really done.

I have no idea what the solution is, and I'm really glad it's not my job to
figure it out.

~~~
ultramancool
The solution is simple: do everything server side. Look at World of Tanks for
a great example of this. They pretty much let people run free with mods in
multiplayer and even the worst of those mods can barely provide an advantage
in online play. They do this by managing absolutely everything server side.
Ammo, direction of aim, position, enemy players appearing and disappearing
etc. Basically your server should be built to send only what's absolutely
necessary and your client should be built to use absolutely everything so
cheats can't provide much advantage rather than vice versa.

~~~
swombat
An even better solution: let the competitive advantages be given only by doing
things the computer is bad at and human beings enjoy doing.

If you get a competitive advantage from doing something like grinding, which
computers can do better and more tirelessly than humans, and humans tend to
dislike, then inevitably you will have cheating. (e.g. WoW)

If on the other hand the competitive advantage is from working out a strategy
of in what order to play your cards to maximise your chances of building up
your strength while attacking the opponent and yet keeping something in store
for surprises the opponent may spring on you - a really complex strategic task
- then that's something people enjoy doing and the computer sucks at.
(Hearthstone)

People may be addicted to the first kind of games, but is that something that
should be protected? It's kind of similar to people addicted to slot
machines... sad, a reality we have to accept, but not something we should
expend vast amounts of intelligence trying to sustain.

Of course, there are people who will do it for the money, and don't stop them,
but realise that they are basically just a few steps short of people refining
formulae for meth to make it more addictive - the fact there's a business case
that exploits a human weakness doesn't mean it's a productive use of
intelligent people's time.

~~~
cataflam
There are some things computers are really good at and that human beings also
enjoy doing. Computers are insanely fast, yet plenty of reflex-based games
exist. A lot of high-level competitive games have a dexterity dimension. FPS
games for example are a mix of strategy and dexterity, with a big emphasis on
dexterity. RTS games like Starcraft put a bigger emphasis on strategy but
still require insane finger dexterity at the highest levels. The dexterity
part of the game could be done far better by a simple AI. Aim bots in FPS or
inhumanly micro-management in RTS have been demonstrated, yet those games are
still played, and interesting.

One of the examples you gave, Hearthstone, actually had many bots running on
the ladder for a time. Their goal was grinding, but even so, a number of them
was better than a significant portion of the human playerbase. With the recent
advances in artificial intelligence, computers are not that bad at this kind
of task.

Overall, I don't disagree with your main point though. I have no qualms
automating boring and repetitive tasks in non-competitive games. And if a
competitive game has those parts, I'd probably avoid playing it. I'd rather
spend my time coding than wasting it grinding because it's part of some game I
otherwise enjoy. I feel it is indeed a more productive use of my time. I
wouldn't go as far as judging other people who like the grind though.

------
MBCook
Since this is a multi-player game this is obviously a big problem but I've
always wondered how you'd handle this in a single player game with a
leaderboard.

Let's say you have a Bejeweled clone/match-3 game and a global high score
board. What can you do to prevent fake scores? You can obviously obfuscate
things, sign requests, etc but at some point the client needs to sign the
score it's sending up so the client has the key.

You could use this model of recording inputs and playing them back on the
server but that seems like it would be a ton of work if your game is popular
and an extraordinary cost for keeping a simple score board clean.

Do you try and rely on subtle math tricks that mean that certain score numbers
are de-facto invalid because no combination of scores could end up with that
as a final total?

Is there any way to handle the issue other than deleting scores that seem
likely fake (large round numbers, for example, or those orders and orders of
magnitude higher than other scores)?

~~~
vvanders
If you have a deterministic game model(RNG seeded properly, pure functions,
etc) you can actually just send the user inputs(which are very small compared
to the whole game state) and replay the whole game to verify.

This is the common model in RTS games called Lock Step since all clients are
advancing a number of frames based on all inputs from other clients in "lock
step".

It's also very common way to implement replay and debugging for reproducing
rare network bugs.

~~~
heartbreak
Using the Bejeweled example of a single-player game, and assuming it's
properly deterministic, could a malicious actor not just falsify the steps
taken as well as the score? It would be more difficult, but not that much more
difficult considering the challenge is the time limit and lack of undo. Both
would be removed if you were simulating steps.

~~~
lobotryas
They could, but they can't falsify the game seed - the server determined order
for what gems spawn when. This way I can send actions (whether real or hacked)
that would be executed in the server against the "real game". If the client's
move did not result in a score then no score is recorded, even if the client
thinks it should.

~~~
hinkley
But it would mean a bot could get the highest possible score for that seed by
going back in time and trying again.

------
vvanders
They missed one key bit on CS's network model(which is what made it so great)
is that they would actually re-wind the gamestate to do hitchecks so you could
have a latency of 200ms+ and have a reasonable experience.

Also the common term for resolving these types of things on the client is
called "dead-reckoning". You've always got a diverging states from latency so
you're continuously trying to reconcile this on the client.

Simple Newtonian physics based games(Subspace, etc) are famous for being
latency tolerant since the simulation is incredibly deterministic and are
based on the players extrapolating where things will be in X time rather than
twitch responses(which is also why fighting games are so hard without using a
solution like Counter-Strike). You could play SubSpace on a 500ms dial-up
connection and still be competitive without needing to lead for lag.

~~~
zubspace
There's something I always wondered about the gamestate rewind thing:

Is it correct that the gamestate rewind only works for hitscan weapons
(instant hit bullet)? What about slow moving projectiles or physically
accurate bullets? Can the server spawn the projectile 200ms in the past at the
time you pressed the button? But then other clients would receive the
information about 400ms later after the initial keypress, which does not seem
right...

~~~
vvanders
The projectile issue actually is much better.

Since as a player you're trying to shoot the projectile to "predict" where it
will intersect you're actually playing to the strength of latency and dead
reckoning(that's the SubSpace case above).

You want to segregate your action into twitch(reactionary: hitscan, parry,
etc) and predictive(slow projectile, > 250ms ttl, etc) and only rewind/replay
for the twitch case.

There's also the case that when you rewind and do confirm a gamestate change
you need to gracefully handle resolving it on all clients. That's why you'd
see people warp back around corners when shot in Counter-Strike. Their player
movement speed slowed after being hit and the server reconciles it and then
the clients interpolate the result.

------
stefs
another recent example of "never trust the client" is the amazon kindle
unlimited debacle.

in case you missed it: the client syncs the last page read. so you can buy a
book, jump to the last side and amazon marks it as read, without checking for
intermediate states.

the problem is: amazon pays authors by "pages read". so people publish fake
books with 3000 pages, let sweat shops try the book for free and just jump
from the first to the last page, sync and it registers with "this customer
read all 3k pages".

now create 20 fake author accounts and 30 fake reader accounts, publish 20
fake 3k books (one for each fake author account and they're practically filled
with 3k pages of randomly scraped web content) and you got 20 _30_ 3000 =
1.800.000 pages read.

if you are an aspiring author and publish a novella with, say, 50 pages and
10.000 readers devour every single page of it, you've got 500.000 pages read.

the faker can do this in a week and take about 4 times more than, even though
it took you 2 months to write it. payouts are out of a pot shared between all
authors. thus, authors lose in the short run (less money now), cheaters win
big time (a lot money now), amazon doesn't lose money (now) because the payout
is the same.

in the long run it's still problematic because small authors are unhappy once
they realize what's afoot (and earn 1k instead of 10k). big name authors don't
care as much if they earn 1m or 1.1m. customers aren't likely to buy the fake
books anyway (cheaters take them offline before the free trial period ends) so
they're not really affected.

~~~
chris_wot
That could reasonably be considered fraud. If I was Amazon, I'd pursue them,
publicly.

------
minimaxir
The Division is an interesting game from a QA perspective. While client-side
architecture is one thing, blocking bugs are present in the game and have been
a strong deterrent to community stability.

Case in point, it was recently discovered by the community that gear with the
"Protection from Elites" bonus actually _increased damage taken_ from Elites:
[https://reddit.com/r/thedivision/comments/4g6lnk/tested_conf...](https://reddit.com/r/thedivision/comments/4g6lnk/tested_confirmed_protection_from_elites_increases/)

And that's not even getting into the loot quality issues and the complete lack
of a carrot-on-a-stick that has been codified in many, many games. (Massive
actually _made the game grindier_ because people were hitting end-game content
too fast. Which only punished people who did not hit the content yet.)

~~~
stefs
coincidentally i re-read the post about the riot games league of legends
automated testing infrastructure yesterday. it's fascinating (and honestly,
i'm even a bit envious!).

in case you haven't read it yet:
[https://engineering.riotgames.com/news/automated-testing-
lea...](https://engineering.riotgames.com/news/automated-testing-league-
legends)

guess it's down to the value of competitive multiplayer. e-sports are the
focus of games like LOL (or, i guess, CS), instead of throw-away gaming.

the division won't have international competitions in 3 years - and that's ok
with ubi soft. because they'll have a couple of new games out by then and
players will have bought those instead of still playing the division.

------
MustardTiger
I'm still constantly shocked at how many games do this. In the MMOG world it
is really easy to do everything right, UO came out in 1997 and the devs
outright told everyone working on similar games "never trust the client" and
"the client is in the hands of the enemy". Yet games like FFXI and WOW that
came out many years later have the client setting the position of the player
and the server just blindly accepting it, allowing for common and popular
speed/pos hacks.

~~~
throwaway13337
UO had rubberbanding rampant - that is, you'd move on the client, then warp
back to an old position after the server said you actually didnt move there.

It was terrible.

Most all MMOs with WSDA movement trust the client to set the player's position
and it mostly works. Cheating through warping or speed is easily detectable
through other means of post-verification and banning players that fail that
verification. It makes the game feel more fluid to the player while still not
really allowing cheating.

Everything else is generally handled serverside which is fine because it
mostly doesn't need as high a reaction time as movement to feel natural.

If speed/warp hacks are a problem, it's not because the developers that use
client-set-movement aren't able to fix it, it's because they don't care.

~~~
MustardTiger
>UO had rubberbanding rampant

Only if you had high latency and tried to move somewhere you couldn't actually
move.

>If speed/warp hacks are a problem, it's not because the developers that use
client-set-movement aren't able to fix it, it's because they don't care.

That's precisely what I said. Obviously they can fix it, since it was fixed
before they started development. They did it wrong and don't care.

~~~
daemin
Yep, I was playing it from Australia on dialup, had about a 200+ms ping. What
ended up happening is my character would be running in the wilderness, then
after a few seconds it would snap back because I hit the side of a
house/castle.

------
bdavisx
I doubt that the developer's didn't know not to trust the client - they likely
made a business decision.

I (used to a lot more) play Elite: Dangerous. A space sim that has multi-
player. They made the decision to use (mostly) p2p/client networking to save
money - they wouldn't need nearly as many servers. This has caused other
issues in addition to cheating - a low limit on the number of players in the
same "instance" of the universe for example.

But it was a business decision - imo the wrong one, but what do I matter?

~~~
mentos
Yea and I'll go out on a limb and say it was the correct one. The shelf life
of a popular game like this is probably shorter than most people think so it
makes sense to offer an amazing experience that you couldn't necessarily
achieve with an authoritative server model and then by the time hackers have
defeated the game players have probably moved on to the next.

~~~
floodyberry-
Guaranteeing your multiplayer blockbuster game designed to have "infinite
gameplay" will be dead in the water the instant hackers discover its highly
flawed networking model doesn't sound like a "correct business decision".

~~~
mentos
It was marketed to have infinite gameplay. What game do you know that honestly
has achieved that?

------
rexpop
How do we reconcile sentiments like "never trust the client" alongside this
community's hope for decentralization, á la "The Internet Has Been Stolen From
You. Take it Back Nonviolently"?

~~~
zaphar
In decentralization you are always the server and everyone else is the client.
You have ways to validate what they send to you. It works out because they all
think of themselves as the server and everyone else as the client too.

This is of course a vast simplification but the trust part is the same. No one
trusts anyone but themselves and validates everything everyone sends them.

------
morgante
Obviously, this applies to all networked applications—not just games.

It's embarrassing how often I see web apps which use something like Firebase
with absolutely no validation or access control. Often the developers behind
them don't even realize they have a problem. Their excuse is that "most users
wouldn't have any reason to hack it, so why does it matter?"

Developers need to realize their clients are inherently in the hands of
hackers. Any security needs to be done on the server if it is going to succeed
at all.

~~~
cced
Can you recommend a go-to book for this kind of stuff?

------
secoif
Ubisoft appears to have some institutional problems. Their other recent Tom
Clancy title, Rainbow Six Siege, is hands-down one of the most rewarding &
intense shooters I've ever played, however it too is yet to realise its full
potential due to amateur-hour quality issues and ongoing troubles with
hackers.

------
kristofferR
The Division also uses TCP instead of UDP so they're clearly totally
incompetent when it comes to networking.

------
wodenokoto
Does anyone have a mirror for the "this is super bad" video?

~~~
pgrote
I couldn't find it, but I did find a reddit thread with over video examples:

[https://www.reddit.com/r/thedivision/comments/4gblza/hackers...](https://www.reddit.com/r/thedivision/comments/4gblza/hackers_are_currently_making_the_dz_unplayable/)

------
nicops
I would think that in 2016 this should be something everybody knows....

~~~
milesokeefe
Yeah, isn't this basically the first thing you learn in game networking?

~~~
cortesoft
Or in any sort of networking work where you don't control both ends of a
connection.

~~~
neohaven
Even if you do, you still should never trust the client. Who knows when you
might lose control of one end?

------
outworlder
Trusting the client, in essence, turns a game into one of your childhood:
shouting matches of "I hit you!" "No you did!" "I totally did".

------
trjordan
I think there's a false dichotomy here. You can't simply ignore the client, so
no matter what, you have to have a set of server-side checks that ensure the
client's inputs are valid.

Part of what this article offers is "you can't fix it by adding server-side
checks", guessing that if they haven't done it already, it's impossible to do.
I think it's entirely possible that their model is reasonable (takes movement,
fire events, etc. instead of position, inventory, etc.), and they simply
haven't guarded against malicious inputs. Adding server-side checks is
_exactly_ what you need to do to make movement/firing inputs safe!

The existence of a certain class of bug doesn't mean the whole architecture is
broken. It may simply mean they shipped the game without trying to detect
cheaters. That may be a bad idea, but if the design is right, it's certainly
fixable.

~~~
kcbanner
I think you might be misunderstanding the scope of an "input". It's something
like "the player pressed the key/button that maps to move forward", not "move
to position X". Teleport hacks wouldn't be possible in the former case because
the player has to actually move their avatar to the desired position by
traversing the game world through a series of inputs. The server would then
process the movement inputs, and the avatar would say, bump into a wall,
preventing it's movement. There would be no input for "my position is now X".

~~~
trjordan
Is an input always valid? "Firing once" is an input: you have to validate that
it's been long enough since last firing that you should act on it (limit the
rate). "Moving left" is an input, but by how much? Can I send "Left, 9999999"
and replicate a teleport hack by pre-calculating these inputs?

You could bound the inputs to a simple stream of enums (FIRE, LEFT, RIGHT,
etc.), but it wouldn't be crazy to batch it up a little bit and send some
magnitude data as well (FIRE 2, LEFT 50, etc.). It's a little harder to
validate, but depending on the specifics of implementation, that could be a
win.

~~~
Ogre
"Moving left" is NOT an input. An input is "The controller is pushed to the
left" (with perhaps a numeric value of how far it is pushed to the left).
"Firing once" is similarly NOT an input. "The fire button is down" and "The
fire button is up" are inputs. That is how well implemented first person
shooters have worked since the Quake days (I think Quake 2 actually).

The point of the article is actually that the server doesn't validate
anything. There's nothing to validate. The only data it gets from the client
is the state of the controls. Then it runs the real simulation. The client
only ran a prediction of what it thought the server would do. Sometimes, it's
wrong and corrects itself when it gets new values from the server.

------
primigenus
The server-side network model he describes here is the same architecture
Meteor is designed with (see this page: [https://www.meteor.com/why-
meteor/features](https://www.meteor.com/why-meteor/features)). With Meteor,
you get client side prediction and latency compensation (they call it
"optimistic UI" now) for free. I've always been impressed they decided to
build that, because I sure never would have myself. In fact the Meteor team
has always said you need this kind of architecture in order to build true
real-time applications. But I haven't seen other web-oriented platforms take a
similar approach. Did the Meteor team just know something no one else has
picked up on (despite it apparently being common practice in the games
industry)? What gives?

~~~
emerongi
Even though I don't use Meteor anymore, I still like DDP[1] a lot. It's simple
(can be implemented in 50-100 lines), but it is the core of Meteor. There's a
specification for it in their Github repo. Meteor's "optimistic UI" is
basically an abstraction on top DDP, IMO.

[1] [https://www.meteor.com/ddp](https://www.meteor.com/ddp)

------
majewsky
One can easily observe this "server is the actual game" property in Minecraft
when running on an underpowered server.

At home, Minecraft runs on my home server (with a small Athlon CPU) and it
fails to keep up with the required speed of the game, so the server log will
show messages like "server clock running behind, skipping 58 ticks" every so
often.

An observable effect of this is that you have to hold the right mouse button
just a little bit longer than the actual animation when eating stuff. And when
you look at the sun, you will see it moving forward continuously (as
calculated by the client), but skipping back a tiny bit every few seconds
(when the client adjusts for the server's time drift).

------
notacoward
I'll bet there was at least one developer who saw they were doing it wrong,
and told them, and was overruled. I'd love to hear his/her account.

~~~
majewsky
That type of story grows old after the 100th time you hear it.

~~~
notacoward
Also, get off my lawn.

------
awalton
Console port gone horribly wrong? I mean, how else do you violate the most
simple of game rules - never trust the client?

~~~
minimaxir
The irony is that the console versions are the least-impacted, as the client-
side hacks are only loadable on the PC version.

~~~
csours
Walton's phrasing is kind of ambiguous, Console port could mean port from
console or port to console.

It could have been developed for console first.

~~~
awalton
"Console port" in the gaming world unambiguously means "a game that was
originally developed for a console, then ported to the PC." The opposite would
be "PC port."

------
partycoder
If you don't know this you should not be working on software. It's a latent
risk to customers.

~~~
rhinoceraptor
It's a video game, not a life critical system.

~~~
partycoder
A game can include a payment system, or processing of personally identifiable
information. Security is important.

------
forgotmypassw
I thought it was a common sense to treat the client as merely the data
representation and nothing else, sad to see that multi million dollar
companies make rookie mistakes.

~~~
abraae
I guess in the eyes of a corporate, it's only a mistake if it harms the bottom
line.

Sounds like this would though, if players can't enjoy a fair experience.

------
andersonmvd
I'm surprised that "never trust the client" isn't among the basic concepts
that everyone should know, at least on HN. It's such a basic premise.

~~~
Aelinsaar
If you ignore the context, "Never trust the client/patient/enduser/etc..." is
probably equally true.

------
stefs
now that i think about it - doesn't GTA5 suffer from the exact same problem?
cheaters spawing tanks above people practically must be a client game state
problem.

~~~
oridecon
Yes and that is why I think it's a new business model to make games not last
long on the PC. At least the multiplayer part.

They promote the shit out of games before release and expect them to expire a
few weeks after. Reducing operational costs. They also don't have to develop
and maintain an anti-cheat, a better server-side structure, and they can close
down most servers.

I think it was a unexpected consequence of a bad console port to pc that
proved to be quite lucrative. And now they know people will keep buying no
matter what, hence the new disposable pc game business model.

Basically, if you play on a PC you're screwed. Another reason not to buy a
console or support games like the division on the pc.

------
andrewclunn
I was not interested in this game, but now knowing that it can be hacked in
cool ways makes me interested. I only care about playing with friends, and
this would allow for some super cool custom game modes. It's not a bug it's a
feature!

