
Building multiplayer games with Socket.io and HTML5 Canvas - gfysfm
https://github.com/sgoedecke/socket-io-game/blob/master/BLOG.md
======
synthmeat
I'm doing similar, it's so much fun, and here are my takeaways:

\- Ignore socket.io, do pure binary websocket

\- _Do not_ serialize JSON, or use msgpack or protobuf - write your own
protocol.

\- Write tests for protocol, you need fast iteration. It's super easy to write
tests for codec.

\- Quadtrees are kinda lousy compared to r-trees.

\- Measure everything with realistic load - i.e. I've found out that it's more
costly to calculate world entity deltas to send to client, than to send
everything in his view.

\- Keep client updates under MTU payload. This is basically impossible with
JSON.

\- Mutability is the way to go for performance.

\- Share code with backend and frontend, you'll need it for client prediction.

\- Prefer fixed entity speeds.

\- Don't log per-frame in production, journalctl will eat up your memory.

\- Write game bots to test your load non-synthetically.

\- Test on your VPS, you'll be surprised how underperforming CPUs are in most
providers. Neighbours are probably serving some CRUD and are very likely not
to be as noisy as you are.

Now, all this might not apply depending on exact gameplay mechanics you have,
but it works for me. I have Node.js serving 200 clients and 10k entities with
vision (can run at or from you) at 20 frames per second that keeps under 100MB
RAM with no hiccups. I think I could use something like Golang for some more
advanced physics though.

A question to all reading - I'm considering adding an option to mine Monero in
my next game. It'd probably be a flip-switch between ads and mining right
there on the home screen, with some intelligent throttle/thread decisions
depending on the kind of device game is running on. I'd appreciate any
feedback regarding the sentiment on that.

~~~
CJefferson
Are you sure about no JSON? My experience is JSON is fairly small, and faster
than anything else because it has browser support.

~~~
synthmeat
For my use case, replacing JSON with custom binary protocol, some old version
being here [1], had savings of about 2 orders of magnitude in bandwidth and at
least 1 order of magnitude improvements in server CPU usage. Some significant
memory savings too. There's also space for more optimizations to be done, but
I consider it viable enough for now.

ArrayBuffer and DataView are fairly fast for my standards.

[1]
[https://gist.github.com/synthmeat/d23595db732d366f658a1981e9...](https://gist.github.com/synthmeat/d23595db732d366f658a1981e92429b1)

------
coldcode
Having worked on a commercial networked first person shooter, I can verify
that making it smooth and coordinated is a massive pain in the ass. Having a
game like World Of Tanks where the server makes all the decisions only works
if the game entities are big and don't change much (i.e. tanks). There is a
reason why none of the World games have people. Nothing is harder than trying
to make a knife fight work over the internet without it looking spastic.

~~~
arca_vorago
I'm working on a project I hope to make multiplayer eventually, but the issues
surrounding it scares me a bit. I wish more companies doing the hard
multiplayer work (fps's mostly) would reveal more of their back end and how
they do things.

I don't think anyone has really done it well yet but everyone is hiding what
they have figured out so far. I have noticed that certain engines tend to have
better success at this, for example source engine seems to have drastically
improved, allowing games working on this issue like Insurgency to make some
noticeable strides towards being better.

This has me remembering UT over dialup lag...

~~~
andai
This is a great writeup by VALVE on the subject:
[https://developer.valvesoftware.com/wiki/Source_Multiplayer_...](https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking)

------
oboopfmlrmnmn
Just wondering, why are people still using socket.io and not standard
websockets, as available in every browser since several years now?

~~~
XCSme
If you don't mind performance, using socket.io gives you stay-alive on the ws
connection. If you don't mind writing the 5 lines of code it takes to
implement stay-alive for normal websockets then socket.io makes no sense
anymore.

------
bsaul
In case you haven't reached that point yet : also evaluate how much network
badnwidth is going to cost you if you ever host your servers on aws or gcloud
and encounter average success. You may be in for a (bad) surprise.

Now compare that to how much you'll gain from ads. And soon you'll realize one
huge issue you're going to have is to maintain your own servers a little bit
everywhere in the world.

~~~
synthmeat
I've carefully measured and worked toward 1TB/month/VPS at full load. Didn't
get there, but now I'm at 2TB (still estimates). With socket.io and JSON,
that'd easily blow up to few hundreds TB/month. DigitalOcean for now (but that
might change in the future) still doesn't charge on overages, but you do get a
slap on the wrist of some kind if you go way overboard, so it's preferable
solution to other VPS providers.

It's good that you mention this, since I've heard "horror" stories of people
working in this space earning about $10k/mo but spending as much and more on
AWS.

~~~
nickrio
Wow, I'm not a game dev pro. But 200 clients + 10K entities could cause that
amount of traffic?

~~~
synthmeat
Yeah, I've choked one office network on a client project (they went through
VPN though) with bot tests. :D

But single-digit TB monthly outbound per VPS was not that surprising to me.
Almost triple-digit one in naive implementation, scared the hell out of me
first time around. Spent hours double and triple-checking and measuring. :D

------
neillyons
This is a great write up. I've had similar experiences, and agree, have the
server-side game state be the single source of truth.

I basically stuck some visuals on top of the example "counting" snippet on the
Elm site ([http://elm-lang.org/examples/buttons](http://elm-
lang.org/examples/buttons)) and made it multiplayer. First player to count to
50 wins.

I didn't use a tick approach, instead the state of the game was only updated
on an event sent from one of the clients. I then broadcast the new state of
the game to the other clients.

This is what I made
[http://retrorace.neillyons.io/](http://retrorace.neillyons.io/)
[https://github.com/nwjlyons/retrorace](https://github.com/nwjlyons/retrorace).
Something is not quite right in the backend code. I think there are problems
with waiting for the mutex to unlock. The game feels like it freezes
sometimes, but it could also just be a latency issue. The server is in London
and I've only tested it with people who are several hundred miles away.

I also had plans to build multiplayer snake but felt like latency would make
the game too slow.

~~~
synthmeat
I try to squeeze a little Go side-wise, so I'll be sure to check this out in
the future. Thanks.

------
baffo
If you have problems with lag and conflicts resolution of state, take a look
at this article, it may be useful. [https://0fps.net/2014/02/17/replication-
in-networked-games-l...](https://0fps.net/2014/02/17/replication-in-networked-
games-latency-part-2/)

------
dkersten
On the topic of multiplayer HTML5 games, I've recently started playing with
Lance[1] and its pretty good. Its main draw is that it largely does the
networking and latency compensation (it has options for interpolation and
extrapolation) for you. The only requirement is that you write your game
(client and server) in javascript and run the server on node, so that the
gameplay logic can be run both in server and client (game is simulated both on
client and server, but the clients are periodically synced with the server,
using interpolation to make it look smooth).

[1] [http://lance.gg](http://lance.gg)

------
zubspace
If you are writing a game server with web, mobile and desktop clients, is
socket.io the way to go?

AFAIK, socket.io is a wrapper around websockets. This makes it unsuitable for
some fast paced games with alot of players. Is there a library which abstracts
away tcp and udp connections?

And what can you do about tcp attack vectors? Are lamers trying to take down
servers a real threat? How do you prepare for this? I therefore advise against
writing your own network stack if you intend to make the service available to
the public. Or am I too cautious?

~~~
gfysfm
As for udp connections, WebRTC looks like the only mature solution (but it's
not designed for games and is way more complicated than game devs want it to
be). Alternatively, I know about
[https://github.com/networkprotocol/netcode.io](https://github.com/networkprotocol/netcode.io)
but it's very much in the early stages - afaik you even need a Chrome
extension to use it on the client. It's not easy to do udp on the web.

~~~
lxtx
An "UDP" server for browser clients:
[https://github.com/seemk/WebUDP](https://github.com/seemk/WebUDP)

~~~
emj
I get 60ms RTT for the largest bin of packets with WebUDP many requests goes
as high as 140ms (positive skew of the distribution), while ICMP never goes
above 29ms RTT. I know UDP can be badly prioritized in some networks but I
don't think it can only be that...

This needs more investigations imho, I wonder where this fails so badly; is
WebRTC aiming at ~100ms latency?

~~~
lxtx
Definitely a bit weird. Where are you located? The echo demo server is hosted
in the Netherlands. That's how the round trip histogram looks for me in
Estonia: [https://i.imgur.com/iBoqwe4.png](https://i.imgur.com/iBoqwe4.png)
I'll take a look later if there's any noticeable delays caused by the server.

~~~
emj
I left it running for some hours and got alot more 30ms responses:

[https://imgur.com/a/SdWtc](https://imgur.com/a/SdWtc)

~~~
emj
Chrome use ~100% cpu transceiving ~83 packets per second, and I think it sends
packets at a slower rate later on, that's why we get so many 30ms responses
when we wait some hours.

This is a tcpdump after loading the page:

    
    
      \time sudo tcpdump -c 1000 -ni eth0 udp and port 9777
      1000 packets captured
      1000 packets received by filter
      0 packets dropped by kernel
      0.02user 0.05system 0:12.45elapsed 0%CPU (0avgtext+0avgdata 5472maxresident)k
      0inputs+8outputs (0major+1508minor)pagefaults 0swaps
    
    

This is just a flood ping, interesting that the UDP version shoots more
packets.

    
    
      \time sudo ping -f -c 1000 188.166.46.128
      PING 188.166.46.128 (188.166.46.128) 56(84) bytes of data.
      ....    
      --- 188.166.46.128 ping statistics ---
      1000 packets transmitted, 996 received, 0% packet loss, time 14195ms
      rtt min/avg/max/mdev = 28.839/38.641/50.433/5.152 ms, pipe 5, ipg/ewma 14.209/41.787 ms
      0.02user 0.13system 0:14.31elapsed 1%CPU (0avgtext+0avgdata 2248maxresident)k
      0inputs+8outputs (0major+1183minor)pagefaults 0swaps

------
bullen
[https://github.com/tinspin/fuse](https://github.com/tinspin/fuse)

TLDR: Websockets are overengineered and waste CPU cycles doing nothing but
marshalling. HTML is better than canvas because it can be GPU accelerated and
you get components and z-order for free.

------
Entangled
Any not so expensive hosting options for sockets-based apps? Heroku? Digital
ocean? AWS?

Also, what are the limitations by the hosting provider regarding concurrent
connections, data transfer, etc.

I am currently working on a poker app using free Heroku dynos but would like
to have a budget in mind for when going live.

~~~
synthmeat
On DigitalOcean, I'm planning a following setup to launch:

1\. $20 staging at all times, already live [1]

2\. $20 static nginx

3\. 3x$5 for socket servers on North America / Europe / Asia

4\. $5 master server with WS connections to full mesh of socket servers /
orchestration / future.

All Debian.

Have no clue what's Heroku like for that, but last time I've checked, it
wasn't as good choice as DO was and is for me.

[1] WARNING: might break at any time, by choice or not
[http://staging.munchies.io](http://staging.munchies.io)

~~~
explodingcamera
For that money you might as well get a beefy dedicated server that's multiple
times as powerful and not have to worry about all these small servers.

~~~
dkersten
The bottleneck likely isn't the server here, but geolocation. He has 3 Digital
Ocean socket servers in three regions. A single dedicated server will only be
in one region. For latency-sensitive applications like realtime games, having
servers close to your users is a must.

------
unoti
The author mentioned that using a control scheme with small acceleration leads
to better synchronizing with the server. Another approach to this is control
schemes where the client indicates to the server "I'm planning on being at
this location at this future time."

~~~
synthmeat
That's then, logically, open to exploitation from user and client-side. Unless
deeper logic layers are made to prevent that.

Rabbit-hole, I tell ya. They're fun, though. :)

------
pdfernhout
I wrote something similar as a very simple demo multi-player tank
game/simulator with socket.io and canvas:
[https://github.com/pdfernhout/TanksInYourBrowser](https://github.com/pdfernhout/TanksInYourBrowser)

~~~
errozero
Do you have a live demo somewhere?

~~~
pdfernhout
No, sorry -- but it is easy to run it locally if you already have nodejs
installed. It's not very fancy -- just shows the basics.

------
majc2
Great fun - one suggestion would be if it could tell you who you are when you
start.

~~~
gfysfm
Good idea!

~~~
synthmeat
...and then someone takes over your discussion. :P

Sorry, gfysfm. I hope you enjoy the discussion though!

~~~
gfysfm
Haha, no worries. I did.

------
joe_carrot
"io.call('up')" broke your server. Sorry :(

~~~
gfysfm
Just woke up and it seems to be working now. No harm done, I guess?

~~~
joe_carrot
Well not really, anyone could type in 'io.call()' and it would add a new
player to your server with no controller. 'while(true) { io.call() }' would
kill your server in about a minute

