
League of Legends Chat Service Architecture - bandris
http://engineering.riotgames.com/news/chat-service-architecture-servers
======
scrollaway
I didn't know Riot kept an engineering blog. I'm so happy to find this out!

I _really_ wish more game studios would do this (Blizzard especially).

~~~
bargl
I agree, I really like seeing the EVE development blog. There are a few blogs
up about major battles where their servers are just being thrashed and how
they were able to stay up.

They also mention that their users warned them about this battle before hand
which was really cool.

~~~
simoncion
> There are a few blogs up about major battles where their servers are just
> being thrashed and how they were able to stay up. They also mention that
> their users warned them about this battle before hand which was really cool.

So, I _know_ that I'm expecting too much, but remember that the _only_ way CCP
can handle large fights is to simulate them at _1 /10th_ realtime, _and_ get
advance notice of said fights so that they can move the simulation for the
target system to a single, beefy server ahead of time.

It's a goddamn shame that they don't have the resources available to figure
out the -admittedly difficult- problem that is

* Reworking their simulation to be parallelizable across nodes

* Dynamically shifting slices of a battle _within a system_ across physical nodes

------
anthonyrrubin
Here is a related talk from Strange Loop last year:

"Scaling League of Legends Chat to 70 million Players" by Michal Ptaszek
[https://www.youtube.com/watch?v=_jsMpmWaq7I](https://www.youtube.com/watch?v=_jsMpmWaq7I)

------
BuckRogers
My languages of choice are Python(2, of course) and Erlang. Most of my fellows
seem obsessed with multicore programming, which IMO is only useful for latency
bound situations such as an OS or game engine. For everything else, multinode
wins handily. I'm pleased to see yet-another one of my favorite applications
building on the fantastic BEAM VM.

~~~
sanderjd
> Python(2, of course)

Sorry to veer off topic a bit, but as a Python-lover-but-outsider I've long
been mystified by the almost _proud_ unwillingness of the community to embrace
Python 3. It seems like a better language to me. What am I missing?

~~~
strong_ai
afaik the two main reasons are 1) the expense of porting from 2.* to 3.* and
2) some libraries do not yet support 3
[http://python3wos.appspot.com/](http://python3wos.appspot.com/)

------
mentos
As an indie game dev I would like to deal with as little of a stack as
possible. So what high level services could you use to achieve a similar chat
feature without having to roll and maintain all of your own infrastructure?

To me something like Firebase could work for this?

It seems like when game dev companies grow they love to blow out all of their
own in house solutions so they can have complete ownership and control. But is
the cost of ownership of your stack really worth the maintenance costs? Why
not outsource as much of the stack to another company that can take advantage
of economies of scale?

~~~
TimothyFitz
Chat at "indie scale" is fairly easy. 10-20k concurrent users fit into one or
two machines with off the shelf software (ejabberd) with minimal expertise.
1k-2k concurrent users could easily be done in almost any stack.

If you're doing more than 20k concurrent users, odds are you can afford to
hire someone to scale up or build out a team to do it, or find a 3rd party
service and pay them to host your chat.

~~~
Thaxll
Indie don't have 20k CCU.

Openfire is a known server for XMPP:

[http://www.igniterealtime.org/projects/openfire/](http://www.igniterealtime.org/projects/openfire/)

~~~
scott_karana
> Openfire is a known server for XMPP

Yes it is, and so is ejabberd, in GP's comment. (And the most popular iirc)

They were talking about indie _game studies_ , not indie chat protocols.

------
traduz
It's nice to see how the backend of my favorite game works. Specially this
article, right now I'm developing a chat service for a company and god I hate
openfire.

~~~
kaylarose
My team also recently built a chat service into a mobile app for a cruise
line, and had to have high resiliency to network drops and offline users -
while having a mission-critical "guaranteed delivery" requirement. We also
chose OpenFire since it was an off-the-shelf XMPP server, and (presumably) was
easily extensibile via plugins.

After working with (hacking around) OpenFire for two years, I 100% agree with
your statement. The clustering plugin routinely fails (to the point where we
have actually investigated not even clustering it anymore), the admin
interface routinely displays "wrong" data, and other fun bugs we found along
the way.

Does it work for us (with a lot of client XEPs and additional custom plugins)?
Yes, it does an acceptable job. Would I choose the same product again, if
given the choice? No.

------
k__
I thought LoL didn't have a chat because it "increases toxicity by 250%"

~~~
eterm
Officially this is why they don't support voice communication.

~~~
Omniusaspirer
Which is also a complete cop-out meant to save $$$ that lacks any basis in
fact or reality.

Dota 2 is churning along quite comfortably with integrated voice chat despite
having more inter-region mixing. Any time this subject comes up the Dota 2
community nearly unanimously agrees that voice chat in no way encourages
"toxicity" and more likely actually prevents it by assigning a human voice to
the player as opposed to just disembodied words.

~~~
iN7h33nD
Not to mention the fact that most toxic people can only spam the chat wheel
commands and cannot actually talk as they are chat banned for X hours.

EDIT:

Are there any articles like this on the architecture of Dota 2 or Steam?

~~~
Namidairo
And if players start turning to auto-matching voice services (Curse Voice and
Razer Connect did this at one stage), they'll go and threaten bans for all.

------
tuukkah
For comparison, is there info on how Facebook is using MQTT for chat? EDIT:
here's something, but not on backend architecture:
[https://www.facebook.com/notes/facebook-
engineering/building...](https://www.facebook.com/notes/facebook-
engineering/building-facebook-messenger/10150259350998920)

------
togilvie
All this fancy architecture doesn't change the fact that....I CALLED TOP
FIRST!

~~~
nanny
>I CALLED TOP FIRST!

We'll never know for sure. There have been many times where I'll type in chat:
"top", my message will appear, and then somebody else will say "top"
immediately after. But, a asking a third person reveals that the second
player's message actually arrived first, even though my client shows my
message first.

The chat architecture is terrible.

~~~
forrestthewoods
Generally speaking users expect instantaneous feedback. It would feel super
shitty if every time you hit enter there was lag until a server roundtrip
posted the message.

~~~
thedufer
Messages that the server saw earlier could easily be inserted partway up,
right? We're talking about ~100ms most of the time; that's not going to cause
you to miss messages.

~~~
lfowles
Some chat service I use does this and it's confusing because I don't expect
it. I check the last message to see if there's anything new, not necessarily
the scrollback.

Basically, it triggers an annoying O(N) operation whenever it happens :)

------
burrox
Ah this is the reason why I hate that my workplace blocks gaming websites.

~~~
lectrick
I used to work for those sorts of companies.

I fired them because I'm a mature professional who can figure out how best to
utilize my time, TYVM.

Find a company that people actually enjoy coming to work and collaborating in.
I mean, if you're an engineer, you're already doing the #1 job in the country,
why poison the well at a company with a mistrustful IT policy?

Sorry but I have no end of disrespect for that kind of bullshit. If you're at
a job and you're jerking around, people will eventually figure it out
regardless of stupid content filtering. And in this case, this is a lost
learning experience for you, as the article is very interesting.

------
bandris
An Erlang success story.

------
Senlaar
Thanks for posting this! I help to produce the blog at Riot and I really hope
the tech community finds it useful.

In case anyone's interested, we just posted new content this week - an
overview of our RFC process, and the continuation of a Docker&Jenkins
tutorial: [http://engineering.riotgames.com](http://engineering.riotgames.com)

------
mattlong
What happens if a server get partitioned for an extended period of time? In
their example, it sounds like Alice and Bob would be isolated from everyone
else until connectivity to the cluster is re-established. Is there a mechanism
that makes clients connected to an isolated server reconnect to the load
balancer?

~~~
lectrick
Assuming state isn't stored in the processes (which it typically isn't in
Erlang) and is instead in Riak, I believe it would be Riak's responsibility to
resolve the split. Which it is likely capable of doing.

------
runholm
Message ordering is still messed up causing a lot of confusing situations
ruining games. "Rito plz."

------
lectrick
TL;DR Erlang and distributed Riak using CRDT's (but the article is
interesting!)

------
debacle
Why not just use IRC?

~~~
baldfat
I seriously don't understand how IRC has withstood the test of time and gets
relegated to the back room. 3d.tv (Twitch before there was twitch) Used IRC in
a way I could just go to the IRC channel and connect to it and the people
could write IRC programs. Twitch.tv IRC is a little more hidden.

[http://help.twitch.tv/customer/portal/articles/1302780-twitc...](http://help.twitch.tv/customer/portal/articles/1302780-twitch-
irc)

~~~
debacle
It only really confuses me because there's not really a lot of value in
writing your own "not quite IRC" type chat system. Just think about the
development and administration time that went into setting something up that's
probably not as good as something you can apt-get install.

~~~
mentos
I can't agree more.

Are there any great examples of IRC being used for the backend of a games chat
system?

How would you envision using IRC in this case?

Would have been great to see RiotGames write their own custom framework to
interface between their game and an IRC backend, open source it for other
companies to contribute to and prevent further reinventing of the wheel.

~~~
6581
Worms Armageddon and some other Worms games use IRC for lobby chat.

[http://worms2d.info/WormNET_(Worms_Armageddon)](http://worms2d.info/WormNET_\(Worms_Armageddon\))

------
sng
don't really have to deal with scaling when so many players are chat
restricted.

