
How Discord Scaled Elixir to 5M Concurrent Users - b1naryth1ef
https://blog.discordapp.com/scaling-elixir-f9b8e1e7c29b
======
iagooar
This writeup make me even more convinced of Elixir becoming one of the large
players when it comes to hugely scaling applications.

If there is one thing I truly love about Elixir, it is the easiness of getting
started, while standing on the shoulders of a giant that is the Erlang VM. You
can start by building a simple, not very demanding application with it, yet
once you hit a large scale, there is plenty of battle-proven tools to save you
massive headaches and costly rewrites.

Still, I feel, that using Elixir is, today, still a large bet. You need to
convince your colleagues as much as your bosses / customers to take the risk.
But you can rest assured it will not fail you as you need to push it to the
next level.

Nothing comes for free, and at the right scale, even the Erlang VM is not a
silver bullet and will require your engineering team to invest their talent,
time and effort to fine tune it. Yet, once you dig deep enough into it, you'll
find plenty of ways to solve your problem at a lower cost as compared to other
solutions.

I see a bright future for Elixir, and a breath of fresh air for Erlang. It's
such a great time to be alive!

~~~
janekm
I read it a little differently. The whole article is about how Erlang/Elixir
fails at its core reason for existence (fast message passing between
distributed processes) and all the complicated work-arounds they had to
implement to avoid actually using this core feature of Erlang.

~~~
josevalim
People tend to forget that scalability is not a binary property. You always
scale up to some users, up to some architecture, up to some amount of nodes.
There is no system that will scale to infinity without requiring developer
intervention once business needs and application patterns start to settle in.

Distributed Erlang/Elixir _has known limitations_. For example, the network is
fully meshed, which gives you about 60 to 200 nodes in a cluster. Or don't
send large data over distribution, as that delays the other messages, etc.
Some of those are easily solvable. For example, you can rely on your
orchestration tools to break your clusters in groups. Or you can setup an out
of band tcp/udp socket for large data. Others may be more complex.

The important question, however, is how far you can go without having to
tweak, and, once you reach those roadblocks, how well you can address them. In
many platforms, writing a distributed system is a no-no or, at best, they
require you to assemble and tweak from day one. In this case, the ability to
start with Erlang/Elixir and tweak as you grow _is a feature_.

And if you never run into those roadblocks, then you can happily continue
running on the default stack. Just look at the many companies using Phoenix
PubSub and Phoenix Presence, both distributed, without having to worry about
fine-tuning the distribution.

~~~
majidazimi
What difference does it make? Erlang/OTP distribution doesn't have pluggable
architecture. Sooner or later you will reach a point that you have to modify
it. Then you are diverging from the original branch which makes it even more
difficult to maintain it. You have to merge your additions into every release
(minor or major) and test it thoroughly.

A better architecture for a distributed system has a strong composability
property. It should be possible to modify every possible aspect of it on a
running cluster without introducing downtime.

Write your own standard well documented distribution layer and become
independent of underlying technologies.

~~~
josevalim
What do you mean by architecture here? If you mean the roles different nodes
take and their topology, I actually doubt you can be decoupled from your
architecture in a distributed system because they directly affect your design
and capabilities.

You can move away from fully meshed for topologies but how does this choice
affect node ups and node downs? Rebalancing can affect how you store data in
the cluster.

How many connections should you have between nodes? A single connection makes
the ordering guarantee straight-forward. Multiple connections is more
performance but requires care if you need ordering and is more efficiently
done along side your application.

And what about process placement? On which side of CAP do you want your
registries to sit?

If you and your team is capable of "writing your own standard well documented
distributed layer" upfront, then you are in a better position than most to
take those decisions. But writing a distributed system is hard, so I will
gladly start with a well-designed system, especially at the beginning of the
project, when it may be unclear which patterns I will need as my application
and business grow.

And most times, it will be good enough.

As far as OTP goes, you can plug your own discovery/topology mechanism as well
as your own module for handling the connection between nodes. But, as
mentioned in my previous reply, some of those issues may be better solved on
the side, e.g. a different tcp/udp connection for data transmitting.

~~~
majidazimi
I did also mention that for fast time to market it is indeed a good idea to
use already available tools. By the way writing a distribution layer is not
that difficult. Many companies/startups with scalable back-ends that are not
using Erlang/OTP have already done it. The point is scalable software requires
knowledge and Erlang/OTP is not going to magically solve it. But it seems many
fans are trying to promote it such that it is a magic tool that is going to
make a shit software become a hyper scalable one. Just look at the comments
below that how people have gone crazy.

~~~
jlouis
Erlang (and by extension Elixir) definitely provides a set of tools which are
good for building highly concurrent distributed systems. And your systems are
likely to have few errors as well as being resilient, if you know what you are
doing.

But indeed, you need to know your tools like every other part of computer
science. Storing 9 billion elements in an array, doing linear search and then
complaining your linear search is too slow and needs to run faster will be
disastrous to your architecture. Likewise, assuming communication is free is
equally disastrous.

The problem with building a distribution layer in another language is the
effort involved. Most of the companies which pull that off and get stability
in addition are usually large, multinational, and has ample amounts of
engineering resources to throw at the effort required.

~~~
majidazimi
Consider for some reason (either technical or political), this company decides
to migrate their web socket servers to Akka/Rust/Go/NodeJS. Integrating these
new servers into their core cluster is going to be deadly painful. They rely
heavily on Erlang/OTP internal clustering. This is not even considered as a
challenge in distributed systems but still really painful to implement because
of their design decision. This is what they did wrong. Clustering and message
routing part of the application should be technology agnostic.

~~~
brightball
That depends what you're going for. Moving web sockets elsewhere are pretty
simple regardless of whether you're using Erlang/OTP or not because it's a
frontal layer. If you do that all you are doing is creating another layer to
hold connections that needs to relay to the rest of the application. The
application behind it still has the responsibility to receive and relay
everything from those web sockets, determine what message goes to who, etc.

You can move it out just the same as you can elsewhere, you just add an
unnecessary layer of complexity and lose a lot of the capabilities already
built into the language. It boils down to a simple question of, what would be
the purpose behind that decision? What improvement would you get from deciding
to move any of that elsewhere? Do you gain speed at the cost of complexity?

Clustering is a non-trivial problem. It can't happen naturally in any language
that includes mutable data without explicit oversight and thus most of the
needed tooling can't be built into the system and guaranteed to work
everywhere. As soon as some part of the system goes from "send message, get
response" to "send message, change variable referenced in memory on this
machine, get response" you break your ability to naturally cluster.

That leads to a dependency in central relay points for things like setting up
web sockets and then having code behind them communicate. The code behind them
in whatever language is usually going to be talking to other central relay
points like load balancers or pub sub databases rather than directly to other
specific servers.

As soon as you have central relay points, they become something else that you
have to monitor and scale even if they are very high volume.

There's nothing wrong with this approach and it will certainly work but if you
want to avoid that and build a distributed system in another language, you've
got to invest a lot in other areas.

~~~
majidazimi
> Moving web sockets elsewhere are pretty simple regardless of whether you're
> using Erlang/OTP or not because it's a frontal layer.

It is supposed to be easy. But they are not using language agnostic messaging.
Instead they rely on Erlang distribution protocol. This is what makes it
difficult to integrate.

> what would be the purpose behind that decision?

1\. To achieve composability. I can replace one web socket server with a C++
implementation to see if I can handle 10 million connections per server. If I
fucked it up return back to Erlang implementation otherwise fuck BEAM. I'm
moving to native code. The beauty of it is that I can do it one step at a
time. Even I can implement web socket layer in multiple languages and
experiment all possible options in production and no one even notices it.

2\. I don't have to pay for 20 Erlang guys who are mostly senior devs. Get 5
of them to write the core and the rest of the team NodeJS guys.

~~~
mononcqc
You're acting like the distribution and serialization formats of Erlang are
not open or documented.

If you're going for your C++ layer, try
[https://github.com/saleyn/eixx](https://github.com/saleyn/eixx) for example.

------
jakebasile
I'm continually impressed with Discord and their technical blogs contribute to
my respect for them. I use it in both my personal life (I run a small server
for online friends, plus large game centric servers) and my professional life
(instead of Slack). It's a delight to use, the voice chat is extremely high
quality, text chat is fast and searchable, and notifications actually work.
Discord has become the de facto place for many gaming communities to organize
which is a big deal considering how discriminating and exacting PC gamers can
be.

My only concern is their long term viability and I don't just mean money wise.
I'm concerned they'll have to sacrifice the user experience to either achieve
sustainability or consent to a buyout by a larger company that only wants the
users and brand. I hope I'm wrong, and I bought a year of Nitro to do my part.

~~~
lordCarbonFiber
A closed source walled garden chat service that survives purely on the free
flow of VC capital 100% has no long term viability. No federation means as
soon as the "next thing" shows up and wins way the VC dollars they'll
disappear. It's so incredibly frusterating that so many companies and users
make/support these closed environments even as we enter a new golden age of
open sourced and federated technologies.

~~~
jakebasile
If there was an open source and federated equivalent to the features Discord
provides I'd use it. There is no such product. Matrix is interesting, but the
experience is no where near as polished as Discord and friends and that
matters for mass adoption.

~~~
aphextron
>If there was an open source and federated equivalent to the features Discord
provides I'd use it.

But will you pay for Discord though? The features and quality Discord is able
to provide are artificially propped up by VC funding. When it runs dry, we
will be left with open source offerings, or the next product to take it's
place and repeat the cycle.

~~~
connorcpu
I already pay $5/mo for Discord. Animated avatars and cross-server custom
emoji were enough to entice me

------
Cieplak
I know that the JVM is a modern marvel of software engineering, so I'm always
surprised when my Erlang apps consume less than 10MB of RAM, start up nearly
instantaneously, respond to HTTP requests in less than 10ms and run forever,
while my Java apps take 2 minutes to start up, have several hundred
millisecond HTTP response latency and horde memory. Granted, it's more an
issue with Spring than with Java, and Parallel Universe's Quasar is basically
OTP for Java, so I know logically that Java is basically a superset of Erlang
at this point, but perhaps there's an element of "less is more" going on here.

Also, we're looking for Erlang folks with payments experience.

cGF0cmljaytobkBmaW5peHBheW1lbnRzLmNvbQ==

~~~
kevincennis
Off topic, but what's the story behind base-64 encoding your email? Is that
just a spam prevention measure?

~~~
Cieplak
Yes

~~~
Corrado
I actually really like this idea and am going to steal it. :)

------
rdtsc
Good stuff. Erlang VM FTW!

> mochiglobal, a module that exploits a feature of the VM: if Erlang sees a
> function that always returns the same constant data, it puts that data into
> a read-only shared heap that processes can access without copying the data

There is a nice new OTP 20.0 optimization - now the value doesn't get copied
even on message sends on the local node.

Jesper L. Andersen (jlouis) talked about it in his blog:
[https://medium.com/@jlouis666/an-erlang-
otp-20-0-optimizatio...](https://medium.com/@jlouis666/an-erlang-
otp-20-0-optimization-efde8b20cba7)

> After some research we stumbled upon :ets.update_counter/4

Might not help in this case but 20.0 adds select_replace so can do a full on
CAS (compare and exchange) pattern
[http://erlang.org/doc/man/ets.html#select_replace-2](http://erlang.org/doc/man/ets.html#select_replace-2)
. So something like acquiring a lock would be much easier to do.

> We found that the wall clock time of a single send/2 call could range from
> 30μs to 70us due to Erlang de-scheduling the calling process.

There are few tricks the VM uses there and it's pretty configurable.

For example sending to a process with a long message queue will add a bit of a
backpressure to the sender and un-schedule them.

There are tons of configuration settings for the scheduler. There is to bind
scheduler to physical cores to reduce the chance of scheduler threads jumping
around between cores:
[http://erlang.org/doc/man/erl.html#+sbt](http://erlang.org/doc/man/erl.html#+sbt)
Sometimes it helps sometimes it doesn't.

Another general trick is to build the VM with the lcnt feature. This will add
performance counters for locks / semaphores in the VM. So then can check for
the hotspots and know where to optimize:

[http://erlang.org/doc/man/lcnt.html](http://erlang.org/doc/man/lcnt.html)

~~~
jlouis
It isn't that likely the OTP20 optimization helps here. If the process never
sends a message containing the literal value, then there is no benefit in the
optimization. What `mochiglobal` and friends are good at is when you have a
large set of data (A ring, say) which update rarely, so you can treat it as
semi-static data in the system. But then you shouldn't really send that ring
data around in the system too much, although it will now be free. [There is a
nice subscription-based approach to updates which are now feasible in OTP20,
but that is more for convenience]

if send/2 takes 30us to 70us, I'm guessing blocking as well, either on
distributed communication or something else along those lines. For local
message passes to take that long, my something-is-amiss-sixth-sense is
tingling.

~~~
rdtsc
> It isn't that likely the OTP20 optimization helps here

Ah good point. I didn't look at the code much. I was thinking of cases of
passing any of those literals in gen_server calls and such and just getting
extra performance from upgrading to OTP20 as a side-effect.

------
mbesto
This is one of those few instances where getting the technology choice right
actually has an impact on cost of operations, service reliability, and overall
experience of a product. For like 80% of all the other cases, it doesn't
matter what you use as long as your devs are comfortable with it.

~~~
kornish
Not sure why this comment saw a couple downvotes earlier. mbesto is correct:
for most startups, most of the time, competitive advantage doesn't come from
the underlying tech stack. To make a general statement, most things could be
done similarly on any of several platforms. However, when product requirements
match exceptionally well with a specialized technology, you can see things
that would simply be infeasible or extremely tough using a different stack.

WhatsApp + Erlang was one of those cases (watch this talk and imagine trying
to recreate that system with only a handful of server engineers using any
other tech:
[https://www.youtube.com/watch?v=c12cYAUTXXs](https://www.youtube.com/watch?v=c12cYAUTXXs)).
Discord + Elixir appears to be another.

Curious if anyone has any examples that spring to mind from outside the highly
concurrent messaging space.

------
jlouis
A fun idea is to do away with the "guild" servers in the architecture and
simply run message passes from the websocket process over the Manifold system.
A little bit of ETS work should make this doable and now an eager sending
process is paying for the work itself, slowing it down. This is _exactly_ the
behavior you want. If you are bit more sinister you also format most of the
message in the sending process and makes it into a binary. This ensures data
is passed by reference and not copied in the system. It ought to bring message
sends down to about funcall overhead if done right.

It is probably not a solution for current Discord as they rely on
linearizability, but I toyed with building an IRCd in Erlang years ago, and
there we managed to avoid having a process per channel in the system via the
above trick.

As for the "hoops you have to jump through", it is usually true in any
language. When a system experiences pressure, how easy it is to deal with that
pressure is usually what matters. Other languages are "phase shifts" and while
certain things become simpler in that language, other things become much
harder to pull off.

~~~
mononcqc
The true evil approach is to send the socket around, not the message, so that
there is no copying required no matter what ;)

~~~
rdtsc
Wah. Easy there, Satan :-)

That is cool trick though. So it's basically sending the port itself around
and changing its ownership, with something like port_connect(Port,NewOwner)?

And btw, thank you for writing [https://www.erlang-in-
anger.com](https://www.erlang-in-anger.com) and
[http://learnyousomeerlang.com](http://learnyousomeerlang.com) !

~~~
mononcqc
The trick is more commonly used when writing to sockets. A socket owner is
required for reading, not for writing.

The trick then is that when you need to write lots of data to a socket to just
send a copy of it to the writer so they can dump all their data for cheap, but
without changing ownership (which is costly).

Also recently I've gotten
[http://propertesting.com/](http://propertesting.com/) out, you might enjoy it
:)

~~~
rdtsc
Thanks for explaining. I'll have to remember the socket trick.

> Also recently I've gotten
> [http://propertesting.com/](http://propertesting.com/) out, you might enjoy
> it :)

It might be just what I need to understand and start using property tests.
I've tried twice and gave up.

Oh and recon! Thanks for that too. Use it almost every day.

------
danso
According to Wikipedia, Discord's initial release was March 2015. Elixir hit
1.0 in September 2014 [0]. That's impressively early for adoption of a
language for prototyping and for production.

[0] [https://github.com/elixir-
lang/elixir/releases/tag/v1.0.0](https://github.com/elixir-
lang/elixir/releases/tag/v1.0.0)

~~~
quaunaut
If I remember right, they were using Erlang at the beginning, and moved slowly
to Elixir as they got more comfortable, and the ecosystem built up around
them.

~~~
Vishnevskiy
We started with Elixir, some of our engineers have used Erlang prior though.

------
didibus
So, at this point, every language was scaled to very high concurrent loads.
What does that tell us? Sounds to me like languages don't matter for scale. In
fact, that makes sense, scale is all about parallel processes, horizontally
distributing work can be achieved in all language. Scale is not like
perforance, where if you need it, you are restricted to a few languages only.

That's why I'd like to hear more about productivity and ease now. Is it faster
and more fun to scale things in certain languages then others. Beam is modeled
on actors, and offer no alternatives. Java offers all sorts of models,
including actors, but if actors are the currently most fun and procudctive way
to scale, that doesn't matter.

Anyways, learning how team scaled is interesting, but it's clear to me now
languages aren't limiting factors to scale.

~~~
dmix
> So, at this point, every language was scaled to very high concurrent loads.
> What does that tell us?

Just like any language vs language debate each one has benefits for various
particular use-cases. Any meaningful comparison of languages must be prefaced
with the use-case scenario.

One of the strongest use-cases of Erlang/Elixir has always been building large
distributed apps that need to scale (async web apps, telecom, chat servers,
messaging mobile apps, etc). The ability to build these large distributed
systems are baked into the very primitive parts of the language and standard
library - to a degree that few other languages can compare to it, if any.

With Erlang/Elixir you design ALL applications in a way where scaling is
rarely an after thought but rather a natural extension of the program.

> Beam is modeled on actors, and offer no alternatives. Java offers all sorts
> of models, including actors, but if actors are the currently most fun and
> productive way to scale, that doesn't matter.

People often make the mistake of trivializing Erlang/Elixirs as merely
programming with actors. It's development not only predated the actor model
but it also goes well beyond that to being the standard programming style you
use when developing any program when using the language - the same way Rails
embraces MVP. When this is fundamental part of every Erlang application then
the means of scaling to a large distributed system are also a fundamental part
of each program.

This built-in scaling is gained without any significant costs in terms of
development time but also provides many benefits beyond scaling, such as
highly modular and extensible code. There are real benefits even if you don't
plan to scale to a large distributed system. similar to Rails it creates a
predictable program design which makes joining new projects easier and deters
NIH syndrome that is far too common in Java/C++/etc. And ultimately,
regardless of what you are building, it provides very high performance by
default for the type of async style applications that are popular on the web
today.

So the key point here is not that the end goal was achieved (that you _can_
scale) but how you get there.

------
jmcgough
Great to see more posts like this promoting Elixir. I've been really enjoying
the language and how much power it gets from BEAM.

Hopefully more companies see success stories like this and take the plunge -
I'm working on an Elixir project right now at my startup and am loving it.

------
ShaneWilton
Thanks for putting this writeup together! I use Elixir and Erlang every day at
work, and the Discord blog has been incredibly useful in terms of pointing me
towards the right tooling when I run into a weird performance bottleneck.

FastGlobal in particular looks like it nicely solves a problem I've manually
had to work around in the past. I'll probably be pulling that into our
codebase soon.

~~~
pmarreck
Note that Erlang 20 may have solved the problem that FastGlobal tries to fix
(that of _not_ copying large amounts of data unnecessarily)

~~~
ShaneWilton
Erlang 20 fixes the case where you're copying a constant literal, but
unfortunately won't help if you're sharing a dynamically generated, but
infrequently modified, term; like Discord does in this post.

------
joonoro
Elixir was one of the reasons I started _using_ Discord in the first place. I
figured if they were smart enough to use Elixir for a program like this then
they would probably have a bright future ahead of them.

In practice, Discord hasn't been completely reliable for my group. Lately
messages have been dropping out or being sent multiple times. Voice gets
messed up (robot voice) at least a couple times per week and we have to switch
servers to make it work again. A few times a person's voice connection has
stopped working completely for several minutes and there's nothing we can do
about it.

I don't know if these problems have anything to do with the Elixir backend or
the server.

EDIT: Grammar

~~~
Vishnevskiy
The messages struggles have been sadly due to issues with Cassandra and GC
pauses caused by bugs within it. We have been trying to work with the
Cassandra developers to resolve these.

Voice issues should not be happening. Please contact our support with more
information and we will gladly investigate.

~~~
joonoro
Thanks for the response, it's good to know what's causing the problems with
messages and that it's being worked on. I'll try to contact support next time
I have voice issues with my group.

~~~
Vishnevskiy
:) np

We love Cassandra, and hate it at the same time.

Check out this nasty bug we got.
[https://issues.apache.org/jira/browse/CASSANDRA-13004](https://issues.apache.org/jira/browse/CASSANDRA-13004)

~~~
jjirsa
Two things are true about Cassandra:

It is by far the best at doing what it does

It has plenty of room for improvement

~~~
Vishnevskiy
You are 100% right that it is the best at what it does.

And thank you again with helping us with that bug :) we really appreciate it.

~~~
tormeh
I'm currently far down the database rabbit-hole and have to ask: What's so
great about Cassandra that you can't get with CouchDB or other AP (yeah, I
know...) databases?

~~~
jjirsa
Solid ingestion story. Very very good write throughout. Linear scaling. Easy
expansion / contraction. Complete flexibility in consistency vs availability
tradeoff.

And most importantly:

It actually works at scale. Huge scale. Thousand node cluster and hundreds of
thousands of instances scales.

Because a good chunk of the active maintainers actually run this shit in prod.

------
ConanRus
I do not see there any Elixir specific, it is all basically Erlang/Erlang
VM/OTP stuff. When you using Erlang, you think in terms of actors/processes
and message passing, and this is (IMHO) a natural way of thinking about
distributed systems. So this article is a perfect example how simple solutions
can solve scalability issues if you're using right platform for that.

~~~
nichochar
You're right. Elixir doesn't pretend to do anything except make using the
wonderful erlang VM/OTP stuff easier.

The VM is an absolute marvel of engineering, and it's insane to me that it
doesn't have more adoption yet in big tech companies.

My best theory is that engineers in top engineering companies are actually not
the best engineers but simply career engineers that learn one skill
(python/java/C++) and then explain to every employer that this technology is
the best for the problem they have, over and over again.

~~~
Anderkent
Coordination problems get more difficult in larger teams/companies. Getting
everyone to use a particular non-standard language is a coordination problem.
Thus large companies are unlikely to experiment with languages.

It makes sense too - say it's 10x easier to write something in language X than
Y. If there's only 10 other people that might interact with the thing / have
to read the sources, that's a great tradeoff. If there's a thousand other
people that might have to at some point understand how some part of your code
works, suddenly all of them have to learn the new language X.

------
majidazimi
It seems awkward to me. What if Erlang/OTP team can not guarantee message
serialization compatibility across a major release? How you are going to
upgrade a cluster one node at a time? What if you want to communicate with
other platforms? How you are going to modify distribution protocol on a
running cluster without downtime?

As soon as you introduce standard message format, then all nice features such
as built-in distribution, automatic reconnect, ... are almost useless. You
have to do all these manually. May be I'm missing something. Correct me if I'm
wrong.

For a fast time to market it seems quite nice approach. But for a long running
maintainable back-end it not enough.

~~~
toast0
The erlang team has been through 20 major releases, I think I've run all of
them since r13 or r14? Being able to upgrade your distributed system is
important and they care about making it possible. Generally you upgrade one
node at a time, until they're all upgraded and only then can you use new
language features. Sometimes you find a build you like and stick with it until
something comes up that causes you to upgrade.

There are many ways to interoperate with other languages. Including libraries
for other languages to claim to be erlang nodes (they will have to upgrade too
when you want to upgrade to a newer version of erlang with distribution
protocol changes, for example when maps we're introduced in r17).

You can easily do standard dist messaging within your erlang cluster and
serialize to whatever makes sense at the boarder. You can serialize to erlang
term format, if you like; it's well specified, but not terribly compact.

You're going to have the same questions with any other language too. Very few
companies get to write clients, servers, and everything else in one language
that never updates.

~~~
majidazimi
> You're going to have the same questions with any other language too. Very
> few companies get to write clients, servers, and everything else in one
> language that never updates.

Then I don't see much difference between Go, Java, ... vs Erlang except they
are simpler to learn plus finding Java, NodeJS, ... devs is much easier. What
was the point of using Erlang? It was supposed to solve a problem for us, but
we end up of solving the problem ourselves.

What I'm really trying to say is: integrating a mainstream language like Go,
Java, C++, ... with a messaging layer like ZeroMQ (or something else) and
adding some reliability features is going to be easier than introducing a
totally new language (with a totally different paradigm) into the stack.

~~~
toast0
> Then I don't see much difference between Go, Java, ... vs Erlang except they
> are simpler to learn plus finding Java, NodeJS, ... devs is much easier.

For the first several years, my team of Erlang devs had zero Erlang experience
prior to being hired; they were just smart and flexible developers. I was
ahead of the curve because I sort of remembered seeing a Slashdot post about
Ericson opening the language way back when.

A lot of people end up using RabbitMQ as their distributed message queue,
which is built in Erlang. If you went with that instead of ZeroMQ; and then
slowly added more things, there's a reasonable path to writing more Erlang.

I'm not sure if you can really bolt on messaging and reliability features and
get the same results; just like bolting on security later, if it's something
that you need, it works better if you have it from the beginning.

But certainly, if you're happy with your stack, don't change it.

------
_ar7
Really liked the blog post. Elixir and the capabilities of the BEAM VM seems
really awesome, but I can't really find an excuse to really use them in my day
to day anywhere.

------
StreamBright
Whatsapp's story is somewhat similar. Relevant read to this subject.

[http://www.erlang-
factory.com/upload/presentations/558/efsf2...](http://www.erlang-
factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf)

------
brian_herman
I love discord's posts they are very informative and easy to read.

------
OOPMan
5 million concurrent users is great and all, but it would be nice if Discord
could work out how to use WebSockets without duplicating sent messages.

This seems to happen a lot when you are switching between wireless networks
(E.g. My home router has 2Ghz and 5Ghz wireless networks) or when you're on
mobile (Seems to happen regularly, even if you're not moving around).

It's terribly annoying though and makes using the app via the mobile client to
be very tedious.

~~~
k__
Coming from IRC with netsplits and stuff, this seems like a first world
problem to me hehe

~~~
OOPMan
You know there are discord bots for bridging between IRC and Discord?

Pretty funky :-)

------
sriram_malhar
I really like elxir the language, but find myself strangely hamstrung by the
_mix_ tool. There is only an introduction to the tool, but not a reference to
all the bells and whistles of the tool. I'm not looking for extra bells and
whistles, but simple stuff like pulling in a module from GitHub and
incorporate it. Is there such documentation? How do you crack Mix?

~~~
amorphid
The docs for Mix are decent. You can start here:

[https://hexdocs.pm/mix](https://hexdocs.pm/mix).

When you are trying to get help w/ a specific task, you check checkout the mix
tasks docs:

[https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content](https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content)

And honestly, I often just look at the source code:

[https://github.com/elixir-
lang/elixir/tree/master/lib/mix](https://github.com/elixir-
lang/elixir/tree/master/lib/mix)

The Elixir Slack channel is pretty amazing, too:

[https://elixir-slackin.herokuapp.com/](https://elixir-slackin.herokuapp.com/)

------
renaudg
It looks like they have built an interesting, robust and scalable system which
is perfectly tailored to their needs.

If one didn't want to build all of that in house though, is there anything
they've described here that an off the shelf system like
[https://socketcluster.io](https://socketcluster.io) doesn't provide ?

~~~
lostcolony
Yes. Discord is served over HTTPS. :P (your link is broken; socketcluster.io
doesn't serve over HTTPS)

But seriously, Discord actually benchmarked 5 million concurrent users,
horizontally distributed, and having to ferry messages across the cluster,
with specifically tailored fanout patterns (rather than just a global pub/sub.
I.e., who a message goes to varies, rather than just "everyone").

Socketcluster.io only has benchmarks for a single machine, capped at 42k
concurrent connections (though to be fair that was due to them running a
single client, rather than a limitation of the server). They don't out of the
box support horizontal scaling; you're required to spin up your own message
queue solution for that.

So, basically, you're advocating a technology that solves -the simplest part
of the problem-, and nothing else. Whereas Phoenix + Elixir, even without any
of the custom tweaking Discord describes, solve that AND more of the actual
problem Discord had. So...yes, and no. Yes, there is plenty here they've
describe that is not available in socketcluster.io, but no, nothing they've
done here is no generally solved by an off the shelf system, because they're
-using- an off the shelf system, Elixir + Phoenix.

~~~
jondubois
Hi, main author of SocketCluster here.

SC does support automatic horizontal scaling across any number of machines out
of the box if you're running it on Kubernetes.

There's also a CLI tool to deploy it automatically to any Kubernetes cluster:
[https://www.npmjs.com/package/baasil](https://www.npmjs.com/package/baasil)

See
[https://github.com/SocketCluster/socketcluster/blob/master/s...](https://github.com/SocketCluster/socketcluster/blob/master/scc-
guide.md#scc-guide)

~~~
lostcolony
I just meant you still need a third party MQ to be spun up (per docs here -
[http://socketcluster.io/#!/docs/scaling-
horizontally](http://socketcluster.io/#!/docs/scaling-horizontally)). Without
that, there is no distribution happening.

From my understanding, you're basically saying "You can combine SocketCluster
with the MQ of your choice (the installation and configuration of which is
left as an exercise to the reader) and then between Docker, Kubernetes, and
Baasil you can orchestrate and deploy it across a cluster". That sounds a bit
more complex than just using SocketCluster, which is what the OP seemed to be
indicating was all you needed, and is also including the DevOps story, which I
don't think either he or I was intending to include.

I was not trying to indicate that SocketCluster can't be -used- to scale
websockets horizontally, but that it's not just an off the shelf solution that
would have solved Discord's problem either. It requires other parts, as both
the docs and you mention.

I'll also reiterate from my post, SocketCluster has no benchmarks pertaining
to what happens when you -do- scale horizontally (per docs here -
[http://socketcluster.io/#!/performance](http://socketcluster.io/#!/performance)
). That lack alone would kill my interest in it (as would scc-state being a
single instance, which would make fault tolerance a real concern to me, but it
looks like you know that already). Is performing horizontal scalability tests
on the roadmap?

~~~
jondubois
If you use SCC, then you don't need a separate MQ - That is only if you want
to do things yourself manually. I will update the docs to make that clearer.

It should only take a few minutes to deploy a cluster across hundreds of
machines. The only limit is the maximum number of hosts that Kubernetes itself
can handle (which is I think is over 1000 now)? SCC is self-sharding and runs
and scales itself automatically with no downtime.

You can easily handle 5 million concurrent users with a small cluster. SC's
problem isn't scalability, it's marketing.

~~~
lostcolony
That's perfectly fair; fix the marketing then. :P In evaluating a solution,
the marketing is the -first thing- anyone looks at. And how it currently
reads, "SocketCluster only provides you the interface for synchronizing
instance channels - It doesn't care what technology/implementation you use
behind the scenes to make this work" definitely reads as "You need a
technology/implementation behind the scene" rather than "we provide you a
default one, and you can swap it out".

For me to pick Socketcluster for a distributed solution (or more broadly, what
I'd want for -any- technical solution) I'd want to know what else I need to
pair it with (which the docs actually mislead me on), what else I can benefit
from (which the docs don't tell me, but which does exist per your links), and
what benefits I stand to get from using it (the docs tell me only marketing
claims, but with no metrics, performance, data, etc, for what happens in a
distributed context, well, I would avoid it).

Ideally, set up a clustered performance test, and then make as many of the
artifacts (docker images, configs, readme, etc) available so others can
conduct the same performance test themselves (as well as have a reference
architecture for their own solution). Heck, if you're doing it in AWS,
consider making the AMIs available along with whatever modifications need to
happen. -That- would be very convincing for someone looking to adopt a
solution in this space, if they could literally just spin up some EC2s and
immediately start throwing load at a fully configured cluster.

Also, to make it clear, is this handling message passing between instances in
the cluster?

~~~
jondubois
Thanks for the advice.

Yes, it handles message passing between instances in the cluster. That means
if you publish a message on a channel whilst connected to one host, the
message will also reach subscribers to that channel which are on any other
host in the cluster. It shards all channels across available brokers, when you
scale up the number of brokers, it will automatically migrate the shards
across available brokers with no downtime.

~~~
lostcolony
Okay, nice. Then yeah, given some performance benchmarks showing some numbers
at increasing number of nodes, with messages being broadcast across 1-to-1
channel pairings (i.e., direct message), 1-to-100 (for groups), and 1-to-all,
I think it could sell itself as a pretty compelling turnkey solution (barring
the scc-state concern which you're aware of).

Possibly also consider some testing and documentation around geographic
distribution; what happens if the nodes are located in different datacenters
with non-trivial latency between them? Is that an issue? In the event of
netsplits, does it split brain (probably not, given scc-state, but addressing
that might cause it to)? That might be fine, it might not, depending on the
use case, and just documenting what happens (by default, at least, if it's to
be tunable) would be helpful as well.

------
etblg
Reading posts like this about widely distributed applications always gets me
interested in it as a career path. Currently I'm working as a front-end dev
with moderate non-distributed back-end experience. How would someone in my
situation, with no distributed back-end experience, break in to a position
working on something like Discord?

------
omeid2
I think while this is great, it is good to remember that your current tech
stack maybe just fine! after all, Discord start with mongodb[0].

[1]. [https://blog.discordapp.com/how-discord-stores-billions-
of-m...](https://blog.discordapp.com/how-discord-stores-billions-of-
messages-7fa6ec7ee4c7)

------
alberth
Is there any update on BEAMJIT?

It was super promising 3 or so years ago. But I haven't seen an update.

Erlang is amazing in numerous ways but raw performance is not one of them.
BEAMJIT is a project to address exactly that.

[https://www.sics.se/projects/beamjit](https://www.sics.se/projects/beamjit)

~~~
jlouis
Still ongoing work. My personal bet is a bit more on modernizing HiPE however
(by using the LLVM backend more).

~~~
alberth
Amy ETA on when we can start using beamjit?

~~~
jlouis
Given that it has been postponed a couple of times, no. JITs are hard to pull
off and it will probably have a period of worse stability as well before it
matures. Another problem is getting a JIT to be faster than the interpreter.
Erlang's BEAM is threaded code and also macro-instructions, so it almost looks
like a JIT internally.

The big gains would be in inlining across module boundaries and type
speculation. But I hold that if we could compile bundles of modules in HiPE,
we would have the same gain for a fraction of the development and maintenance
effort.

The biggest lure of native code generation would be that we could get rid of a
lot of C code in the system as the native cogen would be able to rival the C
code in speed. Many Erlang programs spend shockingly little time in the
emulator loop, especially if they are communication heavy.

If you need speed today, don't underestimate a port-program. My test is that
you can pipeline about a million requests back and forth to an OCaml program
per second per core. So if your work is on the order of 1+ milliseconds, this
is usually a feasible strategy. Espcially because OS isolation means you can
handle exceptions in the OCaml program from the Erlang side by restarting the
port.

~~~
rozap
Would you say the OCaml-port-as-an-optimization-strategy only makes sense if
the program is compute bound, though?

The reason I ask is because we're running a Erlang+JInterface program and the
performance advantage the JVM has over BEAM is less than I would have
expected. Even batching requests up into big pieces, we still see it's about
30% slower than running the same stuff in Elixir, without so much copying. But
the reason we're doing JVM stuff at all is so we can re-use a whole bunch of
code we already had written, and I would have expected it would have been a
marginal performance win as well, but it's not.

Perhaps we're doing it wrong, too.

------
ramchip
Very interesting article! One thing I'm curious about is how to ensure a given
guild's process only runs on one node at a time, and the ring is consistent
between nodes.

Do you use an external system like zookeeper? Or do you have very reliable
networking and consider netsplits a tolerable risk?

~~~
jhgg
We use etcd.

------
myth_drannon
It's interesting how on StackOverflow Jobs Elixir knowledge is required more
often than Erlang.

[http://www.reallyhyped.com/?keywords=erlang%2Celixir](http://www.reallyhyped.com/?keywords=erlang%2Celixir)

~~~
cutler
Despite its appeal to HN geeks I doubt if Elixir will ever achieve mainstream
adoption. Searching Indeed.co.uk's API by title, there are only 5 Elixir jobs
in London, compared with 445 Python and 171 Ruby. I also attended a Silicon
Roundabout jobs fair recently and was disappointed to find Elixir wasn't even
listed in the literature.

~~~
itbeho
It's still early. I've bet on the wrong horse (stack) before, so take my
comments with a grain of salt, but I know several companies that are currently
adopting Elixir/Phoenix with the same level of excitement that I recall from
the early days of Ruby/Rails. It may never be "Rails big", but there is
definitely some momentum building.

~~~
bigtunacan
When Rails arrived on the scene it was very different from everything else,
but also there weren't 1000 new languages/ frameworks popping up all at once.

I'm not implying in anyway that Elixir is bad. I just think there are too many
horses these days to know which to bet on. Elixir? Node? Go? Rust? Something
else?

~~~
vertex-four
Depends on what you're doing, really.

Rust is lovely for security-sensitive code - I'm writing a customer identity
management system with Rust+Postgres+Redis.

Go is... acceptable... for "glue" code where PHP would've been used a decade
ago and Perl before that - all the successes of it I've seen fall into that
sort of pattern.

Elixir is great when you need to think about networked, stateful systems on
the scale of a rack of machines - it provides many of the components to help
you design systems at that scale.

So... as ever, they all do quite seriously different things. I don't think
many people need to build the sorts of systems Elixir is good for - it'll
always have its niche in large-scale communications systems, though. A good
many webapps fall into the patterns that Go is good for - take user input,
munge it, send it to some backend system. A fair bit of code that drives the
foundations of what those webapps are built on will eventually be written in
Rust.

~~~
bigtunacan
I understand where they are supposed to fit, but the problem is there are so
many existing tools to fill the same needs already.

As an example Erlang and Elixir both fit essentially the same bill so it seems
to me what was already a small niche is just getting fractured.

~~~
vertex-four
Elixir is just alternative (Ruby-ish) syntax for Erlang, as much as people
have hyped it up - Erlang code can call Elixir code and vice versa with
essentially no abstraction cost. In fact, the most popular web framework for
Elixir is heavily built on Erlang code.

~~~
bigtunacan
That was my point exactly. Before if you really needed the benefits of running
on BEAM then you would have chosen Erlang. Now that subsection will fragment
between Erlang and Elixir. Today Elixir developers still will lean heavily on
the interaction of Erlang libraries until someone in Elixir-land needs
something that isn't supported so then they will re-invent that particular
wheel in Erlang and thus the cycle continues ever on.

Then as you stated yourself Go is being used in areas where PHP & Perl were
used before (as were also Ruby, Python, and Lua). So now there is just one
more fracture there. Of course this is especially interesting since Go was
supposed to be a better C or C++ then that should have placed it pretty
squarely in systems programming land, yet is has managed to gain more traction
as a glue and web services languages, ergo it ends up competing in this space
when it probably should have competed more with C/C++/Rust.

Rust; as a systems programming language still has to compete with C/C++/ObjC
in this space and considering that "all the systems" already run on these
languages that is one huge challenge to change the guard there. I'm not saying
that Rust is bad, rather just that I think it's long term outlook may actually
be rather bleaker than other languages due to these challenges. Quite frankly
D had some great concepts and improvements as a systems language over C++, but
it is still barely a blip on anyone's radar.

If an individual is just trying to learn and expand their horizons then any of
these languages are great to pick up. If, on the other hand, they are banking
their future career on one then none is a sure fire bet right now. One would
honestly still have more luck with a tried and true like Java.

Personally I enjoy picking up new languages just to see things from a
different perspective and continue learning. Heck I spend much of my free time
coding in Crystal which hasn't even hit a 1.0 launch so that adds pretty much
zero improvement to my career prospects. :)

~~~
vertex-four
I wouldn't bank a career on a language anyway, so.

However, as Elixir is just another skin for Erlang, switching from Elixir to
Erlang or LFE or whatever the next language for that runtime is will be simple
- the issue is really BEAM and OTP, as few of the skills learned around them
are cleanly transferrable to other runtimes and frameworks.

I suspect that Go will be around for a while by sheer force of inertia at this
point - we're 5 years since 1.0 now and it still seems like there's new major
projects being started in it every day.

Rust is... interesting, but I think it'll always have its niche. As much as
people like to riff on the RIIR crowd, there's actually a somewhat decent
ecosystem of reimplemented libraries and systems in Rust already, and now that
Rust code is part of Firefox it seems unlikely that Mozilla will stop
supporting it for a long time to come. It also seems to not be a zero-sum game
- rather than stealing devs from C++, it's brought a lot of developers from
"non-systems" languages into the "systems" space.

------
andy_ppp
Just as an aside how would people build something like this if they were to
use say Python and try to scale to these sort of user levels? Has anyone
succeeded? I'd say it would be quite a struggle without some seriously clever
work!

~~~
MusaTheRedGuard
Yeah pretty much impossible with python.

------
neya
Hi community, Let me share my experience with you. I'm a hardcore Rails guy
and I've been advocating and teaching Rails to the community for years.

My workflow for trying out a new language involves using the language for a
small side project and gradually would try to scale it up. So, here's my
summary, my experience of all the languages so far:

Scala - It's a vast academic language (official book is with ~700 pages) with
multiple ways of doing things and it's attractiveness for me was the JVM. It's
proven, robust and highly scalable. However, the language was not quite easy
to understand and the frameworks that I've tried (Play 2, Lift) weren't as
easy to transition to, for a Rails developer like me.

Nevertheless, I did build a simple calendar application, but it took me 2
months to learn the language and build it.

GoLang - This was my next bet, although I didn't give up on Scala completely
(I know it has its uses), I wanted something simple. I used Go and had the
same experience as I had when I used C++. It's a fine language, but, for a
simple language, I had to fight a lot with configuration to get it working for
me - (For example, it has this crazy concept of GOPATH where your project
should reside and if your project isn't there it'll keep complaining).
Nevertheless, I build my own (simple) Rails clone in GO and realized this
isn't what I was looking for. It took my about a month to conquer the language
and build my (simple) side project.

Elixir - Finally, I heard of Elixir on multiple HN Rails release threads and
decided to give it a go. I started off with Phoenix. The transition was
definitely wayy smoother from Rails, especially considering the founding
member of this language was a Rails dev. himself (the author of "devise" gem).
At first some concepts seemed different (like piping), but once I got used to
it, for me there was no looking back.

All was fine until they released Phoenix 1.3, where they introduced the
concept of contexts and (re) introduced Umbrella applications. Basically they
encourage you to break your application into smaller applications by business
function (similar to microservices) except that you can do this however you
like (unopinionated). For example, I broke down my application by business
units (Finance, Marketing, etc.). This forced me to re-think my application in
a way I never would have thought and by this time I had finished reading all 3
popular books on this topic (Domain Driven Design). I loved how the fact that
Elixir's design choices are really well suited for DDD. If you're new to DDD I
suggest you try giving it a shot, it really can force you to re-think the way
you develop software.

By the end of two weeks after being introduced to Elixir, I picked up the
language. In a month and a half, I built a complete Salesforce clone just
working on the weekends. And this includes even the UI. And I love how my
application is always blazing fast, picks up errors even before it compiles
and warns me if I'm no using a variable I defined somewhere.

P.S there IS a small learning curve involved if you're starting out fresh:

1) IF you're used to the Rails asset pipeline, you'll need to learn some new
tools like Brunch / Webpack / etc. 2) Understand about contexts & DDD
(optional) if you want to better architect your application. 3) There is no
return statement in Elixir!

As a Ruby developer, here are my thoughts:

1\. So, will I be developing with Rails again? Probably yes, for simpler
applications / API servers. 2\. Is Ruby dying? No. In fact, I can't wait for
Ruby 3.

Some drawbacks of Elixir: 1\. Relatively new, so sometimes you'll be on your
own and that's okay. 2\. Fewer libraries as compared to the Ruby eco-system.
But you can easily write your own. 3\. Fewer developers, but should be fairly
to onboard Ruby developers.

Cheers.

------
oldpond
When have you ever read, "How Acme scaled J2EE to 5M Concurrent Users"? I
became an IT architect in 1998 at IBM, the year Sun released j2ee and IBM
released Websphere. I have experienced 20 years of enterprise Java and object
oriented computing, and I was thrilled when Elixir came out. I was a mainframe
programmer before OO became all the rage, so I never really felt at home doing
objects. Functional programming feels completely natural to me though.

What I like about this article is that they shared everything they learned
with the community. Thank you for that excellent experience report.

------
grantwu
"Discord clients depend on linearizability of events"

Could this be possibly be the cause of the message reordering and dropping
that I experience when I'm on a spotty connection?

------
agentgt
I realize this is off topic but how does Discord make money? I can't figure
out their biz model (I'm not a gamer so I didn't even know about them).

------
jaequery
Anyone know if Phoenix/Elixir have something similar to Ruby's bettererror
gem? I see Phoenix has a built-in error stack trace page which looks like a
clone of bettererror but it doesn't have the real-time console inside of it.

Also, I wish they had a ORM like Sequel. These two are really what is holding
me back from going full in on Elixir. Anyone can care to comment on this?

~~~
_asummers
Ecto is hands down the best "ORM" I've used. At no point has it ever gotten in
the way of us dynamically creating queries on the fly, just following the Ecto
internals (not recommended, but totally doable) and has an extremely
expressive syntax. The team behind it has been rolling out new features and
are extremely responsive to requests on the mailing list. I encourage you to
give it a chance.

The mental shift is that "schemas" are not objects in an ORM sense, but
instead are just data, or views over data. The functions come from taking in
data or a Changeset and manipulating those, versus calling a function on a
class.

As far as errors, Elixir 1.4.5 has much better error messages, specifically
printing the args to crashes and such, and OTP 20/ Elixir 1.5 should
drastically improve Dialyzer error messages. I am not a Ruby guy, so perhaps
you can say what you feel is missing from Elixir's messages and how the Ruby
gem improves it. Also you can literally inspect running state on the fly with
the BEAM tooling, so the need for things like debuggers goes down.

~~~
denisw
Could you explain which changes to Elixir 1.5 and OTP 20 should result in
better Dialyzer error messages? I didn't find anything relevant to that in the
respective changelogs.

(For context, Dialyzer is a static analysis tool detecting type errors which
is part of the core Erlang distribution.)

~~~
_asummers
It's kind of cryptic, but the Erlang release notes say:

    
    
      OTP-14369    Application(s): compiler, dialyzer, stdlib
                   Related Id(s): PR-1367
    
                   The format of debug information that is stored in BEAM
                   files (when debug_info is used) has been changed. The
                   purpose of the change is to better support other
                   BEAM-based languages such as Elixir or LFE.
    
                   All tools included in OTP (dialyzer, debugger, cover,
                   and so on) will handle both the new format and the
                   previous format. Tools that retrieve the debug
                   information using beam_lib:chunk(Beam, [abstract_code])
                   will continue to work with both the new and old format.
                   Tools that call beam_lib:chunk(Beam, ["Abst"]) will not
                   work with the new format.
    
                   For more information, see the description of debug_info
                   in the documentation for beam_lib and the description
                   of the {debug_info,{Backend,Data}} option in the
                   documentation for compile.
    
    

There aren't specific notes in the Elixir 1.5 release notes, but given that it
is OTP20 compatible, I assume it would be able to leverage this, perhaps with
some work by Dialyxir.

[http://erlang.org/download/otp_src_20.0-rc2.readme](http://erlang.org/download/otp_src_20.0-rc2.readme)

------
zitterbewegung
Compared to slack discord is a much better service for large groups . Facebook
uses them for react.

~~~
btown
I just wish they were more professional in their communications. Service went
down hard this weekend and their response was a cat GIF. Hard to justify to
anyone in business that they should take it seriously. A shame when their tech
is such a great match for professional collaboration, and all it would take to
grow it organically would be _less_ marketing copy not more.

~~~
notnight
We take outages seriously internally, but users generally don't care about the
nitty-gritty. For a full, detailed postmortem on that outage you can check out
our status site:
[https://status.discordapp.com/incidents/ywdwttd6b0hg](https://status.discordapp.com/incidents/ywdwttd6b0hg)

------
concatime
Sad to see some people taking raw and insignificant benchmarks to evaluate a
language[0].

[0]
[https://news.ycombinator.com/item?id=14479757](https://news.ycombinator.com/item?id=14479757)

------
framp
Really lovely post!

I wonder how Cloud Haskell would fare in such a scenario

------
brightball
I so appreciate write ups that get into details of microsecond size
performance gains at that scale. It's a huge help for the community.

~~~
_asummers
Erlang and Elixir measure in microseconds, so it'd be them throwing away
information if they did otherwise!

------
dandare
What is the business model behind Discord? They boast about being free
multiple times, how do they make money? Or plan to make money?

~~~
satsuma
They currently have a Nitro service that's $5/month for little features, but I
don't know how far that takes them towards profitability.

------
KrishnaHarish
Scale!

------
KrishnaHarish
What is Discord and Elixir?

------
marlokk
"How Discord Scaled Elixir to 5M Concurrent Users"

 _click link_

[Error 504 Gateway time-out]

only on Hacker News

------
orliesaurus
Unlike Discord's design team who seem to just copy all of Slack's designs and
assets, the Engineering team seems to have their shit together, it is
delightful to read your Elixir blogposts. Good job!

~~~
orliesaurus
yeah go on with the downvotes, you know everyone's thinking what I said!
Copying your way to success is a strategy and it's not the first time either
way :)

------
khanan
Problem is that Discord sucks since it does not have a dedicated server.
Sorry, move along.

~~~
vultour
That's actually why it doesn't suck for the vast majority. Not everyone wants
to pay $ every month so they could have their own voice / chat server.

