
Real time bidding in Erlang (2012) - dorsatum
http://ferd.ca/rtb-where-erlang-blooms.html
======
BuckRogers
I'm primarily a Python programmer, and while a great gen-purpose language,
Erlang has been what I've found most appealing as best-in-class serverside.
From the reading and research I've done, it seems like an excellent next-step.

On the implementation side, like Python it's weaker with CPU heavy tasks, but
a normal program maintains low latency, unbeatable uptime, hot code swaps. For
CPU heavy work, I presume most Erlangers do what we all do- C (or maybe Rust)
libraries. On the language side, it's terse like Python and a small language.

I fail to see why this shouldn't be the server language of choice. Other than
tougher recruiting. Learning about FP would be interesting too.

I've dabbled in Go and Node.js, and keep up on developments with Rust, but
keep coming back to Erlang as what I'd like to deep dive on next.

~~~
bad_user
For CPU intensive operations, the JVM is much better than Erlang or Python and
with the new breed of languages I feel more productive with it than I've ever
felt when using Python. Speaking of RTB platforms, the JVM is the other high-
level server-side platform that can handle the high loads.

Erlang sounds nice, but I don't like platforms that are too specialized for
certain use-cases. I also never understood the "C libraries" argument - this
is what made Python non-portable, this is what will kill it.

~~~
cmrdporcupine
I fell into doing RTB (buyer side) with the JVM (Java/Scala) many moons ago,
and this was a mistake. The GC isn't cooperative enough to meet the expected
latencies in the 95th percentile of requests and above. There were erratic
pauses, even after extensive GC tuning. I wouldn't do it again.

After coming to Google and working on the other side of the RTB process for a
while.. well, I just wouldn't do it in anything other than something like C++
or Rust -- a systems level language where I can fine tune everything and not
be interrupted by a GC. I just wouldn't mess around -- not only is it
important to have low latencies, it's important to have _consistently_ low
latencies.

~~~
kasey_junk
I think that using a manual memory language is fine if that is the engineering
decision you want to make.

I'd point out that people can and do use GC'd languages in environments that
are dramatically more latency sensitive than RTB. For that matter, in latency
sensitive applications allocation/deallocation after startup is a no-no, so GC
tends to not be the major reason not to use the JVM (memory layout/unfettered
access to system calls/etc tend to drive that decision).

~~~
cmrdporcupine
From my experience, Java doesn't really offer a true realtime friendly non-
blocking collector which can offer consistent latencies. There are a million
flags you can do set the GC in Java, but nothing is going to stop GC pauses
from happening entirely.

95% of the time the latency is acceptable. The problem is that under heavy
throughput the latency spikes periodically. I wish I still had some of my old
graphs to show.

I just wouldn't do it again, the time spent futzing trying to tune the JVM to
avoid this would have been better spent on writing code in a systems level
language where allocation is predictable.

I was a big booster of the JVM as a mature platform for a lot of things. But
for this purpose... nope.

~~~
kasey_junk
Understood.

My point is that most truly latency sensitive applications, in GC'd languages
or not, don't do any allocation/deallocation on the hot path. Memory
management is just too slow.

So GC twiddling is a red herring in those environments, though using manual
memory techniques on the JVM may obviate the major reasons to use it in your
particular cases. Lots of teams make the engineering decision the other way
and use the JVM in usages that require more consistent latencies than RTB
requires.

------
kator
FYI Current peaks for RTB are about 2.5m qps world-wide today. Some exchanges
are seeing as many as 100B impressions per day with mobile and display
impressions combined.

RTB is an amazing challenge and way too much fun for those inclined to bang
their heads against all sorts of problems you never run across in the
"regular" world.

I've done some work on these systems, one of my favorite challenges is writing
tools that segment impressions in real time, you have only 10ms to respond
because you're augmenting the bid before others can bid. A RTB bidder
typically has about 100ms to respond. When you're shooting for 10ms at 2.5m
qps peaks you have to think about everything from kernel versions, network
settings, pinning to processors, avoiding GC, logging to disk is too slow etc.

~~~
kasey_junk
As someone close to the RTB world observing from the outside, what amazes me
is not the throughput or latency numbers but that it all happens on technology
stacks that seem inappropriate for the problem. Everything from broadcast
messages over TCP/IP or request/reply semantics for messages with no useful
replies, to having to scale connections even though the number of participants
is fixed. Its a whole world built on http that doesn't seem like it should be.
That you can scale qps in that world while maintaining a semblance of response
time is really something.

As a complete tangent, as someone who has worked in environments that had at
least an order of magnitude tighter latency requirements than the RTB world,
and who sees this misconception a lot, logging to disk on a modern OS is not
"too slow". There may be very valid reasons not to log to disk, most obviously
operational concerns, but latency/throughput aren't one of them.

------
getsat
Were these guys a very small shop? Any info on which exchanges they bidded on?
The RTB company I worked at was considered "small" and we were handling around
300,000 requests/second with 15ms max time per request spent in our systems
between just Google and Facebook's exchanges.

~~~
ddorian43
How do you do load-balancing there? Meaning, can 1 box of nginx/haproxy handle
that many requests ?

~~~
chupy
Even if a single box would be capable to handle that many requests, you should
never have just one box to be the main entry point to your server (failsafe).

I also suspect he refers to 300k distributed across multiple datacenters.

~~~
getsat
Yes, you are correct. The 300k was total across all datacenters. 85% of that
was in the US, 10% in Europe, and 5% in Asia.

