
5M Bid request/s, 2ms max response time – The Road to Damascus - sanjayts
https://github.com/eloraiby/fs-pacer/blob/master/fs-pacer.md
======
mej10
Interested to know what Rust was missing. I built an ad exchange last year and
it has been great. I have been using nightly builds, mostly for access to
async/await, and it has been very fast and stable.

I have had to submit a few pull requests to various projects along the way,
but didn't find the ecosystem prohibitively lacking.

~~~
coolsunglasses
Would you mind sharing what libraries you are using?

~~~
mej10
actix, cdrs, rusoto, cadence, diesel, futures/tokio, serde, chashmap, slog,
postgres, criterion, r2d2, chrono

That isn't all of them but those are the main ones.

~~~
coolsunglasses
Thanks, that helps!

------
sebastienros
TechEmpower's Plaintext scenario is currently limited at 7M RPS due to network
limits, though it uses a 10Gb NIC. Knowing that the Plaintext scenario is a
very simple HTTP request (standard headers) that returns "Hello World!", how
close to network saturation are you with 5M in this case with only "2 Gigabit
Ethernet cards"?

------
nullwasamistake
Did you consider Vert.X? it's built on Netty and has it's own Linux epoll
driver, async, and fiber support. It's impossible to know if it would be
faster, but likely comparable and way less work than rolling your own.

In techempower benchmarks it exceeds 2 million http requests/second and it's a
full REST framework.

And if you use the fiber support through Quasar you can pretend most things
are normal blocking code.

Have to tried it or is this a case of NIH?

------
reilly3000
I spent a few cycles in media buying and later in sell-side ad tech. Please do
say what you will about advertising and its effects on the web, but I will say
this: it is a world of fascinating tech. As a buyer I experienced janky pacing
all of the time across various platforms, because this is a HARD problem. We
had to manually adjust campaigns on a daily basis to ensure pacing worked
properly. It was common to stop a campaign and overspend by hundreds of
dollars while all of the caching spun down.

I'm fascinated to see they are running that all on a single node. Its a
massive amount of state aggregated from billions of events that needs to be
served at extremely low latency, but couldn't it be partitioned somehow???
Google Fi/Spanner and BigTable have certainly been developed to support these
issues. I've been trying to dig up what infrastructure powers Google AdX, but
I haven't found anything. AdWords seems to be tied to Spanner, but AdX is/was
an entirely different beast. In any case I'm quite certain that it isn't
running pacing on a single, gigantic node.

~~~
endymi0n
As an anecdotal data point, I once configured a test campaign on Doubleclick
Bid Manager (now Google DV360) about two years ago that I needed some quick
exposure on. So I set a budget cap of 100$ just for safety and didn‘t do any
targeting, so I was effectively bidding on half the worlds‘ ad inventory. What
I didn‘t check or notice was that pacing wasn‘t set to even, but to Flight
ASAP.

Suffice to say, I spent 730$ within _seconds_, so fast actually Googles
systems couldn‘t even switch off fast enough to prevent 7,3x overspend, and
the only thing that prevented stupid me from a five digit spend was probably
choosing an unusual ad size.

Fascinating stuff indeed :)

------
ggregoire
Glad to see this kind of stuff written with a strongly typed language instead
of Python.

~~~
Guthur
Why, they wasted months evaluating with an obsession for statically typed
languages that have so far not produced anything more quickly or markedly
better than that which others are producing with less pedantic languages.

~~~
mhh__
Static typing should be viewed as a tool for long term prosperity/health of a
codebase rather than competing in the short term with a Python/s (etc.) -
Modern one's can, but static typing isn't always obviously useful until it is.

------
w3clan
Is this complete note? You didn't mention what you end up using.

Is it golang or pony or F$? CoreFX mention in the end confused me more.

~~~
rkallos
I work with Wael. Development is still ongoing. One implementation uses
Golang, the other uses F# with a library that wraps libuv for faster network
performance. Pony was used to write the stress-testing client for both
implementations.

~~~
deforciant
Do you use libuv on go side as well?

~~~
Thaxll
You don't want to use C lib in Go for performance reason.

------
csdreamer7
Curious if there are numbers for other languages in high performance
applications.

I am learning Clojure so I would like to know if anyone knows of the highest
performant applications written in it.

------
eliasson
> (scala? That's another story for another day)

This makes me curious - was it the language or the runtime characteristics?

------
Nextgrid
It’s kind of sad that all this engineering effort was spent to essentially
make the internet a worse place for everyone and waste users’ time and
attention.

Imagine if a crime syndicate would brag about their efforts to make their
worldwide criminal activities more efficient.

~~~
packetslave
Do you legitimately think the world would be a better place if gmail, youtube,
flickr, reddit, EVERY search engine, and basically every web content site
disappeared?

Because that's what happens if you don't have web advertising. Free things
disappear without revenue.

Or maybe you'd prefer to go back to the days of randomly-targeted or "PUNCH
THE MONKEY" ads. Because THAT'S what happens without ad auctions and
targeting.

The reality is: advertisers and ad-supported sites WANT to show you a relevant
ad that you're likely to click (modulo obvious bad actors). That's how they
get paid. Anything else is, by definition, "[wasting] users' time and
attention."

~~~
dooglius
Everything you describe could exist as a paid service, and I suspect would be
much higher quality that way.

~~~
wolco
A paid youtube? Not sure they would be worth it without the endless amounts of
free content by free users.

Facebook? There are plenty of sites like facebook without users. Charge for an
account.. reduce the userbase, the platform loses value without the free
users.

~~~
Nextgrid
A paid Facebook means it would be designed to actually serve you (instead of
wasting your time) and people would use it to connect with friends instead of
liking endless shitposts.

~~~
wolco
All your friends or anyone you want to contact would need to be paying
customer.

If it is just friends why waste money on facebook just to talk. Most would
rather spend that money on a more focused community/hobby like gaming and
connect ingame.

------
llamataboot
The Best Minds of My Generation Are Thinking About How To Make People Click
Ads (or serve them efficently)

~~~
stingraycharles
Take a look at finance / trading. Things are even worse over there, but they
don’t blog about it.

~~~
llamataboot
Used to be a semi-professional poker player, which seems to have a high
crossover with finance/trading types for obvious reasons. I enjoyed the
debates, but sometimes it was a bit overwhelming how everyone had convinced
themselves that they were magically providing a ton of value to the world and
that's why they were filthy rich. Also the massive intelligence that seemed to
only work when applied in one domain. Didn't mind taking their money, but
never pretended I was providing society a useful function while doing so. :)

