
Lessons from Building and Scaling Reddit's Ad Serving Platform with Go - ngaut
https://about.sourcegraph.com/go/gophercon-2018-from-prototype-to-production-lessons-from-building-and/
======
katzgrau
As the founder of a SAAS ad serving company, I can say that as slick as (I
think) my platform is, it's not a problem I would want to tackle again. There
will be a large volume of technical challenges and required features to
support.

It'd be like writing Apache or Nginx from scratch. It seems simple on the
surface and then you wake up to all kinds of inconvenient and unexpected
technical challenges.

I'm not sure Reddit's engineering resources are best allocated to building ad
serving infrastructure and software. It was going to be an ongoing headache
from the start, and worse, it sounds like it was build with a language and
stack that was fairly new to the team.

My prediction is that it will become a high cost and difficult to maintain
project that will also prove difficult to hire for when this engineering team
gets tired of working on the (currently new and fun, soon to be old and scary)
adserver and moves on to other jobs.

My point being that some challenges are best left to third parties who make
solving some problem its full time job. I truly do not say that out of bias,
just experience building and managing such a complex system.

~~~
grey-area
They were already using a third party service. It was slow and opaque, so they
ditched it.

At the scale of reddit, and because this is _how they make their money_ , this
is clearly something they want in house and something it would be a
competitive advantage to control, if nothing else it gives them a much clearer
conduit to their advertisers, and much greater control over things like ad
selection.

~~~
supernovae
The problem with Reddit ads isn't their tech, it's their people. The ad
approval process is a farce - and human-managed. Oh, and those humans are
assholes. You have to go through many denials - for no reason - submit a
ticket to get another approver and approved - for no reason stated. It was the
most upsetting process I've ever dealt with.

~~~
pjc50
It seems odd that there's a stringent ad approval process when the site
proudly lets Redditors post any garbage they want. I'd have expected anything
SFW and not obviously fraudulent to be allowed.

~~~
MoronInAHurry
It's obviously not very stringent. A lot of the ads they run lately are
outright scams, including even advertising Scientology:
[https://i.redd.it/6cd1ricqc8j11.jpg](https://i.redd.it/6cd1ricqc8j11.jpg)

------
bflesch
I advertised on reddit for our startup roughly one year ago and it was the
worst ad management experience I have ever witnessed. For the size and age of
reddit the advertiser backend was a real shame. The kind invoices they provide
you is laughable, it feels like the whole backend was built by amateurs.

On top of that, the traffic we received from reddit was very bad for our
e-sports/gaming-related website.

~~~
jamiequint
Hey bflesch, a year ago we weren’t even running our own ad server and had just
released an early v1 of our new ads management interface. It probably felt
like the “worst ad management experience [you] have ever witnessed” because
the other major ad management platforms have multiple years and hundreds of
engineers/designers on us :) We’ve made substantial progress since then, I’d
encourage you to not judge us too early and give it another shot, the platform
and products are improving rapidly.

p.s. If any engineers or PMs are interested in helping us build out a world
class, high-scale ad platform please get in touch (email in profile). We are
hiring for senior and above positions in NY and SF.

~~~
supernovae
If you work in the ad group, my advice is to fix the ad approval process. I've
never had such a disappointing experience in my life. Seems like the approvers
have editorial control based on personal preference over any objective
evaluation. Since reddit is all about the ebs and flows of whats hot and whats
not, sure is a shame your advertising platform is a joke at monetizing that
because of the terrible approval process.

~~~
jamiequint
Thanks for the feedback! We are completely revamping the approval process over
the next few months so I am expecting substantial improvement in this area.

------
pjc50
A means for rapidly selecting and loading an advert which will then be blocked
by the users' browser.

(Although even without adblock, are there actually all that many overt ads on
Reddit, other than the sidebar ad which nearly always is an ad for a different
subreddit?)

~~~
ufo
There are also promoted posts intermingled with user generated content. They
are specially visible in the mobile app

------
minxomat
> 12M posts per month, 2B posts per month

Curious, which one is it? I suspect the former.

~~~
benjaminjackman
Maybe 12M articles / self posts, 2B comments?

~~~
overcast
Nah, no way on average an article gets 166 comments. That must be page views
or something.

~~~
RandallBrown
Why not? Most front page posts have several thousand posts, which seems like
it might be enough to offset the 0 comments that most submissions will get.

~~~
simias
The front page is obviously not representative in any way of 99% of the posts
made on the platform. Look at the slew of posts made on reddit:
[https://www.reddit.com/r/all/new/](https://www.reddit.com/r/all/new/)

Most will be lucking to reach 10 comments.

------
jdright
> Thrift has been around since 2007, and is one of the first rpc protocols.

Done reading there.

Not sure if bad written or just not knowledgeable, please do your research.

~~~
orf
What's more likely: there is an error in transcribing or the developers of one
of the worlds largest websites think the first RPC protocol was written in
2007 (over, say, REST, which their entire API is built on).

~~~
Twirrim
Given the continual attitudes seen within tech... sadly, the latter.

------
cdnsteve
Go-kit looks pretty nice, anyone else have experience with it?

~~~
avdempsey
It can be surprising how little code you're actually using from the Go-Kit
project, and how it's really more of a structure for your app.

It will not save you time typing: each route can have a transport request
decoder and response encoder (encoder's are often shared), a request and
response type, the actual business logic, a function that takes the request
struct, applies it to the business logic function and returns a response
object, a function that takes the previous function and connects it with the
transport machinery (building the actual endpoint), and possibly middleware
implementations.

However, by teasing all of these things apart from the very beginning it's
very easy to come back later and add a circuit-breaker, or (as the article
says) switch your transport from HTTP to grpc (or support both).

------
acroback
Half of the times it just sits there blank. Sometimes it loads in 2-3 seconds.
Rest of the times it loads within 1 sec. Rarely, it also shows a default ad( I
work in same industry, hence the term).

How is that great?

PS: Not talking about native ads.

------
tablethnuser
Enrichment Service, aka surveillance capitalism center

------
HyperTalk2
Ads on reddit are incredibly deceptive. I'm not sure why advertisers tolerate
it. 98% of the ad clicks on mobile are surely fraudulent through being
intentionally accidental. The ads are designed to look exactly the same as
real posts at a glance, with a username, post date, upvote/downvote arrows,
deceptive title meant to make it look like a real post, comment section, share
link, and image thumbnail. Reddit users are all conditioned to believe that
they can tap a thumbnail and be shown a larger preview while staying on the
site. On these ads, it takes them to the advertiser's site instead.

[https://i.imgur.com/Vm68wLP.jpg](https://i.imgur.com/Vm68wLP.jpg)

~~~
bduerst
Have you actually run ads on reddit, or are you saying this as a user?

If so, what stats have seen (impressions, clicks, conversions, etc.) for your
ads?

~~~
tapirl
I agree with OP. My 99% clicks on reddit mobile ad are unintended. It is very
annoying, for if I press the back button to early, then the firefox browser
will back too the page previous to the page containing the ad I clicked. So
generally, I will wait the ad fully loaded then press the back button.

~~~
ma2rten
You and OP might not be typical users.

------
auslander
> Enrichment service is responsible for getting more data and information
> about the request, user ...

Where do you get info about user and whats in it?

------
somethingroma
Cool. Maybe these guys should get in touch with the redesign guys and get them
fired. Also fire any executive that thought it was a good idea.

Because all the hard work to serve ads more efficiently goes to waste when the
site is a mess and unusable.

~~~
untog
What are the actual complaints with the new design? I've been using it on and
off and I can't say I've noticed any major issues. Though I'll admit most of
my browsing is one on the mobile app these days.

~~~
bmurphy1976
I get obnoxious multi-second lag when entering text into a comment box on my
phone, even on a thread that only has a couple comments. It's... befuddling
how this made it to production and remains. And no, I don't want to use your
stupid app. I'm not a fan of loading 1000's of shitty apps on my phone when a
simple web page is sufficient.

~~~
culot
Somehow the scrolling is also broken on their mobile site, and that can't be
by accident. Is it some sort of javascript middle finger to urge users to move
to their app? I've never experienced scrolling issues like that on any other
site, ever.

That they make their mobile site intentionally unfriendly and frustrating is
infuriating.

------
sonnyblarney
Brilliant, thanks for writing.

------
freecodyx
i will add to not use the default http.Client/Server,

poor performance, instead use fasthttp

~~~
acln
I like my HTTP package to actually speak as good an approximation of HTTP as
it can, so fasthttp wouldn't be my first option.

Furthermore, the HTTP stack doesn't show up on most web servers' profiles.
net/http is better understood by the community, interoperates with libraries
better (e.g. golang.org/x/oauth2), and is better supported.

Use net/http.

~~~
ErikDub
The main issue with net/http is that it generates a lot of garbage which is
definitely going to show up in your profile if you're handling a lot of
requests per second.

~~~
sagichmal
Allocation pressure imposed by net/http has never been a top K source of
profile CPU burn in any service I’ve ever written, and I’ve written plenty of
high-performance, high-QPS services. fasthttp is rarely if ever a good idea.

