
How we built Uber Engineering's highest query-per-second service using Go (2016) - godelmachine
https://eng.uber.com/go-geofence/
======
woeirua
This is a very inefficient implementation. Really, just poor quality work
overall, as anyone with even a basic understanding of spatial indexing would
know that an R-tree would be many times faster, as illustrated here:
[https://medium.com/@buckhx/unwinding-uber-s-most-
efficient-s...](https://medium.com/@buckhx/unwinding-uber-s-most-efficient-
service-406413c5871d)

~~~
dang
This comment is unacceptable on Hacker News, and I'd like to explain in detail
why. Please don't think I'm picking on you personally; we all react like this
sometimes. Rather, I want to drive this point home to the community. It's
important for discussion quality here!

\-----

Please omit swipes like "Really, just poor quality work" and "anyone with even
a basic understanding" from your posts to HN. It's great to add relevant
information, such as an applicable data structure and a link to a good article
on the same topic. But it's not great to put others and their work down, and
HN has at least two guidelines that ask you not to:

 _When disagreeing, please reply to the argument instead of calling names.
"That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."_

 _Please don 't post shallow dismissals, especially of other people's work. A
good critical comment teaches us something._

Actually, a third guideline is relevant too:

 _Please respond to the strongest plausible interpretation of what someone
says, not a weaker one that 's easier to criticize._

The strongest plausible interpretation is not that the engineers lacked a
basic understanding of relevant CS—or, for that matter, how to use a search
engine, since R-trees would pop up nearly any place you searched about this
stuff. The strongest plausible interpretation is that they had some other
reason for choosing the implementation they did. For example, perhaps it was
efficient enough and they were smartly choosing not to build something more
complicated than needed.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

Edit: and it turns out that the article explains why they didn't use R-trees.

~~~
deminature
It's disappointing to see a highly-upvoted comment from someone that didn't
read the article. The article discusses explicitly why R-trees were not used:

>Instead of indexing the geofences using R-tree or the complicated S2, we
chose a simpler route based on the observation that Uber’s business model is
city-centric; the business rules and the geofences used to define them are
typically associated with a city. This allows us to organize the geofences
into a two-level hierarchy where the first level is the city geofences
(geofences defining city boundaries), and the second level is the geofences
within each city.

~~~
icholy
R-tree will easily outperform that approach.

~~~
xtomus
You can either ship a 2 dimensional linear search approach today or ship an R
tree based approach in 2 weeks time. I know which one in choosing if they both
meet all of the requirements

~~~
woeirua
Just about every modern language has a well established R-tree library...

------
hobofan
> High performance in throughput and latency. In our main data center serving
> non-China traffic alone, this service handled a peak load of 170k QPS with
> 40 machines running at 35% CPU usage on NYE 2015. The response time was < 5
> ms at 95th percentile, and < 50 ms at the 99th percentile.

> Geofence lookups are required on every request from Uber’s mobile apps and
> must quickly (99th percentile < 100 milliseconds) answer

For a 100ms total budget, a ~50ms 99th percentile for a single microservice
doesn't sound like something to boast about.

~~~
foobarian
If that's 40 bare metal machines... assuming 60ish HT cores, and 35% CPU...
that's not that earth shattering. We do that in Java :)

~~~
asdfman123
Yeah, boring programming languages are criminally underrated. Fro instance:

> High developer productivity. Go typically takes just a few days for a C++,
> Java or Node.js developer to learn, and the code is easy to maintain.
> (Thanks to static typing, no more guessing and unpleasant surprises).

This is why the rest of the world doesn't use JS for everything.

> There is a lot of momentum behind Go at Uber, so if you’re passionate about
> Go as an expert or a beginner, we are hiring Go developers. Oh, the places
> you’ll Go!

If they did it in Java, they wouldn't have to recruit for programmers quite so
hard -- they could pull them out of a hat, and fairly easily get a few wizards
with two decades' worth of experience in it.

~~~
sidlls
They also would have to sift through some really terrible programmers who have
fully bought into the worst practices of enterprise-y OO development.

~~~
asdfman123
Is that really true? Honest question. Or is that just bias against things that
are not new and fancy?

~~~
sidlls
It's really true, in my experience. I managed a team in a company that used
Spring Boot for services and batch jobs and Python for data and analytics
(including ML). My team had its feet in both worlds so I had to hire for that.
My experience trying to find programmers who could code and engineer first and
not be slaves to the Spring framework was quite terrible in this context.

Another thing of note: the python side of the team could regularly move across
and help with bugs or issues in our services and jobs, but the reverse was
rarely true. In the 2+ years I had that team there was one person hired for
the services side who could do it, and he preferred Go to either Java or
Python.

~~~
shantly
I'm not sure what causes it—and I'm not even sure it's a bad thing, certainly
it must not be hurting these folks careers too much, and maybe it's even
helping them—but there's a certain path for developers that ends up leaving
them helpless outside whatever narrow ecosystem they've grabbed onto. Usually
it's Java or some Microsoft thing.

As someone who very much did not develop (as a person/programmer) that way, it
seems baffling from the outside, but there's this whole world of programmers
who work like that until they retire or are promoted into management. It seems
bizarre to me but it must be working for them. Hell, they might even be the
_majority_ of all programmers. Bigcos, particularly the non-tech ones that
nonetheless employ lots of developers, are full of such people.

~~~
asdfman123
Well, I think the difference between them and Silicon Valley programmers is
that they don't constantly learn new technologies outside of work. It's just a
40-hour-a-week job.

I try to learn new things, but I don't do much programming outside of work. I
could learn Node.js, for instance... but I've got other things I want to do.

Is that kind of what you mean?

~~~
shantly
Nah, I'm a 40-hour-a-weeker myself, I'm not sure it's strongly correlated with
that.

Some folks, you say "do you think you can do [thing you're basically familiar
with] in [language and platform you're not]?" and get a "yeah, probably lemme
check it out... cool, compiler and language support's installed, see a couple
tutorials here, I'll poke around and get something doing [subset of thing you
need] then get back to you on timeline" and very likely it works out fine.

Others, you get "uh, I do (Java, .net), I don't... I don't understand what
this is, is it a JVM language? Is there a jar for it?" and it's not really
worth pushing any harder. And maybe some of them are deflecting because they
just can't be bothered (mad respect), but most seem genuinely nervous and out-
of-their-element at the mere suggestion of doing anything but their Java or
.net thing (usually it's one of those) they're used to.

Though, again, the latter seem to get along just fine, career-wise. It's just
a different sort of path, I guess. Seems really weird to me, but there it is.

------
scarejunba
There's a famous 'rebuttal' post to this here
[https://medium.com/@buckhx/unwinding-uber-s-most-
efficient-s...](https://medium.com/@buckhx/unwinding-uber-s-most-efficient-
service-406413c5871d)

~~~
merb
well basically ubers blog posts are really low quality and from the outside
their engineering descisions are kinda vague.

like the switch from postgres to mysql.

I'm still clueless how you can have so much money and one of the biggest
engineering team, but still can't correctly engineer your stuff. I mean,
everybody makes wrong decisions or errors in production code.

~~~
Slartie
From what I gathered off of several HN comments of Uber insiders, the size of
their engineering team is actually more likely to be one of the causes for bad
engineering decisions. They seem to have too many engineers running around for
too little actual work to be done, which results in those engineers coming up
with stuff to do to keep them busy (and of course to ensure they appear to be
busy and worth their money to their superiors).

One of the best ways to create yourself some work is to not choose an already-
proven path to a solution, but invent a new one just for the sake of inventing
a new one. Of course that's not how this kind of doing is justified - the
justification is usually "the proven path does not scale to our needs" or "by
using a special approach adapted to our needs we can be more efficient" or
"the proven path is too complex, we can get by with something simpler and
easier to maintain". Which might actually all be proper justifications, it's
just that you should have some hard proof for these statements, like benchmark
results of a comparison of different approaches. That part often gets skipped,
which is actually ironic, because doing extensive evaluation and benchmarking
and implementing different approaches first before choosing one for production
actually serves quite well to create even more work to do.

~~~
asdfman123
Also, it's resume-driven development. Five years later they want to be able to
be the guy who can say, "Yeah, I was the original designer behind Kafka," but
substituting in a brand-new Uber technology.

------
dang
Related from 2018:
[https://news.ycombinator.com/item?id=16093192](https://news.ycombinator.com/item?id=16093192)
and
[https://news.ycombinator.com/item?id=16084090](https://news.ycombinator.com/item?id=16084090).

Discussed at the time:
[https://news.ycombinator.com/item?id=11205776](https://news.ycombinator.com/item?id=11205776).

------
cryptozeus
I get all the criticism in comments but imo kudos to the team for going after
the problem with new set of eyes and approaching in new language. At the end
of the day you are under budget and have scalable fast microservice.

------
paggle
"For each lookup, we first find the desired city with a linear scan of all the
city geofences, and then find the containing geofences within that city with
another linear scan."

Why wouldn't this be dogshit slow if they used all of the city geofences at
once? I would think that first they would scan the country geofences, then the
province geofences, then the city geofences, etc...

~~~
mbo
It's almost like they could have used some sort of specialised tree data
structure where each tree node was assigned to some spatial region on the map,
where successive levels were assigned to increasingly smaller regions.

------
rayvy
I think this write up is very fair for a solid engineering team. Is it
groundbreaking and eye opening? Absolutely not, I’d say the most “hmm I didn’t
know that” part of the entire thing was the part about R-Trees and S2. Is that
bad? Absolutely not. These guys did the work, logged their performance and are
sharing their story.

However, and I believe this is where the animosity in the comments is coming
from - given the elitist (for lack of a better term) attitude of these
engineering types at these orgs (think of the poster children of the Valley),
this is pretty...lacking. I mean, the part about using the Read/Write lock on
the second attempt and instead trying to go with an installed package just
screams Node.js, and honestly made me chuckle. I guess Leetcoding and
Production Engineering really are different things. I genuinely expected more.

~~~
0xEFF
> I guess Leetcoding and Production Engineering really are different things.

This is key point.

They delivered value to the business. That's the only thing that matters.

~~~
privateprofile
> They delivered value to the business. That's the only thing that matters.

Except this is an engineering blog post, so the engineering part actually
matters.

And, as demonstrated in the article [1] linked around this discussion, there
were better engineering approaches that would have been even "better for
business", as being more efficient means lower costs per transaction and/or
higher throughput.

[1] [https://medium.com/@buckhx/unwinding-uber-s-most-
efficient-s...](https://medium.com/@buckhx/unwinding-uber-s-most-efficient-
service-406413c5871d)

~~~
elbear
It matters for engineers reading the article. It doesn't matter for the
business. What they delivered was good enough. Could they have delivered a
better solutions? Yes.

Should they have searched for a better solution instead of implementing the
one they found? They could have spent some time researching, but you can
always miss something. It's better to err on the side of delivering something
now with a not-so-good solution than constantly searching for a better one.

I say this as software developer who is obsessed with efficiency. I'm starting
to turn around and focus more on just delivering.

~~~
SilasX
>It matters for engineers reading the article. It doesn't matter for the
business. What they delivered was good enough. Could they have delivered a
better solutions? Yes.

The entire reason they posted the article is to brag about having accomplished
intelligently. It's relevant if their approach was actually not so
intelligent.

"We encountered a standard problem and applied standard solutions" is like
"dog bites man". It's not what they were trying to say with the blog post.

>Should they have searched for a better solution instead of implementing the
one they found? They could have spent some time researching, but you can
always miss something. [...] I'm starting to turn around and focus more on
just delivering.

I think the critics point is that the efficient way was probably also cheaper
than what they did, and would take the same time to implement, and have lower
recurring costs. It would have just been a matter of using off-the-shelf tools
and not reinventing the wheel because that wheel is "complicated" and
"obviously our case is special". (Someone did benchmarks, and their case is
not special.)

You're right, there is a danger to what-if-ing everything and being stuck in
decision paralysis. But the clear subtext is that they merit some kind of
admiration for how well they did. If that subtext is wrong, it is worth
pointing out.

------
kuharich
Prior discussion:
[https://news.ycombinator.com/item?id=11205776](https://news.ycombinator.com/item?id=11205776)

------
gunta
it's sad to see these "moved from X language to Y language and it became
faster!" every-time.

If the CPU is the hugest bottleneck, the best answer is not to optimize the
algorithm by going to a lower level language, but rather to invest in a
different architecture like GPGPU or FPGA.

For example, this paper shows a significant speed-up for PIP( Polygon in
Point) algorithm, going from 15hs (CPU) to a mere 11sec (GPU) in task load-
time.
[https://pdfs.semanticscholar.org/1e51/e3c681e1afc908a41ac253...](https://pdfs.semanticscholar.org/1e51/e3c681e1afc908a41ac253b7ef72f3285d1f.pdf)

~~~
justincormack
That is high throughput design, not a low latency design, which is what they
are optimising for here. It is a very different design space.

------
PunchTornado
To all the people "why not do it in Java / Java could have done it".

Why not do it in a different language? Having Go and Node around is awesome
because we can change the language. Don't you get bored to use just a single
language? I'd get so so so bored if I didn't change languages every year. Glad
that uber offers Go jobs.

~~~
jryan49
Usually when you're trying to ship something, the more boring (predictable)
parts the better.

~~~
PunchTornado
but is that the only goal? or you shouldn't enjoy the life? I find that I
enjoy my life more as a programmer when I change things.

