
Why Google needed to build a graph serving system - pplonski86
https://blog.dgraph.io/post/why-google-needed-graph-serving-system/
======
mrjn
Hi all, author of Dgraph and the article here.

Glad to see this post trending on HN. I'm around if you want to ask any
questions, and will share whatever I can about my experience with graphs and
the decisions at Google.

Since leaving Google, I've built Dgraph [1]. It is open source, designed for
horizontal scalability, ACID transactions and provides incredible performance
for both reads and writes. And of course, it solves the join-depth problem as
explained in the post. So, do check it out!

And do remember to give us a star on GitHub [1]. Your love helps other open
source users give Dgraph a real shot in their companies with similar
leadership struggles as I experienced at Google.

[1]: [https://github.com/dgraph-io/dgraph](https://github.com/dgraph-
io/dgraph)

~~~
throwawaygoog10
I'm sorry your efforts failed internally. Our infrastructure is somewhat
ossified these days: the new and exotic are not well accepted. Other than
Spanner (which is still working to replace Bigtable), I can't think of a ton
of really novel infrastructure that exists now and didn't when you were
around. I mean, we don't even do a lot of generic distributed graph processing
anymore. Pregel is dead, long live processing everything on a single task with
lots of RAM.

I suspect your project would have been really powerful had you gotten the
support you needed, but without a (6) or a (7) next to your name it's really
hard to convince people of that. I know a number of PAs that would benefit now
from structuring their problems in a graph store with arbitrary-depth joins
and transactions. I work on one of those and cringe at some of the solutions
we've made.

We've forgotten what it's like to need novel solutions to performance
problems. Instead, we have services we just throw more RAM at. Ganpati is over
128GB (it might even be more) of RAM now, I suspect a solution like dgraph
could solve its problems much more efficiently not to mention scalably.

Good on you for taking your ideas to market. I'm excited to see how your
solution evolves.

~~~
mrjn
One of the TLs was a T7. Got overruled by T8 :-).

As Google is growing, there are many T7s around. So, you really need T8 and
above to actually get something out of the door, when it comes to impacting
web search.

P.S. Dgraph is Apache license, written in Go/Protobufs/Grpc. Google might have
an appetite for that, worth pushing internally. [https://github.com/dgraph-
io/dgraph](https://github.com/dgraph-io/dgraph)

~~~
biomcgary
My understanding is that by default, dgraph query responses are in json. Is it
possible to get gRPC responses instead?

~~~
mrjn
Dgraph exposes both HTTP endpoint and gRPC endpoint. All our official clients
use the gRPC endpoint, including Go, Java, NodeJS client.

[https://docs.dgraph.io/deploy#ports-
usage](https://docs.dgraph.io/deploy#ports-usage)

------
mark_l_watson
Thanks for writing this up! I worked with the Knowledge Graph as a contractor
at Google in 2013. My manager had a neat idea for adding our own Schema and
triples (actually quads) for a specific application.

It surprises me how many large companies do not have a ‘knowledge graph
strategy’ while everyone is on board with machine learning (which is what I
currently do, managing a machine learning team). I would argue that a high
capacity, low query latency Knowledge Graph should be core infrastructure for
most large companies and knowledge graphs and machine learning are
complementary.

~~~
taherchhabra
Sounds like a good Enterprise saas product idea

~~~
pklee
The beauty of RDF is it can support schema promiscuity i.e. you can have many
schemas for your same data. You can do that with OWL. In typical graph
databases you are fixed on nodes, properties and edges, but in RDF you can
choose what should be node and properties arbitrarily. My issue with RDF has
been performance works great for categorical, relationship heavy data, but not
so much for numerical data.

~~~
mark_l_watson
I am also a fan of RDF and OWL. My two ‘practical semantic web’ books (one in
Common Lisp and one for Jave/Clojure/Scala) are now 8 years old, but you can
get free PDFs from my web site. Those books badly need second editions, the
material is interesting but outdated.

~~~
gibsonf1
Mark, are you up for helping with solid security for cl-solid? (The library
now works with both Neptune and Allegrograph) [https://github.com/gibsonf1/cl-
solid](https://github.com/gibsonf1/cl-solid) \- live use of cl-solid library
here [https://graphmetrix.net](https://graphmetrix.net) , sometimes very
lively discussion of Solid here:
[https://gitter.im/solid/chat](https://gitter.im/solid/chat) (including Tim
BL)

~~~
mark_l_watson
I have no time right now. Contact me in 5 weeks and we can talk then.

~~~
gibsonf1
Great

------
wiradikusuma
OOT, but the illustrations on their home page (
[https://dgraph.io/](https://dgraph.io/) ) is veeery cute, you should check it
out.

~~~
pvg
It makes you wonder how the product handles mushroom and more importantly,
snake.

~~~
mrjn
Badger don't care!

[https://www.youtube.com/watch?v=4r7wHMg5Yjg](https://www.youtube.com/watch?v=4r7wHMg5Yjg)

------
staticassertion
I've been using DGraph for quite a while now in a side project of mine.

I looked at a number of other graph dbs, and landed on dgraph for a few
reasons.

a) I like the query language. It felt natural to me. That's just my opinion
though. Other languages felt cumbersome, and I didn't feel like I was
operating on a graph.

b) I needed write performance. I'd talked to colleagues about their
experiences (I have some friends at graphy companies) and every time I asked
"what about X graphdb? could it handle this write load?" the answer was "no,
definitely not".

I chose dgraph and it's been working like champ. Support via slack has been
solid, they're focusing on the right things (performance, correctness), and I
have a lot of confidence in the future of the project.

~~~
mrjn
Not just you, a lot of Dgraph users appreciate the intuitive model of
GraphQL+- (modification on GraphQL), and how it fits naturally with JSON and
apps. When I started in 2015, I didn't imagine that our decision to use
GraphQL would become such a major "feature." GraphQL has taken off like a
wildfire, which was hard to imagine back in 2015 when it just launched in July
2015.

In fact this year, we plan to align GraphQL+- even more with the official
GraphQL spec. Stay tuned!

re: write perf, it just got a whole lot better with 3 changes I made recently,
and there're still a lot of room for optimization: [https://github.com/dgraph-
io/dgraph/commit/b7189935e6ec93aec...](https://github.com/dgraph-
io/dgraph/commit/b7189935e6ec93aec04762321252b70f98466985)

~~~
staticassertion
Like I said, a big part of feeling confident in my choice to use DGraph is the
direction - the improvements to the QL and performance, as well as the Jepsen
tests, are exactly the work I want to see as a consumer.

And that is one hell of an improvement. Nice work.

------
chao-
This a really neat historical perspective, although I do find one detail odd:

> _If you are sneering at the memory requirement, note that this was back in
> 2010. Most Google servers were maxed at 32GB._

As I recall, Nehalem EP (launched 2008 or 2009?) could handle in excess of
100GB per socket? Not cheap necessarily, but definitely still counted as
"commodity hardware" for that era. I say this recalling that even my mid-tier
workstation from then could handle 48GB (in that wonky era of triple-channel
RAM), though I only had it loaded with 12GB. Then again I could see, if said
servers in 2010 were at the end of a purchasing cycle from 2007 or so, that
they were "maxed" at 32GB?

Anyway, my from-memory nitpick doesn't detract from the article's ultimate
point, though: distribution was an obvious need that would only become more
pressing.

~~~
shereadsthenews
The hardware replacement cycle is much longer than you’ve implied. Servers I
am building today will still be TCO-positive for 15 years. In 2010 Google data
centers would still have been full of their first generation dual-socket
Opteron machine.

~~~
chao-
I will happily stand corrected, because I was only implying based on very
limited awareness. I am only directly familiar with workstations, where I've
seen replacements occur on the order of 4-7 years, not servers or entire
datacenters. However, everything I read seemed reinforce an idea that servers
follow a similar pattern. Maybe I was just wildly misinterpreting?

With the relative stagnation in per-core performance since about 2010 I could
easily see something built then expecting to last to 2025, power consumption
improvements nonwithstanding.

------
geuszb
Insightful article / ad. Maybe the problem with the idea is that in practice
not many users want to do joins at arbitrary depths levels, e.g. "sort all the
children of US presidents by height" is probably not a very common query
needing a massively distributed architecture?

~~~
pacala
"Travel from NY to LA over the weekend" was probably not a very common demand
needing a massive investment in air travel infrastructure and equipment circa
1900. User demand is bounded by the capability of the tools commonly
available.

~~~
geuszb
True, but to a certain extent only. The tail only gets so fat...

------
gibsonf1
For the scalability side of graph, we've just started using Amazon Neptune
RDF, and have been amazed that we can easily and very quickly run sparql
queries on 2.6 billion triples on their smallest 2 core 15 gig machine.
Incredible capacity.

~~~
mrjn
As others mention, Neptune is based on Blazegraph. It is a layer on top of
Amazon Aurora and has the typical graph layer issues I mention my blog post
(in particular the join depth problem).

When Neptune was launched, I wrote another article critiquing its design.
Worth a read:
[https://blog.dgraph.io/post/neptune/](https://blog.dgraph.io/post/neptune/)

~~~
pbowyer
I'd missed the announcement of Neptune, so thanks for alerting me. Have you
reviewed the Microsoft Azure Cosmos DB, with its graph access layer?

~~~
mrjn
I haven't. But, the design is the same. Graph layer on top of another DB,
because they didn't want to invest in building a native graph DB.

In general, that approach doesn't lead to good performance, as I explained in
the blog post.

------
macawfish
I've been looking for an open source graph nosql style database that makes it
easy to do gather-scatter on a GPU. It's been tough.

 _Plug:_ some people are working to fork/restart Blazegraph development, which
is the database Amazon took to make Neptune. Its GPU features are missing, as
they were originally kept proprietary. If you have any interest and know how,
this could be an exciting project to contribute to!

[https://github.com/blazegraph/database/issues/86](https://github.com/blazegraph/database/issues/86)

------
sandGorgon
There is also Janusgraph which Google contributes to
[http://janusgraph.org](http://janusgraph.org)

~~~
ddorian43
If you don't do datastructures right, then it will suck on performance.

Imagine doing a search-engine in a rdbms without special datastructures.

~~~
sandGorgon
janusgraph is usually used with Cassandra. This is the schema -
[https://docs.janusgraph.org/latest/schema.html](https://docs.janusgraph.org/latest/schema.html)

~~~
ddorian43
See mrjn replies. You need native graph db.

------
combatentropy
This reminded me of SQL.

> Say, you want to know [people in SF who eat sushi]....If an application
> external to a database was executing this, it would do one query to execute
> the first step. Then execute multiple queries (one query for each result),
> to figure out what each person eats, picking only those who eat sushi.

A query like that in SQL could also suffer from "a fan-out problem" and could
get slow. It's often faster to put subqueries in the From clause than the
Select. It's certainly faster than an app taking the rows of one query and
sending new queries for each row, as many developers do. For example:

    
    
      select
          p.name,
          (
              select max(visit_date)
              from visits v
              where v.person = p.id
          ) as last_visit
      from people p
      where p.born < '1960-01-01'
    

can be slower than:

    
    
      select p.name, v.last_visit
      from people p
          join (
              select person, max(visit_date) as last_visit
              from visits
              group by person
          ) v on p.id = v.person
      where p.born < '1960-01-01' 
    

In the second example, you first form a new table through a subquery of the
original. This is not what a new programmer would first try. The first
example, with the subquery in the Select clause, is closer to the train of
thought. Also you would guess that getting the last visit dates of each person
is more efficient after you know who to look for (like, only the people born
before 1960). But in my experience, it often hasn't been.

Therefore likewise with this San Francisco sushi query, I was thinking that if
it were SQL then I would (1) get all people in San Francisco, (2) get all
people who like sushi, and then (3) join them, to find their intersection. Lo
and behold, I then read that it is the same solution in this humongous graph
database:

> The concepts involved in Dgraph design were novel and solved the join-depth
> problem.... The first call would find all people who live in SF. The second
> call would send this list of people and intersect with all the people who
> eat sushi.

~~~
mrjn
Now consider doing this in a distributed setting, with SQL tables splits
across dozens or 100s of machines (see Facebook TAO).

~~~
Scarbutt
So are there any benefits in dgraph over SQL/RDBMs(matureness and widespread
query lang) for single machine setups?

~~~
mrjn
I don't want this to become a flame war between SQL and Graph. But, we see a
lot of developers come from SQL to Dgraph, because the join performance of SQL
gets worse as the data size grows, and so devs have to resort to doing
aggressive data normalizations. Even more of a problem when your entire
dataset is on a single machine.

For e.g., Stack Overflow normalizes their data to avoid doing joins:
[https://archive.org/download/stackexchange](https://archive.org/download/stackexchange)

~~~
neilk
Did you mean denormalize?

~~~
mrjn
Yes, I meant denormalize. Thanks.

------
Tharkun
Slightly off topic, but while we're on the subject of graph databases: could
anyone point me at some useful introductions to the subject? Thank you.

~~~
mrjn
I'd suggest starting with [https://docs.dgraph.io/get-
started/](https://docs.dgraph.io/get-started/), and then going to
[https://tour.dgraph.io](https://tour.dgraph.io). It gives you all you need to
understand the query language (based on GraphQL) and get you up to speed
building your first app on a graph DB.

~~~
staticassertion
I'll echo this - I learned about graph databases using DGraph.

------
subsaharancoder
How does Dgraph compare in performance, features to something like Neo4j?

~~~
mrjn
Slightly old post, but we did some benchmarks couple of years ago:
[https://blog.dgraph.io/post/benchmark-
neo4j/](https://blog.dgraph.io/post/benchmark-neo4j/)

We also get to hear benchmark conclusions from users who tried out Neo4j,
Janus, Datastax DSE Graph, etc. They typically find Dgraph to be the best
performing one.

There's a comparison chart here: [https://github.com/dgraph-io/dgraph#dgraph-
compared-to-other...](https://github.com/dgraph-io/dgraph#dgraph-compared-to-
other-graph-dbs)

~~~
subsaharancoder
Thank you for the comparisons. I think the approach you've taken with the
documentation and the support out of the box for libraries in Go, Python etc
is excellent, the playground is definitely a winner!!

------
agumonkey
[https://archive.fo/IsWTe](https://archive.fo/IsWTe)

~~~
bufferoverflow
Thank you. That page freezes Chrome, ironically.

------
marknadal
This was a great article, and I'm a competitor! :) Learned a lot about the
history.

My one nit-pick is they say distributed joins are hard though. But they don't
really justify it other than the assumption other engineers have come from a
traditional DB background. It is not a hard problem if you try to solve it on
your own though: You do the join on the client, incrementally (or debounced if
necessary) as the data is streamed. This works very well at scale in
production (TB/day) for our system.

~~~
politician
There are a variety of distributed join scheduling models to consider
depending on the latency of the network and the size of the data sets.
Techniques like Bloom filters used to accelerate joins by minimizing the data
transmitted are novel to many developers. Layer in HA and transactions, and it
becomes hard.

------
taherchhabra
We are using AWS Neptune in building our marketing analytics product. What I
have realized is that it has best of both worlds, the querying power of SQL
databases and schemaless functionality like nosql databases.

~~~
mrjn
Consider Dgraph :-).

[https://blog.dgraph.io/post/neptune/](https://blog.dgraph.io/post/neptune/)

~~~
taway483
I loved working with Dgraph, but I hate doing system/database administration
work.. do you have any plans of providing a hosted solution?

~~~
mrjn
No concrete plans yet, but that's a direction we want to go in.

