

Dynomite – Making Non-Distributed Databases Distributed - stefanwild
http://techblog.netflix.com/2014/11/introducing-dynomite.html

======
void_star
It's a little unfortunate that consistency is only mentioned as a footnote. If
this is being used internally at Netflix, how are they dealing with
consistency issues in the face of failed or conflicting writes?

~~~
antirez
This currently looks like the Dynamo-alike sharding without the read-repair
part making values eventually convergent in all the replicas. This latter
feature is much needed, but they are already planning it apparently.

~~~
void_star
Except no mention of vector clocks, so it's unclear how one would detect or
repair a conflict. Maybe that is an implementation detail that is intended to
be ironed out later...

~~~
jedberg
With a replication factor of 3 or more you don't need a vector clock, you can
just choose majority wins.

That being said, I believe it is using a vector clock, but I'm not sure.

~~~
grogers
You still need ordering amongst writes. Without that you will get into a
situation where there is no majority, all three nodes have different data.

~~~
antirez
Converging to a single value (which is a liveness property) and what the
values will contain (which are usually safety properties) are different
stories... when the partition heals, if the nodes detect they have a different
version for a given piece of data, they can just remove it, and the system is
convergent, which means it is eventually consistent (a liveness property).
_What_ happens in the merge function instead provides additional safety
properties. For example in the case of a CRDT that models a Set type, with the
merge function that is "union of all the values in the nodes", you can state
the safety property that if a Set element X was added into a node, at no time
will happen that the nodes will be able to talk again, and the system will
converge into a Set which excludes such value X.

------
nteon
Netflix is a Java-heavy shop (the client talked up in the blog post is the
Java client). I was pleasantly surprised to see that dynomite is implemented
in C with an eye towards both BSD + Linux. I'm curious if this was because a
GC'ed language like Java or Go was tried and proven not up to the task, or
what.

~~~
viksit
My guess would be that the ecosystem in their (seed) target set of databases
and protocols is written in C. In one case, they're using Twitters twemproxy
to provide connection counts for redis/memcached.

~~~
zerobfd
Not just connection counts; when I first looked at the github I was surprised
at how close the code organization was to twemproxy, and when I looked further
a huge amount of the code is built on top of existing twemproxy code.

Netflix probably saw an opportunity to built the Dynamo layer on top of what
Twitter had already written, and decided to just keep building on in C instead
of rewriting a fairly large codebase.

------
personZ
Netflix always makes for an interesting case, but what always strikes me is
how incredibly "trivial" their application really is: It's a simple
application at a very large scale. This doesn't dismiss their accomplishments,
of course, but often it does seem like Netflix spends an enormous engineering
effort on fairly low value aspects of the service.

~~~
themartorana
Well, they make it seem trivial, while dealing with every watch and every
pause, every rating updating complex recommendation engine algorithms. They
deal with licensing and region awareness and proxying by ISP and running their
own CDNs and...

And that's the magic.

To boot, they do it at _enormous_ scale on a multitude of platforms all while
their own destructive code (Chaos Monkey) runs around destroying servers.

It's true, some things like "stream me this" are maybe not rocket science, but
at Netflix scale, everything becomes a bit more difficult.

~~~
jcampbell1
Netflix is good at providing the service on any device, and having 100%
uptime.

Their requirements are 100% availability, and an API's that never breaks, but
consistency rarely matters.

The engineering effort is cool, but it is solving a problem that _I_ don't
have. The only thing that matters to netflix is they list the catalog somewhat
correctly, and always deliver a stream.

I have a different problem, thus I am uninterested in their _magic_. If I had
a similar problem or they signed my paycheck, I'd care.

~~~
res0nat0r
Not to deride what you are saying, but when I see comments that dismiss
Netflix as a site that just simply streams movies consistently; it tells me
that the commenters haven't seen and appreciated the difficulty of the
problems they are actually solving under the hood as evidenced by their github
page.

[http://netflix.github.io/#repo](http://netflix.github.io/#repo)

~~~
personZ
Neither that or the number of open engineering positions tells you the
difficulty of the problems they are solving. It might just tell you the
difficulty of the problems they are _making_ : If Netflix has a technology
side with a history and the executive ability to overcomplicate, they might be
making little problems into big problems. My perspective as an outsider is to
look at the complexity of their app platform (again, separating the video
streams which happen from systems that never seem to be the foundation of
these tech blog entries), and it is unarguably not a complex system.

Early on I remember the heroics that Netflix went through running on EC2,
building numbers of complex tools for varied performance, reliability issues,
etc. At the time many much larger sites just quietly _worked_ 24/7, minus the
heroics, and minus the effort, often by using purpose-suited dedicated
servers. Later, once Amazon rolled out SSDs, Netflix triumphantly announced
how much of an improvement it was to their product, again demonstrating that
they were creating a problem (huge numbers of horrible I/O machines) that they
then solved with gusto, albeit unnecessarily.

I'm not trying to be overly down on Netflix, but it is a company that seemed
to make tech blog entries a product of the company years ago, and the result
is that people have bought into this notion that they're doing some hugely
complex task. They aren't.

~~~
res0nat0r
Sorry but you are just flat out wrong. Operating at the scale that they are is
in fact complex. They aren't creating all of this software just because they
are having fun in their free time.

~~~
personZ
_They aren 't creating all of this software just because they are having fun
in their free time._

Sorry, but this is an invention of yours. No one said they're doing anything
flippantly.

But they may be doing it _ignorantly_ : Success in one field (in this case
turning a mail subscription business into a stream service) doesn't imply
technical innovation or leadership, and often is despite it.

This industry is absolutely rife with people creating solutions to problems
they themselves invented and caused. In the case of Netflix, an enormous
amount of their solutions have been founded around the notion of deploying on
huge numbers of miserable Amazon EC2 instances, and then dealing with the
problems related to that. Others simply deployed distributed data centers
hosting their own purpose suited, reliable hardware with big fat storage
arrays, and the problem is solved. It's like trying to build a car out of
toothpicks and then detailing the innovations you created in toothpick
redundancy and robustness.

~~~
res0nat0r
Sorry but this just reeks of naiveté. When the few areas of tech that I have
actually been involved with in the real world come up on HN there are always
comments stating that things are just so simple and why so and so is doing it
wrong and why it so much easier if they just do something simplistic...

That just isn't the case in the real world when working with tens of millions
of customers. This will become apparent when you actually get out there in the
field and realize systems involving tens of thousands of machines hosting
dozens of different services to millions of people is actually not simple.

~~~
personZ
Starting with a patronizing "sorry", looking down from heights, doesn't
somehow give your comment more credibility.

There are a large number of services of the user scale of Netflix, with
_dramatically_ more complexity, that don't seem to have the heroic issues that
Netflix does. Facebook and Reddit both are of a complexity scale multiple-
magnitudes greater than Netflix. Someone else absurdly tried to draw Apple and
Google in, but again, what they do absolutely dwarfs the entirety of Netflix's
operation, to the point that it becomes almost laughable.

 _Netflix is a very simple application_. Scaling it can be difficult, of
course, but this is hardly some new grounds. Further, the particularly make-up
of Netflix makes it one of the most profoundly scalable platforms going
(silos, little to no need for transactional integrity. It is profoundly
simple). You don't have to believe this, but your comically patronizing
responses just sound...laughably unskilled.

~~~
res0nat0r
> Netflix is a very simple application. Scaling it can be difficult, of
> course, but this is hardly some new grounds. Further, the particularly make-
> up of Netflix makes it one of the most profoundly scalable platforms going
> (silos, little to no need for transactional integrity. It is profoundly
> simple). You don't have to believe this, but your comically patronizing
> responses just sound...laughably unskilled.

Have you actually work with any of their backend engineers, seen any of their
infrastructure and why they made the design decisions they did, or actually
talked to anyone at Netflix, or is this all just speculation?

I'm sure they'd love to hire you to to greatly simplify their overly
complicated architecture otherwise...

------
jedberg
When I saw this internally a few months ago, I said that I haven't been more
excited about a new project in a long time. I personally think this will solve
a lot of scalability problems people have while letting them keep their
favorite database.

~~~
stefanwild
It looks very promising. I like the fact that it doesn't only work for AWS
environments.

------
lobster_johnson
Funnily enough Dynomite was also the name of an Erlang implementation
([https://github.com/moonpolysoft/dynomite](https://github.com/moonpolysoft/dynomite))
of Dynamo that seemed to get off to a very promising start before it was
abandoned a few years ago.

------
NKCSS
Cool, it's something that Microsoft SQL Server (Enterprise edition) offers out
of the box (Peer-2-Peer replication). The only problem with P2P replication is
that, at large scale, requires massive amounts of bandwidth to keep everything
synced, and it gets worse the more nodes you add.

~~~
diptanu
Depends how things are replicated. In Dynamo style replication everything
isn't replicated everywhere. Based on consistent hashing, and how vnodes etc
are configured, a key lands itself in more than one nodes so that things are
generally available. So scale-out in a dynamo-like system happens more
gracefully.

------
ppearcy
Any ideas if this supports Postgres?

Also, I noticed comments about an architecture document here:
[https://github.com/Netflix/dynomite](https://github.com/Netflix/dynomite)

Anyone know where the doc they are referencing is located?

~~~
jedberg
It doesn't support Postgres yet but that is on the (longer term) roadmap.

However, since it is open source, contributions are welcome! :)

------
diltonm
I don't know if it was intentional but the name reminds me of the character JJ
in the TV series Good Times. Awesome show that was. It sounds solid overall
but one thing caught my eye, "A client can connect to any node to read the
data. Similar to writes, a node serves the read request if it owns the data,
otherwise it forwards the read request to the data owning node in the same
rack.". I wonder if that could lead to an owning node arbitration storm when
more than one node in a rack owns the data?

------
fizx
For projects with similar goals, see also Twitter's Gizzard (abandoned) and
LinkedIn's Voldemort.

~~~
wowmsi
Also Riak. However, one interesting aspect of Dynomite seems to be offer
default support for clustering across multiple data centers with pluggable
stores.

~~~
darkr
Riak (enterprise version) does that, and has pluggable stores (albeit limited
to in-memory, bitcask + eleveldb)

------
RealCasually
Perhaps I missed it, but is there any discussion on how their sharding
mechanism works? How do they define a shard and where is ownership of various
shards managed so a "non-owning node can redirect to an owning node"?

~~~
marktangotango
I questioned tbat as well, could be a property of the underlying store
(memcache hash). How would mysql sharding work without being application
managed?

------
jlafon
It's interesting that Netflix decided to write this rather than using Amazon's
own DynamoDB. I wonder specifically if DynamoDB was too expensive (as I have
found), or was there some other reason?

~~~
jetblackio
This is completely different than DynamoDB. Essentially this is a framework
for sharding or horizontally scaling multiple datastores (MySQL, Memcache,
Redis, etc).

~~~
amelius
Why would you use multiple datastores? Isn't it much more difficult to
_atomically_ store data this way? Assume a transaction succeeds in store 1,
and fails in store 2, you'd need to roll back the transaction in store 1 after
it has committed (!)

~~~
earless1
Because there is no such thing as a one size fits all datasotre. Some data may
make way more sense stored in a relational database, while some data makes
more sense as key:value pairs. Some data may be infrequently accessed and
stored to disk while others are stored in memory for quick access. Trying to
make a single datastore work for many use cases will cause more pain than it
is worth.

------
halayli
nitpick, the code check for NULLs at a low level but misses checking NULLs at
higher levels like here:

[https://github.com/Netflix/dynomite/blob/master/src/dyn_dnod...](https://github.com/Netflix/dynomite/blob/master/src/dyn_dnode_msg.c#L601)

------
evantahler
Are there any deployment notes/examples?

~~~
nrbafna
Setup and Installation Wiki -
[https://github.com/Netflix/dynomite/wiki/Getting+Started#set...](https://github.com/Netflix/dynomite/wiki/Getting+Started#setup-
and-installation)

