

Migrating From MongoDB To Riak At Bump - timdoug
http://devblog.bu.mp/from-mongodb-to-riak

======
stephen
> During the migration, there were a number of fields that should have been
> set in Mongo but were not

Imagine that...this fascination with schema-less datastores just baffles me:

[http://draconianoverlord.com/2012/05/08/whats-wrong-with-
a-s...](http://draconianoverlord.com/2012/05/08/whats-wrong-with-a-
schema.html)

I'm sure schema-less datastores are a huge win for your MVP release when it's
all greenfield development, but from my days working for enterprises, it seems
like you're just begging for data inconsistencies to sneak into your data.

Although, in the enterprise, data actually lives longer than 6 months--by
which time I suppose most start ups are hoping to have been bought out.

(Yeah, I'm being snarky; none of this is targeted at bu.mp, they obviously
understand pros/cons of schemas, having used pbuffers and mongo, I'm more just
talking about how any datastore that's not relational these days touts the
lack of a schema as an obvious win.)

~~~
timdoug
Yeah -- that's one of the huge benefits of marking a field as ``required'' in
a protobuf. The ability to enforce a contract prevents a ton of unexpected and
incomplete data making it on disk (and also, e.g., across the wire to
clients). Having strict types represented in the serialization format is also
handy; when one pulls out an int32 from a protobuf it's going to be an int32,
and not an integer that somehow found its way into being a string.

~~~
lucaspiller
Could you elaborate more on how you have used Protobuffs, as I'm not sure I
fully understand. I've previously used Riak in Erlang and Ruby projects, so am
fairly familiar with how it works. It exposes a HTTP and Protobuffs API which
allows you to store objects of arbitrary types (JSON, Erlang binary terms,
Images, Word Documents, etc). From the sounds of it you are serialising a
Protobuffs packet and sending this as the content of the object. Why did you
choose this, over say JSON, which MongoDB uses?

------
salsakran
I was reading along and nodding my head until I got to the 1000 line haskell
program that handles issues stemming from a lack of consistency.

I'm not exactly a SQL fanboy, but maybe ACID is kinda useful in situations
like this and having to write your own application land 1000 liners for stuff
that got solved in SQL land decades ago isn't the best use of time?

~~~
wpietri
MySQL is circa 1 _million_ lines of code.

I like SQL engines for moderate data sets that fit nicely on one machine and
well within the normal performance envelope. But even there I will often have
to try a few different incantations and cross my fingers that one of them will
perform reasonably because that's easier than trying to figure out what that 1
MLOC engine is up to. And I don't know anybody who does very large MySQL
setups without a lot more hassle than that.

For some things I'd much rather deal with 1KLOC that I had to write myself
than the 1 MLOC that I'm scared to even start digging through.

~~~
salsakran
The question isn't 1MLOC vs 1KLOC.

It's a stable, well understood DB vs an immature, not well understood DB AND
1KLOC to deal with not being consistent.

To be clear, I'm not saying any given DB is the OneTrueWay, just that people
seem to be a bit cavalier in regards to some of this crap and chasing the
newest shiny thing while rediscovering why some of the braindamage in those
1MLOC was put there in the first place.

~~~
wpietri
Oh, I agree that some people are being a bit cavalier about the new shiny.
These folks are migrating away from MongoDB for a reason.

On the other hand, even if half of that 1 MLOC is still relevant to new ways
of building systems (and given that SQL databases are a 70s tech, I doubt it's
that much), that still leaves half that isn't.

The only way we'll find out which part matters is for people to try different
approaches. So I fully support experimentation like this. If we don't
rediscover for ourselves the good parts in the tools we use, then we're just
stuck honoring the traditions of our heavily bearded ancestors.

~~~
etrain
There is an awful lot of 70s tech that is still in constant operation, and is
highly relevant to the types of work you're probably doing today. TCP, C,
Pipes, etc.

~~~
wpietri
Sure. But I think the problem space for databases has changed quite a bit in
ways that aren't true for TCP, and are only partly true for C.

My dad was writing code at the time, and he saw the big benefit as allowing
developers to manage larger amounts of data on disk (10s of megabytes!)
without a lot of the manual shenanigans and heavy wizardry in laying out the
data on disk and finding it again. Plus, the industry thought the obvious
future was languages like COBOL, what with their friendly English-like syntax
and ability for non-programmers to get things done directly.

So little of that is true anymore. For a lot of things that use databases,
you're expected to have enough RAM for your data. We don't distribute and
shard because we can't fit enough spinning rust in a box; we do it because
we're limited on RAM or CPU. A lot more people have CS degrees, the field is
much better understood, and developers get a lot more practice because
hardware is now approximately free. And nobody really thinks the world needs
another English-like language so that managers can build their own queries.

TCP, on the other hand, is solving pretty similar problems: the pipes have
gotten faster and fatter, but surprisingly little has changed in the
essentials.

C is somewhere in between. A small fraction of developers working these days
spend much time coding in plain C, and many of its notions are irrelevant to
most development work.

But unlike SQL databases, you could ignore C if you wanted to; there were
other mainstream choices. That wasn't true for SQL until recently; the only
question was Oracle or something else. I'm very glad that has changed.

~~~
etrain
My apologies for the rant here - it's not directed specifically at you, but
toward a general attitude I see on HN and in the tech community.

I was more commenting on your phrase "if half of that 1 MLOC is still relevant
to new ways of building systems (and given that SQL databases are a 70s tech,
I doubt it's that much)".

There has been a TON of academic and industrial research on SQL databases
since they were invented in the 70s. Calling them 70s tech is akin to calling
LISP 50s tech. The basic ingredients haven't changed much (sets in SQL and
lists in LISP), but the techniques on top have evolved by leaps and bounds.

To your point here - there are plenty of companies that have way more data in
their databases than RAM available. The early users of Hadoop, etc. were
primarily constrained by disk I/O on one machine, rather than constrained by
RAM or CPU on one machine. It is certainly convenient that a distributed
architecture can solve both sets of problems if the problem space lends itself
to distributed computation.

I'm not a huge defender of SQL - I think it has some serious problems - one
fundamental problem is lack of strong support for ordered data, and it can be
a huge pain to distribute. I agree that having some options with distributed
K/V stores is really nice, but you have to admit that much of it hasn't yet
been proven in the field.

I, for one, DO think that the world needs something like an english-like
language so that "managers" can write their own queries. Honestly, the roles
of programmer and data analyst are often wholly different. I think it's a huge
kludge that the only people capable of manipulating massive datasets are
people with CS degrees or the equivalent. Programmers suck at statistics, and
while they're generally smart folks, they often don't have the business/domain
expertise to ask the right questions of their data. Software is about enabling
people to solve problems faster - why should we be only allowing people with
the right kind of academic background to solve a whole class of problems.

Finally, saying that you doubt SQL is relevant to building modern systems is
borderline irresponsible. Experimentation with new tools is good - but you
have to also keep in mind that people were smart way back in the 70s, too, and
that their work may be perfectly relevant to the problems you're trying to
solve today.

~~~
wpietri
No need to apologize! You make excellent points.

There has indeed been a ton of research on SQL databases. But still,
Stonebraker, one of their pioneers, said that they should all be thrown out:

"We conclude that the current RDBMS code lines, while attempting to be a “one
size fits all” solution, in fact, excel at nothing. Hence, they are 25 year
old legacy code lines that should be retired in favor of a collection of “from
scratch” specialized engines. The DBMS vendors (and the research community)
should start with a clean sheet of paper and design systems for tomorrow’s
requirements, not continue to push code lines and architectures designed for
yesterday’s needs." -- <http://nms.csail.mit.edu/~stavros/pubs/hstore.pdf>

I'm sure a lot of their intellectual work is indeed something I could learn
from. But SQL databases are an artifact of a particular moment in
technological and cultural history. The thing I really want to learn about
isn't the residue of their thoughts as interpreted through 30 years of old
code, it's their original insights and how to transfer those to today's world.

Hadoop is a fine example of Stonebraker's point. The original sweet spot of
relational databases was easy queries across volumes of data too large to fit
in RAM. But Google realized early on that they could do orders of magnitude
better with a different approach.

I agree that these new approaches haven't been fully proven in the field, but
I'm certainly glad that people are trying.

As a side note, I think the right way to solve the pseudo-English query
language problem is by making the query language actual English. If you have a
programmer and an analyst or other business person sit next to one another,
you get much better results than either one working alone.

~~~
etrain
Stonebraker's research and commercial ventures the last several years have
been focused on building specialized variants of existing database systems.
Vertica (C-Store), VoltDB (H-Store), Streambase (Aurora), and SciDB are all
specialized DBMS systems designed to overcome the one size fits all nature of
things.

Further, he's been critical of NoSQL/MapReduce recently:
<http://dl.acm.org/citation.cfm?doid=1721654.1721659>

Regardless, there's always going to be a balance between specialized systems
and platforms, but my point is that we should be willing to trust the
platforms that have proven themselves, avoid reinventing the wheel (poorly),
and not be too quick to throw them out in favor of the new shiny.

I agree that programmer/analyst working together is a terrific pair, but the
beauty of software is that we live in a world where we can have 1 programmer
write a platform on which 10 programmers build 10 systems that 1000 users use
to get their jobs done and make society that much more efficient.

~~~
wpietri
Oh, I trust the current stable platforms to be the current stable platforms.
My beef isn't with people who use them. It's with people who don't know how to
do anything else, which was a plague on our industry for at least a decade. At
least the people who get burnt on the latest fad will make new and interesting
mistakes.

I agree that when we can find ways to let users serve themselves, that's best
for everybody. I just don't think universal pseudo-English query languages are
the way to do that, because the hard part isn't learning a little syntax, it's
learning what the syntax represents in terms of machine operations.

Once the programmer and the analyst have found something stable enough to
automate, by all means automate it. Reports, report builders, DSLs, data
dumps, specialized analytic tools: all great in the right conditions. But
people have been trying to build pseudo-English PHB-friendly tools for decades
with near zero uptake from that audience. I think there's a reason for that.

------
timhaines
If you're thinking about using Riak, make sure you benchmark the write (put)
throughput for a sustained period before you start coding. I got burnt with
this.

I was using the LevelDB backend with Riak 1.1.2, as my keys are too big to fit
in RAM.

I ran tests on a 5 node dedicated server cluster (fast CPU, 8GB ram, 15k RPM
spinning drives), and after 10 hours Riak was only able to write 250 new
objects per second.

Here's a graph showing the drop from 400/s to 300/s:
<http://twitpic.com/9jtjmu/full>

The tests were done using Basho's own benchmarking tool, with the partitioned
sequential integer key generator, and 250 byte values. I tried adjusting the
ring_size (1024 and 128), and tried adjusting the LevelDB cache_size etc and
it didn't help.

Be aware of the poor write throughput if you are going to use it.

~~~
bfrog
Riak loves random read/writes, spinny discs do not, try things out with a SSD
sometime and watch things go from a shoddy XXX ops/sec to XXXX(X) ops/sec.

As a simple remark on this, I've gotten 1000+ ops/sec on a single machine
operating as 3 nodes (equating to about 3000 ops/sec per node) when using an
SSD and a measly 150 ops/sec with a spinny disc in the same setup (equating to
about 450 ops/sec per node)

~~~
moonboots
While SSDs will undoubtedly be faster that spinning disk, LevelDB is designed
to address slow random writes by batching and writing sequentially.

~~~
bfrog
That would be true except each vnode (by default 64!) has its own backend
database. That means with 4 physical nodes each one gets 16
leveldb/bitcask/whatever database backends.

LevelDB's write batching and caching is completely void when thats considered
in many circumstances.

Its something that I think Basho should consider changing. Its a trade off of
fault tolerance or performance, and I would personally love to see riak go a
lot faster.

------
tlianza
I find these kinds of stories interesting, but without some feel for the size
of the data, they're not very useful/practical.

I've heard of Bump, and used it once or twice, but I don't actually know how
big or popular it is. If we're talking about a database for a few million
users, only a tiny percentage of which are actively "bumping" at any time,
it's really hard for me to imagine this is an interesting scaling problem.

Ex. If I just read an article about a "data migration" who's scale is
something a traditional DBMS would yawn at, the newsworthiness would have to
be re-evaluated.

~~~
timdoug
I'm not sure what exactly qualifies as respectable scale, but the Mongo master
was running out of space and IO capacity with 24 SSDs and 90 million user
records, and was replaced by a sixteen-node Riak cluster.

I'll happily share any other statistics you're interested in.

Edit: the Riak cluster actually contains lots of other data (communications,
object metadata, etc.); we didn't need sixteen boxes for the user records.

~~~
tlianza
90 million users is a great datapoint, yes! In my book that's more than
respectable.

The only other stat that I'm curious about is the total size of the DB.
Certainly databases with tens of millions of records can be held completely in
RAM these days... but that also depends on how big each record is.

~~~
timdoug
All-told the users database when we started migration was about 600 GB on
disk, so not the most easily stored database in RAM, but not impossible if you
get enough large machines.

In fact, we use Redis a lot at Bump, although almost exclusively for queueing
and transient state, and not as a persistent database. For a period of time we
did store long-lasting metadata in Redis, and as we became more popular
instead of throwing engineering effort at the problem we threw more memory,
culminating with a handful of boxes with 192 GB of RAM each. We've since moved
that entire database to Riak. :)

------
_Lemon_
I have decided on wanting to use riak as well. I was wondering if anyone had
examples of how they used it with their data model?

For example this article mentions "With appropriate logic (set unions,
timestamps, etc) it is easy to resolve these conflicts" however timestamps are
not an adequate way to do this due to distributed systems having partial
ordering. The magicd may be serialising all requests to riak to mitigate this
(essentially using the time reference of magicd) in which case they're losing
out on the distributed nature of riak (magicd becomes a single point of
failure / bottleneck).

Insight into how others have approached this would be awesome.

~~~
reiddraper
There are a several ways to approach this. The simplest is to just take last-
write-wins, which is the only option some distributed databases give you. For
cases where this isn't ideal, you resolve write-conflicts in a couple ways.

One way is to write domain-specific logic that knows how to resolve your
values. For example, your models might have some state that only happen-after
another state, so conflicts of this nature resolve to the 'later' state.

Another approach is to use data-structures or a library designed for this,
like CRDTs. Some resources below:

A comprehensive study of Convergent and Commutative Replicated Data Types
<http://hal.archives-ouvertes.fr/inria-00555588/>

<https://github.com/reiddraper/knockbox> <https://github.com/aphyr/meangirls>
<https://github.com/ericmoritz/crdt> <https://github.com/mochi/statebox>

~~~
stock_toaster
Are there any connector libs that provide "simple" last-write-wins out of the
box?

~~~
bonzoesc
Simple last-write-wins can be configured per-bucket, so client libs can be
naïve: <http://wiki.basho.com/HTTP-Set-Bucket-Properties.html>

------
gizzlon
Would be interesting to see a follow up in 6 months or so..

It doesn't seem fair to compare [ _old tech_ ] with [ _new tech_ ] when you've
felt all the pitfalls with one but not the other.

------
supo
Random thought on proto buffers: OP is advocating using the "required"
modifier for fields and touting it as an advantage in comparison to JSON. I
would move the field value verification logic to the client, because it can
cause backwards compatibility problems if you un-require it.

------
clu3
@timdoug, could you share specific problems with Mongo that made|forced you
switch to Riak please? "Operational qualities" are little vague

~~~
timdoug
We experienced some significant difficulties with sharding; the automatic
methods of doing so only seemed to shard a single-digit percentage of our
data. We've also encountered some wildly unexpected issues with master/slave
replication and related nomination procedures.

You're right that this post is vague with regard to those details; they would
be a good candidate for a future blog post, but the desired takeaway from this
one is that we're quite pleased with the performance and scalability that Riak
provides.

