

Scaling Riak at Kiip - oinksoft
http://basho.com/blog/technical/2012/05/25/Scaling-Riak-At-Kiip/

======
tptacek
It's been a riak-y day for me, so I watched this avidly.

You're no doubt wondering what the issues were; they come 20 minutes in, and
were (paraphrased):

* At scale in production, adding a new node took days to complete all the handoffs; they recommend adding new nodes as soon as it's looking like you need them, rather than waiting until you're redlining.

* 2i is slow, especially in EC2; a straight KV "get" is milliseconds-denominated; 2i index queries were taking multiple seconds. Use 2i, they say, but in background processes.

* Javascript MapReduce is slow; this is well known. They confirm Erlang MR was adequate.

* As the LevelDB keyspace grows, there's a stepping function in latency; 5ms, then 15ms, then 25ms; the solution is to add nodes. (LevelDB is Google's KV store, a new option for Riak, required if you're using secondary indexes).

* Riak Control didn't work for them over low-latency connections.

* Once, a Chef misconfiguration left the whole cluster flapping on and off, which corrupted the cluster; they recovered with Basho support. Be careful about adding and removing nodes rapidly.

* Similarly, flapping a single node caused the cluster to get into a state where it wouldn't converge again; the cluster worked but no nodes could be added until they (presumably?) restarted it.

~~~
dsl
After watching the video, the takeaway for me was don't run $database on EC2
instances.

Last night I just finished replacing 6 maxed out medium instances with one
$100 box from SoftLayer. :/

~~~
bravura
At that price point, it seems that you can get FAR better specs on Honelive.
Kimsufi.ie is even cheaper, if you don't mind your server being hosted in the
UK.

edit:

Softlayer: Intel 4x2.40GHz, 2 GB RAM ECC, $160/mo Honelive: Intel i7-2600 16,
16 GB RAM, $120/mo Kimsufi.ie: Intel i7 4x 2(HT)x 2.66+ GHz, 24 GB, $60/mo

~~~
StavrosK
Has anyone used Kimsufi? Are they any good? I'm at Hetzner right now, but I
never found any use for that box, so I'm just paying for it for nothing...

~~~
bravura
I just signed up for kimsufi. They are the budget brand of OVH. Webhostingtalk
gives them favorable reviews.

------
Scramblejams
Something they (understandably) didn't address in the video was fault
tolerance on the Postgres side. What do people like these days for that?

~~~
mitchellh
I'll address that here: We use streaming replication with a hot standby for
fault tolerance on the PostgreSQL side. For backups in addition to the
streaming replication we do WAL shipping to S3 every 15 minutes or 16 MB, and
we do a full base backup to S3 daily during our low-traffic time of the day,
which uploads around 30 GB to S3 daily (but LZO-compressed, so actually down
to around 3 GB).

~~~
Scramblejams
Thanks, Mitchell. Is your hot standby in a different zone (e.g. one in the
West, one in the East) so you can handle one of their big outages?

~~~
mitchellh
Yes

------
platform
in the last couple of month reading this site, I see more negative MongoDB
reviews than most other 'cool tech' stuff. On one hand MongoDB by carrot-or-
stick was pushed into environments with high-write needs (OLTP kind of
systems). On another hand having true secondary indexes and semistructured
data makes Mongo a 'closest to RDBMS' choice. So things like global write
lock, indexes must fit into memory, auto-sharding questions, single-thread
map-reduce -- all are pretty significant limitations for an OLTP data store. I
wish MongoDB does not get discouraged and instead steps back, reviews academic
foundations of the system and pick a couple of use cases and optimise their
builds for them (similar to datawarehouse vs oltp kind of offerings)

------
minaguib
The glossing-over cassandra (because of Digg history etc..) was the point in
the talk when I started raising eyebrows. Engineers evaluating systems should
let the systems speak for themselves.

We have a fairly small cassandra cluster in production, serving over 50x the
volume they mentioned in their talk, with good latency (real-time bidding) and
not-too-painful operational footprint.

------
gsibble
Great talk! Looking at Riak vs MongoDB right now for a production system in
fact. The data isn't K/V though and we need rich queries so I'm not sure what
our solution will be unfortunately.

~~~
bfrog
Why not just use postgresql?

~~~
taligent
Because Postresql is not a NoSQL database despite the recent lipstick
additions.

I love the database but it is not optimal if you have a fluid schema like many
use cases have today.

~~~
lwat
I recently asked if anyone could point me to an (open source) app on github or
wherever that has a 'fluid schema' on a NoSQL system and nobody was able to
show me one. Do you have a concrete example I can look at please?

------
reality_hacker
Anything concrete about why cassandra is bad?..

~~~
gregburd
One key difference is that Riak will rebalance data across nodes as they are
added or removed automatically, Cassandra will not. You have to manually
adjust the partitioning of data, balancing it by hand.

