
Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS - werner
http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html
======
mark_l_watson
A feature request (I see that Werner is reading this message thread): for
development it woud be very nice to have the Java AWS SDK have a local
emulation mode for SimpleDB and DynamoDB. This would allow development while
flying or otherwise off the Internet. Similar functionality as AppEngine's
local dev mode.

~~~
icey
Since HN doesn't show vote totals I wanted to chime in and say this would be
really useful to me as well.

~~~
viggity
HN doesn't show vote totals, but it does sort posts based on the number of
votes (descending), so the more popular posts rise to the top, just like the
front page.

~~~
scotth
That isn't how the frontpage works, and if it did, we would rarely have news
turnover.

~~~
RyanMcGreal
Well, yes and no. Upvotes push submissions toward the top of the page, but the
ranking algorithm also includes a decay factor so that old articles eventually
fall away.

------
lemming
This looks like a great service - there are some really interesting ideas
here. The provisioned throughput is a great idea. It's interesting to see it
runs on SSDs as well, I wonder if its storage is log based like LevelDB. The
composite hashed indexes are really interesting as well - I guess they
partition the index across nodes, and each node maintains a range of the
sorted index. It'll be interesting to see how usable that is in practise.

I read with interest Steve Yegge's mistakenly-leaked email about the
oppressive conditions for engineers at Amazon. It's hard to reconcile with the
sort of innovation they consistently show.

~~~
jsolson
> I read with interest Steve Yegge's mistakenly-leaked email about the
> oppressive conditions for engineers at Amazon. It's hard to reconcile with
> the sort of innovation they consistently show.

My take on Steve's rant (as an engineer at Amazon) is that a lot of the issues
he pointed out are legitimate, but at an entirely different scale than he was
pitching them at.

Day to day I work on a product with another couple dozen or so engineers. We
build what makes sense for our product, and for the most part we build it in a
way that makes sense for us. Sometimes we are under pressure to leverage other
parts of the platform, and sometimes that does entail a lot more work. Most of
the time, though, it ends up reducing our operational load (because the
systems we depend on support products much larger than ours :) and giving us
someone we can page when things go pear-shaped.

Amazon isn't the perfect place to work, but it's generally not bad (other than
the frugality thing; that sucks as an employee no matter which way you slice
it).

~~~
lemming
Interesting, thanks for the perspective - he definitely made it sound like a
sweatshop, and that's not the sort of environment normally associated with
this kind of innovation.

------
ridruejo
I simply do not see how competing cloud vendors can keep up with this. Most of
them are still struggling to provide anything beyond a simple API to
start/stop machines.

~~~
balakk
Well, Azure has been offering a similar NoSQL service for a few years now.

[http://sigops.org/sosp/sosp11/current/2011-Cascais/printable...](http://sigops.org/sosp/sosp11/current/2011-Cascais/printable/11-calder.pdf)

~~~
ridruejo
No. They offer the equivalent of Amazon S3, SimpleDB and SQS, but nothing
comparable to this

~~~
balakk
Can you elaborate how this is different and not comparable? Azure's table
service offers the same automatic partition management, unlimited per-table
scalability, composite keys, range queries, and availability guarantees. The
linked paper goes into more details.

~~~
vyrotek
Besides the points I already complained about before... How about 200ms
response times even when performing a query using the Row & Partition Keys.
I'm not sure if by composite keys you were referring to something other than
the RK & PK because those are the only indexes you get.

~~~
balakk
ATS response times within the Azure data center are pretty impressive in my
experience.

Your partition keys can be composite, have a look here:

[http://blogs.msdn.com/b/windowsazurestorage/archive/2010/11/...](http://blogs.msdn.com/b/windowsazurestorage/archive/2010/11/06/how-
to-get-most-out-of-windows-azure-tables.aspx)

I agree with your other pain points - in terms of not being able to get
counts, secondary indices etc. However, you can easily simulate some of those
- maintain your own summary tables, indices and so on. These ought to emerge
as platform features pretty soon though. It's not perfect, but its feature set
is close to Dynamo.

As for Mongo DB, I guess this service has been built from ground-up to provide
the availability guarantees and automatic partition management features. I
don't know if Mongo provides those. You could run Mongo yourself on Azure if
you wanted to; there's even a supported solution done recently.

~~~
vyrotek
Hmm, I guess when I think about composite keys I think of ways to indicate a
specific field/column as being part of the key. Data duplication along with
string concatenation aren't really an elegant way to do it. If I remember
right you also can't update the key values once the record has been saved.
This is coming from a big SQL guy though :)

------
Ataraxy
This still seems a bit expensive to me for an application that would require
thousands of writes per second? ie. 5k writes per second is ~$120/day. Using
this for performance based analytics for example would seem out of the realm
of reason for the moment.

~~~
peripitea
Can you explain your use case a bit more? I'm having a hard time imagining
something that does ~430M DB writes/day but can't easily afford to pay $120
for those writes.

~~~
Yrlec
Remember that the throughput is per item, not per query. For instance we have
an indexed query that returns ~1500 rows each time. Just doing that query a
couple of times per second would create that kind of throughput requirement.

~~~
werner
The amount of consumed read units by a query is not necessarily proportional
to the # of items. It is equal to the cumulative size of processed items,
rounded up to the next kilobyte increment. For example if you have a query
returning 1,500 items of 64 bytes each, then you’ll consume 94 read units, not
1,500.

~~~
latch
The official documentation seems to clearly contradict you. The pricing
calculator doesn't let you specify a value of less than 1KB. Who's right? Or
maybe I'm just not understanding what either you or the official pricing doc
is saying :)

~~~
Aqua_Geek
From the pricing page (<http://aws.amazon.com/dynamodb/pricing>):

\---

If your items are less than 1KB in size, then each unit of Read Capacity will
give you 1 read/second of capacity and each unit of Write Capacity will give
you 1 write/second of capacity. For example, if your items are 512 bytes and
you need to read 100 items per second from your table, then you need to
provision 100 units of Read Capacity.

\---

Looks like 1KB is the minimum for calculations.

~~~
latch
Agree, but Amazon's CTO said something different, hence my question.

~~~
davelang
Werner is right. The query operation is able to be more efficient than GetItem
and BatchGetItems. To calculate how many units of read capacity will be
consumed by a query, take the total size of all items combined and round up to
the nearest whole KB. For example, if your query returns 10 items that were
each 1KB, then you will consume 10 units of read capacity. If your query
returns 10 items that were all 0.1KB, then you will consume only 1 unit of
read capacity.

This is currently an undocumented benefit of the query operation, but we will
be adding that to our documentation shortly.

------
mark_l_watson
For about $15/month for minimal "reserved capacity" charges and $1/gigabyte
per month replicated disk space this service looks like it will cure a lot of
deployment headaches for a reasonable cost.

Interesting that when I signed up for the service that they verified my
identity with a cellphone call, like Google sometimes does.

------
dannielo2
"Amazon DynamoDB stores data on Solid State Drives (SSDs)" This is big.

~~~
onk
When Amazon notes "SSD," from a client's perspective, it is only marketing.
The storage media matter when you are managing your own hardware. The storage
media do not matter in SaaS. For example, the storage media could be floppys
and you would get satisfactory performance if there was a memory cache.
Similarly you could get poor performance with SSD media if the networking
layer(s) were slow. Similarly if the storage media were fault-likely CD's that
wouldn't matter either, to us, because of the data replication performed in
"the service." What matters in this case is the reported and actual latency.

------
swatermasysk
I would love the option to add N index (at cost).

My guess is they will add options for additional indexes in the
future...everyone needs start somewhere. Even at Amazon's size and scale.

~~~
jsolson
> Even at Amazon's size and scale.

At Amazon's size and scale, it's all the more important that you start with
something simple with well-understood performance characteristics from day
one. AWS doesn't really get a grace period during which they get to fix
scalability problems.

~~~
adpowers
Exactly, compare this to SimpleDB. SimpleDB started out with an advanced query
language that let query and filter your results in all sorts of ways. And
guess what? SimpleDB is still limited to 10 GB per domain (aka, database).
Want to horizontally scale? The official suggestion is to shard your data
across domains. This is a really messy solution because you have to preshard
based on estimated database size and resharding is nearly impossible (you'd
have to rewrite your entire DB).

AppEngine went the other route and provided a very simple database API at
first and all queries had to be range scans over an index. Any query you
wanted to perform had to be precalculated by defining a composite index and
some things (like inequalities on multiple fields) weren't supported. Over
time they've built upon their basic database and added features such as a
zigzag merge join algorithm which lets you perform queries that were otherwise
impossible with a given set of indexes.[1]

I bet DynamoDB will be going the AppEngine route by starting with a simple,
scalable base which can be used to build more advanced query engines and
features.

1\.
[http://code.google.com/appengine/articles/indexselection.htm...](http://code.google.com/appengine/articles/indexselection.html#Improved_Query_Planner)

------
mikehuffman
While putting my data completely in the hands of another company makes me
nervous, I have used s3 since its release and have had no problems. Amazons
really seems to "get" developer needs. I do wish they would ease down a smidge
on thier pricing in general, but otherwise I really feel about them the way I
feel about google at this point.

------
nosequel
[https://img.skitch.com/20120118-etwh2gwu52isnn47fw2jpcir5p.p...](https://img.skitch.com/20120118-etwh2gwu52isnn47fw2jpcir5p.png)

You could buy a lot of Riak or Cassie for that.

~~~
werner
5000 reads per sec of 64Kb items, would make you stream 2.5 Gbits/sec using
consistent reads and 1Gbits/sec writes, moving close to 1.5TB each hour. At
the end of the month you have read well over 800 TB and updates 160 TB... That
is a substantial application you have in mind... :-)

~~~
tedivm
That may be true for an application with a constant load, but applications
with a less balanced load have to provision for their peaks. My company
(Malwarebytes) has very irregular traffic (at the hour mark we get very big
spikes, but only for a couple of minutes) and it seems like we would have to
provision (for this specific app) that peak for the entire hour. I might be
misunderstanding the billing for this service though- if we ask for more units
for 15 minutes, would the billing be prorated?

This actually hits on my only real issue with AWS in general, which is the
hourly billing. We've used the mapreduce service a bit, and having a cluster
roll up and fail on the first job is heartbreaking when it has a hundred
machine hours associated with it. Obviously that is far, far cheaper than us
building out our own cluster (especially with spot instances, which I can't
even describe how much I love), but for some of the services smaller billing
times would be useful.

------
ytNumbers
Here's a brief overview comparison of DynamoDB vs. BigTable:

<http://vschart.com/compare/dynamo-db/vs/bigtable>

The site is also able to compare many other different databases.

~~~
dhimes
It seems as if this site is a user-edited wiki, and there are a lot of things
that need to be filled in (in case anybody is up to speed on DynamoDB and
wants to help). For instance, the Map/Reduce entry was still '?' when I wrote
this.

~~~
lyddonb
And Big Table (or at least GAE) does support transactions.

------
encoderer
I work for a popular startup that has been privately testing dynamo for the
last few months.

It's a fantastic product and even while in private beta has been stable and
well supported.

------
Loic
1$/GB/month on SSD and replicated. So basically, 0.25$/rawGB/month if they
replicate 4 times.

They are making money on the read/write and are selling the capacity at
current cost. Which — knowing the tendency for AWS to decrease the prices very
slowly combined with the huge decrease of the prices of the SSD drives in the
past months/years — is not a bad strategy to convince us to switch.

------
jbellis
I posted a comparison to Cassandra here:
<http://news.ycombinator.com/item?id=3480480>

------
dannielo2
There are some bugs in signing up for the service. In console, if I go to
dynamoDB tab I am asked to Sign Up first. I click the sign up button and I am
told "You already have access to Amazon DynamoDB". Repeat.

~~~
werner
Can do me a favor and drop that in the AWS DynamoDB Forum so folks can look at
it? <https://forums.aws.amazon.com/forum.jspa?forumID=131>

------
andrew311
Question about Composite Hash Keys that someone might have the answer to (or
be able to relate to other known implementations):

The composite key has two attributes, a “hash attribute” and a “range
attribute.” You can do a range query within records that have the same hash
attribute.

It would obviously be untenable if they spread records with the same hash
attribute across many servers. You'd have a scatter-gather issue. Range
queries would need to query all servers and pop the min from each until it's
done, and that significantly taxes network IO.

This implies that they try to keep the number of servers that host records for
the same hash attribute to a minimum. Consequently, if you store too many
documents with the same hash attribute, wouldn't you overburden that subset of
servers, and thus see performance degradation?

Azure has similar functionality for their table service, requiring a partition
key, and they explicitly say that there is a throughput limit for records with
the same partition key. I haven't seen similar language from Amazon.

Whether you scatter-gather or try to cluster values to a small set of servers,
you'll eventually degrade in performance. Does anyone have insight into
Amazon's implementation?

------
sehugg
Awesome. Anybody want to sublease a 3 yr RDS reserved instance? :P

------
bconway
What's the concern for lock-in here - how difficult with DynamoDB be to
migrate away from? Amazon seems to be increasingly catching the App Engine
Syndrome, though to be fair, it's been mostly network traffic-style
functionality in the past.

~~~
latch
Lock-in? You'll need to get all your data out and transform it into whatever
alternative you pick. I'd say as far as AWS lock-in, this is on the low side
of things.

~~~
justinsb
The lock-in comes not from the data format, but rather from the quirks of the
particular system du jour: API lock-in.

~~~
foobarbazetc
Getting your data out will also cost you a boat load of money. Same as putting
it in.

------
zerosanity
From the limits page(1) I see you can only have 256 tables per account.

(1)
[http://docs.amazonwebservices.com/amazondynamodb/latest/deve...](http://docs.amazonwebservices.com/amazondynamodb/latest/developerguide/Limits.html)

~~~
werner
As with all the other AWS services you can have your limits lifted upon
request.

~~~
yahelc
Does that apply for the 50-per-account cap on S3 buckets as well?

~~~
ceejayoz
I'd imagine S3 falls within the scope of "all the other AWS services"...

------
meta
I read through a number of the docs and can't quite find the answer to this
question, hopefully someone here can help me out quick.

I already have a bunch of (large-ish, deeply nested) JSON objects defined for
my application. I don't really want to go about redefining these since they
work great between my various node processes and the front end. I am saving
them in a nosql database already, I am curious about switching (to save on
devops costs). I only request based on 1 Hash Key (int) and 1 Range Key (int)
for all my current get operations.

Looking through the docs/examples I see a lot of this type of thing:

    
    
        {"TableName":"comp5",
        	"Item":
        		{"time":{"N":"300"},
        		"feeling":{"S":"not surprised"},
        		"user":{"S":"Riley"}
        		},
        	"Expected":
        		{"feeling":{"Value":{"S":"surprised"},"Exists":true}}
        	"ReturnValues":"ALL_OLD"
        }
    

The JSON item has a kinda-of 'type syntax' on it. I really don't want to
redefine my deep objects, but would be willing to redefine the Hash key and
Range key, while leaving the rest of the nested types alone.

Ok, my question: Do my JSON objects need to conform to this 'type syntax' JSON
notation in the examples? Or can I save just any JSON object into this
database and only annotate the Hash Key and Range Key using this special
notation?

~~~
ww520
Their usage of JSON is just incidental to your usage of JSON. They use JSON as
a REST transfer format. You can pretty much ignore their JSON if you use one
of the high level libraries in the SDK.

You can define a table with 3 fields: yourKey, yourRange, and yourJson. Put
your entire JSON data as string in the yourJson field.

------
taylorbuley
Pricing: _$0.01 per hour for every 10 units of Write Capacity and $0.01 per
hour for every 50 units of Read Capacity._

~~~
justinsb
It is amusing that was positioned to try to address SimpleDB's problem of
"pricing complexity". These aren't those complicated "Machine Hours", they're
"Capacity Units"!

As someone who has grappled in the past and is grappling again with the issues
of pricing a database-as-a-service though, this is very much a non-trivial
issue. If you have nice scaling characteristics and you want to charge the
minimal price, your price isn't going to be predictable. Essentially, what
Amazon have done here is to set a cap and then charge you the cap irrespective
of actual usage, which seems to be the model that's winning on the cloud.

~~~
snewman
Capacity Units strike me as a big improvement (for the user) over Machine
Hours, because it's very clear how a given usage pattern will translate into
Capacity Units. I can predict how many Capacity Units I'll need. I've gotten
badly burned over seemingly simple queries using unexpectedly high Machine
Hours in SimpleDB.

~~~
justinsb
My understanding is that DynamoDB's Capacity Units are just a query throttle,
and you get charged based on the throttle you set, whether or not you use that
capacity. It also looks like you can still have one query that consume many,
many Capacity Units (e.g. table scans).

SimpleDB's Machine hours are basically the same units, but without the
throttle.

So, from a technical and value viewpoint, it's a huge step backwards (pay for
capacity rather than for usage), but I'm learning that psychology is perhaps
just as important here.

It seems like what you really want is a throttle with per-query charging, to
cap your bill. Probably you'd much prefer not to be forced to pay your cap
every month, but I don't think that's being offered.

(Edit: Downvotes? Am I wrong here? If so, please contribute to the discussion
and tell me why!)

~~~
snewman
True, it's pay-for-capacity and that's worse for the user. On the flip side,
the constant factor seems to be about 20x cheaper (caveat: this is based on my
personal experience with SimpleDB; since Amazon doesn't seem to explain how
"box usage" is computed, I don't know how broadly applicable my experience
is).

The big plus for Capacity Units is that Amazon actually provides a
deterministic model for figuring out what you'll be charged for a given query.

~~~
justinsb
Ah - that is a fair point, transparency of the "unit". The "box usage" formula
was reverse-engineered and shown to be fairly simple:
[http://www.daemonology.net/blog/2008-06-25-dissecting-
simple...](http://www.daemonology.net/blog/2008-06-25-dissecting-simpledb-
boxusage.html)

This is a big step forward in transparency, although I would suggest that
SimpleDB's pricing shouldn't have been obscured in the first place.

------
latch
I think this is huge. My first wish, before secondary indexes, is that they
don't round up to the nearest 1KB for pricing.

~~~
werner
noted. (both requests that is).

------
ed209
With regard to reliability, how safe would a service like this be to store
your data? They mention "back up to s3" but this sounds more like archiving.
I'm wondering about backups in the case of a problem, or is the data so spread
across all of their physical locations that data loss impossible?

~~~
ww520
SSD are pretty safe in general. Amazon also makes replicate of your data. It's
the same if not safer than other NoSQL solution. Definitely safer than those
RAM-based NoSQL approaches.

The "back up to s3" part is for archiving data, like periodic snapshot backup
of your data so that you can get back the old data in case they are deleted by
app/human.

------
rojabuck
Fast, Consistent, Secure & Replicated.

Game changed.

~~~
nosequel
Not really if you pay attention to the set of NoSQL DB's available already.
They are just hosting their own now.

~~~
rojabuck
Would be interested in details about any with Strong Consistency, decent
replication across fault areas, which get the same level of performance
(single digit ms) at a cost equivalent to dynamodb.

Quite a lot can be done with smart use of hugely scalable simple key-value
maps.

------
Johnyma22
I'm too scared of the lock in. Afaik this isn't like EC3 where you can move a
VM off the EC3 stack. The data is stored in a proprietary format so once your
in, you're in.

Game changer? I need to see some evidence first on how well it performs and
integrates.

~~~
weavejester
It's a database, so you can always get your data out again.

~~~
justinsb
It's a non-standard (NoSQL) database. Even SQL databases, despite years of
standardization and efforts at poaching each other's customers, still have
rough edges that make moving between different SQL database products non-
trivial. Just because you can get your data out doesn't mean you're not locked
in to all of SimpleDB/DynamoDB's quirks, of which there will be a lot more
because it's not following a standard approach. Your code will have to go
through contortions to work around / adapt to the SimpleDB limitations; effort
which might well be wasted or counter-productive on a different system. That's
the lock-in with AWS in general: API lock-in, not data format lock-in.

~~~
weavejester
DynamoDB and SimpleDB are also a _lot_ simpler than SQL databases. As far as I
can tell, DynamoDB is a key/value DB with support for ranges and MapReduce,
and not dissimilar to other NoSQL databases like Riak.

There may be instances where large datasets are hard to migrate from DynamoDB,
but overall it doesn't look to me like lock-in would be that much of a
problem, assuming you have a decent abstraction layer.

------
justinsb
Very interesting that there's no mention of the CAP theorem here, despite
Amazon's heavy reliance on it in the past when marketing their non-relational
stuff.

~~~
chubot
I'm very curious about this too. I posted in another thread:

"Doesn't that just sweep the latency tradeoff under the rug, or is flash
making up the difference? What about the availability tradeoff? (I like the
formulation here, consistency vs availability and consistency vs latency, as
opposed to CAP which never made sense as a 3-way tradeoff:
<http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-an...>)

<http://news.ycombinator.com/item?id=3480805>

------
simonw
I hope this means an SSD option for RDS is coming soon.

------
wagerlabs
I wrote an Erlang API for DDB while it was still private. I'm hoping my client
will let me opensource it in the near future.

------
Smrchy
Why is there a 1MB reply limit on BatchGets and obviously no limit on Gets?
Did anybody find limits on a single item?

~~~
PhrosTT
The items are limited to 64kb so its implicit.

------
n_are_q
Any word on when boto will support this?

~~~
ConstantineXVI
Sooner than you think

[https://github.com/boto/boto/commit/8ef550ec028b379bcd85e550...](https://github.com/boto/boto/commit/8ef550ec028b379bcd85e550eed564e4c9035d56)

------
hiroprot
Is there a Ruby SDK?

~~~
ConstantineXVI
Yes (which I find mildly surprising given how long the Ruby library took to
support batch ops on SQS)

<https://github.com/amazonwebservices/aws-sdk-for-ruby>

------
nirvana
Trying to read thru all the hype, this looks like Riak hosted on machines with
SSDs, with less features, and a nice billing system in front of it.

Of course for people who want a hosted solution, the fact that it is hosted,
is what gives it a lot of value. There haven't been a lot of hosted NoSQL
databases, at least on this scale and availability, out there.

But technologically, what's new here? Is there anything here that is really
innovative? (not a sarcastic question)

~~~
PanMan
For one, it responds quicker than Riak: Riak has (cold) response times of
about 300ms, while this service claims single-digit ms response times. Also
setting up Riak is not exactly trivial, and using this service outsources that
hassle.

~~~
nosequel
You can use Riak Smartmachines on Joyent's cloud that would get similar
performance for an order of magnitude cheaper than what amazon is charging. If
you are seeing 300ms response times, you are not using SSD's and you are not
using a similar number of nodes that Amazon is charging you for.

I'm not hating on Amazon, it is a good move for them and they are doing some
things that Riak cannot do, but cost and response is not one of them.

~~~
PanMan
I both wouldn't bet on Joyent's Smartmachines being a magnitude cheaper and
being as fast: On the speed: We run a Riak cluster, and for us the actual
response times we get from Riak are, as described, slower than what Amazon
promises in its docs.

On the price: If you would get 3x16 GB machines with Joyent, that would cost
you 1400$/month. You can get a lot of resources for that with this new AWS
service.

I don't have any experience with either Joyent or this new DynamoDB, but I do
have some experience with Riak, and from the docs this new service would be a
very viable competitor.

