
Amazon DynamoDB Transactions - leef
https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-transactions/
======
abalone
Any experience with using Aurora in place of DynamoDB?

A couple years ago there was an interesting tidbit at re:Invent about
customers moving from DynamoDB to Aurora to save significant costs.[1] The
Aurora team made the point that DynamoDB suffers from hotspots despite your
best efforts to evenly distribute keys, so you end up overprovisioning.
Whereas with Aurora you just pay for I/O. And the scalability is great. Plus
you get other nice stuff with Aurora like, you know, traditional SQL multi-
operation transactions.

It was kind of buried in a preso from the Aurora team and the high-level
messaging from Amazon was still, NoSQL is the most scalable thing. Aurora was
and is still seemingly positioned against other solutions within the SQL
realm. I sort of get it in theory that NoSQL is still _theoretically_
infinitely scalable whereas Aurora is bounded by 15 read replicas and one
write master.. but in practice these days those limits are huge. I think one
write master can handle like 100K transactions a second or something.

So, I'm really curious where this has gone in the past couple years if
anywhere. Is NoSQL still the best approach?

[1] [https://youtu.be/60QumD2QsF0?t=1021](https://youtu.be/60QumD2QsF0?t=1021)

~~~
coder543
I personally recommend using a SQL database until you're absolutely positively
sure you don't need one, for many reasons.

But, as far as the "you end up overprovisioning" because of hotspots thing,
DynamoDB _does_ offer autoscaling these days, which should alleviate a lot of
provisioning-related headaches and save you money compared to the provisioning
you would have done with DynamoDB, from what I understand.

~~~
orthecreedence
We use a hybrid. We process a lot of incoming data and dump most of it into
dynamo (it's ephemeral so the TTL feature is nice) and if we get capacity
errors (Dynamo takes a while to scale up sometimes) we just dump our objects
in the DB. The end result is we keep a huge amount of writes off our DB for
processing incoming largish objects. The amount of data it stores would cost
an arm and a leg to put into redis.

Granted, I don't think I'd want to use Dynamo for anything other than
temporary data. Lock-in makes me nervous, and the way it scales up/down really
makes it difficult to use it for hourly workloads...by the time it scales up
we're close to done needing more capacity, then it doesn't scale down for like
40m after. We set up caps and the DB overflow machanism keeps things from
grinding to a halt.

~~~
abalone
Why don't you use Kinesis for this? Isn't that what it's made for?

------
piinbinary
My wishlist for DynamoDB is now down to:

* Fast one-time data import without permanently creating a lot of shards (important if you are restoring from a backup)

* Better visibility into what causes throttling (e.g. was it a hot shard? Was it a brief but large burst of traffic?)

* Lower p99.9 latency. It occasionally has huge latency spikes.

* Indexes of more than 2 columns

* A solution for streaming out updates that is better than dynamodb streams

~~~
monstrado
Also, better insight into partition sizes / what's causing hot spotting. The
DB abstracts _a lot_ from the user, which isn't necessarily great, because
it's still subject to the normal pitfalls of a NoSQL database.

~~~
tejasmanohar
Bigtable is a different beast, but it's new "Key Visualizer" is impressive.
Has helped us quickly find anomalies
[https://cloud.google.com/bigtable/docs/keyvis-
overview](https://cloud.google.com/bigtable/docs/keyvis-overview)

Wish Dynamo had something similar

------
jchrisa
Congrats to the DynamoDB team for going beyond the traditional limits of
NoSQL.

There is a new breed of databases that use consensus algorithms to enable
global multi-region consistency. Google Spanner and FaunaDB where I work are
part of this group. I didn’t catch anything about the implementation details
of DynamoDB transactions in the article. If they are using a consensus
approach, expect them to add multi-region consistency soon. If they are using
a traditional active/active replication approach, they’ll be limited to
regional replication.

~~~
erik_seaberg
They warn about other regions seeing incomplete transactions (if you opt into
transactions on global tables), which fits with the current "copy each new
item from the stream" async replication.

------
ryanworl
“DynamoDB is the only non-relational database that supports transactions
across multiple partitions and tables.”

Uh... this is just not true.

~~~
otterley
Can you identify some others?

~~~
stickfigure
The Google Cloud Datastore (formerly the "App Engine Datastore") has had
cross-entity-group transactions since 2011:

[https://googleappengine.blogspot.com/2011/10/app-
engine-155-...](https://googleappengine.blogspot.com/2011/10/app-
engine-155-sdk-release.html)

~~~
airwot4
The cross group transactions are a little limited -
[https://cloud.google.com/appengine/docs/standard/java/datast...](https://cloud.google.com/appengine/docs/standard/java/datastore/transactions#what_can_be_done_in_a_transaction)

I don't think it's fair to compare them.

However, the more recent Google storage offerings based on Cloud Spanner do
seem to offer this. I don't see how Amazon can make this statement - that
doesn't stop it being an excellent enhancement to DynamoDB though.

~~~
wsh91
Cloud Firestore (the next generation of Cloud Datastore) removes those
limitations. [https://cloud.google.com/firestore/docs/manage-
data/transact...](https://cloud.google.com/firestore/docs/manage-
data/transactions)

It also supports the Cloud Datastore API.

(I work on it!)

------
jedberg
This is cool, it lifts the burden of having to bake "atomicity" into your app
if you're using a key/value store like DynamoDB. I can see a nice balance of
combining this with some built in error checking in the app itself.

------
jared2501
I'd be interested to see comparisons/benchmarks against FoundationDB. DynamoDB
transactions make dynamo a serious alternative to FDB now. I can see the two
manage advantages for FDB being: 1) you can deploy it on premise (which is
potentially important for some B2B companies), 2) it shuffles data around so
that hot-spotting of a cluster is eliminated (which dynamo appears to still
suffer from).

~~~
wahnfrieden
#2 appears to be solved now [https://aws.amazon.com/blogs/database/how-amazon-
dynamodb-ad...](https://aws.amazon.com/blogs/database/how-amazon-dynamodb-
adaptive-capacity-accommodates-uneven-data-access-patterns-or-why-what-you-
know-about-dynamodb-might-be-outdated/)

------
psankar
Postgresql gets native JSON support (at least since 9.2 onwards) to store
schemaless free flowing text. Dynamodb gets transaction guarantees.

There is globalization and intermingling happening on technology too.

On a similar thought, a few years back, C# and Java got `Any` generic types,
while Python/JS got static types (via python3 typings, typescript)

~~~
lozenge
C# doesn't have an Any generic type (Foo<?> In java parlance)

~~~
manigandham
C# can use `object` or `dynamic`:

[https://stackoverflow.com/questions/2690623/what-is-the-
dyna...](https://stackoverflow.com/questions/2690623/what-is-the-dynamic-type-
in-c-sharp-4-0-used-for)

------
asien
>If an item is modified outside of a transaction while the transaction is in
progress, the transaction is canceled and an exception is thrown

You are still responsible to implements a Queue or a Lock on the Items you
want to mutate.

That said this is a huge milestone for DynamoDB, we can now safely mutate
multiples items while remaining ACID.

------
grogers
Max 10 items per transaction, that's quite a restriction! I guess you have to
plan all the transactions you would perform and make sure they meet the
bounds.

------
Thaxll
Is the heat map available to customers now or is it still a request you have
to do?

~~~
darkr
Still have to request it AFAIAA

