Hi Pino, you mentioned in the article that Dynamo was built in response to an internal demand. Can you talk at all about the specifics of what sort of information Amazon puts in there?
I've overseen a few big Dynamo installations at this point, but for the most part I wouldn't store any "core" information in there (say like a customer, merchant, or product at Amazon) because after adding data to Dynamo, your ability to query and restructure it is quite limited. We've generally put data in that's append-only, is considerable in volume, and which we only ever need to retrieve in a few particular ways. To go back to the Amazon example, I probably wouldn't put a customer, merchant, or product in it, but I might put in a log entry for a purchase transaction.
What I'm trying to suss out is whether the sweet spot product like Citus is more akin to a Dynamo, or more akin to a traditional RDMS.
Hi Brandur, I'll answer your final question first, then circle back. CitusDB is definitely more akin to a traditional RDMS that to Dynamo because it keeps almost all the goodness of relational databases while providing the ability to scale horizontally via clever partitioning, usually on the tenant and some other primary key (for the multi-tenant, B2B use-cases).
In contrast, Dynamo sacrificed all sorts of guarantees that relational database clients had relied on for decades - partly justifying your comments.
I left Amazon in 2009 so I can't speak to how Dynamo is used internally today (and AFAIK AWS DynamoDB is based on a different design). However, while I was at Amazon the shopping cart data was migrated to Dynamo. That was definitely "core" to the business in the sense that it was a key part of the user's experience and we couldn't lose the contents of a single shopping cart - it's a bad user experience. And for that use-case we could change the per-cart schema on write (with additional scans to clean up carts that were written to old schema versions).
> And for that use-case we could change the per-cart schema on write (with additional scans to clean up carts that were written to old schema versions).
why would cart schema change on a per-cart basis ? I'm kind of wondering what usecase would this solve ?
Frankly, it's been a while since I worked on Dynamo so I can't say for sure. I don't see a fundamental reason not to support null and zero-length. On the other hand it seems more like a convenience than a necessity, since you can encode those.
You may have different use-cases in mind, but it seems that if you're writing nulls you're using the key-value store to distinguish between keys that do or don't exist - basically storing a large set rather than a large map. And in that case you can write a small value to encode null and you won't change the value (rather you will delete the key-value entry).
I think this takes us to an API design discussion - I don't feel strongly about it. But I'd love to hear if that caused you significant burden or trouble in your application.
To be clear, I'm talking about nulls and zero-length strings/bytearrays as values, not as keys. I suppose it does boil down to API design though, yeah.
I guess I'm saying: Querying for something and getting null, "", or bytes("") has a different meaning and - crucially - a different type than undefined or DNE. Supporting these just seems like such a straightforward win in terms of language interop and usability that I was curious what the constraints and such were that caused the current state of affairs. If it's been a while, I certainly understand. Just something that I've wondered for a bit.
