Tangent question. What do you think about the vision of the future of storage from folks at ScyllaDB have?
There is a great presentation from their recent conference [1]. I am asking this because I find it quite similar to what you are doing - decoupling storage from compute and utilising S3. I really appreciate if you could share your insights.
I had a look at this passage of the talk. Generally S3 is more obvious for OLAP where insert are very batchy and reads are large.
For OLTP, the equation is less obvious.
I'd be scared to suffer from the cost associated with PUT/GET requests.
(Pushing once every 30s or so equates to a few dollars per month.)
Since Scylla is based on an LSM-Tree (I think), I would have expected the talk to be about saving sstables on S3 but keeping the WAL on disk... But the slides seem to point to pushing the WAL to S3.
There is a great presentation from their recent conference [1]. I am asking this because I find it quite similar to what you are doing - decoupling storage from compute and utilising S3. I really appreciate if you could share your insights.
[1] starts at 30:45 https://m.youtube.com/watch?v=ZX7rA78BYS0