This is an interesting modern solution. I agree with the JSON flat files on S3 for event storage to start. Athena is cool but I've also always felt that the pricing model is weird being based of amount of data scanned (not data returned by the query). That said I don't have much experience partitioning or bucketing for Athena to optimize this such as the ideas mentioned in [1]. And the whole per query thing, the more you query your data, the more you pay.
It would be fair to argue that the pricing for small amounts of data is low enough similar to Lambda pricing that it's not worth worrying about or that one can optimize costs by using a columnar format like Apache Parquet over JSON. But it's just a concern to be cognizant of vs e.g., Redshift, where you're paying a flat rate instead of per query. Overall, I do think you're right with the "serverless analytics" approach even if not many companies have adopted a "stack" like this yet.
I would be curious to hear how you would evolve your v0 solution into a v1 solution as it grows.
I'm not the OP, but I can report on how we evolved our BI analytics at Appcues.
We started out writing events into a Postgres table, but BI queries were slow and Postgres was an expensive place to put the events.
So then we started writing the events into S3 in batches of 10,000. That number was chosen semi-arbitrarily, intending that any Lambda function would be able to process an entire batch within the 5 minute execution limit. We started also keeping aggregate stats on this data by updating counters and HyperLogLog estimators in a Redis store, updated as each batch hits S3. Athena became our BI tool.
About a year after that, we'd evolved the system so that we were splitting events by customer (instead of being an arbitrary time-slice of a day's traffic). This became a suitable backend for a customer CSV export system, as well as making BI cheaper by allowing us to zero in on the customer(s) we were curious about.
And in the last year, we've begun to use the batched event data in S3 to feed a Snowflake DB (via Snowpipe), which we use for both offline BI and online analytics as part of our product. Snowflake is not free and requires some sophistication, but it supports the leading analytics tools and visualizers, and it's part of a direct evolution from keeping JSON files on S3.
My experience with Athena was that cost was negligible compared to alternative options, but YMMV. As in all things cloud pricing related, you should experiment and measure once price becomes worth thinking about.
Partitioning by date gets you really far because the vast majority of queries are interested in a specific timeframe. If your data is growing, but the ingestion rate is not, v0 might work forever.
v0.5 is simply to add another level of partitioning. This can get you an order of magnitude performance gain in many situations. I've often not needed to go past this.
v1 is a different beast. The nice thing about the simplicity of v0 is that it gives you time to learn the requirements for v1 (including whether you even need a v1) and the data is JSON on S3 so it's never hard to migrate it somewhere else. v1 is usually driven by very specific, known requirements, so there's no generic answer. For many requirements, it's useful to put a queue like Kafka (powerful) or Kinesis (convenient/cheap) in front of S3 and have a Spark job performing actions on that queue.
If speed of a single query is the issue, too many small files could be the root problem. Then, a good option is to read batches from the queue and write them to s3 as a single Parquet or ORC file. This could also be a good option if the issue is the cost of s3 API calls (when you're looking at 10k JSONs per second, PUT calls get very pricy), but at large enough scale, you might be better off using Redshift.
With Postgres or Redshift, you can run into contested resource problems when the number of users querying the data scales. I haven't found that to be a problem with Athena due to separated compute and storage, but if it is, you could maybe replicate data to another AWS account. Or, depending on the query patterns, you can build a higher-level table (e.g. given a series of events, tell me the most current state for every entity) from the raw data, either using ETL or by writing a new Spark job that reads from the Kafka queue, figures out the current state and writes the higher-level data a new s3+Athena table.
If the issue is time-to-answer (e.g. if there is a problem, how quickly will that be apparent in your analytics since batched writing introduces delay), streaming analytics could be the solution. You can also use the lambda architecture, but that's always looked too hard to implement and maintain to me.
If the issue is auth/data visibility in an enterprise context, I would lean towards using a fully featured enterprise solution, preferably one that uses a well-documented format on s3 as the underlying datastore with separated compute and storage. Alternatively, you could build or buy a query layer that sits on top of Athena that handles the permissions (e.g. Looker).
At a certain point, you should consider just adopting Snowflake as they have solved many of these problems already and is building an analytics solution really your core competency?
It would be fair to argue that the pricing for small amounts of data is low enough similar to Lambda pricing that it's not worth worrying about or that one can optimize costs by using a columnar format like Apache Parquet over JSON. But it's just a concern to be cognizant of vs e.g., Redshift, where you're paying a flat rate instead of per query. Overall, I do think you're right with the "serverless analytics" approach even if not many companies have adopted a "stack" like this yet.
I would be curious to hear how you would evolve your v0 solution into a v1 solution as it grows.
[1]: https://aws.amazon.com/blogs/big-data/top-10-performance-tun...