Hacker News new | past | comments | ask | show | jobs | submit login

Do you have recommendations for data warehousing? Our data does have a fixed structure at the moment.



BigQuery and Snowflake are the two managed services I'd recommend today if you'd like good performance and cost-effective storage. They both separate compute from storage so that your cold data isn't sitting on expensive SSDs like your Postgres instances are probably using.

They're both also significantly faster than Postgres at large scans and aggregations.

Snowflake is the most interesting to me because they offer a semi-structured data type called VARIANT which efficiently encodes semi-structured data in a column-wise format while losing only a tiny bit of performance compared to a fixed schema. This could let your customers send semi-structured or variable size data (like arrays or maps with arbitrary keys) and still keep your dashboards fast.

If you'd like to chat more, I just requested to connect with you on LinkedIn.


BigQuery is /terrific/, for in-house analytics. It would very likely not be appropriate for backing a SaaS, at $5 per terabyte scanned.

I would suggest the OP is just fine with Postgres for awhile. They can shard it when needed.

Then eventually they can either get more sophisticated with Postgres sharding, or move to something like TiDB, clickhouse, or another event store.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: