Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Postgres starts to struggle when you push it past 10TB, the details of which will depend on your specific data model and workload. I've seen it pushed to 50TB but I would not recommend it. The architecture simply isn't designed for that kind of scale.

There are fast database engines specifically designed for single servers with 1PB of storage, and these servers commonly have hundreds of cores. This is much more efficient for some use cases. It is the kind of thing you find on edge platforms designed to manage and operate on sensor data models. You build database engine internals very differently when working at this storage density and number of cores; many good architectural ideas in databases at much smaller scales become severe bottlenecks even on a single server.



Thanks for the reply - I'm not super familiar with databases. Do you have a few in mind (preferably open-source for reference)?

Also, are you referring to a single-instance of Postgres, or that Postgres would struggle with 10TB even with something like Citus or TimescaleDB?


I was referring to a single instance of classic PostgreSQL running on a single server. Scale-out databases are a completely different animal, it is difficult to compare them to scale-up databases because they make a different set of tradeoffs.

Extreme scale-up database engines are relatively rare and usually bespoke, I am not aware of anything close in open source. They only have one major advantage -- they are easily deployable outside the data center, e.g. at the edge, because it is a single self-contained box and relatively power efficient. Most people opt for scale-out because 1) there are many to choose from in open source and 2) it will work just as well as extreme scale-up for most applications, though you'll need more hardware for the same workload.

I've started to appreciate the utility of the extreme scale-up engines in real-world applications, though I've mostly worked on scale-out engines. They solve a real problem at the edge for data intensive applications but most currently people deploy in the data center so there isn't a pressing need.




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: