
Ask HN: Is there an equivalent of LLVM for databases? - ThePhysicist
Being very impressed with what LLVM did for compiler generation, I wondered if there is something similar for databases?<p>What I mean is a low-level system that would allow me to make use of some primitive facilities that databases are based on (indexes, heaps, storage engines, ...) and use them to build my own database&#x2F;data processing systems. Is there such a thing?<p>I think this could be very interesting as I often find myself in a situation where I have a good idea of how to access data using a given set of indexes, but am not able to do this reliably because the query planner does give a different, less efficient plan. And please don&#x27;t tell me that the query planner always knows better, as often it does not even provide a way to specify which goal I&#x27;m optimizing for (e.g. predictable performance &#x2F; throughput vs. better but unpredictable performance).<p>Having access to such an open, optimized platform for database generation would make it much easier building systems that are highly optimized for a given use case instead of being designed for general-purpose data storage.
======
brudgers
My 10,000 foot high understanding is that the kind of performance you're
seeking is achieved by the dark arts of database tuning. At a certain level of
abstraction one might think about the query planner's default mode as context
free and your goal as adding context sensitivity to it.

At the other end of the abstraction tree, it might be that a relational
database is the wrong tool or that a row-based relational database is less
right than a column based database. If you can see how the data ought to be
organized for performance, then it might make sense to build project specific
data structures. Or use an in-memory database. There are lots of options
besides relational databases.

Good luck.

~~~
ThePhysicist
Yes I agree with what you say, sometimes it's just inconvenient to manipulate
a query planner using a high-level language instead of using a more direct
approach. Also, while many databases provide a way to work with underlying
representations of their data, it is still quite difficult or impossible to
build something entirely new (I.e. a new application) with them.

~~~
brudgers
Because I don't have details below the level of database and query, I had to
imagine them up to write an answer. It seems like your use case is a bit
unique.

