When the application would be restarted, reading back the log would recreate the in memory data structures. I thought that it was cool, and that databases themselves could be that way instead of using the programming language data structures without a networked API.
That is literally how databases work. In Memory + WAL + Data Files on disk. You could, in theory, live without the Data Files and just a big WAL.
Relational databases (except maybe MemQL) treat the file system as the source of truth. And usually the file system is bigger then the memory so they need to constantly update their cache with more relevant data.
Redis loads everything to memory. And doesn't keep the structure in the file system, only the log, recreating it from log+snapshot.
Except that most databases don't store their data in anything resembling "programming language data structures". You get tables, rows, and columns (or maybe a bit of JSON if you're lucky) instead of native integers, strings, lists, sets, and dictionaries.
The primary purpose of an ORM is to overcome the "impedance mismatch" between relational databases and programming language data structures. There's no need for an ORM if you can store your data structures directly in the database.
> ... if you can store your data structures directly in the database.
ABSTRACT. Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation). It provides a means of describing data with its natural structure only—that is, without superimposing any additional structure for machine representation purposes. Accordingly, it provides a basis for a high level data language which will yield maximal independence between programs on the one hand and machine representation and organization of data on the other.
E.F. Codd. 1970. A relational model of data for large shared data banks. Commun. ACM 13, 6 (June 1970), 377-387.
You propose to reintroduce a problem that they absolutely wanted to get rid off 40 years ago. Just imagine that you first have to figure out how to painstakingly parse serialized Python dictionaries before you can access the data in another program written in e.g. Rust.
It clearly amounts to UNSOLVING a problem that is now SOLVED ALREADY.
Well, Redis allows me to store a JavaScript array, add some more with a Ruby client, remove some items with a PHP client, and finally read it back as a Python list just fine. What's the problem that has been unsolved? :)
https://msgpack.org is absolutely fine. I wasn't criticizing msgpack. You can also serialize the bytes of an array of C structs, if you see what I mean.
If this is a primary purpose of an ORM, then I wish I knew of one that isn't utterly failing at that.
One thing I've learned about this "impedance mismatch" is that it isn't a syntax thing, it's a fundamental difference in the way of viewing the world. The way you store data about the world is different from the way you model that world dynamically, with objects. I find it safer to always split out the "business model" from the storage layer, so that those different views don't interfere - and once you do that, you may as well implement the storage layer in a relational way.
.. and the code you implement to connect that business layer to that storage layer is an ORM.
The idea that the ORM forces the storage layer to a particular representation of the business layer is only true if they implement the ActiveRecord pattern, which isn't universal.
Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data.
Exist 2 main reasons for the "impedance mismatch":
- Paradigms. 2 different paradigms will be at odds. Example: Functional and OO. This is ok.
- Limitations: The relational model is absolutely superior and more expressive at manipulate data than OO/Functional. You need A LOT of machinery to recover that power.
This is not ok.
However, this not change the fact that OO is ok.Similar how a KV store is fine, but certainly, a RDBMS store is much more capable.
> Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data.
This is why I love Clojure (and a particular style of Javascript). Destructuring and a good library of object/array manipulation functions make an environment well-suited to transforming data structures (which is precisely what I want to do in most programs I write), and I find I do not need an ORM where this type of data-focused programming is supported.
You can say OO/Functional lean more to the "Algorithms" side but RM to the "Data Structures". OO/Functional not say much how operate on data, most is a exercise for the reader.
Instead, RM give a clear answer and defined operations for that.
You need to "spice up" things to make the one be more useful to the other part of the equation. RM, too pure, certainly is incredible limited, not even can print "hello world!", but that is just too give a solution about how transform data...
I think you're slightly missing the point -- for me, a certain unique "awesomeness" lies in specifically being able to literally pipe stdout back to stdin and get back the exact same data structures.
That is literally how databases work. In Memory + WAL + Data Files on disk. You could, in theory, live without the Data Files and just a big WAL.