Long ago, when Datomic was only recently released, I saw blog posts (not from Cognitect) claiming that Datomic's transaction log could be used to create a subscription system for arbitrary datalog queries. Basically "tell me when the result of this query changes". In practice, I've yet to find a general way to do that, except for wildly inefficient stuff like keeping query results in memory, requerying on each transaction, and comparing. How does 3DF handle a query like this?:
[:find (max ?num) :where [_ :attr ?num]]
It seems like Datomic's transaction log can only tell us that the result of this query might have changed. Does differential dataflow solve that problem?
I think you would need to use only datomic's transactor and tx log and then build your own incrementally maintained query engine, which is basically what this project declarative dataflows does. Differential dataflow and regular dataflow are both ways to implement this. Differential dataflow has the restriction that your query engine must be implemented in terms of diff-aware primitives (https://news.ycombinator.com/item?id=22095199). Regular dataflow is like Excel; it works with any functions, but is not as good at incrementally maintaining things like large lists and nesting – if you're familiar with React.js, at a high level, regular dataflow works a lot like React.js rendering which can call any function during rendering. (Note React.js is highly specialized to html view maintenance, not general purpose incremental maintenance of any computation). Someone correct me if I've said something wrong.
