
Rich Hickey: Deconstructing the Database - noidi
https://www.youtube.com/watch?v=Cym4TZwTCNU
======
saurik
Watching this talk has so far (I'm halfway through, and now giving up) been
very disappointing, primarily because many of the features and implementation
details ascribed to "traditional databases" are not true of the common modern
SQL databases, and almost none of them are true of PostgreSQL. As an initial
trivial example, many database systems allow you to store arrays. In the case
of PostgreSQL, you can have quite complex data types, from dictionaries and
trees to JSON, or even whatever else you want to come up with, as it is a
runtime extensible system.

However, it really gets much deeper than these kinds of surface details. As a
much more bothersome example that is quite fundamental to the point he seems
to be taking with this talk, at about 15:30 he seriously says "in general,
that is an update-in-place model", and then has multiple slides about the
problems of this data storage model. Yet, _modern databases don't do this._
Even _MySQL_ doesn't do this (anymore). Instead, modern databases use MVCC,
which involves storing all historical versions of the data for at least some
time; in PostgreSQL, this could be a very long time (when a manual VACUUM
occurs; if you want to store things forever, this can be arranged ;P).

[http://en.wikipedia.org/wiki/Multiversion_concurrency_contro...](http://en.wikipedia.org/wiki/Multiversion_concurrency_control)

This MVCC model thereby directly solves one of the key problems he spends
quite a bit of time at the beginning of his talk attempting to motivate: that
multiple round-trips to the server are unable to get cohesive state; in
actuality, you can easily get consistent state from these multiple queries, as
within a single transaction (which, for the record, is very cheap under MVCC
if you are just reading things) almost all modern databases (Oracle,
PostgreSQL, MySQL...) will give you an immutable snapshot of what the database
looked like when you started your transaction. The situation is actually only
getting better and more efficient (I recommend looking at PostgreSQL 9.2's
serializable snapshot isolation).

At ~20:00, he then describes the storage model he is proposing, and keys in on
how important storing time is in a database; the point is also made that
storing a timestamp isn't enough: that the goal should be to store a
transaction identifier... but again, this is how PostgreSQL already stores its
data: every version (as again: it doesn't delete data the way Rich believes it
does) stores the transaction range that it is valid for. The only difference
between existing SQL solutions and Rich's ideal is that it happens per row
instead of per individual field (which could easily be modeled, and is simply
less efficient).

Now, the point he makes at ~24:00 actually has some merit: that you can't
easily look up this information using the presented interfaces of databases.
However, if I wanted to hack that feature into PostgreSQL, it would be quite
simple, as the fundamental data model is already what he wants: so much so
that the indexes are still indexing the dead data, so I could not only provide
a hacked up feature to query the past but I could actually do so efficiently.
Talking about transactions is even already simple: you can get the identifier
of a transaction using txid_current() (and look up other running transactions
if you must using info tables; the aforementioned per-row transaction
visibility range is even already accessible as magic xmin and xmax columns on
every table).

~~~
krosaen
I had no idea one could get a consistent read view across multiple queries
within a transaction using most sql databases. That does poke a hole in a
major benefit that I thought was unique to datomic, great to know!

However, I do think trying to setup a sql database to be able to query against
any previous view of the world based on a transaction id as datomic allows
wouldn't be as "simple" as you make it out to be.

~~~
saurik
The reason I claim this would be simple is that PostgreSQL is _almost_ already
doing this. The way the data is stored on disk, every row has two transactions
identifiers, xmin and xmax, which represent the transaction when that row was
inserted the the transaction that row was deleted; rows, meanwhile, are never
updated in place, so the old data stays around until it is deleted by a
vacuum.

To demonstrate more tangibly how this works, I just connected to my database
server (running PostgreSQL 9.1), created a table and added a row. I did so
inside of a transaction, and printed the transaction identifier. I then
queried the data in the table from a new transaction, showing that the xmin is
set to the identifier of the transaction that added the row.

Connection 1:

    
    
        demo=> create table q (data int);
        CREATE TABLE
        demo=> begin; select txid_current();
        BEGIN
        189028
        demo=> insert into q (data) values (0); commit;
        INSERT 0 1
        COMMIT
        demo=> begin; select xmin, xmax, data from q;
        BEGIN
        189028|0|0
    

Now, while this new transaction is still open, from a second connection, I'm
going to create a new transaction in which I am going to update this row to
set the value it is storing to 1 from 0, and then commit. In the first
connection, as we are still in a "snapshot" (I put this term in quotes, as
MVCC is obviously not copying the entire database when a transaction begins)
from a transaction started before that update, we will not see the update
happen, but the hidden xmax column (which stores the transaction in which the
row is deleted) will be updated.

Connection 2:

    
    
        demo=> begin; select txid_current();
        BEGIN
        189029
        demo=> update q set data = 1; commit;
        UPDATE 1
        COMMIT
        demo=> select xmin, xmax, data from q;
        189029|0|1
    

Connection 1:

    
    
        demo=> select xmin, xmax, data from q;
        189028|189029|0
    

As you can see, the data that the other transaction was referencing has not
been destroyed: the old row (the one with the value 0) is still there, but the
xmax column has been updated to indicate that this column no longer exists for
transactions that began after 189029 committed. However, at the same time, the
new row (with the value 1) also exists, with an xmin of 189029: transactions
that begin after 189029 committed will see that row instead. No data was
destroyed: and this data is persisted this way to disk (it isn't just stored
in memory).

My contention then is that it should be a fairly simple matter to take a
transaction and backdate when it began. As far as I know, there is no reason
that this would cause any serious problems as long as a) it was done before
the transaction updated or inserted any data, b) there have been no vacuums
during the backdated period, c) HOT (heap-only tuple) updates are disabled (in
essence, this is an optimization designed to do online vacuuming), and _maybe_
d) the new transaction is read only (although I am fairly confident this would
not be a requirement).

For a more complete implementation, one would then want to be able to build
transactions (probably read-only ones; I imagine this would cause serious
problems if used from a writable transaction, and that really isn't required)
that "saw all data as if all data in the database was alive", which I also
believe would be a pretty simple hack: you just take the code that filters
dead rows from being visible based on these comparisons and add a transaction
feature that lets you turn them off. You could then use the already-
implemented xmin and xmax columns to do your historical lookups.

P.S. BTW, if you want to try that demo at home, to get that behavior you need
to use the "repeatable read" isolation level, which uses the start of the
transaction as the boundary as opposed to the start of the query. This is not
the default; you might then wonder if it is because it is expensive and
requires a lot more coordination, and as far as I know the answer is "no". In
both cases, all of the data is stored and is tagged with the transaction
identifiers: the difference is only in what is considered the reference time
to use for "which of the rows is alive".

However, it does mean that a transaction that attempts to update a value that
has been changed from another transaction will fail, even if the updating
transaction had not previously read the state of the value; as most reasonable
usages of a database actually work fine with the relaxed semantics that "data
truly committed before the query executes" provides (as that still wouldn't
allow data you update to be concurrently and conflictingly updated by someone
else: their update would block) and those semantics are not subject to "this
transaction is impossible" errors.

Both Connections (setup):

    
    
        demo=> set session characteristics as transaction isolation level repeatable read;
        SET

~~~
krosaen
Thanks for taking the time to elaborate, very interesting. I wonder if the sql
db vendors or open source projects will take the next step to make querying
against a transaction ID possible given the underlying implementation details
bring it pretty close.

I also see Rich has made some interesting points elsewhere in this thread
about consistent views being available outside of transactions and without
need for coordination (within datomic) - seems more appropriate to comment
directly there though.

Overall I think it's important to understand these nuances, and not view
datomic as some revolutionary leap, even if I am excited about the project. I
appreciate your insight into the power already within sql db engines.

~~~
saurik
I am not certain whether your response comes from a reading of my comment
before or after the paragraph I added that started with "for a more complete
implementation", but if it was from before I encourage you to read that
section: the ability to do the query is pretty much already there due to the
xmin/xmax fields that PostgreSQL is already reifying.

------
lhnz
There are two people that I will stop what I'm doing and watch every new
lecture they make: Rich Hickey and Bret Victor. Both are visionaries.

~~~
puredanger
And both will be at Strange Loop this year!
<http://thestrangeloop.com/sessions>

~~~
Sandman
Strange Loop is such a great conference. I wish there was something like that
here in Europe.

~~~
puredanger
What about Devs Love Bacon? <http://devslovebacon.com/>

------
erikpukinskis
Fascinating stuff. Some things that came up for me while watching this and the
other videos on their site[1]:

It's not Open Source, for anyone who cares about that. It's interesting how
strange it feels to me for infrastructure code to be anything other then Open
Source.

I'm sort of shocked that the query language is still passing strings, when
Hickey made a big deal of how the old database do it that way. I guess for me
a query is a data structure that we build programmatically, so why force the
developer to collapse it into a string? Maybe because they want to support
languages that aren't expressive enough to do that concisely?

[1] <http://www.datomic.com/videos.html>

~~~
jkkramer
You are not forced to pass strings. That's merely a convenience (for Java).
You can construct and pass data structures instead.

~~~
nickik
Yes, and in clojure you acctually use data literals not just strings.

------
sriram_malhar
I'm always puzzled when the Datomic folks speak of reads not being covered
under a transaction. This is dangerous.

Here's the scenario, that in a conventional update-oriented store, is termed
as a "lost update". "A" reads object.v1, "B" reads the same version, "B" adds
a fact to the object making it v2, then "A" comes along and writes obj.v3
based on its own _stale_ knowledge of the object. In effect, it has clobbered
what "B" wrote, because A's write came later and has become the latest version
of the object. The fact that DAtomic's transactor serialized writes is
meaningless because it doesn't take into account read dependency.

In other words, DAtomic gives you an equivalent of Read-committed or snapshot
isolation, but not true serializability. I wouldn't use it for a banking
transaction for sure. To fix it, DAtomic would need to add a test-and-set
primitive to implement optimistic concurrency, so that a client can say,
"process this write only if this condition is still true". Otherwise, two
clients are only going to be talking past each other.

~~~
richhickey
Addressed here: <http://news.ycombinator.com/item?id=4448351>

~~~
sriram_malhar
Lovely. That addresses my concern.

------
bsaul
Anyone understands how this system would deal with CAP theorem, in the case of
a regular "add 100$ then remove 50$ to the bank account, in that order and in
one go" type of transaction ? The transactor is supposed to "send" novelty to
peers, so that they update their live index. That's one point where i would
see trouble (suppose it lags, one "add" request goes to one peer, the "read"
goes to the second, you don't find what you just add...) Another place i see
where it could mess things up is the "Data store" tier, which uses the same
traditional technics as of today to replicate data between different servers
(one peer requests facts from a "part" of the data store that's not yet
synchronized with the one a second peer requests). It seems like all those
issues are addressed on his "a fact can also be a function" slide, but he
skips it very quickly, so if anyone here could tell me more...

~~~
plaeremans
Well there is _one_ transactor, the transactor handles each transaction
sequential, so the transactor can abort a transaction when the world has
changed since it got queued to the transactor.

There is (virtually) infinite read scalability. Each datum has a time
associated with it, so you might not see the latest information (yet), you
know the state of the world at a certain point in time.

I think it 's a really well designed system.

------
arscan
I recall Datomic making a bit of a splash on HN when it was announced 6+
months ago, but basically crickets since then. Anybody build something cool
that took advantage of Datomic's unique design?

~~~
Tuna-Fish
People tend to be more conservative about data than they are about the other
parts of their stack -- probably for a good reason.

I don't think datomic (or it's kin) will have that huge of an influence until
years from now.

~~~
kinleyd
I think Datomic is potentially disruptive and represents some great thinking
on the part of an individual. Whether it will be disruptive will hinge on how
well that thinking has subsumed the years of industry experience and
practicalities, not to forget the conservative approach to data. I'd be
interested to see how it pans out.

~~~
nickik
I think if will also be imortend what other databases come out of this. There
is space for other prioritys in terms of CAP in a world were perseption and
process are seperated. Or just other implmentations of the same ideas,
opensource maybe.

------
brlewis
Anyone have a summary for those of us who don't want to watch an hour-long
video?

~~~
breckinloggins
I'm only a few minutes into it, but so far the tl;dr is:

"There are lots of things we do with respect to databases simply because of
the limitations of our databases. Here are some things that could be made far
simpler if you just had a better database."

Yes it's partly a sales presentation for Datomic, but you could do worse than
be sold by Rich Hickey.

~~~
breckinloggins
More notes on the video:

\- Rich's whole view on the world is pretty consistent with respect to this
talk. If you know his view on immutability, values vs identity, transactions,
and so forth, then you already have a pretty good idea about what kind of
database Rich Hickey would build if Rich Hickey built a database (which, of
course, he did!)

\- The talk extends his "The Value of Values" keynote [1] with specific
applicability to databases

\- Further, there is an over-arching theme of "decomplecting" a database so
that problems are simpler. This follows from his famous "Simple made easy"
talk [2]

\- His data product, Datomic, is what you get when you apply the philosophies
of Clojure to a database

I've talked about this before, but I still think Datomic has a marketing
problem. Whenever I think of it, I think "cool shit, big iron". Why don't I
think about Datomic the same way I think about, say, "Mongodb". As in, "Hey,
let me just download this real quick and play around with it!" I really think
the folks at Datomic need to steal some marketing tricks from the NoSQL guys
so we get more people writing hipster blog posts about it ;-)

[1] <http://www.infoq.com/presentations/Value-Values>

[2] <http://www.infoq.com/presentations/Simple-Made-Easy>

------
danecjensen
this reminds me a lot of "How to beat the CAP theorem"
<http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html>

------
sbmassey
How would you idiomatically fix invalid data in Datomic? for example, if you
needed to update a badly entered value in a record, but keep the record's
timestamp the same so as not to screw up historical queries?

~~~
bmurphy1976
You wouldn't. You can't change history either. You would insert a compensating
transaction and the data would be fixed from that time forward when you
inserted the compensating transaction.

~~~
sbmassey
Figures. So I guess that, in the case where you both need to worry about
fixing invalid data, and also need to do historical queries, you would have to
add your own timestamp to the data to represent the actual time of the event,
because the built-in timestamp is just giving you the time of the state of the
database. Hopefully that wouldn't get too hairy.

------
hobbyist
I often wonder, is Phd in computer science really required to do awesome work?

~~~
swannodette
Rich Hickey never went to school for Computer Science as far as I know. He
studied music composition.

~~~
wooby
Rich has a CS master's degree.

~~~
swannodette
Did not know that!

------
duck
I'm getting "This video is currently unavailable"?

~~~
anatoly
I don't know - are you?

