
Datomic: Event Sourcing without the hassle - fnordsensei
https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html
======
codebje
The Events here give up on the naming, granularity, and semantics problem:
they're extremely low level fine-grained changes to fields in a database.

Events themselves are no longer interesting or semantically meaningful,
because they're a single atomic change to a database field. A change in the
way state is represented means a different set of events is produced.
Subscribing to meaningful occurrences in this model is difficult, and probably
will eventually result in the creation of a "meta-event" for each action that
contains the semantic intent of the outcome.

Events are IMO most useful for analytics and processing when they correspond
to meaningful business outcomes: steps in a workflow, consequences of user
actions, and the like - despite this having the problem of making
extraordinary and rare business outcomes more difficult to accomodate.

~~~
seer
That is true, but isn’t it trivially solvable with the datomic model? You just
have a datum “significant event” list and add stuff to it?

You still get all the benefits of event sourcing, but then you can query the
state much more easily.

At least that’s what it looks to me from a cursory look at datomic and daily
struggles with normal event sourcing (kafka)

~~~
pwm
Have you written about your daily struggles somewhere? I, for one, would be
very interested reading about it as I think not nearly enough people talk
about the cons/war stories when it comes to ES (or I just don't find those
blogs).

------
ChrisMalherbe
Great article, but he should have started with the "it's proprietary" bit.

Life is too short to run proprietary software.

~~~
Scarbutt
Many succeed with proprietary software while others worry and don't ship.

~~~
chongli
Proprietary software is a cousin to technical debt. You're trading expedience
for future risk. You simply never know when the copyright holder will fold up
shop and leave you in the lurch.

Edit: If you think this is just an abstract risk, remember that Google shuts
down popular and incredibly useful products so often it's become a meme around
here.

~~~
manigandham
Every decision is a trade-off. Proprietary software with support from a vendor
can greatly speed up your product viability and market profitability.

For most companies, and especially startups, that is far more important than
the very unlikely risk that a vendor completely disappears overnight, and the
even more unlikely risk that their software also stops working completely
before you can migrate to something else.

~~~
chongli
The vendor disappearing overnight is only one of many risks from using
proprietary software. Others include the vendor discontinuing the product,
taking the product in a radically different direction, the company being
acquired, or simply changing the licensing model (see Adobe Creative Cloud) to
dramatically increase the costs of using the software.

Look, right now I am working for a company that is in the midst of attempting
to transition out of a very old source code and release management system and
they're having a hell of a time doing it. That system happens to be
proprietary and the support plus licensing fees are astronomical while the
actual tech support is abysmal.

Yes, the risk may seem like an easy tradeoff when you're starting out and you
need to ship and you don't have any market share to worry about. It's a whole
different story when you're dealing with a very clunky, yet very profitable
legacy system that you're not allowed to fix because it's proprietary and yet
your business depends on it.

~~~
aidenn0
> The vendor disappearing overnight is only one of many risks from using
> proprietary software. Others include...

> the vendor discontinuing the product

This is not a zero risk proposition with open source software either; your
costs go up significantly if you have to start maintaining a legacy codebase.

> taking the product in a radically different direction

See above; if you just want to run the old version, proprietary software lets
you do this as well.

> the company being acquired

This is definitely the biggest risk with Datomic; if Cognitect decides to EOL
Datomic, there is a very high chance that they open source it (see the various
free software they develop already), but if they are acquired by Oracle that
chance becomes zero.

> or simply changing the licensing model (see Adobe Creative Cloud) to
> dramatically increase the costs of using the software.

Datomic licenses are perpetual I believe, so not a risk with Datomic.

~~~
fnordsensei
> Datomic licenses are perpetual I believe, so not a risk with Datomic.

On-prem is perpetual with a year of maintenance (which can be extended), while
Datomic Cloud is integrated with AWS and charged and licensed like other AWS
services: month-by-month.

~~~
pritambaral
Could one transition from Datomic Cloud to an on-prem setup on self-managed
EC2 instances?

~~~
Scarbutt
You can't (at least not with some great effort), cloud depends completely on
various aws services, that's the worst part of datomic IMO, cloud and on-prem
are two different incompatible databases.

------
nathan_long
I like the idea of storing events rather than simply the current, mutable
state of the world.

However, it seems like privacy requirements like "forget you ever knew about
this user" would throw a wrench in the gears.

~~~
valw
That's manageable: [https://vvvvalvalval.github.io/posts/2018-05-01-making-a-
dat...](https://vvvvalvalval.github.io/posts/2018-05-01-making-a-datomic-
system-gdpr-compliant.html)

~~~
nathan_long
How cool that you've already written a whole blog post dealing thoroughly with
that problem. :D

------
z3t4
After implementing "collaboration" features in an editor, I discovered that
functionality can also be used else-where for example with undo/redo. But also
in for example a web app that has many servers, eg a site like Facebook. For
example 1) you get message A, 2) You Read message A 3) You get message A
again, which would usually end up with annoying user interface errors, like
"you got one undread messages", even though you've read them all. But if you
transform the event, that last message, get market as read, because there was
a prior event marking it as read.

------
Kiro
I don't understand how you can be performant with Event Sourcing. So to read
the latest value I need to backtrack all the changes from the start? Every
time?

~~~
jonathanoliver
Almost. Think of it like a bank account. You don't sum up every transaction
amount across all time. Instead, to calculate your balance you start from the
last statement balance (the last memento according to Greg Young) and apply
newer events. So it's VERY fast. You can even compute and save your newer
balance (memento) every few events if you'd like.

------
jfbaro
I really wish Datomic will go Open Source some day! It is such a great
product... a lot of innovation would come from a community driven Datomic.

~~~
amirouche
I am working on a Datomic clone, watch this
[https://github.com/amirouche/asyncio-
foundationdb](https://github.com/amirouche/asyncio-foundationdb)

------
quantum_state
It seems to me content of an event should respect causality: an event simply
encapsulates the time and context something happens without any info about how
it is going to be interpreted semantically in the future by the event
processing components, one would be able to resolve any of the issues. The
design needs to respect separation of concerns at the info level.

------
truth_seeker
The DynamoDB (storage layer) commercial plans for reads and writes is a deal
breaker for me.

ScyllaDB is a much stronger alternative. It is supported but not officially in
Datomic Cloud. It will make Redis/Memcached almost redudant for most of the
applications in the Datomic Cache layer as it has very good latencies for read
and write.

