
Building a CQRS/ES web application in Elixir using Phoenix - tortilla
https://10consulting.com/2017/01/04/building-a-cqrs-web-application-in-elixir-using-phoenix/
======
eric_b
I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all
failed. Each time the people leading the project and championing the
architecture were smart, capable, technically adept folks, but they couldn't
make it work.

There's more than one flavor of this particular arch, but Event Sourcing in
general is simply not very useful for most projects. I'm sure there are use
cases where it shines, but I have a hard time thinking of any. Versioning
events, projection, reporting, maintenance, administration, dealing with
failures, debugging, etc etc are all more challenging than with a traditional
approach.

Two of the projects I worked on used Event Store. That was one of the least
production ready data stores I've encountered (the other being Datomic).

I see a lot of excitement about CQRS/ES every two years or so (since 2010) and
I strongly believe it is the wrong choice for just about every application.

~~~
slashdotdash
I'm the author of this article. It sounds like you have some valuable, real-
world experience with CQRS/ES.

I'd love to read more about the difficulties you've faced, and overcome.

For migration of immutable events, there's a good research paper[1] that
outlines five strategies available: multiple versions; upcasting; lazy
transformation; in-place transformation; copy and transformation. The last
approach even allows you to rewrite events into an entirely new store.

[1] The Dark Side of Event Sourcing: Managing Data Conversion
[http://files.movereem.nl/2017saner-
eventsourcing.pdf](http://files.movereem.nl/2017saner-eventsourcing.pdf)

~~~
karmajunkie
Good link! Looking forward to reading that.

I've been following your projects on Github for awhile, good work—I don't
necessarily agree with all of the design choices but we've built on the
eventstore at work and I'm going to be using it on another project in the near
future.

~~~
slashdotdash
Please do feedback your ideas to improve these open-source projects. I'd be
interested to find out how you're using CQRS/ES in Elixir.

~~~
karmajunkie
We've got a channel (#eventsourcing) on the elixir slack—low volume but
frequently interesting discussions, please feel free to join us there as well.

------
privacyfornow
One way to understand CQRS is to realize that even the traditional model
relies on CQRS underneath, it just so happens that the relational databases
hide some of this behind a blocking wait on commit while a background thread
reads and flushes the event log (consisting of SQL statements written to a
redo log for example). Really what you are gaining by inverting the database
like this is a lot more flexibility and simplicity for initial development and
subsequent change management (maintenance). The flexibility I speak of
manifests in your ability to tweak availability by introducing controllable
circuit breakers with availability at the expense of some latency or
alternatively putting an upper bound on latency at the expense of some
availability (with every tradeoff fully documented and demonstrably adhered
to).

------
maxekman
Very interesting read, and more so the comments here with all the various
experiences. I'm the author of Event Horizon for Go [1] which is currently
being developed along side a project that uses it, in this case a CRM-like
system where the audit log is a selling point. We still have a lot of unsolved
areas but as some have noted it is a very rewarding developer experience. How
it behaves in production is still to be seen, but reading here has given me a
few good insights to prepare.

[1]
[https://github.com/looplab/eventhorizon](https://github.com/looplab/eventhorizon)

------
bebop
If the writer of the blog post is here, the link to the Elm language looks
like it is missing an [http://](http://)

------
sbellware
I've done a number of event-sourced projects, and have built a good amount of
tooling over time to support the work.

There's one overarching issue I see early on in the adoption process.

The difference between what is being called "traditional" architecture here
and event sourcing is in which things that have to be correct from the very
start. I.e.: The things that you can change later in traditional vs event-
sourced are different. If you don't know realize that this is the case, and
proceed with developing an event-sourced system, you'll end up facing
decisions that you'd expected to be able to reverse later which are not
reversible.

Architecture at its root is concerned primarily with "reversibility" \-
understanding which decisions can be reversed and which can't and making sure
that you get the irreversible decisions correct up-front, and focus design
efforts on them.

If you've only ever built systems under a single paradigm, CRUD for example,
then you might not even be aware of the differences in reversibility concerns
between paradigms.

I know folks who've succeeded wonderfully with event sourcing and I know folks
who've failed. I don't know anyone who has failed with event sourcing who were
not themselves to blame for the failure. Some of those have refused to
consider the possibility that they are the failure's root cause, and then
derided others who point that out, as has been done here in this discussion.

Event sourcing is a big leap. There is a lot of presumed knowledge that isn't
easily spelled out in bite-sized chunks like blog posts and tweets. You won't
learn it (and wield it safely) if you believe that all things should be as
readily-consumed as Meteor over Mongo, or Rails, or pick-your-forms-over-data
tool.

If you don't work to become aware of the dominant paradigm that shapes your
preconceptions, then it will be next to impossible to leverage another
paradigm without polluting it with concerns that are counterproductive.

The things that need the application of your intellect in event sourcing are
not the things that need the application of your intellect in "traditional"
(or any other) architectural paradigm. Square pegs, round holes, etc. But
since our perception of the squareness of the pegs is filtered through the
lens of our own predispositions, we may not even know that we're in the
process of attempting to force-fit things that need a tight fit, and that will
cause project failure without the tight fit.

Event sourcing makes things simple - far simpler than ORM, for example. But it
only does so when approaching it from its own predispositions, rather than the
predispositions (especially unrecognized) of some foreign and impertinent
approach.

I find the prospect of working on "traditional" systems depressing now. I find
having to solve problems that shouldn't exist in the first place to be soul-
crushing. I have an expectation of productivity that is sustained and
sustainable over the long haul, and doesn't decrease even as the size of the
system/team/complexity and expectations grow.

But, I've been following event sourcing since it's unnamed, prototypical forms
in 2006. I started putting it into practice many years later. I didn't expect
it to come to me fully formed after watching a handful of conference
presentations, and I didn't expect it to come to me as easily as other
architectural paradigms and approaches I've used in the past.

I'm still experiencing new realizations about event sourcing. The journey
isn't done yet, but I'm far better off than I was.

I've compared notes with colleagues who are also quite deep into the
transition and have heard similar observations repeated back to me: I can do
an event-sourced system today as fast as I could have built the same system in
a rapid-prototyping tool like Rails in years past. However, unlike Rails (for
example), my productivity, and my team's productivity remains stable, and
changes in scale and complexity of the business and its organization don't
induce the panic that it used to.

And my expectation for the quality and the caliber of the implementation has
only increased - and dramatically so relative to the implementations I still
occasionally see in "traditional" systems.

But had I not had to put some distance between my own mind and its hard-won-
yet-entrenched preconceptions I might not have seen the instincts and
subconscious mechanisms which surely would have led me to underestimating how
and where I needed to focus in order to not build elaborate structures that I
would have been crushed by in the end.

If anyone is really and truly interested in digging into event sourcing, and
really seeing it through its own lens (by becoming aware of the existing lens
we all may have), I'd love to help out. Yes, I make money doing this, and yes,
I have a vested interest - but I also spend a lot of volunteer time with devs
who are earnestly trying to get to the point where event-sourcing and all of
its unfamiliar challenges are just another thing that was learned and facility
cultivated over time.

I can also help ween you off a compulsion to adopt event sourcing for a
project that you want to do as fast as you can with a paradigm that you do
already have a grasp of. Either way.

Ultimately, I find the potential of a future with event sourcing far brighter
than one without, but I've had to cross many bridges and dismantle many self-
imposed obstructions to get there. I hope that others are open to these kinds
of experiences and that we have more profound conversations about event
sourcing experience reports from the other side of the chasm.

My 2 cents, anyway.

~~~
codesuki
Do you have recommendations for resources about this topic? I would like to
learn more about it but most of what I read until now were superficial
introductions, word definitions...

Could you exemplify what you mean by: `problems that shouldn't exist in the
first place`?

~~~
sbellware
By "problems that shouldn't exist in the first place", I mean complexity and
lack of clarity that is the result of presuming to reproduce a relational
database schema in an object model. This is the single biggest problem in
application software development today, and unfortunately it's also a default
mode of development.

I don't have any recommendations for resources. I haven't found any that are
as helpful as learning resources as they are helpful for exercising an
author's lexicon of superfluous jargon (especially DDD jargon and patterns).

I prefer to teach interactively, through coaching. If I ever find a resource
that helps people learn, and that doesn't just load the reader up with
distracting vocabulary, it'll be a happy day. We're not there yet.

I have a good deal of sample code at this point, but not yet much in the way
of documentation.

But even before trying to get a grasp on this stuff, it has to become clear
why reproducing a relational database model in code creates the productivity
problems that harm projects in the long term. Until that's a no-brainer, the
solutions to this problem as presented by event sourcing won't click. Until
the partitioning of an application around "root" objects (or just "roots", if
not using OO) is understood, and until the traversal of a web of associations
in order to execute queries is understood to be the magnet that draws
complexity and obscurity, event sourcing might make no sense.

Until it clicks that ORM is as unnatural an abstraction now as server pages
was in the 2000s, event sourcing probably won't matter. And working with event
sourcing might create all kinds of problems for not having recognized how to
partition a domain. You may just end up with a distributed monolith rather
than a service architecture. And at that point, you might just end up blaming
event sourcing as a pattern rather than the unconscious importing of anti-
patterns from "traditional" development.

For myself, I picked up the necessary precursor ideas over many years and from
many disparate sources; integrating and re-integrating new bits of knowledge
until I had refined a working understanding.

It was much more involved that learning "traditional" development. You can
learn traditional development from blog posts. It's kind of trivial in that
way. There are reasons that there aren't three-month boot camps for beginners
that are based on event sourcing. But once you grasp it, it'll fundamentally
re-write your conceptualization of applicative development.

The event sourcing community needs to do a better job with resources, but it's
not there yet. The work is under way, but it's not complete. It will be as
some point, hopefully sooner rather than later, but not in the immediate term.

~~~
codesuki
Thank you for sharing your thoughts!

------
arcbyte
> CQRS library

You missed the whole point of CQRS

~~~
bglusman
Would you care to expand on what "the whole point" of CQRS is, or how they
missed it? You seem to imply that the existence or creation of a library to
support the pattern is, in some sense, against the spirit of the pattern,
which is a) not at all obvious to me, and b) seems like something you should
expand upon if you're criticising the article and/or author for HN's civil
dialogue guidelines.

~~~
realntl1
I think CQRS is more commonly viewed as an architectural pattern, not a code
pattern, despite many definitions floating out there on the internet that
focus on command/query object patterns. See this post:
[http://udidahan.com/2009/12/09/clarified-
cqrs/](http://udidahan.com/2009/12/09/clarified-cqrs/). If we view it as an
architectural pattern, it becomes folly to think that CQRS can be distilled
into a library.

~~~
sb8244
> it becomes folly to think that CQRS can be distilled into a library

Sincerely curious about this statement. I understand that CQRS is an
architectural pattern, but couldn't a library implement that architecture and
then provide ways for you to implement into that architecture? You don't get
to pick the architecture nuances at that point, though.

~~~
karmajunkie
At its most basic level, CQRS is frankly too simple to take on a dependency as
a cornerstone of a project. You can write a single base class that handles all
of the responsibilities needed. Its NOT an architecture—its an architectural
tool.

~~~
kellyleahy
Agreed. This is so true. I've never ended up with a "library" for doing this.
At my previous employer we had three different CQRS-based systems and two of
the three of them even had completely different implementations for 90% of the
CQRS infrastructure. They are both successful implementations and there was no
need to share that bit of the code, as the two teams doing things came to
different conclusions on how they wanted to write their CQRS code. The only
thing we shared was the event store and some of the minor things on top of the
event store.

