
CQRS and Event Sourcing Intro for Developers - witek1902
https://altkomsoftware.pl/en/blog/cqrs-event-sourcing/
======
tekmaven
I've worked on CQRS / ES microservices at scale and have seen how successful
they can be at reliability, scalability and performance.

I don't think people are qualified to reject this pattern unless they've spent
some serious time working in these ecosystems. It took me a long time with a
ton of production experience to evolve my thinking and truly appreciate
CQRS/ES.

~~~
vorpalhex
I've worked in mature CQRS/ES services at scale, I've also helped convert
other systems both into proper CQRS, and out of proper CQRS.

Like anything else, it's one tool in the tool bag, but like that giant pipe
wrench that you're always looking for a reason to use, it's almost always the
wrong tool for the situation. CQRS carries horrific complexity and requires
commitment to a handful of golden rules or the entire thing comes crashing
down.

> I don't think people are qualified to reject this pattern

I don't think most people are qualified to know what the hell this pattern
truly is, much less put it into proper operational use. I've seen plenty of
people attempt to use CQRS when they should have stuck to a simple CRUD model.
At one point I was adjacent to a team that had built 8+ services to handle one
or two trivial business processes.

CQRS isn't a fine wine or stinky cheese. If it smells funny for your case, it
probably isn't the right choice.

~~~
mason55
> _requires commitment to a handful of golden rules or the entire thing comes
> crashing down_

What are the golden rules?

~~~
dmitryminkovsky
Two bigs ones for me:

\- Don't use the read side from the write side.

\- Each aggregate should own its data (or more abstractly, the basics of DDD,
bounded contexts[0]).

[0]: [https://cqrs.nu/Faq](https://cqrs.nu/Faq)

~~~
eithed
Every state change needs to be reflected as event. Does your function need to
save something to DB? Then your functionality should create event which will
perform the DB save.

------
dmitryminkovsky
I'm always excited to see posts about CQRS and Event Sourcing. My current
project is entirely ES/CQRS-driven and it's been a revelatory experience. I
can't understate it. As Charity Majors says:

> And while storing raw events may be "expensive" compared to any single
> optimization, all optimizations can be derived from raw events.

> The reverse is impossible.[0]

But I have to say: the resources that this site links only served to confuse
me. Greg Young may have popularized these concepts, but watching his talks
left me with little practical implementation guidance, and Udi Dahan was even
worse, much, much worse, in terms of leaving me helpless and confused.

What really helped me was "Designing Event Driven Systems"[1] a promotional
book by Confluent that nevertheless has many great sections with practical
advise for implementing these patterns. Likewise the CQRS/ES FAQ[2].

And while this post says you don't need Kafka for CQRS/ES, Kafka sure does
help. Kafka Streams is the ultimate tool for CQRS/ES. It contains all the
primitives you need to do CQRS/ES easily. I am in the process of writing a
blog post about my experience and am looking forward to sharing it. People
love React/Redux, state as a function of reducers, and time travel debugging
on the front-end: there is no reason you can't have all the way down your
stack. Kafka Streams makes this possible, and much easier than you'd think.

[0]:
[https://twitter.com/mipsytipsy/status/1115537408705957888](https://twitter.com/mipsytipsy/status/1115537408705957888)

[1]: [https://www.confluent.io/designing-event-driven-
systems](https://www.confluent.io/designing-event-driven-systems)

[2]: [https://cqrs.nu/Faq](https://cqrs.nu/Faq)

~~~
refset
I agree, the Confluent team paint a very clear big picture. You may be
interested to take a look at Crux [0], a general purpose document database
with bitemporal graph queries, implemented as an ES/CQRS system directly on
top of Kafka (or a local KV store).

We created Crux because we found ourselves routinely needing bitemporal
functionality when building immutable systems that are capable of ingesting
and interpreting out-of-order/late-arriving events [1].

So far we have decided against using Kafka Streams to keep our log-storage
options very pluggable, but we pretty much implement the same mechanics.

Disclosure: product manager for Crux :)

[0] [https://juxt.pro/crux/index.html](https://juxt.pro/crux/index.html)

[1] [https://juxt.pro/blog/posts/introducing-
crux.html](https://juxt.pro/blog/posts/introducing-crux.html)

~~~
dmitryminkovsky
Thank you! I look forward to checking this out. I'd say the picture painted by
Confluent is much rosier than reality, but it definitely is the clearest, most
actionable picture to date.

~~~
refset
To be specific, the picture I am thinking about in particular is described in
a slide titled "Local consistency points in the absence of Global Consistency"
(from this talk:
[https://youtu.be/VYOMmwkdSig?t=1817](https://youtu.be/VYOMmwkdSig?t=1817)),
where it shows how each local consistency point is ideally implemented using
ES/CQRS. I also agree that the tools Confluent are currently providing us with
today do not make this trivial.

Edit: the other big picture Confluent quote I like is "You are not building
microservices, you are building an inside-out database" (Tim Berglund,
Confluent, 2018) which is a perfect answer to this other quote: "The hardest
part of microservices is your data" (Christian Posta, Red Hat, 2017)

------
danielovichdk
I have never understood the effect CQRS seems to have on developers ? It's a
pattern that should be treated and applied with absolute cautiosness. \--- A
command feeds a data store, without ever returning anything back.

Given a query a system returns some relatively useful model back.

Honestly, why is this so interesting ?

Sure, you might want to add some autonomous component(s) that manipulate data
before returning them.

A good example of applying CQRS to a small part of a system is e.g

1\. Data is constantly being written (commands) into the system, excessively.
2\. You want to present a "snapshot" of the data, because trying to do it
"realtime" for all your users will demand too many resources, and your system
will come to a hault. 3\. You create the "snapshot" every 10 seconds, from
code in an autonomous component, and then store it as a serialized object in a
data store. Like a cache sort of. 4\. When the query asks the system, the
system loads up the "snapshot", deserialze it, and returns that.

Here you have two independent read and write systems. That's CQRS for you.

Do not apply this pattern with a loose hand.

~~~
raeldc
We use NestJS/CQRS in our property management app. Here's how we implement it:

1\. There are 2 event handlers for "writes". One handler writes normalized
data in a PostgreSQL DB. The other handler writes denormalized data into
Firestore. 2\. Our frontend uses Firestore so mutations are reflected realtime
in the frontend. We never found a need for the command to return data. There
is also no need for complex queries in Firestore since our data is
denormalized and optimized for reads. 3\. The PostgreSQL DB is useful for
reporting and complex queries. Our frontend app displays this data only in the
reports area.

So far, I don't see how things can get confusing with this pattern.

~~~
stocktech
What problem does CQRS solve in this scenario?

~~~
danielovichdk
This is a good example of What CQRS really is. Separate read/write in a
system.

It's in fact a very simple pattern. But I wouldn't use it system wide because
not everything has to be eventuel consitent.

~~~
he0001
Isn’t all reads and writes already separated?

------
vaultcool
I've seen CQRS pop up over and over, but never seen anyone do a real
application aside from some pseudo-bank account code. Are there any bigger OSS
projects that use event sourcing?

Also the cached version:
[https://webcache.googleusercontent.com/search?q=cache:jZXfTY...](https://webcache.googleusercontent.com/search?q=cache:jZXfTYYOUnUJ:https://altkomsoftware.pl/en/blog/cqrs-
event-sourcing/+&cd=1&hl=de&ct=clnk&gl=de&client=ubuntu)

~~~
mrkeen
Used it. Loved it. It does have its issues though.

I was working on the backend systems for electric car charging. When we heard
about real-world events happen (start-charging, stop-charging, etc.), we wrote
them directly to Kafka. It was up to other services to interpret those events,
e.g. "I saw a start then a stop, so I'm writing an event to say user-has-
debt". Yet another service says "I see a debt, I'm going to try to fix this by
charging a credit card". I guess you'd call the above the 'C' part of CQRS.

But Kafka by itself is not great for relational queries. So we had additional
services for, e.g history. The history service also listened to starts, stops,
debts, credits, etc. and built up a more traditional SQL table optimised for
relational queries, so a user could quickly see where they had charged before.

The issues we had were:

1) Where's the REST/Kafka boundary? I.e. when should something write to the
Kafka log as opposed to POSTing directly to another service? E.g. If a user
sets their payment method, do we update some DB immediatley, or do we write
the fact that they set their payment method onto Kafka, and have another
service read it?

2) Services which had to replay from the beginning of time took a while to
start up, so we had to find ways to get them not to.

3) You need to be serious about versioning. Since many services read messages
from Kafka, you can't just change the implementation of those messages. We
explicitly versioned every event in its class name.

Worth it? For our use case, hell yeah.

~~~
dmitryminkovsky
> 1) Where's the REST/Kafka boundary? I.e. when should something write to the
> Kafka log as opposed to POSTing directly to another service? E.g. If a user
> sets their payment method, do we update some DB immediatley, or do we write
> the fact that they set their payment method onto Kafka, and have another
> service read it?

I believe the term that's emerging for this issue is "collapsing CQRS" and how
you handle this is application-dependent (are you using plain synchronous HTTP
requests? websockets?) In my case, the HTTP server has a producer that writes
to a Kafka topic and a consumer that consumes the answers. The HTTP request
waits until the answer appears on the answer topic.

> 2) Services which had to replay from the beginning of time took a while to
> start up, so we had to find ways to get them not to.

Kafka Streams local state makes this fast, unless you need to reprocess.

> 3) You need to be serious about versioning. Since many services read
> messages from Kafka, you can't just change the implementation of those
> messages. We explicitly versioned every event in its class name.

Yes, this is tricky. In my case, I either add fields in a backwards compatible
manner, or rebase/rewrite event topics and roll them out while unwinding the
previous version of the topic that may still be in use. The former is
obviously the simpler option.

------
Liron
I would _love_ to write my application layer with any kind of declarative CQRS
paradigm, but I did some research and concluded that today's database
technologies don't give us the expressive power to do it. I wrote up my
thoughts here:

[1] [https://hackernoon.com/data-denormalization-is-
broken-7b6973...](https://hackernoon.com/data-denormalization-is-
broken-7b697352f405)

[2] [https://hackernoon.com/when-logic-programming-meets-
cqrs-113...](https://hackernoon.com/when-logic-programming-meets-
cqrs-1137ab2a5f86)

------
brentjanderson
Event sourcing and CQRS in Elixir with
[Commanded]([https://github.com/commanded/commanded](https://github.com/commanded/commanded))
is a treat. If you’re interested in these patterns or Elixir, take a look. The
maintainer is remarkably friendly and helpful on any issues as well.

------
725686
I've never used CQRS or Event Sourcing, but I found it very interesting after
watching some of Greg Young's videos.

He also has this:
[https://leanpub.com/esversioning](https://leanpub.com/esversioning)

This blog by Jay Kreps (Linked-in) is also a good read:
[https://engineering.linkedin.com/distributed-systems/log-
wha...](https://engineering.linkedin.com/distributed-systems/log-what-every-
software-engineer-should-know-about-real-time-datas-unifying)

------
kierenj
We use the concepts of commands and queries, where commands can't return
anything and queries cannot mutate state. No separate stores for each, though.
I think that's either more correctly called CQS, though I remember finding
sources that it's just as valid an implementation of CQRS too.

Some advantages:

\- Command and query handlers being individual (testable) classes is great for
general organisation and neatness

\- Beats 'Transaction Script' as a pattern any day

\- Can wrap command/query execution with various value-add pipelines: logging,
timing, caching (queries)

\- Can direct queries to use read replicas

Some dirty tricks:

\- Service layer (API endpoint controllers) does pre-validation, declaratively
where possible, to enable OpenAPI/Swagger documentation of failure modes

\- We do really care about whether commands finish or not - so they do run
synchronously in 99.9% of cases. They can be put to a queue, but usually the
caller wants to know it finished

\- Also, they can throw exceptions with error results.. it seemed a trade-off
that's worked well enough

Recently had an idea to steal some concepts from serverless leveraging this:
could measure command/query performance, resource use (CPU/RAM) by, based on
some heuristic, farming out execution of one or the other to a separate
process. Commands, queries and query results all being serialisable.

Event Sourcing and separate stores seemed a bridge too far. It definitely
wants loads of expertise/experience and careful design, whereas the above is
easy as pie to get going with.

Anyways. It's proved for some enjoyable enough development.

~~~
jddj
Another advantage:

\- Compared with a repository pattern or typical fat-fingered ORM, separating
command and query models in code can make it easier to let the data store /
search engine do the things it has been optimised to do well on the query end
(joins, views, etc), rather than over-fetching data and doing too much of the
translation in code.

------
andy_ppp
Has anyone tried doing Event sourcing with a redux app?

I'd be pretty interested to see the same "store" working on frontend and
backend and every redux event being queued up and (eventually) synced to the
server to provide a log of what happened. Obviously this leads to building an
analytics system for your product quite quickly.

~~~
superqwert
Redux pretty much is CQRS + event sourcing, as applied using functional
paradigms.

A while ago I built a system that would use a React CQRS plugin and provide
optimistic concurrency - allowing really fast UIs at the expense of users
having to deal with failed commands if a breaking error occurs:

[https://github.com/MichalPaszkiewicz/cqrs-external-
confirm](https://github.com/MichalPaszkiewicz/cqrs-external-confirm)

~~~
Heliosmaster
Same for Re-frame in CLJS: [https://github.com/Day8/re-
frame](https://github.com/Day8/re-frame)

------
Qasaur
I've implemented this pattern a handful of times, and if done in a team I
think it manages the complexity of large software projects in a way the
traditional MVC pattern simply cannot do. The real value of this kind of
architecture is not the reliability, scalability, or performance advantages
(even though they are significant), it is the elegence in the way it manages
complex systems and the associated business logic. The cognitive gains here
are innumerable, and may even increase the velocity of a team provided the
underlying infrastructure is in place.

------
faizshah
Any recommendations on an example driven/practical book on implementing
CQRS/Event Sourcing for web apps?

~~~
tytho
I read part of an early chapter of a book that taught a lot of these concepts
in a practical way. The book isn’t out yet, but will be in the next month or
so: [https://pragprog.com/book/egmicro/practical-
microservices](https://pragprog.com/book/egmicro/practical-microservices)

I think the code examples are out of date so don’t worry about downloading
those until the book is officially released. This guy has had some great
success stories implementing CQRS/Event Sourcing at his place of work in the
last year.

------
mrbanks
This should not be used on 99.999% of real applications. You just end up with
a shit show.

~~~
CuriouslyC
Being able to roll back writes and reconstruct earlier states is pretty key
for any sort of high-compliance application.

That being said, I converted a failed application that needed rollback and
history from CQRS to standard CRUD with trigger based operation logging and
achieved the same effect with about 1/10th the code. The architecture was a
lot more accessible to lower level devs too, greatly reducing maintenance
costs.

~~~
cc81
You can do that with backups and transaction logs.

