
Event-Driven Architecture - pradeepl
https://pradeeploganathan.com/architecture/event-driven-architecture/
======
jeremycw
There's something about event driven architectures that captures our
engineering minds and makes our imaginations run wild. At face value it seems
like the Grand Unifying Theory of software engineering, a Grand Unifying
Architecture. Everything is perfectly decoupled and only reads and writes to
an event bus. Perfect uniformity, everything just reads and writes events.
Perfect open-closed semantics, I can completely add new functionality without
touching any existing components just by having it read from the event bus.

However, like the grand unifying theories of physics so far, it just doesn't
match reality. I have yet to witness a system that fully embraces event driven
architecture that isn't a complete nightmare to read, understand, debug and
modify. Yet we seem unable to shake the idea that it is a panacea. It captures
the minds of each new generation of engineers and gets implemented in the
technology of the era. For me it was applying the observable pattern to
everything in Java. Now it's setting up a bunch of microservices that only
communicate through Kafka. Maybe I'm just not a true Scotsman but this pattern
has anecdotally never worked well in anything I've had to work with. I would
think very hard before applying it as a pattern in my code let alone as the
driving force of my architecture. That's just my experience and 2 cents.

~~~
dkarl
In my experience it's a bit the opposite; the domains I've worked in have
always have events, whether given that name or not, and the systems for
handling these events are usually coded much more optimistically and naively
at first before people break down and start migrating parts of the system to
run in event-driven style for performance and reliability reasons. People who
are aware of event-driven architecture tend to make a smoother transition and
create systems that make sense and are easier to work with. People who get
there accidentally, forced every inch by successive performance and
reliability bugs, end up with a hodge-podge that in retrospect is a poorly
designed event-driven system.

But that's just what I've seen in my experience. I've seen damage from people
being ignorant of event-driven architecture or being in denial about how their
systems are evolving; you've seen damage from people being overeager to use
it. Probably in your shoes I would have seen the same things you have.

One thing I've given up on seeing is the content of the linked article.

~~~
safety-third
In my experience, much more damage has been done by people who implement EDA
and fancy cloud architecture in general. Something like 95% of applications
are simple CRUD with a minuscule amount of custom domain logic. Doing anything
beyond a sanely laid out monolith for these is resume padding and potentially
project killing. I think people just don't want to admit that their job is
validation, presentation, and shuffling data in and out of a database.

~~~
dkarl
Yeah, but very quickly in my experience the simple REST-based CRUD service
starts waking up every 15 minutes to post a batch of rows from the database to
a different REST-based CRUD service (or to itself if you have a monolith), and
when that gets slow it gets scaled by increasing the wake-up frequency, and
then someone starts writing batching logic and trying to scale it
horizontally. Then external services start getting mixed in (and, if you're
doing B2B, customer services) and you have to deal with their performance and
reliability issues. Or maybe not; maybe you're Reddit and can get huge while
still being essentially a CRUD app with a couple of UI front ends.

------
iamspoilt
It seems like the post on event driven architecture couldn't handle the click
event load from Hacker News and is throwing a "Database Error".

~~~
dantodor
Yup, the pipelines are down and CQRS is suffering :D

~~~
ak39
That’s a saga for another time.

------
ch_sm
Here is a cached version of the article:
[https://webcache.googleusercontent.com/search?q=cache:https:...](https://webcache.googleusercontent.com/search?q=cache:https://pradeeploganathan.com/architecture/event-
driven-architecture/)

------
AndrewKemendo
I'm consistently surprised at the negative comments on EDA on Hacker News
because there are so many examples of major organizations successfully
implementing and running EDA at scale. Here are a few examples:

Uber:

\- [https://eng.uber.com/ureplicator/](https://eng.uber.com/ureplicator/)

\- [https://eng.uber.com/reliable-
reprocessing/](https://eng.uber.com/reliable-reprocessing/)

Google:

\- [https://cloud.google.com/blog/products/gcp/implementing-
an-e...](https://cloud.google.com/blog/products/gcp/implementing-an-event-
driven-architecture-on-serverless-the-smart-parking-story)

Twilio:

\- [https://signal.twilio.com/2017/sf/sessions/18530/building-
ro...](https://signal.twilio.com/2017/sf/sessions/18530/building-robust-and-
scalable-data-pipelines-with-kafka)

Stripe:

\- [https://stripe.com/blog/canonical-log-
lines](https://stripe.com/blog/canonical-log-lines)

My hypothesis about why this is, is that most organizations probably don't
need EDA yet. They don't have that many data producers and consumers and don't
have HA and other requirements that drive the need, so implementing it is
overkill, and so their experiences have been bad.

~~~
mst
"Trying to implement google-scale things when you're tiny" seems to be an
extremely common and understandably tempting antipattern.

~~~
nostrebored
But "trying to decouple, modularize, and choose the right tools for the job"
for your applications seems applicable at any scale.

EDA requires a change in testing methodology, in software design, and a bit of
reading, but calling it a "google-scale thing" is pretty fallacious. The idea
that a monolith is easier to maintain or that synchronous inter-component
communication is easier to reason about also seems fallacious.

I'd love to see a chart of developer's perceptions of event driven
microservices and their personal operational expectations.

~~~
mst
My point was meant to be more general, in that people try to implement the
"google-scale" version of any given conceptual model even when they don't need
to (i.e. this is about "choosing the right tools")

For example, one can do a CQRS style database setup just fine with a single
API server, a single worker process system, and a single postgresql db - and
given all changes are already driven by command objects, building out other
datastores later if you need to tends to work out well.

Though I would also point out that _on average_ , communication within a
process is easier to reason about than communication outside because there are
fewer failure modes, and sync is easier to reason about than async because
there are fewer failure modes.

Admittedly, a monolith _does_ make overly tight coupling between components
easier to not notice as you're doing it ... but then again it's depressingly
easy to accidentally end up with what's essentially a distributed monolith
even with a theoretically-microservice-based design (googling "distributed
monolith" will provide a bunch of articles with disagreeing definitions of the
term ;)

------
ak39
This is just an overly elaborate concept of a centralized (or commonly
accessible table) of states that other independent systems can use for their
own “triggers”.

What seems like a lifetime ago, we achieved this with IBM MQSeries allowing us
to send events from a Microsoft transaction server (MTS) object to a COBOL
program that listened for the events to insert records into an order entry
system that was on an AS/400 DB2 db.

Maybe an Event Bus concept carries all the expected “good design aspects” of
this pattern. But it is just another example of an _integration concept_ \-
but please can we stop with evangelizing this as though it ought to be the
linchpin or necessary aspect of a good modern system you are starting with!

Edit: this design concept 25 years ago was the best we had considering the
zero alternatives for guaranteeing that events don’t get lost etc. IOW: this
design itself was a compromise for absence of a cohesive and unified database
(what the Event Bus crew and the Microservices crowd would now call a
monolith)!

~~~
idiocratic
I think this is just a good overview of the architecture and its
characteristics. I'm not sure anyone is trying to evangelize anything.

------
kazinator
Event-driven architecture can be nasty. Ideally, you want the point of origin
of an action, and the point of its execution, to be connected by a chain of
function calls which can be traced in a debugger.

Events are not easily traceable. An event is pulled from some event queue, and
by the time that happens, the thing that put the event into the queue has long
since buzzed off to do something else.

That's just one problem. The other problem is how events can be subject to a
routing, translating, duplicating and splitting labyrinth. Events can
bi{tri,quad,...}furcate. Just because your code is processing the event over
here doesn't mean you were the first to do so, or the last.

I have a good idea! Let's process this event and re-inject it. Two years
later, someone is debugging an event routing loop.

------
aabbcc1241
For people cannot view the article, here's an archive:
[https://web.archive.org/web/20200210123446/https://pradeeplo...](https://web.archive.org/web/20200210123446/https://pradeeploganathan.com/architecture/event-
driven-architecture/)

------
teddyh
Ubuntu’s “Upstart” init system used an event-based design, which was
criticized by the creator of systemd, thus:

—

[Upstart]'s main feature is its event-based approach: starting and stopping of
processes is bound to "events" happening in the system, where an "event" can
be a lot of different things, such as: a network interfaces becomes available
or some other software has been started.

Upstart does service serialization via these events: if the syslog-started
event is triggered this is used as an indication to start D-Bus since it can
now make use of Syslog. And then, when dbus-started is triggered,
NetworkManager is started, since it may now use D-Bus, and so on.

One could say that this way the actual logical dependency tree that exists and
is understood by the admin or developer is translated and encoded into event
and action rules: every logical "a needs b" rule that the
administrator/developer is aware of becomes a "start a when b is started" plus
"stop a when b is stopped". In some way this certainly is a simplification:
especially for the code in Upstart itself. However I would argue that this
simplification is actually detrimental. First of all, the logical dependency
system does not go away, the person who is writing Upstart files must now
translate the dependencies manually into these event/action rules (actually,
two rules for each dependency). So, instead of letting the computer figure out
what to do based on the dependencies, the user has to manually translate the
dependencies into simple event/action rules. Also, because the dependency
information has never been encoded it is not available at runtime, effectively
meaning that an administrator who tries to figure our why something happened,
i.e. why a is started when b is started, has no chance of finding that out.

Furthermore, the event logic turns around all dependencies, from the feet onto
their head. Instead of _minimizing_ the amount of work (which is something
that a good init system should focus on, as pointed out in the beginning of
this blog story), it actually _maximizes_ the amount of work to do during
operations. Or in other words, instead of having a clear goal and only doing
the things it really needs to do to reach the goal, it does one step, and then
after finishing it, it does _all_ steps that possibly could follow it.

Or to put it simpler: the fact that the user just started D-Bus is in no way
an indication that NetworkManager should be started too (but this is what
Upstart would do). It's right the other way round: when the user asks for
NetworkManager, that is definitely an indication that D-Bus should be started
too (which is certainly what most users would expect, right?).

A good init system should start only what is needed, and that on-demand.
Either lazily or parallelized and in advance. However it should not start more
than necessary, particularly not everything installed that could use that
service.

Finally, I fail to see the actual usefulness of the event logic. It appears to
me that most events that are exposed in Upstart actually are not punctual in
nature, but have duration: a service starts, is running, and stops. A device
is plugged in, is available, and is plugged out again. A mount point is in the
process of being mounted, is fully mounted, or is being unmounted. A power
plug is plugged in, the system runs on AC, and the power plug is pulled. Only
a minority of the events an init system or process supervisor should handle
are actually punctual, most of them are tuples of start, condition, and stop.
This information is again not available in Upstart, because it focuses in
singular events, and ignores durable dependencies.

[…]

—
[http://0pointer.net/blog/projects/systemd](http://0pointer.net/blog/projects/systemd)

------
rubiquity
Web servers are event-driven architecture. They listen to events on the HTTP
bus.

------
ykr1
Says database error. Was that an event?

------
stephenwilcock
When I follow the link to the article I get an "Error establishing a database
connection" message.

Delicious irony.

~~~
pradeepl
yes, this is ironical. I was away and thanks for the HN hug of death my blog
went down :-). My blog provider throttled it down severely and I was able to
restore it from backups just now. I blog occasionally to put down notes on
what ever I am working on or thinking through, in the hope that maybe it would
help some one or myself in the future. Apologies if I wasted your time.

