
Building Domain Driven Microservices - grzm
https://medium.com/walmartlabs/building-domain-driven-microservices-af688aa1b1b8
======
francasso
Am I the only architect that hates microservices with gusto?

I've only seen bad implementations out there (tens of products), and have yet
to encounter one that looks reasonable. If your average developer has trouble
implementing something sane when the only problem is splitting data and
functions locally into different components, why in the world would you assume
that putting in a network layer, exponentially increasing the failure rates,
loosing good stack traces, making refactorings that have to move data from one
part of the architecture to another a pain in the butt (because requirements
never change right?), etc... would be a good thing?

The horrors I've seen... oh boy, the horrors.

The people I've heard praising microservice architectures so far fall into two
categories: (1) Ex developers that moved into management and read a few
articles/bought a book on the subject and think it's a great idea (but have no
actual practical experience) (2) developers that have just started working on
a new one and are still in the honeymoon phase (mind you, not developers that
have started working on an __existing__ one that is a few years old, those
figure out the mess they are in pretty quickly).

You can screw up not-so-micro services too, but it's far easier to screw up
microservice architectures and they are far, far harder to fix down the line
when you figure out how you messed up (or re-architect when that thing that
the PM told you was never going to be a use case is going to become a use case
(after they tell you they are sorry of course)).

~~~
andy_ppp
Totally agree - I'm not against services but I say as few services as humanly
possible!

The issues with micro services are just so unbelievably large that it'd take
too long to enumerate them here, but for me the largest issue is you are
putting arbirary network partitions into your application and you CANNOT
possibly know when these network partitions will suddenly make an operation
you need to be atomic impossible (or at best eventually consistent).

In general it is excruciatingly difficult to iterate on micro services as
well.

Then there is deployment, service mesh, k8s etc. etc. that you have to wrangle
with, all because people refused to make bounded contexts in their application
without making them into different services. It is much simpler to just design
your monolith into separate contexts than it is to introduce all the error
handling you need for network partitions.

I personally don't understand why Elixir isn't more loved by these micro
services zealots - it allows you to get extremely far without adding network
partitions and then allows you to switch to message passing when you need and
then networked message passing all extremely smoothly as it makes sense for
your app.

~~~
SomeoneFromCA
The best network layer is the memory controller of the CPU. Never breaks, no
partitioning problems. If you really wnat for whatever reason uservices run it
on one machine,

------
dvancouvering
I have done a lot of work with microservices, and I agree they come with a ton
of cognitive overhead and new ways for things to fail in unexpected and
painful ways.

In particular it now requires software engineers to understand principles and
design patterns for distributed systems that when I was first starting out
were relegated to theory and university papers. Can you guarantee ordering?
Can you guarantee exactly once? How do you get transactional semantics? How do
you handle cascading failures? It's all a bit overwhelming.

I have also seen however that monoliths at some point lock up an organization.
The key measures of software delivery performance are deploy frequency, lead
time from commit to running in production, mean time to recovery and change
failure rate. Monoliths impact all of these when you engineering org starts
scaling past a handful of teams. A monolith also becomes a single point of
failure. One person in one team checks in bad code, and the whole system falls
down. Microservices can give you resilience. You can also scale microservices
independently - a CPU intensive service can get a bunch of pods, a memory
intensive service gets a beefier memory allocation, and other services can run
idle most of the time or even be deployed as FAAS functions that almost never
run.

So I agree it's not a technical solution as much as it is about scaling an
organization. At WeWork the team shrank so much that we began consolidating
small services back into a single service because the operational overhead of
having separate services didn't make sense any more when a single team was
responsible for all of them.

I do think choosing to split out a new service is a big decision to make. You
have to really think through the cost and benefits. You also need to make sure
you have the right tools in place so you can detect and recover quickly.

Things like log aggregation, SLOs defined, measured and alerted on, canary
deploys, progressive delivery, etc. Without that you're running completely
blind.

I also think you want to be thoughtful about keeping your coupling low. If
each service depends on twenty other services then in my mind you now just
have a distributed monolith.

~~~
dmitryminkovsky
> Can you guarantee ordering? Can you guarantee exactly once? How do you get
> transactional semantics? How do you handle cascading failures? It's all a
> bit overwhelming.

I don’t advocate DDD/microservices but if you absolutely must go down this
road, Kafka Streams really neatly answers all these questions. Ben Stopford’s
book[0] and blog posts[1] are really great if you want to know more.

[0] [http://www.benstopford.com/2018/04/27/book-designing-
event-d...](http://www.benstopford.com/2018/04/27/book-designing-event-driven-
systems/)

[1] [https://www.confluent.io/blog/building-a-microservices-
ecosy...](https://www.confluent.io/blog/building-a-microservices-ecosystem-
with-kafka-streams-and-ksql/)

------
jungoolist
Domain Driven Microservices looks good on paper, but its a very very terrible
idea.

1\. You can't run join queries across domains

2\. You can't migrate your schema across domains

3\. You will need to sync and map a db with domain dbs for reporting

4\. You need to write api/client server for every domain and maintain this
with db schema changes

~~~
UK-Al05
It seems like database schema and domain api are very coupled. They should
have nothing to do with each other. You should be able to change your database
schema without changing your api....

If you want to expose extra features or change the api design, you version
your api.

Reporting is a problem, but most large business have some kind of
datalake/datawarehouse where all of it comes together anyway these days.

~~~
chrisan
> If you want to expose extra features or change the api design, you version
> your api.

But you are still coupled. If you have to version your API that means it is
relied on by something else and in order to use that extra feature still
requires both (or more) services to all be on the same page.

Microservices let you deploy and maintain at different times (if that is
appealing to you) but in order to realize a new business feature you are still
coupled.

Now you have x tests for x versions as well, make sure the old version remains
old and the new feature works in the new version.

~~~
UK-Al05
I mean yes, if you want a use a new feature using the new api. Then your going
to have to move onto the new api. But that would happen regardless of using
microservices.

If you version it though, services that don't care can cary on regardless.

------
soonnow
I think that many people look at Microservices from a purely technical
standpoint. How much compute power do I get per dollar spent. What about the
monitoring complexities and the logging problems?

Microservices are actually often driven by business requirements. According to
the Myhtical Man Month communication channels follow n(n − 1) / 2\. So a team
of 4 developers has 6 possible communication channels, a team of 5 already 10.
This eincreases sharply as team sizes increase and is one of the reasons big
enterprise projects suffer compare to startups.

The business benefit of Microservices (done right) is that you build islands
of communication, that communicate using only one outside channel. Now
communication increases less sharply as teams are separated by their
boundaries.

According to the guardian the team size and structure is one of the secret for
Amazons success. [https://www.theguardian.com/technology/2018/apr/24/the-
two-p...](https://www.theguardian.com/technology/2018/apr/24/the-two-pizza-
rule-and-the-secret-of-amazons-success)

For smaller teams the communication benefit will often not outweigh the
additional complexity.

~~~
jensneuse
You're making a point that Microservices help teams to solve the problem of
increasing communication needs for larger teams. While your statement makes a
lot of sense I'd like to ask why this problem cannot be solved with packages
or classes within a monolith? A team can own a Microservice and expose an api
to others the same way they could expose an api on a class or package, no?
Microservices are a way of deploying small parts of software. It's not about
communication across teams. It can help with communication but this is not
only true for Microservices.

~~~
jmchuster
I believe it could be done via classes or packages, but only if the providers
have the discipline to design it like a rest service, and the users then have
the discipline to only use it like a rest service. Each package needs a small,
consistent interface, with no way to interact except through that interface,
with that same style of consistency across all libraries in the same way. The
code must be isolated at runtime, so it can't steal resources from other
pieces of code in the monolith or cause anything to crash. It can only respond
to requests along the interface, either successfully or unsucessfully. So,
theoretically possible, but in practice difficult once you add humans to the
mix.

~~~
livesinhel
You can use ArchUnit[1] or similar frameworks to impose the constraints.

[1]: [https://www.archunit.org/](https://www.archunit.org/)

------
remote_phone
Bleah. I hate articles like this. They regurgitate all the well known talking
points about enterprise software development but none of the realities of
micro services. It sounds like they haven’t actually worked in the trenches
supporting and coding real microservices at scale.

The realities are that going from monolith to microservices is a huge
investment, and only large engineering teams should do this migration.

The reason is that once you go microservices You instantly require an
immediate investment in devops which is something you didn’t need as much
before with a monolith. Oncall rotations and pager duty schedules become
imperative because services will go down. Also monitoring is another
investment that you immediately need otherwise you’re completely blind.

The article talks about things that are obvious, like designing the apis etc,
but those are the same regardless of if it’s a module within a monolith or a
microservice. All the characteristics they list at the beginning is no
different than what a team that owns a module in a monolith would require as
well. It feels like this article isn’t very useful because it regurgitates
concepts from regular enterprise development and packages as new for
microservices when it’s not.

What the article lacks is the difference in architecting for a real
microservice environment.

One of the biggest missed issues is never mentioning dependencies. Sure it’s
great to talk about how to model services with eventual consistency blah blah.
But if you already have a large engineering team, which you must have
otherwise why have microservices, then there will be a dependency between
services and to believe otherwise is crazy. You will probably have one or two
services that when they go down, your entire company is down. It’s imperative
that every service is architected with that in mind and what happens when it’s
down. Hand waving “eventual consistency” won’t cut it in the real world, it’s
the biggest problem that each team needs to focus on early otherwise it’s a
ticking time bomb.

~~~
rumanator
> Bleah. I hate articles like this. They regurgitate all the well known
> talking points about enterprise software development but none of the
> realities of micro services.

I disagree. The post is a good summary of all the key features and main
concepts of a microservices architecture developed with domain-driven design.
It does what it says on the tin, and you don't have to read three books to get
there.

> It sounds like they haven’t actually worked in the trenches supporting and
> coding real microservices at scale.

I have no idea where you got that idea. It certainly wasn't from the blog
post, though.

> The realities are that going from monolith to microservices is a huge
> investment, and only large engineering teams should do this migration.

Your rant makes no sense at all. You're just mindlessly repeating cliches
about issues that have absolutely nothing to do with the subject. I mean, the
blog post is about microservices software architecture with domain-driven
design and clean code, but here you are whining about how you believe
refactoring projects designed following an entirely different architectural
style is hard.

~~~
remote_phone
My point is everything they talk about is common to any type of enterprise
software development. It’s not new. And then they slap on “microservices” at
the end so they can claim to have written something new.

If they want to write something new then talk about the realities of how
microservice development is different than regular enterprise software
development. Not regurgitating the same old concepts and pretending it’s eye
opening and new.

~~~
vmurthy
Um, it may not be new to you but it is new and eye opening to me and there
might be hundreds or thousands of people who haven't read about this. From a
newbie's POV, I quite like how the author explains the concepts with the
example of an e-commerce company. Sure there will be advanced concepts but the
author makes no claims that he is addressing that audience :) .

------
tannhaeuser
> _Hide implementation detail and expose functionality through intention-
> revealing interfaces._

Except "REST APIs" (what microservices expose almost always) don't reveal
anything most of the time, wanting to hide internal state behind a beautiful
and uniform facade essentially representing a two-tier model. For example,
orders and order line items can't be added/deleted at will, but depending on
whether the order has been processed or not, should expose a "cancel" action
(with partial refund/with penalty for cancelling), etc. Sure you can hide that
behind a DELETE method, or "reify" a "cancelled" status, but why?

> _Services are resilient to failures._

In the case of "REST APIs", this means manually and labouriously attempting to
reconcile state across multiple services in the client.

Compare this with state changes though ordinary RPC, which can expose a
meaningful state model, give contextual diagnostics/exceptions, and can have
transaction context flowing with RPC calls.

------
yeswecatan
I've been interested and confused about DDD, hexagonal architecture, clean
architecture etc. for a couple of years now but unfortunately have not had
time to dedicate towards reading up on all of them. Has anyone actually
implemented any of this as part of their jobs? I guess it all depends what you
do. I mainly do web stuff, and Django/DRF lets us go fast. Out of the box, I
think this goes against the aforementioned architecture as the ORM couples the
domain model with the data model. Besides, gathering requirements (at least at
my company) is unfortunately an ongoing thing which is just the nature of a
startup I suppose.

~~~
rumanator
> Out of the box, I think this goes against the aforementioned architecture as
> the ORM couples the domain model with the data model.

It doesn't. You're confusing the persistence model with the domain model. Your
persistence model is dictated by your need to persist your data while meeting
the requirements of your domain model. Your domain model does not care about
your persistence requirements. Your persistence model is bounded to your
persistence service and should not leak out, while your domain model is
ubiquitous to your system.

Let's put it this way: with a Clean Architecture you are able to add/replace
any persistence service you wish, and that has no impact on any other service.

~~~
yeswecatan
Can you provide a concrete example? I would assume if you wanted to decouple
the persistence model from the domain model then you would need to write a
separate domain model in Django since the domain and persistence model are
tightly coupled.

~~~
rumanator
I'm not very familiar with Django thus my ability to provide relevant examples
using this particular framework is limited.

However I direct your attention to the fact that Django docs make it very
clear that to them in general each model maps to a single database table.
That's quite Spartan even for an ORM, and obviously the point of a domain
model is to model a problem domain, not to model the database.

------
karmakaze
This talks about the thing that makes microservices fail most often, getting
the bounded contexts wrong. It doesn't go into enough detail to show you how
to do it right but does tell you what to watch for when you're doing it wrong.

The biggest, most common mistake is when a bounded context is called "<noun>
service". This should be a huge red flag. Unfortunately, most books (and this
post) always use this bad naming starting everyone down the bad path. At the
very minimum call it a "<noun> management service" if it is the
primary/general create/update activity. e.g you might have a checkout service
that creates most orders and a order management service that can make drafts
or adjustments. Order management (being the decoupled source of truth) might
even be downstream from checkout service. Other services that need this
information should get it's own copy asynchronously, i.e. decoupled.

The best way to learn how to do bounded contexts right is to study ones that
worked well. Most writing tries to describe this in an abstract way using
placeholder widget service things that helps no one.

The BFF pattern is good but doesn't solve everything. If you're accessing
multiple services for a synchronous mutation, it's likely not highly reliable
but can get by for an interactive service that can tolerate some errors. A
messaging service is not one that should, they should always accept, store and
forward with the lowest possible error rates.

------
hestefisk
I might be a bit cynical / neckbeard, but the tech industry tends to go in
circles with good engineering principles sold as the newest fads / hype.
Before all the microservices craze it was REST and APIs, before that web
services, SOA and ESBs, and before that XML-RPC / CORBA, EAI, and enterprise
Java beans. And before that we had, we someone we had RPC over the network
(‘open systems’).

Similarly DDD promotes separation of concerns and test driven development.
That is exactly what eXtreme Programming told us in late 90ies when OOP and
unit testing was in. Before that it was structured programming and software
design specs, 4GL etc.

Bottom line is I wish people would start calling a spade a spade and indulge
in less fads. Let’s just call it structured programming over the network with
clear separation of concerns and clean interfaces. That’s good engineering.
Test early and often. Write good documentation.

~~~
eurekin
I'm completely on the same page with you on that one.

I understand though the marketing needs. IT has this steady influx of
beginners that don't have the insight yet. Since there are so many of them, at
any given time, it's reasonable (or maybe it _actually works_ ) to use
standard advertising techniques at them. "Secret sauce", "The missing puzzle
piece", "10 things that make you a better dev", "Dev done correctly".

Developers are kind of a dream target group: lots of disposable income,
technically adept (i.e. at the cost of social skills), excited to experience a
latest unicorn.

I'm seeing this whole "hype cycle" thing as a competition among more senior,
adept in marketing, developers. Since there is such a high demand, the supply
is also rich.

~~~
jka
Could/should there be a 'wikipedia for software development practices'?

~~~
steve-s
What about [https://wiki.c2.com](https://wiki.c2.com)

~~~
jka
Something along exactly those same kind of lines of intent, yep.

A wiki-like reference guide that can assist architectural and technology
selection recommendations.

Potentially with support for argument mapping[1] and decision trees[2] to deal
with contentious or context-sensitive topics.

[1] - [https://en.arguman.org/](https://en.arguman.org/)

[2] -
[https://en.wikipedia.org/wiki/Decision_tree](https://en.wikipedia.org/wiki/Decision_tree)

------
asim
Good article. This is what micro was built for. [https://go-
micro.dev](https://go-micro.dev)

------
corpMaverick
Microservices are a good way to scale up development teams. So if you have 16
developers you can partition the work in two teams with one microservice each.
If you have 80 devs you can have 10 teams with one microservice each team. Use
Conwayś law to your advantage. Use DDD to decide where the boundaries are.

The advantage of having each team own a microservice is that each team can
deploy their software independently from the rest of the teams. And it works
as long as the APIs are honored. So Microservices are way to decouple the work
of different teams allowing them to work independently from each other.

~~~
dakiol
This only works for the simplest cases. If the two microservices must share
data and direct calls (API calls) are not allowed (because they make your
services highly coupled to each other to the point where if one of them ia
down, then the other cannot work properly, or simply because your use case
cannot tolerate the latency overhead) then each service must 1) replicate the
data needed locally by listening to interesting events of the other service
and b) emit interesting events so that the other service can update its local
db. In other words, async communication.

While this let's you scale up teams, it's a huge overhead. So, unless you have
over, let's say, 500 devs, it's not worth it.

~~~
corpMaverick
> This only works for the simplest cases. If the two microservices must share
> data and direct calls (API calls) are not allowed

Actually, one assumption is that the microservices don't share data. Only API
(Rest, SOAP, GraphQL, etc) calls are allowed. And the boundaries are created
minimizing the coupling between microservices.

> While this let's you scale up teams, it's a huge overhead. So, unless you
> have over, let's say, 500 devs, it's not worth it.

As opposed to Monolith ? I think you will have problems collaborating in a
Monolith at around 30 devs.

What I am opposing is the idea that you can have 30 devs and 50 microservices.
Which I have seen in real life.

------
tveita
I see the writer is from "WalmartLabs", so I assume this is advice for
"Walmart scale" applications. For the rest of us it's probably good to
remember that "you are not Walmart."

> Now, we know that in any modern application, sacrificing availability is not
> a good idea either.

I feel like this will apply to less than 10% of applications. Normally you'd
rather deal with the occasional downtime than dig through the fallout of
broken attempts at distributed transactions. Ask yourself "What amount of
$X,000 are we losing per minute of downtime" and see if it rounds down to
zero.

> Another way to model this system is to separate, or group related models
> into separate microservices. In DDD, these models — Price, Priced Items, and
> Discounts — are called Aggregates.

And this must be the case for less than 1% of readers - it would take a
massive system before I'd consider having prices and discounts in separate
databases. You're taking on the overhead of writing an additional service for
what could have been a database join.

------
fulafel
Wow, microservices sound like a very painful trade-off to make. Is it really
worth it?

~~~
oftenwrong
Microservices are for when you have no other choice. If you truly cannot build
a monolith, then build microservices. If you have merely chosen to adopt a
microservice architecture when a monolith would have been suitable, then you
have made a huge mistake.

------
fmakunbound
This is pretty frequently discussed at Java developer conferences for some
reason.

------
eecc
> Hide implementation detail and expose functionality through intention-
> revealing interfaces.

Great, then a new US comes in that changes the intention and you enjoy the
pleasure of changing the interface. Enter versioning. You start crying.

~~~
rumanator
> Great, then a new US comes in that changes the intention and you enjoy the
> pleasure of changing the interface.

That's not a microservices problem, not even a software architecture problem.
That's at best a project management problem. In fact, pretty much every single
engineering project there ever was would struggle to meet any form of neck-
breaking change in design requirements.

~~~
quonn
But with microservices you have more supposed independent interfaces. With a
more monolithic approach you might just change a system internally and deploy
it. Internal interfaces are generally easy to adapt.

~~~
rumanator
> But with microservices you have more supposed independent interfaces.

You don't. You have more services accessed over a network, and might not have
control over all services. But that's already par for the course in nonoliths.

If you were to refactor your components by changing classes around (say, a
major version update in a dependency) while not updating the components that
used them then you would also have problems.

That's why versioning and service discovery and HATEOAS are a thing.

~~~
quonn
But the compiler will find that. It‘s very different from deploying services.
(And I‘ve been doing both, for years.)

------
kcen
One of the problems with forcing strictly disparate domain/service is that
each one introduces its own security context and authZ handling. That's a lot
of places for mistakes.

~~~
rumanator
You're just stating that distributed services need to have security.

~~~
elygre
I think they stated that distributed security is more work / harder than
monolithic security. I would tend to agree.

~~~
Tsarbomb
In my experience, the reason it is more work is because you end up with
something much more well defined and robust.

Each service having well scoped and defined RBAC or AuthZ for its focused set
of features makes the whole architecture as a whole much easier to reason
about from a security standpoint. I've done successful pen testing and
auditing of some monoliths in my time where the critical security issues arose
out of untested and unexpected execution paths that were only possible because
the surface area is so large.

Maybe in theory a "well written" monolith would be superior but I'm only going
but what I have seen in practice. I think the extra work is worth the trade
off.

------
donbox
[https://outline.com/ws4FFC](https://outline.com/ws4FFC)

------
Copenjin
I'm really having a hard time finding the willpower to read articles that just
even mention "microservices" in the title after working on such architectures
(with Java) for a while.

Nice idea but most of the times horribly implemented, with no care about
monitoring and logging as someone has already said.

It requires a degree of attention to details during design and development
that few are willing to put in. It's easy to go for a prepackaged
solution/framework thinking that you are building something that could be
considered state of the art, when instead you end up with something that takes
multiple tries to even boot correctly or 10 minutes or so to "stabilize" a
garbled mess of inter-dependant components.

Designing these systems require a whole new set of skills that were not
required when designing monoliths.

~~~
jordanbeiber
You really point out that you’re going to require a dedicated team to manage
all the things you’ll need. I you’re not at this scale there will be a lot of
overhead to manage for the dev teams.

I’m in a team like this, supporting a bunch of developers - we build a CLI, an
API (in our domain! It’s just another domain with the devs as consumers) and
juggle approx. 16 different tools (monitoring, metrics, tracing, service
discovery, etc etc).

Doing it through custom tooling that enforces conventions is, IMHO, the way to
do it successfully.

~~~
jon-wood
This is precisely what the team I lead is responsible for as well. We’ve got a
set of standardised libraries, deployment conventions, and a CLI which
encapsulate almost everything needed to have a service coexist within the
wider platform in a consistent manner.

We’re also responsible for wider reaching services within the platform such as
an API gateway, authentication, and event ingestion from IoT devices. As the
team that has an on-call rota we also give the final go/no-go on services
going into production.

------
ivan_ah
Off topic but wanted to flag:

> [...] we intend to cover our experiences with Event Storming in a separate
> blog.

In case anyone is not familiar with this usage, these days "blog" is often
used instead of "blog post." I was thoroughly confused by this when I first
ran into it. I tried fighting against this usage by colleagues, but it seems
to be widespread enough now that it cannot be fought... I guess language
evolves, and I have to get with the times.

I suspect one or more of the following aspects might be at play: • general
loss of identity (since lots of posts are on medium, and less on a personal
and work blogs) • "blog" is short for "blog post" • desire to distance oneself
from "post" which is more associated with facebook and instagram posts

~~~
hnick
I don't read any more into it than someone saying "put it on the USB" instead
of spelling out USB stick/drive/etc. It's just a lazy abbreviation that caught
on.

------
banq
anti-agile

~~~
pragmaticdev
That's a strange claim to make. Care to elaborate on how making things easier
and faster to change is less agile?

