
Domain-Oriented Microservice Architecture - amadk
https://eng.uber.com/microservice-architecture/
======
sagichmal
> In other words, organizations adopt microservices for an operational benefit
> at the expense of performance.

...no, not at all. There are a couple of operational benefits, but vastly more
drawbacks, and on balance microservices are phenomenally harder to operate
than monoliths.

Organizations adopt microservices when the logistical overhead of coordinating
teams against a monolith becomes so large that it starts affecting product
velocity. Microservices are a way to release that organizational/logistical
friction, at great technical cost.

With that said, the domain-oriented bounded context is indeed the right way to
think about service delineation.

~~~
scarface74
The last company I worked for was very much an “API first” company. We were a
B2B company and our APIs were used by our relatively lightly used website. But
they were also used for sporadic and large ETL jobs when files from customers
came in and most importantly we sold access to our APIs that were used for our
customer’s mobile apps and heavily trafficked website.

We went microservices for the canonical reason you should - to be able to
release and scale independently.

~~~
sagichmal
That reason is oft touted but experience has pretty unambiguously revealed
that it never pans out in practice. I mean you do get that benefit but the
price you pay for it is absolutely disproportionate if that's the only thing
you want. Opting in to microservices purely for presumed technical benefits is
absolutely a mistake.

~~~
scarface74
How is it a “presumed” benefit when we actually did sell access to our APIs to
clients? Some of our APIs initially had very low usage during the day until a
batch job came in, others saw a spike in usage by over 100% when we bright a
new client in.

We were used by health care networks. Can you imagine the increased use post-
Covid?

We even had some that were both hosted on Fargate (Serverless Docker) with
lower latency, more expensive but slower scaling for online use and hosted on
Lambda for internal batch use - higher latency, faster scaling, less expensive
for batch use. The CI/CD pipeline deployed to both.

~~~
sagichmal
My point is that you don't need to rearchitect your application in an entirely
different style to be able to scale in response to load.

~~~
scarface74
If only part of your “application” has more load than others what do you
suggest?

That instead of being able to granular take part of the application that had a
larger load, and run it on a Firecracker micro VM (the underlying VM for
lambda and Fargate) with 256MB RAM and 1 core, we add enough VMs to scale a
monolithic app - even the parts that don’t need it on a full scale VM with 8GB
RAM and four cores?

We did actually have to do something similar for a legacy Windows app. We
scaled the entire process up based on the number of messages in the queue. It
was extremely wasteful. It required at least a 4GB/2 CPU VM compared.

~~~
sagichmal
Scale the whole thing. The cost of doing that is usually orders of magnitude
less than the true CapEx and OpEx of microservices.

edit: If breaking out one service from your monolith worked for your use case,
that's great. I'm not trying to deny your experience. It is atypical, however.

~~~
scarface74
We are talking about a 16x difference in resources. Would you also suggest
scaling a database that was more read heavy than write heavy instead of
splitting reads and writes when you can deal with eventual consistency and
just autoscale the read replicas?

~~~
sagichmal
The comparison is not equivalent.

~~~
cutemonster
Agreed, I think the comparison is not equivalent,

because a read replica actually is the same database application (same
"monolith") just started with different parameters, to behave as a replica

~~~
scarface74
The database is the same but you have to then separate your code into reader
and writer services with different connection strings, you have to make sure
that anything that can’t be eventually consistent uses the writer connection
string, etc.

It’s not just a matter of spinning up a database.

Also, since many enterprise apps live stores procedures and putting business
logic in the database, that’s another ball of wax you have to untangle.

------
Twisol
Depending on what angle you come at this from, you could say that DOMA groups
services into clusters ("domains"), as Uber has done here, or that services
always should have been domain-driven, and DOMA welcomes the networking layer
inside the bounded context as well.

I've spent a lot of time trying to understand when a microservices
architecture makes sense, what the caveats are, and what philosophy one should
take to building services. All the material I've read seems to point in the
direction of services being ideally coupled to domain boundaries.

It seems to me that Uber's services proliferated beyond the framing of bounded
contexts, and DOMA is their attempt to reign it back in again. I think it's an
excellent strategy, and arguably a very good approach for other companies who
find themselves in this position.

I don't think DOMA is a good place to _stay at_. The network should only be
tolerated as long as it provides benefits that outweigh the costs. "Monolith"
is not synonymous with "poor design". Seeing that these enclaves of services
sitting within a domain depend on eachother in the way the OP describes, it
really makes me think that they'd find further benefits by expelling the
network from each domain.

~~~
adamkl
> The network should only be tolerated as long as it provides benefits that
> outweigh the costs.

I agree. There should be no need to use network calls to enforce interface
boundaries if you have a cohesive bounded context.

Or to be snarky about it, welcome to 2004 Uber! Eric Evans sends his regards!

~~~
gen220
Unfortunately, picking and enforcing bounded contexts is hard work in a big
organization, even with strong code review processes.

There will always be people who don’t want to respect where the boundaries are
drawn (not in a constructive, “it could be better” kind of way, but in a “but
this works, too” kind of way). If a group of such people get together,
microservices compartmentalizes their capacity to drag down the ship, so to
speak. I think this risk compartmentalization is a benefit that must be
weighed against the costs (in terms of latency, maintenance of shared libs,
opentracing, etc). These days the costs are vanishing, as tools are quite good
and becoming easier to manage.

All that said, if you’re a small team of senior engineers working with a
shared mental model, a single binary with internally-bounded contexts works
really well and I agree with you, having seen it done well.

~~~
adamkl
I was lucky enough to be in the right place, at the right time, to lead a
group that scaled this approach across half a dozen different development
teams.

Fortunately everyone bought into the architecture, and respected the
boundaries. Not everyone was senior, and not all the code was great, but we
adopted the viewpoint that so long as the bad code is in the right spot (and
not talking to things it shouldn’t) everything would be ok in the end. And it
was.

Half a dozen teams working in one codebase was definitely pushing it though,
and the need to scale much beyond that would have definitely required some
service-level compartmentalization to keep the ship from sinking, as you said.

Even then, it would still be a far cry from the “microservices should be small
enough to re-write in 2 weeks” approach.

------
systematical
I'm looking forward to the article about the problems DOMA introduced for them
and what they came up with next. Wonder if we will see a full circle back to
monoliths.

I know I'm being snarky and I have used micro-services myself, but only when
it was smacking me in the face as the best tool for the job. Is the fools-gold
rush still on to do everything as a micro-service from the get-go?

Side note, I can't wait to hear my boss use "DOMA" in a meeting in the coming
months. FML.

~~~
pjmlp
Sun RPC marketing message, "The network is the computer".

So yeah, we keep going at this, and then people discover that monoliths
written in a correct modular way, with libraries, happen to be easier to debug
and reason about without a network in the middle.

Main problem seems to be that not many developers bother to read about modular
programming, large scale development (like Lakos books) and what features
their language of choice offers for such endeavours.

~~~
eternalban
Are you sure that was just a marketing line for "RPC"? I thought the scope was
broader.

IMO Micro-Services were addressing the skill gap in designing comprehensive
schemas, not so much the object layer between user and data. So not modular
"programming" but rather "modular design".

~~~
pjmlp
Might have been, I just remember it from being printed somewhere on the page
on the programming manuals.

------
kopos
Feels like a lot of old knowledge was discounted / ignored and the mistakes
are being rediscovered.

Feels a bit like the Hexagonal Architecture being rediscovered.

Micro service with half life of 1.5 years? Does that mean that enough planning
is not being done? Or leadership failure at software planning level?

Collaboration between teams at scale is very hard but that is what the
leadership layer is for - to collaborate more not build more micro services.

~~~
sixdimensional
Or how about plain old onion architecture... even older than hexagonal.

This is literally scaled/distributed domain-driven design (DDD).

I have felt strongly that folks got so caught up in the hype of yet another
new thing they forgot how to extend what came before.

It feels like a reinvention of existing ideas at larger scale, and a pattern
we keep repeating.

I'm not complaining, of course - new things are possible and being learned
through this innovation. I do feel we should be more careful on the cutting
edge, to see how it relates to where we came from.

Put another way, reinventing the wheel is not pointless if you come out with a
better thing, or better wheel. But don't forget what was good about the
previous wheel before you throw it out?

~~~
staticassertion
I'm confused. You're saying that people are reinventing existing ideas at
larger scale, but you also are saying that people don't know how to extend
what came before? Those sound like the same exact thing.

~~~
sixdimensional
You're not confused, but there is a nuance. I'm trying to walk a fine line of
not criticizing them for discovering this late into their process (maybe they
knew all along but were busy inventing), but also questioning why they
couldn't see this sooner. Trying to get to the heart of what took this
innovation/recognition/learning to happen.

I am making an observation - when each new "fad" or "hype cycle" tech starts,
it seems as though the pattern knowledge of what came before is discarded, or,
disregarded as "legacy" or possibly even just forgotten. It feels like a
knowledge transfer is missing. It would be terrible if, we, as an industry
aren't passing down knowledge and reinventing hard won pattern discoveries
efficiently.

Did this pattern Uber discovered come from studying onion architecture, DDD,
etc first, finding the limitations, and then scaling them?

Or did this arise from throwing away everything that came before (or not
knowing about it), forging an undiscovered path, and then rediscovering the
old patterns could be applied? If the latter, what can be learned to make this
process of discovery and linkage to existing patterns more efficient?

I think this article shows innovation is tricky, or, the risk (and potential
reward) at the bleeding edge. Leaving behind design constraints of what came
before might be necessary.

Maybe I'm trying to say, as an industry we need to balance exploitation of
previous knowledge and our attitudes about how we feel about "legacy", with
the unquenchable thirst for the next new innovation?

------
SergeAx
I felt bad for Uber engineering force while reading whole article. They
swamped themselves into that microservices hell due to pressure for teams
velocity and TTM, and created enormous tech debt in the process. Now they are
trying to patch it up with ridiculous means like putting JSON inside of
protobuf messages. I really hope this article could be a warning for
management of growing startups for what could happen if you squeeze features
out of your team, but managers usually speed-reading those articles just to
order "do like Uber did".

------
harel
I'm not entirely sure why this is sold as innovative new approach. Over the
last few years, I've seen and developed micro services that are indeed cut
down to specific functionality, and I've done ones where the domain was the
scope. The latter has proven to produce bigger services, but a more
maintainable ecosystem. Also, the division for micro services should also be
dictated by the context of the business, and not just the current whims of the
devs.

------
xcskier56
While I work on much smaller codebases and so don’t have any particular
insight into how DOMA will work for such large organizations, the piece that I
really liked here is how the separate their different general layers of
services.

The key to any service being usable outside of the exact context it was first
written in is to ensure no product specific business logic is added to it.

By splitting out infrastructure from general business from product specific
services, you can do a much better job of understanding, and therefore
controlling where where product specific logic is allowed. This in turn will
make your lower level services far less coupled to the exact context they are
first used in.

For smaller orgs, this is by far the more useful information, rather than how
to deal with 2200 microservices.

------
levischoen
Yup. So micro services of micro services. Clearly defined interfaces. Single
entry point for other services/domains to interact with. seperation of
concerns. The fundamental law will always be true :
[https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_softw...](https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering)

------
dkersten
Somewhat off topic but maybe someone can enlighten me:

> Uber has grown to around 2,200 critical microservices

Even thinking about the very largest systems I’ve worked on, I can’t think of
what could possibly be split into 2000 separate individual services. What are
these thousands of microservices and how micro are they?

~~~
jmchuster
Hmm, would it be easier to imagine PayPal or Stripe requiring a lot of
services? Given its volume of payments, Uber has basically had to re-implement
PayPal internally, custom-tailored to its own flow. Companies at this
size/scale basically contain multiple whole companies within them. Think of
like all the SAAS services that your company relies on, and then imagine that
your company now owns all of them and all the services that come with them.

~~~
clintonb
Uber uses Stripe and PayPal to process payments. They don’t need to
reimplement either, and doing so would be a bit of a waste.

It’s worth pointing out that not all of the 2200 services may be user-facing.
Some may be internal, such as admin tooling or CI services. That said, 2200
seems like a lot!

~~~
jmchuster
Sorry, I think you are a bit mistaken about how payment platforms work. Stripe
and PayPal provide an interface to a country's banks. In exchange for
abstracting away the underlying bank infrastructure, you pay them a fee for
every transaction. Stripe and PayPal support a limited number of countries and
a subset of the bank's functionality. So if you want to interact with any new
countries or access any non-supported financial instruments, you'll have to
directly integrate with the banks and implement those yourselves. To give you
a sense of scale, Uber's payments volume is like 10% of Stripe's.

~~~
wpnx
Curious; where did you get the 10% number from

~~~
jmchuster
Here's an estimate of Stripe's yearly transaction volume at $200 billion
[https://www.forbes.com/sites/christianowens/2019/09/21/strip...](https://www.forbes.com/sites/christianowens/2019/09/21/stripe-
isnt-overvalued-at-35bn-you-just-dont-understand-twenty-first-century-
commerce/)

Here's Uber's announced results of $18 billion in gross bookings
[https://investor.uber.com/news-events/news/press-release-
det...](https://investor.uber.com/news-events/news/press-release-
details/2020/Uber-Announces-Results-for-Fourth-Quarter-and-Full-Year-2019/)

Round that to significant figures and you get 10%.

------
exnuber
They say they draw inspiration from DDD and CQRS, yet seem to miss one crucial
factor. And no, I'm not talking about bounded contexts. They still focus in
re-use of commodity rather than addressing why their microservice approach
didn't work. Their strategy for defining how functionality moves down the
stack is not bad, but it's not anchored towards the right thing. It's the
consistency boundary (or aggregate root if you will). It's the circle you draw
around your business rules and say "this group needs to remain consistently
enforced". This defines your unit of scale, not re-use. So while they are
headed in the right direction, I fear that they are still missing a
fundamental piece of information to steer on.

I was employed by Uber (I quit after a couple months), and the idea's that
they now hint towards were largely rejected by the engineers, and that was
less than a year ago. Uber is just in the position to throw a large sum of
money at making wrong decisions and getting away with it, because it's not
their money. It's VC funny money.

DISCLAIMER: Yes, I did indeed create this account to be able to reply
anonymously.

------
vmurthy
I recently read Domain Modelling made Functional [0] and one of the under-
recognised benefits of using Domain Driven or oriented designs is that (if
done right), _business workflows_ and processes become well explained for
developers and minimises surprises from business closer to release dates. For
e.g., an order processing system will have lesser assumptions by developers
and tech leads because someone with enough knowledge of the business process
from order processing will be able to guide you.

[0][https://pragprog.com/titles/swdddf/](https://pragprog.com/titles/swdddf/)

------
nfoz
I don't like that they _define_ microservices as components communicating via
RPC, because this precludes other models like pub-sub.

------
ianamartin
I despise Uber as a company, but I've always loved them as a technology group
ever since I used to go to their meetups in NYC. You can always count on Uber
to throw a great party and also come up with the worst possible technology.

If you want to know how not to do things, Uber is a very good place to look.

~~~
hinkley
And all the people who worked there will consider the fact that they did 1)
something and 2) were successful means that that 'something' had anything to
do with it, and therefore we should keep doing it.

Not that this is unique to Uber, mind. I just described half of my coworkers
too.

------
etxm
I’m honestly not trying to be snarky here but I didn’t know there was any
other way to design microservices (with intention) that weren’t domain
oriented.

~~~
timclark
I’ve seen one service per database table, and have had architects seriously
think that a table equals a bounded context. Then they try to work out how to
do two-phase commit across micro services.

~~~
etxm
Oh that. I’ve had to merge services like that.

People are always grabbing nouns and thinking they’re microservices.

------
sdan
Basically clumping microservices

~~~
deltaeerie
I believe this is only the first point. It's also establishing a hierarchy of
these "clumps" (layers), strict APIs representing a single clump for other
clumps to consume (gateways), and a pattern for clumps to extend functionality
of other clumps without polluting their models (extension architecture).

------
keyraycheck
> as Uber grew from 10s to 100s of engineers with multiple teams owning pieces
> of the tech stack, the monolithic architecture tied the fate of teams
> together and made it difficult to operate independently.

I think this is key sentence. Microservices does not make a lot of sense for
teams of 10s, while are a great tool for team of 100s.

------
recursivedoubts
Seems to me that big progress will be made when we can decouple system
boundaries from network calls.

~~~
bob1029
I still think the approach of baking the entire enterprise into 1 gigantic
binary is the best. Stack traces are so much easier to work with compared to
distributed logs and side effects.

We are already at a point where you could hypothetically do this for a
reasonably-sized organization. A 64 core CPU can support a huge number of
clients. Stuff like .NET Core scales really well if you want to build
something complex like this. One big binary that occupies an entire physical
host is an extremely compelling development model. Literally everything
becomes a direct method invocation. You can also have type enforcement and
atomic releases for the entire enterprise. Also makes a monorepo an obvious
choice for source code management.

~~~
jzoch
This falls apart when you care a lot about availability and therefore want a
particular service to be stateless. Then you add some feature that needs a
tiny bit of state and suddenly you can't scale up and down as easily and need
to start paying attention. This rolls forward into a big ball of mud - a
monolith can be tricky too at scale

~~~
bob1029
One trick is to leverage the fact that you have 1 type domain, and to develop
a common persistence abstraction for the entire enterprise. Then, all of your
business entities can be pushed through it for a consolidated replication
flow. This can be [a]synchronously replicated to additional node(s) as
required, potentially with replication rulesets defined in application code as
well.

It sounds like a complex monster until you build it one time. Then you are
basically done. The leverage you get when you have 1 way to do everything is
extremely powerful. I do recognize there are scenarios where you cant force
one persistence abstraction on all use cases, but there's no reason you
couldn't have a TimeSeriesEntity (keyed by time) in addition to a typical
BusinessEntity (keyed by a unique integer). Both could have unique replication
implementations, but it would still be standardized and all nodes would be
speaking the same protocol because they all derive from the same source.

The monolith only remains a monolith in operational terms until the
engineering team develops some imagination.

------
kerng
Very interesting and seems like a logical next step with things get a bit too
messy with micro services.

One thing that was not touched on - how is security done in DOMA?

* Is auth happening at domain gateways or at each service?

* Similarly for encryption, does it terminate at Domain GW or at each service?

~~~
alpmeow
Hopefully both

------
stevenalowe
Nice to see Uber rediscovering domain-driven design principles.

------
mbrodersen
If you can't make a monolith work, then you will fail even harder trying to
make a Microservice Architecture work.

------
pgt
Microservices are about scaling teams, not tech.

------
sroussey
Uber thinks they only have that many services? Yikes, double, or triple the
number... several years ago. Maybe they have fewer now?

