
Under Deconstruction: The State of Shopify’s Monolith - kenrose
https://engineering.shopify.com/blogs/engineering/shopify-monolith
======
lmarcos
Great article. Main takeaway: microservices is not the only option when
managing big codebases. In a parallel universe I imagine that the coolest
trend in software development right now is a tool for monoliths: all code in a
single repo, independent deployable components, contracts in the boundaries
and mockable dependant components where needed. As opposed to our universe in
which building microservices is the non-official way to go.

~~~
didibus
Ignoring some of the deployments and dependencies related aspects of
microservices Vs monolith, one aspect that has me convinced against my own
ideals is that a micro-service has a "strong" boundary in that it is actually
difficult and effortful for a developer to cross it.

This in turn has a positive effect in maintaining proper boundary and putting
the right amount of thought about the interfaces and responssability of each
component.

~~~
exterm
It's all tradeoffs. You get a stronger boundary, but you also get a
distributed system.

Also, the first try of drawing boundaries will always be varying degrees of
wrong. If you have very strong boundaries at this stage, iterating on them,
moving responsibilities around, can be harder.

Also, with the right tooling it's definitely possible to harden monolith
internal boundaries to a comparable level.

I can see though how many smaller companies would not be in a position to
build that tooling.

Anyway... there is no either / or here, as I've explained in another comment.
What if you have components within a monolith, but each component has its own
database, for example? What if test suites are completely isolated, so that
tests for component A can not access code in component B?

You can get pretty strong boundaries with a few comparably simple tricks.

------
straws
A number of years ago, I worked on a team (~20 engineers in total) that
successfully carved off two relatively independent portions of a large Rails
app using engines. I'm happy to see that Shopify is also using that strategy.

I'm curious to know more what sorts of challenges they have around managing
dependencies across engines — I think what we were doing was fairly vanilla
Rails, and we didn't have the opportunity to run into those sorts of issues.

~~~
exterm
The answer to that question could probably fill another blog post :D

Long story short, Rails and dependency inversion equals lots of friction. The
whole framework is built on the assumption that it's OK to access everything
from everywhere, and over the years we've built lots of tooling on top of
those assumptions.

E.g. we heavily use
[https://github.com/Shopify/identity_cache](https://github.com/Shopify/identity_cache)
with active record associations that cross component boundaries.

We also have a GraphQL implementation that is pretty closely coupled to the
active record layer and _really_ wants to reach into all the components
directly.

All of those problems can be overcome, but this is definitely an area where we
have to working against "established" Rails culture, and our own assumptions
from the past.

~~~
straws
I hope to hear more in the future!

Do you envision any extension points to the way engines are implemented that
could better enforce boundaries? In our engines, there was nothing that
referenced another engine's resources, leaving the main application to handle
route mapping and ActiveRecord associations between app models and engine's
models.

I feel like the use-case for engines has long been around supporting framework
like functionality (Devise, Spree, etc), but I wonder if there are changes to
be made that better support modularization for large apps.

~~~
exterm
> extension points to the way engines are implemented that could better
> enforce boundaries

Can you expand on that? I'm not sure I follow.

------
joelbluminator
2.8 million lines , 100 billion business. Rails can scale.

~~~
mandelbrotwurst
100 billion? That seems like a lot of businesses per capita!

~~~
shwoopdiwoop
Fairly certain GP referred to the market cap, not the number of businesses on
Shopify's platform.

~~~
khendron
He might be referring to Shopify GMV (Gross Merchandise Volume —the value of
commerce facilitated by the platform), which is probably approaching $100B per
year.

~~~
csomar
No shopify market cap is $100bn; which is much higher than I expected. So I
looked up their revenue, which is $1.6bn and they have an income deficit.
so...

------
sandGorgon
This is a brilliant brilliant article.

Does anyone know how Shopify created it's Architecture Guild and grew it ? The
author talks about "should have done it earlier"

~~~
exterm
As the author, I would know :)

Thank you for the praise.

Ours kind of organically grew over time, but as I've been keeping it alive for
the last few years I have a pretty good idea of how I would start it fresh.

You probably have some people in the company who either know much more about
architecture than others, or are working on projects that are more interesting
in terms of architecture. Find one of them, convince them to give a 15 min
talk.

Announce the talk widely within the company, tell people to come to the new
"architecture guild" slack channel you created to get the details / invites.

Schedule an hour to give plenty of time for discussions after the talk.

Repeat biweekly.

~~~
sandGorgon
Thanks for replying.

How would you do it in a remote-first world? A zoom talk ?

How does this go beyond that one talk - would you incorporate aspects of this
into official rewards/recognition ?

Or is gratification good enough. Getting a zoom audience is gonna be hard.

~~~
exterm
Shopify has been a fully remote company for a few months now.
[https://financialpost.com/technology/shopify-is-joining-
twit...](https://financialpost.com/technology/shopify-is-joining-twitter-in-
permanent-work-from-home)

We're not using zoom, but google meet - but yes, these happen completely
online now.

I find that people that are doing interesting stuff often _want_ to talk about
it. However, a big part of Shopify culture is "do things, tell people" \- it
is definitely encouraged to spend time spreading context.

It's not directly part of any rewards framework, but one metric that goes into
promotions is the area of impact. By giving a talk to the guild, you can have
impact on a group that's larger than your team, potentially the whole
organization. It counts.

But another reward is the positive feedback, interesting discussions and new
connections that you make through this.

------
octernion
we are actually doing precisely the same thing at instacart (breaking our 1+
million lines of code monolith into discrete components, which we call
"domains"), and typing the boundaries and as much of the internals of these
domains as possible with sorbet types.

this has the benefit of ruby dynamicism (fast development within domains, you
can use all the nice railsy tooling, activerecord, and all the libraries we've
built over the years), with type safety at the boundaries (we've also put in
timeouts, thread separation, and error handling at the boundaries).

the additional benefit for using sorbet is that it makes making typed RPC
calls (over twirp or graphql) much easier as you can introspect the boundaries
trivially.

really cool to see other companies evolving similarly given the same starting
conditions!

~~~
dragosmocrii
Slightly off topic, but does anyone know if this "component based" development
is what umbrella applications are in Elixir?

~~~
ravenstine
Am I the only one who has a distaste for this phrase "component based
development"? It just seems like a fancy way of saying object oriented
programming without an overarching design pattern.

~~~
aidos
Sounds like the “components” described above are much larger than classes.

~~~
octernion
that's correct, at least for us a domain encapsulates many response types and
dozens of different APIs that wrap various datastores, business logic, etc.

------
treis
Have y'all seen any issues around autoloading of classes/modules in
development? I've been working on a rails app composed of a handful of engines
and I've noticed that every so often classes aren't loaded. 6 seems to be a
lot better about it than 5 was.

~~~
mhoad
Rails 6 has a totally new code loader that was built specifically to address
those issues called Zeitwerk. Some details here if you're interested
[https://blog.bigbinary.com/2019/10/08/rails-6-introduces-
new...](https://blog.bigbinary.com/2019/10/08/rails-6-introduces-new-code-
loader-called-zeitwerk.html)

------
gregkerzhner
Interesting article. We use a similar approach for our mobile apps to allow
multiple teams to develop their own modules independently.

Can anyone speak to what the advantages and disadvantages to such an approach
are as opposed to going full Kubernetes / Microservices? Is it that deploys
are riskier and you can't scale separate pieces independently?

------
kawsper
Does anyone know if the Storefront rendering described here[0] is running
Rails or something else?

[0] [https://engineering.shopify.com/blogs/engineering/how-
shopif...](https://engineering.shopify.com/blogs/engineering/how-shopify-
reduced-storefront-response-times-rewrite)

~~~
rafaelfranca
The application is a Rack application reusing some of the components of Rails,
but it is not a conventional Rails application given it doesn't need most of
the framework.

------
banq
DDD aggrgates: loose coupling with high cohesion!

------
throwaway691999
I think it's kind of bad that we have this trend to use "walls" to enforce
modularity. This whole thing about using "walls" to enforce "developer
behavior" is, in my humble opinion, the wrong direction.

If you think about it, almost all lack of modularity comes from shared mutable
variables. Segregate mutability away from the core logic of your system and
the smallest function in your architecture will become as modular as a
microservice.

Really, any function that is stateless can be moved anywhere at anytime and
used anywhere without fear of it being creating a permanent foothold in the
architectural complexity of the system. So if the code is getting to
structured where you become afraid of moving things... do this rather than
build classes and walls around all your subroutines.

Remember as long as that add function doesn't mutate shared state you know it
has zero impact on any part of the system other than it's output... you can
replace it or copy it or use it anywhere.... this is really all you need to do
to improve modularity of your system.

>Again and again we pondered: How should components call each other?

I think this is what's tripping most people up. They think DI IOC and OOP
patterns are how you improve modularity. It's not. Immutable functions are
what improves modularity of your program. The more immutable functions you
have and the smaller they are the more modular your program will be. Segregate
IO and mutations into tiny auxiliary functions away from your core logic which
is composed of pure immutable functions.

>Circular dependencies are situations where for example component A depends on
component B but component B also depends on component A.

I've never seen circular dependencies happen with pure functions. It's rare in
practice. I think it occurs with objects because when you want one method of
an object you have to instantiate that object which has a bunch of other
methods and in turn dependencies that could be circular to the current object
you're trying to call it from. In essence this kind of thing tends to happen
because when you call a method you're actually calling a group of methods and
state within a class and upon all those dependencies as well increasing the
chances of a circular dependency.

Still I've seen this issue occur with namespacing when you import files. Walls
aren't going to segregate this from happening. You need to structure your
dependencies as a tree.

~~~
lmm
> Really, any function that is stateless can be moved anywhere at anytime and
> used anywhere without fear of it being creating a permanent foothold in the
> architectural complexity of the system.

That's not really true. A pure function can still be coupled to a particular
internal data representation. It can still assume particular invariants that
you may not want to maintain. Namespacing functions together with the data
structures they operate on is still a good idea, and helps with keeping a
coherent model at each level - e.g. if your business logic is calling a
function that's about the specific mechanics of encoding data for Redis,
you're probably using the wrong abstraction.

Pushing mutability to the edges is good and useful but it's not the be-all and
end-all of decoupling. Enforced walls are a much better idea than spending
your discipline budget on maintaining decoupling by hand. A lot of the time a
pure function can actually be decoupled completely from the datatypes it's
operating on by using parametricity (and maybe a standard typeclass that the
datatype it operates on conforms to), but you may not notice that unless
you've got some module boundaries that nudge you to think about that kind of
thing.

------
ryanmarsh
There’s so much truth in this. It’s full of lessons I tell clients at the
outset of similar endeavors yet they often do not heed until they experience
the pain first hand.

------
banq
in Shopify, they actually applied DDD bounded context and aggregate ,but they
maybe don't know DDD!

------
ToJans
I can imagine that this has been a huge effort, and kudos to the team, but
this is a solved problem; there are ample methodologies to resolve the big
ball of mud.

IMHO the shopify team could have saved a lot of time by getting some schooling
about strategic DDD, and consulting one or more DDD experts to draw a first
version of their context map.

~~~
yeswecatan
Do you know any DDD experts that offer consulting services?

------
meesterdude
Interesting read. I've seen a component based rails architecture work wonders
for cleaning up a codebase and allowing for the benefits of a SOA
encapsulation while still keeping everything under a monolithic architecture
(and avoiding the networking nasties). Not such a fan of sorbet though, but
hopefully something better comes along.

------
mochii
Very interesting read! Thank you for sharing.

------
bori
I like that they completely dodged the term "microservice" in the whole post.

~~~
exterm
you should read the first post in the series if you want to read about
microservices.
[https://engineering.shopify.com/blogs/engineering/deconstruc...](https://engineering.shopify.com/blogs/engineering/deconstructing-
monolith-designing-software-maximizes-developer-productivity)

------
leafboi
I think it's kind of bad that we have this trend to use hardware to enforce
modularity. If it's a performance issue, sure break it up into more hardware.
If it's just code modularity than by shifting to microservices you are adding
additional complexity of maintaining multiple services on top of modularizing
the system. In short it's overkill. This whole thing about using hardware to
enforce "developer behavior" is stupid. You can use software to enforce
developer behavior. Your operating system, your programming language is
already "enforcing" developer behavior.

Additionally, your microservices are hard lines of modularization. It is very
hard to change a module once it's been materialized because it's hardware.

If you think about it, almost all lack of modularity comes from shared mutable
variables. Segregate mutability away from the core logic of your system and
the smallest function in your architecture will become as modular as a
microservice.

Really, any function that is stateless can be moved anywhere at anytime and
used anywhere without fear of it being creating a permanent foothold in the
architectural complexity of the system. So if the code is getting to
structured where you become afraid of moving things... do this rather than go
to microservices.

>We can more easily onboard new developers to just the parts immediately
relevant to them, instead of the whole monolith.

Correct me if I'm wrong but don't folders and files and repos do this? Does
this make sense to you that it has to be broken down into hardware?

>Instead of running the test suite on the whole application, we can run it on
the smaller subset of components affected by a change, making the test suite
faster and more stable.

Right because software could never do this in the first place. In order to
test a quarter of my program in an isolated environment I have to move that
quarter of my program onto a whole new computer. Makes sense.

>Instead of worrying about the impact on parts of the system we know less
well, we can change a component freely as long as we’re keeping its existing
contracts intact, cutting down on feature implementation time.

Makes sense because software contracts only exist as http json/graphql/grpc
apis. The below code isn't a software contract it's only how old people do
things:

    
    
       int add(x: int, y: int)
    

Remember as long as that add function doesn't mutate shared state you know it
has zero impact on any part of the system other than it's output... you can
replace it or copy it or use it anywhere.... this is really all you need to do
to improve modularity of your system.

Editing it on the other hand could have some issues. There are other ways to
deal with this and simply copying the function, renaming and editing it is
still a good solution. But for some reason people think the only way to deal
with these problems is to put an entire computer around it as a wall. So
whenever I need some utility function that's located on another system I have
to basically copy it over (along with a million other dependencies) onto my
system and rename it... wait a minute can't I do that anyway (without copying
dependencies) if it was located in the same system?

>Again and again we pondered: How should components call each other?

I think this is what's tripping most people up. They think DI IOC and OOP
patterns are how you improve modularity. It's not. Immutable functions are
what improves modularity of your program. The more immutable functions you
have and the smaller they are the more modular your program will be. Segregate
IO and mutations into tiny auxiliary functions away from your core logic which
is composed of pure immutable functions. That's really the only pattern you
need to follow and some languages can enforce this pattern without the need of
"hardware."

>Circular dependencies are situations where for example component A depends on
component B but component B also depends on component A.

I've never seen circular dependencies happen with pure functions. It's rare in
practice. I think it occurs with objects because when you want one method of
an object you have to instantiate that object which has a bunch of other
methods and in turn dependencies that could be circular to the current object
you're trying to call it from. In essence this kind of thing tends to happen
with exclusively with objects. Don't group one function with the instantiation
of other functions and you'll be fine.

Still I've seen this issue occur with namespacing when you import files.
Hardware isn't going to segregate this from happening. You need to structure
your dependencies as a tree.

~~~
look_lookatme
>>We can more easily onboard new developers to just the parts immediately
relevant to them, instead of the whole monolith.

>Correct me if I'm wrong but don't folders and files and repos do this? Does
this make sense to you that it has to be broken down into hardware?

This entire post is literally about using folders (directories) and files to
enforce boundaries...

~~~
leafboi
they use the term breaking down a monolith and "architecture" so from that you
can derive that it's literally about using an entire VM or computer to enforce
boundaries.

Folders and files are used in "monoliths" anyway. Nothing new to talk about
that here. Are you implying that their monolith is just one big file and
they're beginning the process of breaking that thing down into multiple files
and different folders?

I don't know about you but that doesn't make any sense to me.

~~~
exterm
Hey Leafboi - I recommend reading the first post in the series for some
background
[https://engineering.shopify.com/blogs/engineering/deconstruc...](https://engineering.shopify.com/blogs/engineering/deconstructing-
monolith-designing-software-maximizes-developer-productivity)

We don't use "hardware" or "VMs" to facilitate modularity.

~~~
leafboi
All right. I'm wrong. Didn't know this. Thanks for linking. Still can't
exactly fault me on that. It's not easy to find the contextual blog post if
this post doesn't easily say it's part of a series.

Still though, my expose is still relevant, those are some hard lines that can
easily be gotten rid of if your functions were immutable and not part of a
class.

Any internal private function is safe to use anywhere in the system as long as
it's not attached to a class and it doesn't modify shared state. If your
systems were modelled this way there would be no need to really think about
modularization as your subroutines are already modular.

For example:

    
    
      class A:
         def constructor:
             //does a bunch of random shit
    
         def someMethodThatMutatesSomething() -> output
    
    
    
    
       class B:
    
           def someOtherFunctionThatNeedsClassA:
               //cannot call someMethodThatMutatesSomethingwithout doing "a bunch of random shit" or even possibly modifying or breaking something else. Modularity is harder to achieve with this pattern. 
    
    

versus:

    
    
       def somePureFunctionWithNoSideEffects(input) -> output
    
    

somePureFunctionWithNoSideEffectsabove does not need any hard lines of
protection. There is zero need to use the antics of "deconstructing a
monolith" if you structured things this way. Functions like this can be
exposed publicly for use by anyone with literally zero issues.

Shared muteable state and side effects is really the key thing that breaks
modularity. Everyone misses it and comes up with strange ways to improve
modularity by using "walls" everywhere. It's like cutting my car in half from
left to right with a wall and calling it "modularization." When you find out
that the engine in front actually needs the gas tank in back then you'll
realize that the wall only produces more problems.

~~~
richardlblair
I think what's really unfortunate here is you started pretty pointed in what
you were saying, and you've stayed pointed. It reads as confrontational.

It's unfortunate because you make a good point. Pure functions do not get the
attention they deserve. However, no one will read that because you just sound
like you're attacking for no real reason.

I'm only saying this because if you're this way here there is a solid chance
you're like that in other areas of your life. What you have to say is
important, but if you approach your conversations this way people won't
listen.

Why did I take the time to write this? Because sometimes those closest to us
won't give us the feedback we need.

~~~
leafboi
Thanks. But this is the internet. I use a bit of aggression experimentally at
times. Overall though, it sounds confrontational but I'm actually pretty
factual and I never attacked anyone personally, it's all about the topic and
idea. I actually admit when I'm wrong (see above, and who does that in life
and on the internet?).

What's going on is I'm spending zero energy in attempting to massage the
explanation with fake attempts to be nice. I'm just telling it like it is.
Very few opportunities to do this in real life except on the internet.

In the company I work for do I spend time to tell my coworkers that pure
functions are the key to modularity when classes and design patterns are
ingrained in the culture? Do I tell them that their entire effort to move to
microservices is motivated by hype and is really a horizontal objective with
no actual benefit? No. I don't. People tend to dismiss things they don't agree
with unless it's aggressively shoved in their face. They especially don't
agree with ideas that go against the philosophies and and practices and
they've been following for years and years.

Thus if I'm nice about it, I'm ignored, if I'm vocal and aggressive about it,
I'm heard but it will also hurt my reputation. It's HN feel free to experiment
just don't try it at work.

Yeah my attitude isn't the best, but honestly, if I was nice about it, less
people would read this or think about it. By doing this on the internet I can
raise a point while not ruining my rep. (And I'm not actually aggressive as
there are no personal attacks unless someone said something personal about me)

Tell me, in your opinion, how would you get such a point across in a culture
where the opposite is pretty ingrained? I'm down to try this, I can repost my
original post with the errors corrected and a nicer tone to see the response.

~~~
richardlblair
I appreciate the point you're trying to make, but the truth is that you can
make factual arguments without being so aggressive. Whether the aggression is
targeted at a person doesn't really matter. It's unnecessary, disrespectful,
and just feeds into the general toxicity that plagues our culture.

> Thus if I'm nice about it, I'm ignored, if I'm vocal and aggressive about
> it, I'm heard but it will also hurt my reputation.

I think the fact we are talking about your tone and not your points about
functional programming speaks to this by itself. You weren't heard. You were
felt, though.

> I'm not actually aggressive as there are no personal attacks

Aggression without a target is still aggression. If I aggressively take the
recycling out, that aggression is still experienced by people around me.
Probably my partner, who will inevitable have a little talk to me about it,
lol.

> Tell me, in your opinion, how would you get such a point across in a culture
> where the opposite is pretty ingrained?

Engage in an intellectual conversion based off mutual respect. You will never
change someones mind on the spot, intellectual people will often mull things
over for a while. In the process you may learn a few things yourself. I've
worked in places that excelled at this, where respectful discourse was
promoted. Conversations revolved around facts, but respect was maintained.

Sidebar: Shopify doesn't really have microservices. They have a few services,
but they are entire services which serve an entire business unit. They are the
exception. When I worked there I worked on one such service. What I'd tell
people is if you couldn't start a whole new company with the service you were
building, don't build it as a service.

~~~
leafboi
I think you missed my point. I'm saying when you aren't aggressive people tend
not to want to intellectually engage with you. People are emotional creatures
and what doesn't excite them emotionally they don't engage. I'm saying I used
the aggression on purpose for my own ends, but I caveated by saying that no
actual attack occurred.

I think you need to think deeper than the traditional "mutual respect"
attitude and generally being nice. Not all great leaders acted this way
either. It's very nuanced and complicated how to get people to change or
listen. The internet is an opportunity to try things out rather then take the
safe uncomplicated "nice" way that we usually try in the workplace.

>Engage in an intellectual conversion based off mutual respect. You will never
change someones mind on the spot, intellectual people will often mull things
over for a while. In the process you may learn a few things yourself. I've
worked in places that excelled at this, where respectful discourse was
promoted. Conversations revolved around facts, but respect was maintained.

Right except this is exceedingly rare. Most people do not act this way.
Respect was maintained but the point is instantly forgotten and dismissed.
Likely the respect covers up actual misunderstanding or disagreement. I find
actual intense arguments open people up to say what they mean rather than
cover up everything in gift wrapping.

Think about this way. The reason why Trump won the election is not because he
was nice. The complexities of human relationships goes deeper then just
"mutual respect" There are other ways to make things move. The internet is
often an opportunity for you to try the alternative methods without much risk.

>I think the fact we are talking about your tone and not your points about
functional programming speaks to this by itself. You weren't heard. You were
felt, though.

The world moves through feelings. Not for all cases but oftentimes to get
heard you need to get "felt" first.

~~~
webmaven
_> >I think the fact we are talking about your tone and not your points about
functional programming speaks to this by itself. You weren't heard. You were
felt, though._

 _> The world moves through feelings. Not for all cases but oftentimes to get
heard you need to get "felt" first._

This is true, but you have options in terms of what feeling you're aiming for.

There is a world of difference in the response you're likely to get from "When
Z you should do X because Y" vs. "We had a Z problem, it turns out that Y was
the issue, so we did X."

The former will probably get you an "uh-oh" and the latter an "a-ha" or "hmm".
Big difference.

