
Microservices Ending Up as a Distributed Monolith - adamnemecek
http://www.infoq.com/news/2016/02/services-distributed-monolith?utm_source=infoq&utm_medium=popular_widget&utm_campaign=popular_content_list&utm_content=homepage
======
paulddraper
There's compile-time DRY, and then there's run-time DRY.

* Compile-time DRY allows me to change my code quickly and reliably.

* Run-time DRY allows me to deploy services independently.

Sometimes these objectives are at odds. Sharing code among my services, can
make my code less repetitive at compile time, but it can also lead to
duplicated execution at run time. And vice versa.

The important thing (besides reading the latest blogs and following the latest
tech trends, of course) is to figure out what _your objective is._

For example, at my company, we want (1) to improve the ease of code
development, and (2) to increase production stability for critical
functionality.

So we have microservices that allow us to prevent non-critical issues from
taking out critical functions. And we use extensive share code for HTTP, JSON,
templating, logging, some business logic etc. in order to make development
easier. We don't care about decoupling releases or segmenting production
ownership. In other words, run time DRY isn't important to us. Our services
are simply ways to intelligently distribute execution across limited
resources.

Assess your objectives, and then choose the most solution that will meet them.
Never do the reverse.

If you need separation of concerns and information hiding at runtime, this
article has some good tips.

------
agentgt
I find this pretty hilarious coming from the guy who wrote Hystrix of which
was extremely tightly coupled to another library called Archaius of which
pulls a gazillion other dependencies (that being said I still highly respect
Ben).

I had to spend a couple of days decoupling the mess [1].

The other annoying thing with Hystrix is that its basically Request/Reply
pattern except its even worse because it uses inheritance (command pattern)
and a massive singleton (HystrixPlugins). Furthermore although it uses a
streaming library (rxjava) it sort of disingenuously provides very little
actual support for streaming.

You don't need to have separate executables to make things decoupled but
rather a good build system and architecture. Some of the ways are avoiding
Request/Reply, singletons, and the command pattern.

We achieve fairly good decoupling by using a language agnostic custom message
bus (it's over RabbitMQ which is AMQP... tons of clients) but you could use an
actor framework or as another poster (@platform) mentioned Storm and Erlang
process if you want to stick to one language.

[1]:
[https://github.com/Netflix/Hystrix/pull/1083](https://github.com/Netflix/Hystrix/pull/1083)

~~~
fritzo
Do you have some good examples of why not to use the Request/Reply pattern?
I've been trying to understand why some distributed dataflow systems prefer
pushing rather than pulling.

~~~
gizzlon
Let's say you want to push data from the server to a mobile/web app in "real
time". Examples I've come across lately would be a chat application and real
time map updates (a la Uber).

Now your clients could either: (1) Pull the server every 0.5 second, or (2)
Open a connection that stays open and let the server push messages across the
connection.

Request/Reply and pulling is not bad in itself, but in some cases it has two
downsides: (1) It delays everything because you can only pull so often. (2)
I's bad for performance and scalability because all those pulls have to be
handled, even when there's nothing to return.

Now you want to create a backend that also avoids pulling in the relevant
places.

~~~
kasey_junk
I think it is important to always remember that all push models, under the
covers, are just pull models at a lower level of abstraction. This goes all
the way down to the interrupt level.

So when we say pull models are bad for performance and scalability, what we
really mean is that the abstractions we are putting around our pull model at
this level are more costly than the one level down (ie your message bus is
pulling off a tcp/ip socket but avoiding http).

Depending on what kind of performance and scalability you are talking about,
you can either address that by going an abstraction down (which is nearly
always a performance booster that comes with a development time cost) or you
can pull more, less often, at the high level of abstraction (smart batching
protocols also exist at every level of abstraction).

~~~
gizzlon
Your point is valid. But I was under the impression that lower level stuff
often did _not_ pull. At point does a TCP/IP connection pull for example?
Bellow epll/kqueue? Or the keyboard->machine connection? I haven't done
anything that lowlevel in a while though, so I could be wrong.

~~~
kasey_junk
Remember that epoll/kqueue are just more efficient abstractions on top of OS
level event loops. Those event loops are polling from driver* level queues and
interrupts.

Interrupts are just* the cpu polling on interrupt lines in hardware.

------
iand
I think the article misses some important context about scale and economic
efficiency.

When you have 10,000 engineers then decoupling everything behind protocols and
formats makes a huge amount of sense because the coordination costs are
astronomical.

When you have 10 engineers then you need to standardise tooling and libraries
because the costs associated with diversity in the codebase swamp the
coordination costs.

------
wcummings
I've worked on what I think could be called a "distributed monolith" and its
not bad, services and clients can share interfaces, common build and
deployment infrastructure for everything etc.. You still get the benefit of
being able to scale out, separate failures and deploy components easily, w/
the benefits of standardizing around one language.

~~~
maxxxxx
We do the same. A monolith gets copied to all stations which then run
different parts of it. That way they all have the same infrastructure code
available.

------
yahyaheee
I love this article. Microservices are as much about coupling as they are
about scaling. It reminds me of the recent notion of disposable code. We need
to get more comfortable with the idea that decoupling likely has more long
term benefits than reusablilty.

------
deepsun
I don't understand how Guava couples my services, could anyone explain me?

Let's say I have service A that uses Guava v18, and service B that uses Guava
v19. Neither of them care what other one use, all they see is RPC (or HTTP)
calls.

~~~
Randgalt
I was the guy at Netflix that had the task of moving the Company from Guava 10
to Guava 11. Guava 11 was backward-incompatible with Guava 10. On the Platform
team (produced libraries the entire company consumed) we could only move to
Guava 11 if EVERYONE moved to Guava 11. The only alternative would have been
to shade Guava 11 into the platform library which wasn't pleasant.

~~~
BurningFrog
That sounds like a real impractical way to run a platform team.

Moving the entire company codebase at the same time may be possible with a
dozen developers, but when you reach Netflix size it seems impossible.

What am I missing?

~~~
Randgalt
It wasn't practical. But that's what we had at the time.

~~~
hardwaresofton
Just to chime in here -- I think this is a relatively common happening. I've
found that companies with lots of teams developing software often look to
"standardize" the software being used to create things like web
services/pieces of infrastructure.

This often starts with creating a local nexus server (in java land), then is
followed relatively quickly by creating a "X company commons" for "everyone"
to use.

------
benjaminwootton
A lot of Microservices advocates talk about abandoning DRY to reduce code
coupling.

I think that's completely impractical and that we should still make heavy use
of shared abstractions in modules and libraries.

If I have a routine in two application services, e.g. a currency converter,
and that code has to be changed, I have a few options:

With code duplication (Abandon DRY):

1\. If the domain demands, I have to change the routine in multiple services
and build, test and deploy both of those services in lockstep.

2\. If the domain allows, I can change it in one service and build, test and
deploy that one service in the short term. The other service will likely need
to be updated at some point in the future. This is hard to manage and brings
duplication of effort.

Without code duplication (DRY):

3\. If the domain demands, I have to change it in _one shared library_ and
then test and deploy both of the dependent services in lockstep.

4\. If the domain allows, I can change _one shared library_ and build test and
deploy _only the one service_ that I need to change in the short term. The
other service will likely need to be deployed at some point in the the future.

Shared libraries are orthogonal to the question of loose or tight coupling. We
can have no shared libraries but still have two services which are tightly
coupled and chatty. _That_ is the situation we need to get away from, and it
is acheived through good DDD and service boundaries, but really doesn't seem
to have much to do with shared libraries.

~~~
pm90
I think the problem comes in when the shared library is used to provide some
kind of baseline service, so that if changes are made to the library and 2
services using them are out of sync, then chaos will result.

The solution you advocate is correct. Service boundaries need to be well
defined and well separated.

~~~
ahallock
> so that if changes are made to the library and 2 services using them are out
> of sync

Wouldn't versioning help with this?

~~~
pm90
In what way? Versioning helps pin shared libraries so they are not
inadvertently updated. However, when you have 2 services A and B using the
same library, changes to the library, if they are not updated in both
services, may lead to the services being unable to talk to one another. e.g.
Suppose service A and B use library L. But the authors of library L decided to
rename a certain field in the JSON of HTTP calls that are crafted by the
library. Now, unless both A and B update to the new version, they may not be
able to talk to one another, so each has to coordinate their efforts to either
hold off using the new version, or deploy the new version at the same time.

~~~
ahallock
> But the authors of library L decided to rename a certain field in the JSON
> of HTTP calls

The API endpoints should be versioned as well. Once all services have been
upgraded to use the next version of the API, the previous endpoint can be
deprecated.

------
neuroid
I had a quick look at the transcript of Ben's talk, and it seems that one of
his main issues with microservices which share the same code base (and
therefore dependencies) is the overhead of upgrading their dependencies. I do
understand that in some cases this might be problematic. However, if the
microservices are written using the same language (or runtime), there is a
huge benefit of reusing some abstractions. These might include things like
validation schemas, serialisation formats, etc. These are the things that most
likely would need to be created anyway for every microservice written in a new
language. As usual, there has to be some balance in the amount of coupling
between microservices. But there is nothing inherently wrong with reusing code
between them. I think the most important factor in a microservice ecosystem is
having well defined protocols and APIs.

------
austingunter
You can also view the original talk at www.microservices.com/ben-christensen-
do-not-build-a-distributed-monolith

------
jonesb6
Isn't that what's supposed to happen? You don't distribute small systems, you
distribute big systems, otherwise the benefits of microservices are way fewer.

------
bkovacev
People jump on the microservices bandwagon way too often. I'm glad that Jan
pointed out that not everything that is broken down is a micro service.

------
tunesmith
So, if someone creates a DropWizard endpoint, and someone else creates a
Spring @RestController endpoint, and someone else creates a Scala Play
endpoint, then the problem becomes that someone that knows Spring but not
Scala might find themselves tasked with supporting the Play codebase, which is
scary. So then the employer seeks to standardize on Spring, but that's
apparently bad because if every new endpoint requires Spring then that's a
Distributed Monolith...

------
platform
I have used Apache Storm and Erlang OTP. And cannot still grasp how micro-
services are different from Storm's bolt's or Erlang's processes.

While subjective, for a JVM-based solution stack, Apache Storm appears to be
more elegant, complete and efficient than Linkerd, as an example.

May be advantage of microservices vs Storm becomes more apparent with scale,
and I just simply had not had the experience at that level.

~~~
kashif
Something I posted to clarify microservices and SOA -
[https://medium.com/@kashifrazzaqui/will-the-real-micro-
servi...](https://medium.com/@kashifrazzaqui/will-the-real-micro-service-
please-stand-up-16bef3ed72ec)

~~~
agentgt
Message Bus != SOA. What the prior poster is alluding too is the Actor pattern
and/or Message passing pattern.

My problem with microservices is that your often picking subpar
solutions/technologies particularly if low latency is a priority. For example
HTTP is not always the best protocol and Request/Reply is not always the best
messaging pattern but this is what most people equate to microservice (ie HTTP
1.1 REST a lightweight list of servers (etcd, zookeeper, consuler) or a load
balancer).

