
Why we ditched Django for Microservices - tmwatson100
http://blog.hubblehq.com/ditching-django-for-microservices-1/
======
umurkontaci
More often than not, I start my projects thinking that they would be small and
don't need a comprehensive framework like Django. Then I figure I need an
authentication mechanism, SQL abstraction, an admin site, a pipeline to
handle, verify the requests and so on. Then I switch back to Django mostly
because I don't want to rediscover the world again. Surely given enough time I
could rediscover, but why would one bother?

As per staying _lean_ as stated in the article. I don't think that's staying
lean at all. I must mention that I have no knowledge of the framework or
codebase of the company in question, although rewriting a codebase is almost
never a good idea. Well, it will require a lot of rewriting which is
redundant, you would be ditching a lot of testing and bug fixing you have done
ever since, and you will probably create a space a lot of new bugs.

I believe a better approach would be componentize your structure in a way
that's still governed by django, every component split into apps. And if there
are outlier apps that do not need to be within Django at all could be used by
redirecting the requests to those services on the load balancer / reverse
proxy level.

~~~
tmwatson100
We're lucky enough that we decided early enough for it not to be too big an
issue.

Also we split up the re-write over stages and haven't done it all in one go.
I'm gonna write more about it in another blog post but essentially we still
have our monolithic Django system.

It currently powers the majority of the services except we have changed the
architecture so we can split things out overtime. This allowed us to achieve
what we wanted to in 4 weeks yet lay the groundwork for the future.

------
bsdpython
This makes no sense from a business perspective. Letting developers work using
any languages and technologies sounds like a good idea to individual
developers maybe but how do you intend to maintain it? If you stick to a few
core languages and technologies and hire / train for those tech then you
reduce your risk if one or more developers leave and reduce the total
complexity of your stack. You should have a good reason for introducing a new
language or technology to your stack and think carefully before diving in.
Just create separate Django apps and only switch to a different tech if Django
is particularly ill suited for the task.

~~~
tmwatson100
I perhaps should have explained that a bit more. Technology choices have to
make sense before implemented but giving developers the flexibility opens a
lot of doors for creativity and exploration.

I strongly believe that learning new languages help people become better
coders overall as it increases their knowledge of different problems and
solutions which can be applied across most stacks.

Also no one developer should ever be solely responsible for a particular
service. But that should also be the case for apps within a Django project.

~~~
bsdpython
As a business if you want to encourage your developers to improve by exposing
them to more different types of languages and technologies then I am all for
it. You can fund a weekly Haskell class. You can implement a 20% time
initiative and let developers work on prototypes using whatever tech they
want. For your production systems, however, you need to have some boundaries
to serve the business even if you are a tech company run by engineers.

------
drbryanadams
I don't get it: why does moving to microservices mean ditching Django?
Couldn't you take your Django monolith and break it into separate (still-
Django) microservices? I get that Django is seen as "too heavyweight" to do
such a thing, though there are literally books
([http://shop.oreilly.com/product/0636920032502.do](http://shop.oreilly.com/product/0636920032502.do))
written about how to slim down Django.

I get that monolith to microservice is all the rage these days, I just don't
understand (and the article doesn't really say) why you have to "ditch Django"
to do that.

~~~
eliben
Seconding this. Could someone clarify?

I thought Django and "microservices" are orthogonal. Can't Django be used to
create any service, even a micro service? I certainly used Django in the past
just as a REST server, without any "web site" to speak of. It just reduces
some boiler plate.

~~~
adambratt
Using Django Rest Framework is practically the EASIEST tool for building
microservices I've ever used. It's incredible how awesome it is for building
APIs. We were recently looking at building out some new services in Flask or
Express. We compared the advantages of each to DRF and quickly concluded that
it wasn't worth it. DRF has so much commonly used stuff done for you and it
sets up one of the best standards for REST API's I've ever seen.

~~~
karuth
Flask is so minimal. Curious what were the disadvantages that weighed against
chosing Flask.

~~~
rattray
Flask is too minimal. Using DRF is like having code written for you. Because,
you know, the code was written (and battle-tested) for you. Why start from a
log every time when you can get a wheel for free?

------
programminggeek
I think the biggest reason people are switching to microservices is that it
enforces protocols and documentation between components that would otherwise
never be created.

Microservices create a boundary and that forces you to create a
request/response protocol to enforce that boundary. You also have to create
documentation for it to be useful.

There is nothing stopping you from creating your app as a bunch of small
services without the added complexity of microservices/SOA. Each slice of your
system could be an isolated library with a stronger enforced protocol and
documentation and you'd get largely the same benefits without having to deal
with the deployment headache.

The reason people don't do that is if they build isolated components in the
same monolitic system, nobody takes the time to create a proper protocol and
documentation.

The only way people seem to be willing to give separate services the
appropriate design respect is to physically separate them and solve them as an
isolated problem. The physical separation forces better behavior for
microservices to even be useful.

It doesn't help that most languages have really weak capabilities to enforce
strong protocols/boundaries beyond a simple type checker via complier. A
language like JS, Python, PHP, Ruby, etc. don't even have a compiler so the
ability/desire to create strong protocols in most code is almost nonexistent
to most developers.

------
davidw
> A microservices architecture is (fairly) new and it doesn't tie you to a
> particular language. If you want to build a service in Lisp, Go, Elixir, etc
> then you can (assuming it makes sense). Good devs love to learn, experiment
> and hate being bored. A microservices approach encourages all of that

The developer in me kind of likes that - fun new toys! But from a business
point of view, I'm not sure it sounds like such a good idea: lots of weird,
unmaintainable stuff.

~~~
raverbashing
And of course, ignoring the problem of duct-taping all the services together.

"Oh we'll just RabbitMQ everything together", sure, great, until you
discovered certain services needed a smaller latency, more bandwidth, etc

And RabbitMQ libraries are a pain and are not smart

~~~
sandGorgon
actually this is a question that I have - how do you render the final output ?

I mean lets say I request a web page which depends on 5 microservices. What
happens ? does rails or whatver block till all five have returned the output -
or do you use something like ESI ... how are the microservices composed
finally ?

~~~
chrisvxd
Adopting a microservice architecture does nothing to define how you "render
your output"\- the devil is still in the details, but that is why a
microservice is so interesting and flexible.

The way you will tackle this will largely depend on many questions like what
services are safe to be exposed publicly, what services are dependant on other
services, do you have a public API?

For example, you could have a public facing API on each service, or endpoints
that render HTML templates. The services may or may not talk amongst
themselves, either via the same public API, or a private one. If you make a
request to one of the services, and that service needs to make a request to
another service to generate the response, you've got to make a call on whether
you want a blocking or non-blocking approach. Consider:

1) Blocking until the other services have responded

2) Pinging the server to check when it's done

3) Using web sockets

4) Whatever the hell else you can think of

Alternatively, you might want not want to expose your services publicly, and
choose to have a single service that talks to the services on the clients
behalf, a middleman. Your client speaks to this service, and the service sends
of various requests to the other services. But the problem still stands. To
block, or not to block?

AFAIK some big names like Amazon, Spotify, (Twitter?) have multiple public
services. You may sometimes notice individual components on their [web]
applications failing without bringing down the whole site, since the failures
are isolated to the different services. Neat, huh?

------
mml
Judging by the following statements: "we don't worry much about code quality",
and "lots of moving parts isn't without its complexity", I'm just going to
very politely wish him luck in his endeavors, and maybe make some popcorn.

------
Mc_Big_G
One of my previous employers had let each new incoming VP of engineering use
whatever he wanted to build out new parts of the app. With the high turnover
rate they had, the app turned into the worst Frankenapp piece of crap
imaginable. Significant portions of the app were built in php, java, python,
javascript, perl, ruby, ruby on rails, mysql, sybase, and a few more I'm
forgetting.

The result was that changing the title on the homepage took two weeks of
manual regression testing and it took a new developer a minimum of two weeks
to get a development instance up and running. I felt so sorry for the Ops
guys, I can't even imagine how terrible it was for them. There was no one, ops
or dev, that knew even 25% of how the whole system worked. It always felt like
the leaning tower of pisa made toothpicks and popsicle sticks held together
with bubble gum and boogers.

The cost to run that Rube Goldberg machine monstrosity ruined what could have
been an amazingly profitable business and great place to work. I don't
recommend it.

------
falcolas
> it was becoming incrementally larger and more complicated for new team
> members. Because of this, the monolithic Django system was already proving a
> headache to get quickly set up, running and understood by our new hires.

So, now you have thirty smaller services for each new hire to install and link
together? I can all but guarantee that this particular problem is better
solved with Vagrant and some orchestration scripts. Takes your install from a
list of packages to a single "vagrant up" command.

------
roberthahn
I've been a big fan of microservices for years. I believe that a good
microservice has a lot in common with traditional Unix commands. And I find it
funny that no one here has gone off on a tangent to talk about how horrible
those are.

Unix's tools and good microservices have the following things in common:

1\. They are small 2\. They are loosely coupled 3\. Shared-nothing state 4\.
They enforce standards for consuming and emitting data 5\. They enforce a
single standard for piping data from one process to another.

That said, I strongly believe that this won't work for every class of problem.
And while I may be putting words into Tom Watson's mouth here, I believe he
would say the same. But hey, guess what? This WOULD work for quite a few
problems!

Some of the sentiment in the comments here feel defensive and knee-jerk. And I
don't understand why. Tom Watson isn't telling you that you should use
microservices. He's only explaining why he's using it at Hubble. I'm pretty
sure how it turns out is going to have no impact on anyone else here, except
(obviously) his customers.

But I hope it succeeds. I think there's a lot we can learn by applying the
lessons Unix taught us to web-scale utilities.

~~~
lumpypua
_1\. They are small 2. They are loosely coupled 3. Shared-nothing state 4.
They enforce standards for consuming and emitting data 5. They enforce a
single standard for piping data from one process to another._

This can all be done via libraries with clear APIs, with none of the
deployment headache of a microservice.

------
speg
Doesn't Django already encourage this type of design? Each Django project is
composed of several installed 'apps'. Each app should be portable enough that
it could be the only app in any project.

~~~
collyw
I have a large app in Django at my work. The thing is it doesn't make sense to
split this into smaller apps (except for the shopping cart equivalent). It
tracks the workflow of our organization, so all queries need to be related
from the very start to the very end of the process. (Ok, not all queries, but
a lot of them).

~~~
kyllo
Yeah, you wouldn't really want to split a Django app across foreign key
relationships unless you want to be doing your joins on the client side with
REST API calls, which is more work than letting the ORM do it.

------
jexus2
> We don't worry too much about code quality as I believe that that is fixed
> at the recruitment level. Good people write good code and, more importantly,
> ship good products.

Good people make mistakes too. Having good tools to ensure code quality
doesn't make it impossible to ship bad products, but it's definitely
important.

~~~
tmwatson100
Not worrying too much doesn't mean not worrying at all! We still have tools in
place to ensure silly mistakes aren't made or are at least rectified quickly.
Just we don't expect 100% test coverage on every single feature or fix we
ship.

This approach will probably change as we get larger but being a small
development team means that it is fine for now.

~~~
jexus2
Thanks for the clarification! This sounds quite a bit more reasonable. :)

------
statictype
What about the headache of monitoring 10s of microservices, making sure they
stay up and running and healthy?

I assume that REST or Thrift (or something similar) is the way most
microservices talk to each other. What if the latency isn't acceptable?

I've thought about switching our architecture to microservices but these two
potential issues are making me hesitate.

How important are these issues in practice?

~~~
tmwatson100
It depends on what you're building. For us, we haven't noticed any issues with
latency just yet and there are plenty of steps you can take to optimise this
down the line.

As for monitoring there are plenty of tools to do that. You have bigger
problems if you expect your services to go down often.

~~~
statictype
_As for monitoring there are plenty of tools to do that. You have bigger
problems if you expect your services to go down often._

Yes, even in your monolithic app, you probably used some tooling to monitor
it. But now that single service being monitored has multiplied into 10-20
services being monitored.

Nobody expects their services to go down, but if you don't plan for it, then
that means you haven't thought about high availability or disaster recovery
enough.

------
mike_ivanov
It's amazing how easily people swap "I can't see how to use this in the
contexts I'm familiar with" to "it doesn't make any sense at all".

Being myself - just a few years ago - a long time strong proponent of tightly
coupled monoliths, I can't believe how blind and foolish I was in my
fundamentalist rejection of loosely coupled architectures.

Monoliths do just fine up to a certain level of structural complexity. Above
that, asynchronous service-like architectures are the only viable way to go.

Microservices is an attempt to see if the services patterns work below that
waterline, down to the function level. That's why I find microservices at
least interesting.

At any rate, it's a cost/benefit balance game, not an ideology.

Edit: clarity.

------
merb
We had similar issue's. Our Designers couldn't design with that big Django
app, so we split it up. Everything talks ReST to a cached AngularJS Client. We
still have Authentication in Django and two of our Microservices are running
with Django aswell. Some things were reimplemented in Golang, as we tried some
new things out.

What people here are talking is mostly dumb, since Microservices are better in
a lot of ways especially when it comes to multiple Developers. Currently my
company struggled a lot with Code that got changed by two guys at the same
time, which was hard to fix.

We had less unit tests than now, since they were too complicated at a certain
time.

------
fubarred
Standardization is, generally, good for maintenance, learning curve and
security, not always for retention (fun) though. Too much standardization is
like the flip-side of too many one-offs: an extreme that can lead to
unintended consequences. It's much easier to accrue technical debt and
organizational bifurcation by having too many languages, with a big, painful
shake out (technical debt repayment) into just a 2-3 (say Ruby|Python|Node &
Go|Scala|Haskell|Erlang|C)... (I've seen this happen at a number of shops go
through growing pains at scale).

Overall though, services should expose either a well/self-documented RESTful,
zmq or similar API such that they can be rewritten in (your favorite whatever
here). Being able to discover the API meta information automagically can be
helpful for autogenerating clients.

The other aspect is that services have a risk budget: if the service is
conventional and not as crucial, increase the experimentation. If the
business/process model is risky/uncertain, use common, stable technologies.

(Gist: Pick from the toolbox, carefully... apply common-sense... Don't get
stuck on perfect now or perfect waay later... Anticipate a little now to avoid
pain of changing later.)

PS: _Enterprise Rails_ is a great read. It's safe to ignore the Rails bits,
because the architecture bits are universal and excellent.

Update: RESTful API metadata formats [http://apiux.com/2013/04/09/rest-
metadata-formats/](http://apiux.com/2013/04/09/rest-metadata-formats/) (and
also
[https://en.wikipedia.org/wiki/HATEOAS](https://en.wikipedia.org/wiki/HATEOAS)
)

------
kyllo
Every time I see microservices, I think of Martin Fowler's First Law of
Distributed Object Design: "Don't distribute your objects."
[http://martinfowler.com/bliki/FirstLaw.html](http://martinfowler.com/bliki/FirstLaw.html)

Splitting your app into multiple apps that send each other HTTP requests or MQ
messages seems likely to just make your app slower.

~~~
angersock
If speed is your only requirement, write the thing in assembly.

~~~
kyllo
If speed is not a requirement, just write the thing in Django (or Rails)?

I'm just not compelled by the reasons the author gave for needing to switch
from Django to microservices.

------
crdoconnor
Can I be the first to say that this is a terrible idea?

>We have a multitude of parts that each could easily be self contained as
their own service. For example: Messaging, Search, Authentication, etc. These
all have different requirements and focuses and therefore lend themselves
better to different languages.

Yeah, not really. Python is not inherently less suitable for authentication
than, say, Java. Both languages are designed for general purpose programming.
The only real drawback to python is that it is relatively slow and this isn't
even likely to be an issue at all if you are using django (you'll most likely
bottleneck on I/O).

>Also, looking at our roadmap, we have lots of sprints and projects coming up
that could be implemented in their own silos. By splitting these into services
this allows us to quickly test and iterate without worrying about the rest of
our product.

And splitting them into separate libraries is not sufficient because?

>Whereas in a monolithic system these services are bundled together and can
create a spaghetti mess of workarounds and packages.

Newsflash: you're MORE likely to make a spaghetti mess of workarounds if you
use a microservices architecture because of the impedance mismatch between API
layers. You'll have to create a representation of, say, a user object on both
sides of the microservice divide and write code to serialize it and
deserialize it. Hello massive code repetition and obscure bugs.

>Having been a dev team of 1 for a long time I'd never experienced the
headache of installing all the dependencies required by our Django system. It
never really occurred to me that as I installed more packages, it was becoming
incrementally larger and more complicated for new team members.

If you think your headache is bad with django (which has consistent opinions -
e.g. on how templates should work - that makes it possible for modules to work
together), get ready for a nightmare when you start using a hodgepodge of
different systems written in different languages.

>We don't worry too much about code quality

This has officially moved this blog post from bad idea to outright parody.

>However, most of the cons are part and parcel with programming.

Here's a small laundry list of the exciting issues you'll now get to deal
with:

* Writing serializers and deserializers for your objects so they can be passed from one service to another. And debugging them!

* Babysitting all the new services you will need. You'll need to keep them up and running, monitor them, have a backup plan for all of them going down and lots of lovely extra error conditions.

* The headache of trying to figure out how to scale your network of microservices - probably meaning lots of time configuring load balancers.

And much, much more.

~~~
chhantyal
This is exact my thought as well.

If OP is more worried about maintenance of project, Django gives foundation to
build upon with sane conventions.

Anyone who reads few pages of Django Docs should be able to pick up quickly.

PS. I happen to currently maintain project written by other people with no
docs.

------
knicholes
Adding packages doesn't make it harder to onboard new developers. Tools like
virtualenv handle this for you.

------
jdoconnor
make sure to document well. When you're scaling up a team and moving from a
coupled code base to independent codebases a good amount of assumption
happens, especially around how mature each codebase is.

------
omegote
Fixed 500px width...

------
mreiland
> For example: Messaging, Search, Authentication, etc. These all have
> different requirements and focuses and therefore lend themselves better to
> different languages.

What?

Every single one of those is a solved problem that has bindings to most major
languages. Why in the world would you switch languages for an Auth mechanism
(for example)?

------
_pmf_
> Having been a dev team of 1 for a long time I'd never experienced the
> headache of installing all the dependencies required by our Django system.
> It never really occurred to me that as I installed more packages, it was
> becoming incrementally larger and more complicated for new team members.
> Because of this, the monolithic Django system was already proving a headache
> to get quickly set up, running and understood by our new hires. Splitting it
> up into more digestible chunks (services) therefore should allow us to scale
> the team a lot quicker this year and beyond.

Yet, the dynamic language community constantly laughs at Java developers and
their high overhead dependency management solutions, which just happen to have
these kinds of problems solved since more than a decade.

Pathetic.

~~~
iends
I don't believe he's talking about _code_ dependencies but rather things like
LDAP, mySQL, MongoDB, etc which Java dependency management tools don't solve
either. Managing code dependencies is solved in Python (as it is in most
dynamic languages) using pip and virtual environments.

~~~
lmm
IME, encouraging people to only run part of the system results in a more
fragile environment, and devs who can't fix problems in the wrong part of the
system or change boundaries that were put in the wrong place. Everyone should
be encouraged to be full-stack; specializing on one part or the other is ok,
but you should be able to chase things into other components where necessary.

From the talk of "packages" I think the original _was_ talking about code
dependencies. But it's worth saying that you can manage external dependencies
the same way, with something like puppet.

~~~
iends
I wasn't trying to advocate only running part of the system.

Of course you can use things like puppet, but the comment I was replying to
was saying these were solved problems in the Java world for more than a decade
which isn't exactly true.

