

How we build microservices - klaaspieter
https://blog.yourkarma.com/building-microservices-at-karma

======
al2o3cr
"By just going ahead and building the app, we became familiar with the problem
we were trying to solve, and the more familiar we were with the problem, the
more obvious it was where we needed boundaries between aspects of the app.
Every time we encountered something that clearly looked like it should be a
separate piece, we turned it into a service."

THIS. This should be printed in 36-point font on every microservice article.

------
Nelkins
"The biggest challenge with microservices is testing. With a regular web
application, an end-to-end test is easy: just click somewhere on the website,
and see what changes in the database."

I find this surprising. I would've thought having a more modular system would
lend itself to easier testing. Have others had this same experience?

~~~
Arie
Every little piece is easy/easier to test, but testing end-to-end over
multiple micro-apps is hard in my experience.

~~~
vertex-four
Why is this? Is it simply a deployment issue - that it's hard to get all the
pieces running together in a test environment?

~~~
xetorthio
In my experience even getting all the pieces installed in a testing
environment is hard. Specially when you need to make sure that they are all
wired correctly, with the correct version, with all their dependencies (maybe
different kind of databases, queues, caching layers) and on top of that if
services are written in different languages, that adds another level of
complexity. Eventually when the system grows big enough and the team of people
that needs a stable test environment grows big enough, it just becomes too
expensive.

~~~
AdrianRossouw
this is where docker and the ecosystem around it starts filling in the pieces.

also, it makes failures hurt less, and rollbacks much easier.

~~~
edwinnathaniel
Docker answers half of the issues: deployment.

The other halves: distributed logging (how do you trace a 'transaction' that
is being executed against multimachines/multiplaths?)

What about rolling releases? what about versioning?

What about service discovery (still immature field, look at how many
products/tools out there trying to be THE service discovery choice).

~~~
sborsje
We actually use syslog-ng + Loggly for distributed Loggly. We partially solve
the transaction problem by slapping a request ID on every action that's
initiated by a web request. We then carry this request ID from service to
service, which gives us the ability to trace exceptions cross-service. It's
obviously not the perfect solution, but it has helped us many times when
debugging customer-facing 500 exceptions.

We solved the service discovery problem by simply using DNS. Every service
runs behind an (internal) Elastic Load Balancer, and we let the load balancer
configuration figure out which instance is up and which one is down. Again,
not the perfect solution, but it works great for now and is very easy to
setup/maintain.

------
Smudge
> We have an in-house tool we use called Fare which reads this configuration
> and sets up the appropriate SQS and SNS queues.

Would love to see this released publicly. As stated in the post, there aren't
a ton of (public) examples of microservice architectures, so everyone seems to
be solving the same problems independently. It would be great if we could
start pooling more of our resources together.

~~~
iain_hecker
We will! There are however a couple of really nasty issues that we don't feel
comfortable to release to the public yet.

------
cwt137
I've been thinking a lot about SOA. Articles like this and Living Social's SOA
blog posts series are a big help. But, I feel something is still missing. For
example, no one talks about security of the various services. Are there any
good books someone can recommend on SOA?

~~~
iain_hecker
We have all services running inside a VPN (see one of our older posts:
[https://blog.yourkarma.com/building-private-clouds-with-
amaz...](https://blog.yourkarma.com/building-private-clouds-with-amazon-vpc))
and we also use HTTP Basic token auth that are configured upon deployment.
Every app gets its own token, so we can trace which app does what.

~~~
cwt137
I'd figured the services were on a private network (or at least the app is
listening on an interface on a private network), but it is good someone
confirmed it. Thanks for your insight on security.

------
pdx

        The shipment app listens to the messaging system, sees an 
        order take place, looks at the details, and says, "Okay, 
        I need to send two boxes to this person." Any other 
        services interested in an order happening can do 
        whatever they need to with the event in their own 
        queues, and the store API doesn't need to worry about 
        it.
    

I'm curious how you guarantee that all the systems that need to see an event,
will actually see it, before it is removed from the event queue. I assume the
shipment app, in your example, is responsible for removing the event from the
queue. So, what if it removes it before the mailer app or the "make cash
register sound" app, sees the event?

~~~
iain_hecker
Each app gets their own queue. So there is a queue for the shipment app and
for the mailer app. When an event is published to SNS, it gets added to both
queues.

~~~
pdx
I see. So you never add a listener, without also modifying the emitter to send
to an additional queue.

~~~
futurepaul
We don't change the emitter, the listener changes the SNS config. The emitter
publishes to SQS under a certain topic. Listeners that interested in that
topic subscribe to it by changing the SNS config, basically saying "give me a
copy of those events too and put it onto my own queue". SNS copies the
messages to every queue.

------
tiglionabbit
The text makes it seem like they're using SQS for many things, but the diagram
only shows it between Orders and Shipments. I guess everything else
communicates through the database and HTTP requests?

Using SQS means you're either polling or long-polling with hangups every 20
seconds. This seems pretty shitty to me. Also how do you structure one of
these services around polling or long-polling to get its info? It sounds like
they're using Rails. Does Rails have something that makes this easy to do?

At first, reading this reminds me of the Blackboard pattern from The Pragmatic
Programmer. This pattern seems like a neat way to separate an application into
different agents.

~~~
iain_hecker
We use SQS for basically everything that happens asynchronously to the main
flow. So when a user signs up there are a bunch of things that need to happen
straight away (create an account, give 100MB to every new user), these go via
HTTP. The others (email, give the owner of a Karma an additional 100MB and a
push notification) goes through SNS and SQS.

All these async things don't impact the main flow of the user, so it's no
problem that it isn't instantaneous. We are running background processes that
do this.

We only use Rails for our frontend applications, they don't contain any
business logic. Our backend processes are usually Sinatra for HTTP requests,
or custom daemons.

------
jessaustin
What's the advantage for your situation of publishing to SNS which is in turn
publishing to SQS vs. just publishing directly to SQS? Are you forking to
multiple SQS queues?

~~~
iain_hecker
Yup we are. There might be multiple applications interested in the event and
they each get their own queue.

~~~
marceldegraaf
Have you looked into a "real" messaging queue for this, like RabbitMQ or
another AMQP based system for this? Or are you bound to SNS/SQS for a
particular reason?

~~~
amstrad464
"Real messaging" aside, this is an interesting question. It'd be great to hear
from iain_hecker if they considered using RabbitMQ (or similar) and if yes,
why SQS/SNS were chosen.

Also, I'd be interested to read more about how you handle authentication
across web/mobile. Thank you for the blog post and for taking the time to
answer questions here. :-)

~~~
futurepaul
We have considered stuff like RabbitMQ. They would be excellent solutions too
and we are working on a part of our architecture with MQTT/protobuf (more on
that later, I'm sure). We went with SNS/SQS because we don't have to build a
cluster with that. Building clusters is hard and Amazon takes care of that.
Since we were already using a bunch of AWS products, this was a nice fit and
we're quite happy with it.

Authentication is a good idea for an article too. Thanks :)

------
MattGrommes
Is there a difference between microservices and SOA? I've used architectures
just like this in the past but never heard of "microservices" until recently.

~~~
dragonwriter
AFAICT, "microservices" is exactly SOA in its original sense; my guess is the
rename is because SOA has become so attached to particular XML-based
implementations and standards (e.g., the WS-* series of standards).

~~~
iain_hecker
This. Also SOA means STD in Dutch, so we don't want to use it that often ;)

------
phazmatis
Every time I cludge new functionality into an existing repo, I wish
microservices were more generally acceptable.

