Hacker News new | past | comments | ask | show | jobs | submit login
Untangling microservices, or balancing complexity in distributed systems (vladikk.com)
156 points by ablekh 87 days ago | hide | past | favorite | 69 comments



Some other commenters have discussed it, but one thing to make very clear that I think a lot of people are missing:

1. I fervently believe, after having lots of experience with both microservices and monoliths, that microservices do not provide any technical advantage over monoliths. If anything, they can make a lot of technical details more difficult.

2. Microservices, however, can solve some particularly thorny organization problems for large teams with respect to how these large teams build complex software systems.

Thus, in my experience microservices can serve a useful purpose, but they will only be as successful as your organizational-wide communication is.

Furthermore, for the developers replying "I don't see why I'd ever use microservices for project xyz", well, if you're a team of 1, there is hardly ever a need to use microservices.


I wonder if there is a difference in terminology between the way I use microservices and the industry? I have worked on a number of systems, both monoliths and microservices, and when a service gets big enough (think Spotify or Uber) monoliths simply don't scale. I will agree that doing microservices before you have to is unnecessary complexity and what I would consider overengineering, however if the consensus is they provide no technical advantage I have to disagree, here are some that come to mind:

1) A monolith will have to consume a large amount of resources per request due to having all the models available at runtime. Lets say I have a system that handles 100 RPS but only 1 RPS uses that model, and that model is GB's of RAM and requires GPU, that means I will have to have this memory and GPU allocated for every request, making scaling much more costly. (Note: your mileage will vary depending on your complexity)

2) A monolith is a central point of failure, if I make a mistake in that service it can affect all of my requests, rather than just the request that calls out to that microservice, this is a difference between my entire service going down and just a feature.

These are just 2 I thought of off the top of my head, but I wonder why I seem to be out of step with HN consensus here.


There's a difference between a microservices architecture and putting in services where they make sense. We build mostly monoliths at work but we still split things into services where necessary. Necessary meaning the benefit is both large and obvious for the company.

Not all systems that use services is a microservice architecture.

#2 makes sense if you're taking advantage of that. In contrast, I've seen lots of microservices code that does little to no error handling. And if it's unicorn code that has error handling, they haven't really tested all the ramifications of their services going down in all the ways they can go down. Instead of having a system with a single point of failure, they have a system with N single points of failure. Their microservices are less fault tolerant than a monolith.

The backlash against microservices on HN is unsurprising. Over the years a ton of articles have been written vaguely about microservices and people cargo culted the architecture based upon these vague articles. Even reading comments people talk vaguely about "scaling" - what do people really mean? Before this backlash there had been non-ironic articles posted AND upvoted on HN about using microservices to scale to... a hundred requests per minute!

Lots of people seem to think that your architectural choices are either a single threaded rails app vs an eventually consistent distributed transaction CQRS nanoservice system that needs 99.999999% uptime. Meanwhile that eventually consistent distributed transaction CQRS nanoservice system that needs 99.999999% uptime is, for some reason, only running in one availability zone and falls down and corrupts your database as soon as any of your services fall over.

Oh, and it serves pictures of cats and it has 10 users and they're all family members. But someday that architecture will come in handy.


> There's a difference between a microservices architecture and putting in services where they make sense. We build mostly monoliths at work but we still split things into services where necessary.

It seems that your problem has more to do with semantics than with what microservices are.

Arguably, the definition of a microservices architecture is "not a monolith", in the sense that it's a distributed system not only with regard to support and infrastructure services but also with regards to business logic.


> 1) A monolith will have to consume a large amount of resources per request due to having all the models available at runtime. Lets say I have a system that handles 100 RPS but only 1 RPS uses that model, and that model is GB's of RAM and requires GPU, that means I will have to have this memory and GPU allocated for every request, making scaling much more costly. (Note: your mileage will vary depending on your complexity)

Needing a heavy model per-request is very much the exception rather than the rule; even if you're doing ML, for most cases you'd do that with a single shared copy of your model. If you're somehow loading a model for each request, surely you can just not do that for requests that don't need it?

> 2) A monolith is a central point of failure, if I make a mistake in that service it can affect all of my requests, rather than just the request that calls out to that microservice, this is a difference between my entire service going down and just a feature.

I can understand people doing that in languages like Python or especially Ruby, where metaclasses / monkeypatching mean that it's easy for one part of the code to interfere with another. But I think it's a mistake to apply that to more structured languages where it's relatively easy to have an enforced separation between different modules within a single deployable.


Counter points, based on experience:

1. Yes, if you have an application that works this way you will have to manage your deployment to compensate. But you can use configuration as feature flags to keep a monolith but when you deploy it to X servers you enable one feature set and to Y servers you enable another. So you get the benefits of developing in a monolith but the isolation of microservices.

2. See #1, but also use proper error handling. It's been a while since I've seen a single request take down the whole app.


I wonder if this is why my issue is with the terminology, what you have described in 1 to me is a microservice architecture. I currently have a monorepo deploying microservices in order to get the benefits of developing in a monolith but with the isolation of microservices, but I do consider it a microservice architecture on the whole.


There are definitely ways each approach can approximate the benefits of the other:

- Microservices can be deployed as processes on the same machine and communicate over the loopback interface or domain sockets without much network-related risk. This is still more complicated than function calls, and I wouldn't quite call it a monolith.

- Monoliths can have traffic for different endpoints partitioned over different deployment groups. So the behavior of one endpoint can be changed independent of the others. But a module deep in the stack, potentially consumed by many endpoints, cannot be deployed on its own in a way that affects all its consumers. So I wouldn't quite call this microservices.


Nah, it can be one service just deployed with different configuration. Or as another commenter said, a different binary based on compilation flags/config.


You can also compile different binaries from a single monorepo


Good solution too!


Monolith doesn't mean you couldn't have fault tolerance, e.g. if the monolith is distributed. Monolith is just a term that means "all encompassing", or that "you've thought of everything". In contrast, a microservice is basically "I can't think of everything so I break up my system into smaller, easier to understand pieces, and hope for the best".


YMMV and also depending on what counts as a microservice but I currently write and maintain (as part of a very small team) several services that run on 200+ (physical, virtual and cloud) devices around the world and a monolith simply wouldn’t be able to do the job for boringly real technical reasons.

Another team writes and maintains a few more.

We also have some monoliths that provide customer facing UIs and aggregate data for our support teams and they are perfectly fine for that.

Right hammer for the right nail and all that.


You can have a distributed monolith. Monolith doesn’t mean there’s a single deployment. It’s about the interaction between domains. So if you have an app with a database on each device and the app layer is 1 application, one deploy, that’s still a monolith.


> 1. I fervently believe, after having lots of experience with both microservices and monoliths, that microservices do not provide any technical advantage over monoliths. If anything, they can make a lot of technical details more difficult.

Sure, monoliths are certainly easier to work with, and I think they should be the default starting point for most companies.

But regarding technical advantages, what if different parts of the program have different performance profiles and scale differently?


If a single code base has multiple non-isolated deployables designed to be published simultaneously, and/or having a shared data store, is it a collection of microservices or a monolith? I've taken to calling it a scalable monolith.


> that microservices do not provide any technical advantage over monoliths

Sometimes there is a real benefit of having a service that can fail without impacting everything else. That is a technical upside, but it doesn't apply in most cases. But it do apply in some cases where you just want to limit any problems from impacting too many part of the platform at once. The service could of course still be both micro or not.


if you dont see the technical advantages you simply arent at the scale. Its that simple. Speaking as someone who works at a company with one of the largest monoliths in the world, microservices become necessary lest you deal with incredibly difficult problems at massive scale.


Can you expand on this for us who do not work at large companies with massive scale services?


You ship your org structure



Hah, I have seen this before but this was the first time I actually notice the PRODUCT SPEC at the beginning:

GOAL: Surprise and delight users by displaying their birthday on the setting page.

DELIVERABLES: Display User Birthday on Settings Page

"Surprise and delight" has definitely become one of my new bullshit bingo favorites.


I knew this would be in the comments before even opening the page. This thing has been getting shared a ton lately since it was just uploaded 2 weeks ago. I think it kind of goes to show how many people empathize with the video.


Great satire! haha

There should be one of these videos for reactive programming as well. Misuse of observable pattern can quite quickly turn an app into a mess, making code navigation / minor change requests a nightmare.


Great video, also in a couple of clicks I got to this, which is equally hilarious: https://www.youtube.com/watch?v=5WL_jkFS2gw


Hilarious video (and most likely very true). Thank you for sharing.


Do you think he ever found love?


I was referring to technology aspect only. Regarding love, you'd have to ask him. :-)


This made my day, thank you.


I find the discussion of monolith vs microservices to be very unhelpful. It is a discussion about how to split the code.

But what you really want to understand, when building a distributed system (i.e. system to process large amounts of data), is how to partition the data, so it could be processed in parallel.

Essentially, there are two options, vectorization (single instruction multiple data) and pipelining (multiple instruction single data). They both have different uses in different scenarios, based on the critical path dependencies in the data processing.

Monoliths are easier to vectorize, microservices are easier to pipeline. But to choose which one you need before you understand the nature of data processing you need is a wrong way to do it.


The "vectorizing" and "pipelining" here seem to work when describing changes/deployments made to the system, but that seems orthogonal to the data processed by the system.

If one part of your workload is suited to pipelining, and another part of your workload is suited to vectorizing, then that might be a reason to split the workload into different processes running on different clusters. Few but beefy nodes for the vectorized part. Many smaller nodes for the pipelined part.


> The "vectorizing" and "pipelining" here seem to work when describing changes/deployments made to the system

That's not what I mean.

But you're correct, what you want to do with the data informs your decision of what should be separate processes, and you once you know, you might decide to split (modularize) the processing code accordingly.

Doing that other way around (i.e. to design the modules before you understand the data flow) is just going to cause more trouble.


Monolithic architecture implies the company is only willing to manage one production deployable. Someone solving a specific problem cannot introduce new processes / network boundaries even if warranted based on the characteristics of their problem.


Uber is not reorganizing their microservices into "macroservices". The original tweeter refuted it, but it's still being propagated as fact. Granted the original tweet was poorly written but in the thread itself, he refutes it later on.


Thank you. This is the core of it[1], IMO:

> Back in the day, we’d spin up a microservice that did one, small thing just like that. We had a bunch of small services built and maintained by one person. This was great for autonomy, iteration speed, learning and making devops a no-brainer. You could spin up a service anytime: but you’d be oncall for it.

> Now, as my area is maturing and it’s easier to look ahead, as we create new platforms, we’re doing far more thoughtful planning on new services. These services don’t just do one thing: they serve one business function. They are built and maintained by a team (5-10 engineers). They are more resilient and get far more investment development and maintenance-wise than some of those early microservceis.

[1] https://lobste.rs/s/mc3k1c/at_uber_we_re_moving_many_our


> We had a bunch of small services built and maintained by one person.

That just seems like it'd be a disaster at any non-trivial scale.


I've never bought into this idea of microservices. Always wrote high performance native servers and have yet to encounter the situation when well designed monolith running on multicore monster with gobbles of RAM failed to satisfy client. Granted it will not work for Google but what does the world care. Most of the businesses (read potential clients) will never come anywhere close to those scales.

Long time ago when computers were slow I did design some distributed applications myself - reliable IP multicast server, business process servers, map reduce type solutions like bill generation etc where it did make sense. That was long time ago and now most of those at their old scale could run on a single reasonable priced server.


Microservices can be useful in medium/large companies where you have many teams of devs and it becomes cumbersome to have them all work on one monolith. As long as there's good communication and understanding of needs between teams, each team can develop their services how they want, in whatever language/platform, with their own review and QA and deploy processes.

For smaller teams or small projects it isn't really super useful. Also, my personal experience is that when you start making microservices, the monolith doesn't go away. It also isn't so cumbersome if each team is in charge of one or a few services and you only split things if there's a clear benefit.

I'm probably saying things you already know. My point is that I think for multiple cooperating teams, monoliths and microservices are complementary.


I think a lot of this also becomes a question of where you choose accomplish the same root goal: if your team can maintain sound boundaries in either a large codebase or a bunch of microservices, test & tune well, you’re likely to have good results either way. A lot of people tend to see something working or failing and attribute it to the technology rather than the social dynamics behind it.


My current set of experiences say:

Microservices (where it makes sense) + Macrorepos.

Microservices are really handy when you're at scale and/or the problem you're solving works well with the paradigm. I'm thinking back to a gig I had where each of our vendor integration points (sending the same data, just formatted for that vendor) was a good use case even for a smaller company. We used Scheduled tasks to launch the application that did what was basically an ETL.

Oh, that vendor is down? Great, just have the sysadmin stop the thing.

No database muckery required. No need to build a fancy front end. Easy to delegate to/train on helpdesk type folks.

But any microservice is part of a larger ecosystem. It's critical to have a workflow that doesn't make refactoring the domain a huge chore/lift. IOW: Keep the related microservices in a single repo.

Really, unless you've got a use case like the above (processes that need granular control) or HA/Scaling concerns (since in that case you'll be able to better focus on the parts that need to scale), A Monolith or domain based services are the way to try to go.


I myself directed teams of developer up to 35. Do not know beyond that scale. Teams simply worked on modules and main designers made sure they complied to a shared interfaces / state machines in more complex cases. Not a big deal. The law was: I trust developer so here is your module interface and constrains. Now go away and come back with implementation (which would still compile to a single monolith). As I was lucky to work with very good programmers bar couple of bad apples I've never had any real problems with the delivery and its quality


Its clear that you could afford to not use microservices (which is great! You didnt burden yourself with the cool approach since it wasnt necessary). Not everyone has those requirements though and may NEED them


I completely fail to see what is so "cool" about increasing system complexity beyond of what is actually enough for most of the real life cases. To me: reliably working product is what is "cool" and that is what I've always pursued.


You don't need to "buy into" an idea to realize that there are advantages and disadvantages to a given approach.

Take one example: The single server idea is going to suffer outages. You could at the very least have two monoliths either load balanced or in some kind of failover setup.

But, what distributing some of the components into so-called microservices would give you is a graceful failure scenario where even if all the instances of a specific component fail, the rest of the application still works. Maybe only one function stops working temporarily but most work continues on.

This doesn't mean everything must be microservice koolaid all the time all the way, but I'm sure you can conceive of that being an advantage under certain use cases that would be worth other possible trade-offs.


I have done extensive work with monoliths, and extensive work with microservices. I quite literally have never seen microservices be a benefit in the way you are describing, and if anything microservices can make the problem of robustness more difficult, because in many workflows there are one (or many) services that are critical. If they're down, everything is down, and if anything distributing these dependencies across different systems can make it more difficult to determine where the critical points of failure are.

As you point out, there is nothing inherent in a monolith that makes it less robust. I currently am building a monolith that is deployed on auto-scaling servers on a cloud provider, so if one server dies it just spins up other instances.


I have also done extensive work on both (I'm much older than the word microservices) and have found advantages to both approaches depending on requirements.

I've also found that most so-called monoliths still often end up breaking apart components anyway. I've only seen true monoliths as described above in the mom and pop type IT shops where there's only a couple support staff and they don't have the resources or experience to properly break things out into separate resilient components. And yes supportability is definitely a valid part of that design call.

Again that was just one example of a potential advantage to it while admitting there are trade offs. Others here have given better examples such as at larger orgs where multiple independent teams get involved with a single app.


This is what I do, I just develop monolithic stateless API's in Docker containers and run them as serverless Cloud Run apps or as Kubernetes pods, if one of them fails it's no big deal.

For particular usecases that fall out of the wheelhouse of my preferred language, I'll typically use single OpenFaaS functions + endpoints for these things (think doing ML with Python, or really heavy computation with Go/Rust).

This combination of containerized Mostly-Monolith with single-purpose endpoints for particular usecases works really well and is very productive. If something is much easier in another language (due to tooling/ecosystem) for example, then I can just build that method or sub-system out in whatever language it's fastest in with this architecture.


> But, what distributing some of the components into so-called microservices would give you is a graceful failure scenario where even if all the instances of a specific component fail, the rest of the application still works. Maybe only one function stops working temporarily but most work continues on.

The important thing to note here is that microservices give you one way to build a graceful failure scenario. You don't get it for free. Often that 500 will propagate all the way back! So you have all the same failure modes + network failure modes!

You could also build a monolith that could handle component level failures, for that matter. Tooling doesn't seem to be as common, though.


I don't disagree with you at all. There are certainly different ways of thoughtfully building things.

The point being you don't need to be a paying subscriber to microservices(tm) to recognize there can be positives to the approach, which you describe in your last paragraph

A monolith that can handle component level failures is very close to the same idea infrastructure-wise minus a lot of baggage implied by the buzzword microservices.


"A monolith that can handle component level failures" - being able to handle component level failures I think is more related to business logic design rather than being a monolith/microservice implementation. If your business process depends on some login microservice and that one is down it does not matter that the other parts are functional.


So just to play to your hypothetical. If there were two login methods, and they were separate microservices, if only one failed and the other remained up, that would be the graceful failure scenario.

I'm not telling you that you must to do this. Others may have requirements that go beyond yours.


"running on multicore monster"

Just curious, but how do you take advantage of the multiple cores? Are you using a language that handles concurrency directly, with something like threads? If you use a language that has historically been single threaded (Ruby, PHP, partially Python) do you use multiple processes? If you use multiple processes, what manages the processes? What can spin up additional processes? And when you talk about a "multicore monster" what are you actually running there, the application or the web server or both? If the web server runs elsewhere, how does it connect to the app? Do you have something else to load balance the web server itself?


I write multi-threaded servers in languages like C++/C/FreePascal/Java. As how to properly write multi threaded applications that take advantage of multiple cores? Well I think it is beyond the scope of this discussion. I have no desire to write lengthy articles. It takes experience anyways and ff you do not already know how to do it putting few words in newsgroup is not going to help.


You can have a monolith distributed across many machines, all your microservices running on the same machine, cluster per microservice, all your microservices running in a shared cluster, etc. Those axes are only tangentially related.


Do your clients not have availability needs? Even if one machine can handle the traffic levels, what if there are hardware failures? How do you do patching?


I've always had failover functionality built in. As for patching - it very much depends on the nature of the patch, type of application and business constraints. Too much to write here. For the most part it never caused me any particular grief. The most generic case for server type is: all new requests are denied with appropriate error code, all requests in process are either gracefully completed or aborted if business logic allows for that. Meanwhile the background process is migrating/upgrading in place data model if there are any changes. Upon finishing the upgrade the new server takes over.

I do not thing upgrading microservices is any simpler. It could actually be way way worse.


Nothing stops you from running multiple instances of a monolith on different servers.


Right, but the person I was commenting was not really talking about microservice vs monolith as much as "I only need one server"


If you dont have a team large enough or a business big enough to need multiple individual services than you aren't the target market. Unless you care about high availability (which small services can help you achieve) then a single server is fine.


Everyone wants to believe they are the target market. That’s the problem.

What you should be doing is telling people what they need to hear and let the biggest layers figure out for themselves that they don’t work that way.

But we want to know what the big people are doing as if that will let us become big too.

But the bulk of what we do is reacting to situations, not creating them.


May I ask what high availability has to do with the microservices? Since many of my servers were designed for relatively large companies (like TELCO) high availability/reliability requirements were always there. Running 2 or more monoliths in parallel with the appropriate logic built in does not make them into micro services


if they are large enough and have components that need to scale independently than a bottleneck in one of those components may affect availability of the service.


Or may not. I design and create products that serve particular problem for customers while satisfying reasonable constraints. If it comes to a situation you've described than it will be spun into a separate component. I already told that I did and know how to do distributed applications. The keywords here are "WHEN NEEDED". And on modern hardware it is not in my cases.


also running two very large pieces of complex software in parallel is not always easy without breaking out the things that cant always be run in parallel :)


Whatever problems you MIGHT have with the monolith - the same problems will hunt you in your microservice implementation. You'll just have a lot more microservice related problems as an extra toppings. Also it sounds like you're bringing straw arguments without having practical experience


> You cannot build a system out of independent components!

I disagree, the Unix philosophy proves otherwise. You can build very powerful applications, just by combining grep, sed, ls, ... and the like.


Opinions are welcome on whether this kind of approach could help move forward from the microservice/monolith debate:

- Teams commit to individual repositories so that team membership, changeset review, and issue management are kept close to the relevant code, and checkout sizes are minimal

- Strong language support for inter-repository dependency management is introduced to enable effective rollback and rollforward of commits that affect other repositories

- The function call becomes the unit of deployment -- not a microservice, and not a monolith. This allows for low bandwidth deployments and for small amounts of highly-utilized code to migrate to the devices where they are needed for computation

- Function calls and return values are signed cryptographically so that inter-function calls have integrity, regardless whether within a single device or across the network

- Functions that perform I/O calls are tagged with the virtual storage and/or network interfaces that they require access to

- Kubernetes-like infrastructure manages the deployment, scaling, and resource allocation of functions to devices

This seems to me like it would provide the organizational benefits of microservices while reducing developer hand-wringing over where to define service boundaries. It just becomes a question of 'should this be a function?'.

Good tooling around binary-dependency-or-source-code retrieval could make it easy for developers to work on large applications built like this and inspect / debug / modify code as-needed without having to check out or download all the source code and binaries at once.


This oversimplifies the measure of complexity by focusing on the difference between distributed and monolithic systems. A potentially interesting example I recently encountered was a monolithic system that used internal interfaces. Transitioning these internal interfaces from a synchronized design to an unsynchronized design effectively introduced much of the complexity of a distributed system into monolithic design.

Ultimately everything about data flow, typing, abstraction, and so on has some effect on performance, encapsulation, complexity, organization. In a way all aspects of development from onboarding to builds and tests to release distribution need to be measured the same way we take into account of resource usage and basic performance criteria. If a piece of software masterfully meets all needs, but has internal interfaces that are simply too complex and brittle to be worth learning then that software will quickly decay and need to be replaced as the dynamic world alters the constraints of usage.


The backlog example in the article is crazy! Do they really do services like that in the big world? Backlog service which encapsulates all the services I can follow (partly) but all these separate ones




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: