I don't think I blame the author at all. I'm not sure why you would start with microservices, unless you wanted to show that you could build a microservices application. Monoliths are quicker and easier to setup when you're talking about a small service in the first place.
It's when an organization grows and the software grows and the monolith starts to get unwieldy that it makes sense to go to microservices. It's then that the advantage of microservices both at the engineering and organizational level really helps.
A team of three engineers orchestrating 25 microservices sounds insane to me. A team of of thirty turning one monolith into 10 microservices and splitting into 10 teams of three, each responsible for maintaining one service, is the scenario you want for microservices.
We’ve done exactly this - turned a team of 15 engineers from managing one giant monolith to two teams managing about 10 or so microservices (docker + kubernetes, OpenAPI + light4j framework).
Even though we are in the early stages of redesign, I’m already seeing some drawbacks and challenges that just didn’t exist before:
- Performance. Each of the services talks to the other service via well-defined JSON interface (OpenAPI/Swagger yaml definitions). This sounds good in theory, but parsing JSON and then serializing it N times has a real performance cost. In a giant “monolith” (in the Java world) EJB talked to each other, which despite being java-only (in practice), was relatively fast, and could work across web app containers. In hindsight, it was probably a bad decision to JSON-ize all the things (maybe another protocol?)
- Management of 10-ish repositories and build jobs. We have Jenkins for our semi-automatic CI. We also have our microservices in a hierarchy, all depending on a common parent microservice. So naturally, branching, building and testing across all these different microservices is difficult. Imagine having to roll back a commit, then having to find the equivalent commit in the two other parent services, then rolling back the horizontal services to the equivalent commit, some with different commit hooks tied to different JIRA boards. Not fun.
- Authentication/Authorization also becomes challenging since every microservice needs to be auth-aware.
As I said we are still early in this, so it is hard to say if we reduced our footprint/increased productivity in a measurable way, but at least I can identify the pitfalls at this point.
The first thing I start trying to convince people to ditch on any internal service is JSON.
There are only two times when JSON is a particularly good choice: when it's important for the messages themselves to be human-readable, or when it's important to be able to consume it from JavaScript without using a library. Any other time, something like protocol buffers is going to give you lower latency, lower bandwidth requirements, lower CPU costs, lower development effort, less need for maintaining documentation, and better standardization.
If you ditch the HTTP stuff while you're at it, you can also handily circumvent all the ambiguities and inter-developer holy wars that are all but inherent to the process of taking your service's semantics, whatever they are, and trying to shoehorn them into a 30-year-old protocol that was really only meant to be used for transferring hypermedia documents. Instead you get to design your own protocol that meets your own needs. Which, if you're already building on top of something like protobuf, will probably end up being a much simpler and easier-to-use protocol than HTTP.
Not to mention, JSON APIs start out simple because you just call `to_json` on whatever object you need to share and then move on. Except nobody’s ever documented the format, and handling ends up taking place in a million different places because you can just pass the relevant bits around. The API is “whatever it did before we split it up”.
Now when someone goes to replace one side, it’s often impossible to even figure out a full definition of the structure of the data, much less the semantics. You watch a handful of data come across the pipe, build a replacement that handles everything you’ve seen, and then spend the next few months playing whack-a-mole fixing bugs when data doesn’t conform to your structural or semantic expectations.
JSON lets you get away with never really specifying your APIs, and your APIs often devolve to garbage by default. Then it becomes wildly difficult to ever replace any of the parts. JSON for internal service APIs is unmitigated evil.
You could just make a shared object module that is versioned and have all the microservices use those objects to communicate. Or you could implement avro schemas. Many ways around this issue.
I don't know if I would recommend NOT using json when starting out (I'm on the side of pushing for a monolith for as long as possible), but yes omg I've been there.
I've migrated multiple services out of our monolith into our micro service architecture and oh boy, it is just impossible to know (or find someone who knows) what structure is passed around, or what key is actually being used or not.
Good luck logging everything and pulling your hair documenting everything from the bottom up.
I think the point the OP was making was more about the tooling than the format. Having a library that forces you to define the format you're using, instead of dumping to_json whatever internal representation you had, also doubles as documentation.
But then, of course, that can be considered boilerplate code (and, in the beginning and most of the time, it actually is just a duplication of your internal object structure).
That, and having the format definition file support comments means that you have a convenient all-in-one place where, at lest in the more straightforward cases, you can handily describe both the message format and the service's behavior in a single file that's easy to share among all your teams.
The easiest path with JSON is to do none of this, and so the majority of teams (particularly inexperienced ones) do none of it. With protos, someone must at least sit down and authoritatively outline the structure of any data being passed around, so at a minimum you’ll have that.
But even just forcing developers to do this generally means they start thinking about the API, and have a much higher chance of documenting the semantics and cleaning up parts that might otherwise have been unclear, confusing, or overly complex.
Right, but I'm asking if FlatBuffers' approach to encoding is similar to Cap'n Proto. It seems like the answer is "yes", but I might be missing something.
1) I'm the author of Cap'n Proto; I'm probably biased.
2) A lot could have changed since 2014. (Though, obviously, serialization formats need to be backwards-compatible which limits the amount they can change...)
Cool, thanks for this! I'm happily using Cap'n Proto in a side project, and so far have really enjoyed working with it. It's a really impressive piece of engineering.
(personal opinion, not speaking on behalf of my employer)
Start out with protobufs, so you can take advantage of gRPC[1] and all of its libraries and all the other tooling that is out there.
If you profile and determine that serialization/deserialization is actually a bottleneck in your system in a way that is product-relevant or has non-trivial resource costs, then you can look at migrating to FlatBuffers, which can still use gRPC[2].
> So naturally, branching, building and testing across all these different microservices is difficult. Imagine having to roll back a commit, then having to find the equivalent commit in the two other parent services, then rolling back the horizontal services to the equivalent commit
that should not happen.
if it does you don't have a microservice architecture, you have a spaghetti service architecture.
How does one know if X is the wrong solution or if X is the right solution but the shop is "doing X wrong"? This also applies to monoliths: maybe they can scale, but one is doing them wrong. Changing from doing monoliths wrong to doing microservices wrong is obviously not progress.
The same issue appeared when OOP was fairly new: people started using it heavily and ended up making messes. They were then told that they were "doing it wrong". OOP was initially sold as magic reusable Lego blocks that automatically create nice modularity. There was even an OOP magazine cover showing just that: a coder using magic Legos that farted kaleidoscopic glitter. Microservices is making similar promises.
It took a while to learn where and how to use OOP and also when not to: it sucks at some things.
Uh, my post was snark towards engineers' general tendency to champion and defend their own favourite architecture over competing approaches rather than focusing on the most suitable architecture for the circumstance, in response to this question:
> How does one know if X is the wrong solution or if X is the right solution but the shop is "doing X wrong"?
Sorry, I just misread your post as being more of an attack. As I had said, I dislike the constant "hype means something is bad" posts that I've been seeing for years - I think it's really unfortunate.
No worries, I can see how it might have been read that way in the wider context of the conversation. And I totally hear you on the "hype equals bad" thing. If something's popular, it's popular for a reason. That reason MIGHT just be trendoids jumping on a hype train, but it might also be because the thing is good.
I have posted about this before. OP is describing nano-services, not micro-services. Nano-service is micro-service that provides leftpad via JSON api.
You have an auth.yourapp.com and api.yourapp.com and maybe tracer.yourapp.com and those three things are not a single app that behaves like auth, api or tracer depending on setting of a NODE_ENV variable? If so, you have micro services.
> that should not happen. if it does you don't have a microservice architecture, you have a spaghetti service architecture.
A "service" is not defined principally by a code repository or communications-channel boundary (though it should have the latter and may have the former), but by a coupling boundary.
OTOH, maintaining a coupling boundary can have non-obvious pitfalls; e.g., supported message format versions can become a source of coupling--if you roll back a service to the version that can't send message format v3, but a consumer of the messages requires v3 and no longer supports consuming v2, then you have a problem.
you should never actually remove an api without a deprecation process.
basically a message format is part of your api.
actually in a microservice world your interfaces/api should be as stable as possible.
if something gets changed, it needs to go trough a deprecation, that means not removing it until a certain amount of time.
private apis exist for a reason. going through the deprecation cycle repeatedly sounds like a waste of time if you control all sides of the system. which is to say, not every interface needs to be a service boundary. you do need to keep versioned apis, but there is a cost to doing so.
IMHO, part of a successful microservices program is treating your "private" APIs like your public ones, with SLAs, deprecation protocols, etc. Otherwise, you end up with tight coupling to the point that you should just go with a monolith.
When you are still figuring out the problem space and message formats are being created removed extended and reverted several times a sprint, and you have already gone live with third parties depending on stable interfaces so that you cannot make the changes you need in a timely fashion, the warm arms of a purpose built monolith look particularly attractive.
Indeed. At that early stage of development, a monolith (or at least monolith-style practices, such as tight coupling between API providers and consumers) is definitely simpler and more efficient. But it wouldn't hurt to take steps from the beginning to make it easier to break the monolith apart when/if it becomes pragmatic to do so.
> private apis exist for a reason. going through the deprecation cycle repeatedly sounds like a waste of time if you control all sides of the system. which is to say, not every interface needs to be a service boundary.
The whole point of a microservice is to create a service boundary. If you have a private interface where both sides are maintained by the same team, both sides should be in the same service.
All interfaces are service boundaries. The only difference is whether you controll all clients and servers or not. Either way, API versioning is a must and trivial to implement, and there is really no excuse to avoid it. All it takes is a single integration problem caused by versioning problemd to waste more time than it takes to implememt API versioning.
The accept header is a mechanism for communicating formats you support; it doesn't do anything to address the problem of managing change to supported versions, which is a dev process issue, not a technical issue.
> The accept header is a mechanism for communicating formats you support; it doesn't do anything to address the problem of managing change to supported versions
That assertion is not true. Media type API versioning is a well established technique.
It sounds like you're paraphrasing the great Steven Jobs: "You're holding it wrong"
Be that as it may, I believe mirkules's issue is not an uncommon one. Perhaps saying "building a microservice architecture 'the right way' is a complex and subtle challenge" would capture a bit of what both of you are saying.
Something being complex and therefor easy to mess up does not mean it's a great system and the users are dumb, especially if there are other (less complicated, less easy to mess up) ways to complete the task.
> Perhaps saying "building a microservice architecture 'the right way' is a complex and subtle challenge" would capture a bit of what both of you are saying.
Supporting API versioning is not a complex or subtle challenge. It's actually a very basic requirement of a microservice architecture. It's like having to test null pointers: it requires additional work but it's still a very basic requirement if you don't want your app to blow up in your face.
You are right. It should not happen. It is difficult to see these pitfalls when unwinding an unwieldy monolith, and, as an organization all you’ve ever done are unwieldy monoliths, that have a gazillion dependencies, interfaces and factories.
We learned from it, and we move on - hopefully, it serves as a warning to others.
To handle deploys and especially rollbacks you need a working CI, or rather CD chain. Where eveything is automated. In there are dependencied in your architecture, all of that needs to be part of the same deploy so you can redo the deploy before that one. With a monolith, you get a simple deploy of all dependencies, as they are baked into one thing.
There are down sides of everything baked into one thing. Having all your "micoro" servies deplyed as one package, would make the dependency you have act as before the monolith change.
Seeing this kinds of dependency imho even in a monolith are an architechture problem, that in the monolith shows up at code changed and fixes can't be handles by localized fixes, but changed have to go into large parts of the code base.
I think a bigger thing here, is that each deploy should have a REALLY good reason to not be backwards compatible for some pretty long time period. If that requirement is painful for you, then you probably have two pretty tightly coupled services.
Monolith has huge advantage when maybe your code is like 100k lines or below:
1. Easy cross module unit testing/integration testing, thus sharing components is just easier.
2. Single deployment process
3. CR visibility automatically promotes to all parties of interests, assuming the CR process is working as desired.
4. Also, just a personal preference, easier IDE code suggestion. If you went through json serializing/de-serializing across module boundary, type inference/cohesion is just out-of-reach.
And it is not like monolith doesn't have separation of concern at all. After all, monolith can have modules, and submodules. Start abstracting using file system API, but grouping relevant stuff into folders, before put them into different packages. After all, once diverging, it is really hard to go back.
Unless you have a giant team and more than enough engineers to spare for devops. Micrservices can be considered as a organizational premature optimization.
JSON parsing IS expensive. Way more expensive than many people realize. There is actually an almost-underground JSON parser SCENE in the .Net ecosystem where people develop new parser and try to squeeze the maximum performance using all sorts of tricks: https://github.com/neuecc/Utf8Json . Here there is discussion of needing JSON support in .Net core that's faster than Json.Net: https://github.com/dotnet/announcements/issues/90 .
And people say web applications are never CPU bound :)
one could argue that if serialization/deserialization is the majority of what your api is spending time on then you’ve got distributed monoliths and shouldn’t be making the call in the first place. also ensuring you have the fastest json parser before having a real problem is premature optimization
"turned a team of 15 engineers from managing one giant monolith to two teams managing about 10 or so microservices"
Knowing absolutely nothing about your product, this sounds like a bad way to split up your monolith.
Probably something like 5 teams of 3 each managing 1 microservice would be a better way to split things up. That way each team is responsible for defining their service's API, and testing and validating the API works correctly including performance requirements. This structure makes it much less likely services will change in a tightly coupled way. Also, each service team must make sure new functionality does not break existing APIs. Which all make it less likely to have to roll back multiple commits across multiple projects.
The performance issues you cite, also seem to indicate you have too many services, because you are crossing service boundaries often, with the associated serialization and deserialization costs. So each service probably should be doing more work per call.
"all depending on a common parent microservice"
This makes you microservices more like a monolith all over again, because a change in the parent can break something in the child, or prevent the child from changing independently of the parent.
Shared libraries I think are a better approach.
"Authentication/Authorization also becomes challenging since every microservice needs to be auth-aware."
Yes, this is a pain. Because security concerns are so important, it is going to add significant overhead to every service to make sure you get it right, no matter what approach you use.
Probably more like do your best to reason about the boundaries of each bounded context and pick an appropriate number of services based off that analysis?
Surely splitting up your application along arbitrary lines based on advice of an internet stranger whose never seen the application and doesn't know the product/business domain just isn't sound way of approaching the problem.
Conway's Law is profound. Lately I realized even the physical office layout (if you have one) acts as an input into your architecture via Conway's Law.
We use grpc instead of rest for internal synchronous communication but we've also found that by using event pub/sub between services, that there are not many uses cases where we have direct calls between services.
We used to have a parent maven pom and common libraries but got rid of most of that because it caused too much coupling. Now we create smaller more focused common libraries and favor copy/paste code over reuse to reduce coupling. We also moved a lot of the cross cutting concerns into Envoy so that the services can focus on business functionality.
I would say it depends on the stability of what's being copy/pasted. If it's just boilerplate, it's less concerning.
In my opinion, decoupling should be prioritized over DRYness (within reason). A microservice should be able to live fairly independently from other microservices. While throwing out shared libraries (which can be maintained and distributed independently from services) seems like overkill, it seems much better than having explicit inheritance between microservice projects like the original poster is describing.
For any non trivial code, which needs to be maintained and be kept well tested, to the contrary of the OP, I would favor shared libraries over copy/paste.
How do you handle cases where your client is awaiting a response with a decoupled pub/sub backend? E.g a user creates an account and the client needs to know their user id.
Would that user object be the responsibility of one service, or written to many tables in the system under different services, or...?
For one, you could use something like snowflake IDs so that whatever server receives the user data first can generate and return an id for that user before tossing the data on a queue to be processed.
How would you approach a situation where a client updates a record in service A, and then navigates to a page whose data is returned by service B, which has a denormalized copy of service A's records that hasn't consumed and processed the "UpdatedARecord" event?
Do we accept that sometimes things may be out of sync until they aren't? That can be a jarring user experience. Do we wait on the Service B event until responding to the client request? That seems highly coupled and inefficient.
I'm genuinely confused as to how to solve this, and it's hard to find good practical solutions to problems real apps will have online.
I suppose the front end could be smart enough to know "we haven't received an ack from Service B, make sure that record has a spinner/a processing state on it".
You use eventing only when eventual consistency is acceptable. In your scenario, it sounds like it is not. So then you should use synchronous communication to ensure the expected experience is met. However, that also means that now you can't do stuff with service B without service A being up. So you're trading user experience against resiliency.
Also, you should check your domains and bounded contexts and reevaluate whether A and B are actually different services. They might still legitimately be separate. Just something to check.
Some people advocate that microservices own their data and only provide it through an API. In this scenario, Service B would need to query Service A for the authoritative copy of the record. I think the standard way to deal with the query and network time is, yes, to wait until Service A provides the data and timeout if it takes "too long".
Then your question is about optimizing on top of the usual architecture which hopefully is an infrequent source of pain that is worth the cost of making it faster. I could imagine some clever caching, Service A and Service B both subscribing to a source of events that deal with the data in question, or just combining Service A and B into one component.
I would create the account directly from the initial call and return its ID and then publish an account created message. Any other services could receive the message and perform some action such as send a welcome email or do some analytics.
I know copy paste is looked down upon, but it's even suggested as a good practice (a little bit) in golang. A little copying is better than a little dependency.
Consider larger services than a typical microservice.
For the same reason monoliths tend to split when the organization grows, it is often more manageable to have a small number of services per team (ideally 1, or less).
It's ok if a service owns more than one type of entity.
It's less good if a service owns more than one part of your businesses' domain, however
> Consider larger services than a typical microservice.
People seem to forget that there’s a continuum between monolith and microservices, it’s not one or the other.
Multiple monoliths, “medium”-services, monolith plus microservices, and so on are perfectly workable options that can help transition to microservices (if you ever need to get there at all).
That's fine, but reorganizations happen, teams can grow, and there is an advantage to having things be separate services in cases like this.
Definitely don't just stuff unrelated stuff into a service since a team that normally deals with that service is now working on unrelated stuff. If the unrelated stuff takes off, you now have two teams untangling your monolithic service.
That said, I'm a big fan of medium sized services, the kind of thing that might handle 10 or 20 different entities.
I'm going to go out on a limb here and suggest that parsing (and serializing) JSON is unlikely to be the actual problem, performance-wise. (Although "OpenAPI/Swagger" doesn't fill me with enthusiasm.)
More likely, I suspect, is that either you are shipping way too much data around, you have too much synchrony, or some other problem is being being hidden in the distribution. (I once dealt with an ESB service that took 2.5 seconds to convert an auth token from one format to another. I parallelized the requests, and the time to load a page went from 10 sec to <3; then I yanked the service's code into our app and that dropped to milliseconds.)
Performance problems in large distributed systems are a pain to diagnose and the tools are horrible.
The whole point of doing microservices is so that you can split up processing responsibility boundaries reasonably, and each team is responsible for being an "expert" in the service it's responsible for.
This also means that each service should have no other services as dependencies, and if they do, you have to many separate services and you should probably look into why hey aren't wrapped up together.
Using a stream from a different service is one thing: You should have clearly defined interfaces for inter-service communication. But if updating a service means you also need to fix an upstream service, your doing it wrong and are actually causing more work than just using a monolith.
EDIT: and because you have clearly defined interfaces, these issues with updating one service and affecting another service literally cannot exist if you've done the rest correctly.
Even just defining what “internal communication” means is difficult. We definitely suffer from the what-if syndrome- “what if some day we want to expose this service to a client?”
Our product is a collection of large systems used by many customers with very different requirements - and so we often fall into this configurability trap: “make everything super configurable so that we don’t have to rebuild, and let integration teams customize it”
Ah. In that case, you can expose your gRPC endpoints as traditional JSON/HTTP ones with gRPC-Gateway, which supports generating OpenAPI documentation too! Best of both worlds.
Sounds like you're suffering from not using YNGNI enough: you're not gonna need it. Build what you need now. When that changes, you can change what you built. That was the original intention of Agile methodologies and BDD or TDD. When the tests pass, you're done.
You must be doing something wrong. In a company I work for, we clearly have separated microservices by bounded context, thus making them completely decoupled.
This is the key - decoupled. When you have to rollback commits across multiple service, they are not decoupled, and you're doing something wrong.
Each service should be fully independent, able to be be deployed & rolled-back w/o other services changing.
If you're making API changes, then you have to start talking about API versioning and supporting multiple versions of an API while clients migrate, etc.
Sure it does. Once you have a large monolith, the same coordination problems hit. Just ask the Linux kernel developers: https://lwn.net/Articles/769365/
> This sounds good in theory, but parsing JSON and then serializing it N times has a real performance cost.
It's not just the serialization cost but latency (https://gist.github.com/jboner/2841832) as well, every step of the process adds latency, from accessing the object graph, serializing it, sending it to another process and/or over the network, then building up the object graph again.
The fashion in .net apps used to be to separate service layers from web front ends and slap an SOA (the previous name for micro-services) label on it. I experimented with moving the service to in process and got an instant 3x wall clock improvement on every single page load, we were pissing away 2/3rds of our performance and getting nothing of value from it. And this was int the best case scenario, a reasonably optimized app with binary serialization and only a single boundary crossing per user web request.
Other worse apps I've worked on since had the same anti-pattern but would cross the service boundary dozens/hundreds/thousands of times and very simple pages would take several seconds to load. It's enterprise scale n+1.
If you want to share code like this then make a dll and install it on every machine necessary, you've got to define a strict API either way.
- Logging. All messages pertaining to a request (or task) should have a unique ID across the entire fleet of services in order to follow the trail while debugging.
I don't think it's necessarily the best idea to immediately have almost as many services as you have engineers. There are usually more gradual and logical ways to split things up.
> We also have our microservices in a hierarchy, all depending on a common parent microservice.
Can you explain this a bit more? I thought the point was to have each service be as atomic as possible, so that a change to one service does not significantly impact other services in terms of rollbacks/etc.
If I'm wrong here let me know, our company is still early days of figuring out how to get out of the problems presented by monolith (or in our case, mega-monolith).
These are excellent points. I was just having a conversation yesterday about system design and how there seems to be a tipping point after which the transactional/organizational costs of segmenting services outweigh the benefits.
My unscientific impression is that some of the organizational costs - just keeping the teams coordinated and on the same page - can become even more "expensive" than the technical costs.
Yes - both the protocol and tooling are expensive to support, even by 90s standards. Unless you have to use it for compatibility with a service you can’t fix it’d be much better to start with a better implementation of the concept.
ASN.1 has a number of problems with equivalent parsings (very bad in a security context, has been the source of a number of TLS vulnerabilities), as well as the fact that even discovering if two ASN.1 parsers will give the same result is undecidable.
I agree with all those drawbacks, but auth is something that can be handled with a bit of one time engineering effort. Where I am all traffic to microservices comes through an API gateway which is responsible for routing traffic to the correct service, and more importantly authorising access to those endpoints. Once the gateway has completed auth it places a signed JWT in the Authorisation header, at which point the microservice's responsbility goes from handling the entire auth process to checking the signature can be verified.
There are always going to be cross-cutting concerns. You don't want to have every microservice implement it's own authentication, auditing, etc. from the ground up.
I really hate the term 'microservice', because it carries the implication that each service should be really small. In reality, I think the best approach is to choose good boundaries for your services, regardless of the size.
People forget the original 'microservice': the database. No one thinks about it as adding the complexity of other 'services' because the boundaries of the service are so well defined and functional.
I really like this example. A lot of databases have very good module separation internally. However, you don't often see people splitting out query planning, storage, caching, and etc into separately hosted services forced to communicate over the network; even in modern distributed databases.
Meanwhile, you also don’t see a lot of people claiming you should have one single repository that stores the source code, configs, CI tooling, deployment tooling, etc., for Postgres and Mathematica and the Linux kernel and the Unity engine, or that operating any one of these kinds of systems should have anything to do with running any other system apart from declared interfaces through which they might choose to optionally communicate or rely on each other as black box resources.
Funny, but we saw a debate around monolithic codebases and the monolithic image in Smalltalk.
A team of three engineers orchestrating 25 microservices sounds insane to me. A team of of thirty turning one monolith into 10 microservices and splitting into 10 teams of three, each responsible for maintaining one service, is the scenario you want for microservices.
A team size of 10 should be able to move fast and do amazing things. This has been the common wisdom for decades. Get larger, then you spend too much time communicating. There's a reason why Conway's Law exists.
I don't think Martin Fowler realized when he wrote the first microservices article that he'd stumbled upon a technical solution to a political problem. He just saw it work and wanted to share.
I don't think Martin Fowler realized when he wrote the first microservices article that he'd stumbled upon a technical solution to a political problem. He just saw it work and wanted to share.
The generation of programmers that Martin Fowler is from, are exactly the people from whom I got my ideas around how organization politics effect software and vice versa. There was plenty of cynicism around organization politics back then.
And it's not as if Martin Fowler came up with the idea as an original either, QnX, Erlang and many other systems used those basic ideas much earlier (sometimes decades earlier). But this is the web, where the old is new again.
Indeed! Conway's the man. Unless your "service" corresponds to an actual existing team who has the time and authority to focus on it, you are asking for trouble and/or wasteful busywork. I curse misapplied microservices.
By the way, for a relatively small service to be shared by multiple applications, try RDBMS stored procedures first.
What reasons do you have for making that link? What are you refering to?
It's possible to load some code and snapshot as a Smalltalk image; then load some different code and snapshot as a different Smalltalk image.
It's a different story when you're working on a team, and a different story when there are two or more teams using the same repository. Sure, you still have the image. The debate had to do with how the Smalltalk image affected the community's relationship to the rest of the world of software ecosystems, and how the image affected software architecture in the small. That "geography" tended to produce an insular Smalltalk community and tightly bound architecture within individual projects.
People at least played around with that as a research project. There's one that showed up at the Camp Smalltalks I went to, with a weird-but-sensible sounding name. (Weird enough I can't remember the name.)
There would have been great utility in such a thing. For one thing, the debugger in Smalltalk is just another Smalltalk application. So what happens when you want to debug the debugger? Make a copy of the debugger's code and modify the debugger hooks so that when debugging the debugger, it's the debugger-copy that's debugging the debugger. With multi-image Smalltalk, you could just have one Smalltalk image run the debugger-copy without doing a bunch of copy/renaming. (Which, I just remembered, you can make mistakes at, with hilarious results.)
If you do the hacky shortcut of implementing one Smalltalk inside another Smalltalk (Call this St2), then the subset of objects that are the St2 objects can act a bit like a separate image. In that case, the host Smalltalk debugger can debug the St2 debugger.
When I bring up Smalltalk, I get an all-in-one environment from the image I load. Its live and any code I add goes into that image. Now I can use code control and build specific images, but it pretty much is a one image at a time world.
What I'm talking about is loading up multiple images into the same IDE and run them like fully separate images with maybe some plumbing for communication and code loading between them. You can sorta pull that stunt by, as stcredzero mentioned, running Smalltalk in Smalltalk, ut I want separate images.
In this example on Ubuntu "visual" is the name of the VM file, and there are 2 different image files with different code in them "visualnc64.im" and "quad.im".
That's created 2 separate OS processes, each OS process is running an instance of the Smalltalk VM, and each Smalltalk VM opened a different Smalltalk image containing different code.
I’m not sure if we are talking past each other or you are ignoring the whole IDE thing. Yes, I can run multiple VMs on the same machine, but you are missing that I want to spin up these VMs in my Smalltalk IDE and not via some terminal launch script. I want my environment there for me to edit and debug code. I’m pretty sure you cannot do that in VisualWorks.
> I want my environment there for me to edit and debug code
Both of those instances of the Smalltalk VM, the one in OS process 8689 and the one in OS process 8690, are headfull — they both include the full Smalltalk IDE, they are both fully capable of editing and debugging code.
(There's a very visible difference between the 2 Smalltalk IDEs that opened on my desktop: visualnc64.im is as-supplied by the vendor; quad.im has an additional binding for the X FreeType interface library, so the text looks quite different).
(iirc Back-in-the-day when I had opened multiple Smalltalk images I'd set the UI Look&Feel of one to MS Windows, of another to Mac, of another to Unix: so I could see which windows belonged to which image.)
Yeah, but they are 2 IDEs not a single IDE. You are running two copies not one copy with two instances. I then need to jump between programs to edit code.
So when I asked "What will that let you do?", the only "benefit" you-can-think-of is the possibility of switching from editing code in visualnc64.im to editing code in quad.im without a mouse-click ?
So when I asked "What will that let you do?", the only "benefit" you-can-think-of is the possibility of switching from editing code in visualnc64.im to editing code in quad.im without a mouse-click?
No, that would not be enough to make anything work. What I can think of is an IDE that had access to all the VMs running and some plumbing for the VMs to communicate. I would love to be able to spin-up Smalltalk VMs so I can simulate a full system on my desk. Having separate IDEs running means I don't have any integration so I have to debug in multiple different IDEs when tracing communications. I can imagine some of the debugging and code inspection that could be extended to look at code running simultaneously in multiple VMs.
But they enable something really powerful: continuity of means, continuity of responsibility, that way a small team has full hand of developing AND operating a piece of a solution.
Basically, organization tends to be quite efficient when dealing with small teams (about dozen people, pizza rule and everything), that way information flows easily, with point to point communication without the need of a coordinator.
However, with such architecture, greater emphasis should be put on interfaces (aka APIs). A detailed contract must be written (or even set as a policy):
* how long the API while remain stable?
* how will it be deprecated? with a Vn and Vn-1 scheme?
* how is it documented?
* what are the limitations? (performance, call rates, etc)?
If you don't believe me, just read "Military-Standard-498". We can say anything about military standards, but military organizations, as people specifying, ordering and operating complex systems for decades, they know a thing or two about managing complex systems. And interfaces have a good place in their documentation corpus with the IRS (Interface Requirements Specification) and IDD (Interface Design Description) documents. Keep in mind this MIL-STD is from 1994.
According to Wikipedia, Military Standard 498 has been replaced with ISO/IEC/IEEE 12207. Do you have any experience with that? Do you have experience with any other modern standards for software development?
Not really, it's something I was confronted to when I was working on military contracts a few years ago.
From what I recall, it's very waterfall minded in term of specification workflow, it's also quite document heavy, and the terminology and acronyms can take a while to get used to.
I found it was a bit lacking regarding how to put together all the pieces into a big system, aka the Integration step. IMHO It's a bit too software oriented, lacking on the system side of thing (http://www.abelia.com/498pdf/498GBOT.PDF page 60).
10 teams of 3 each owning their own little slice of the pie sounds like an organizational nightmare; mostly, you can't keep each team fully occupied with just that one service, that's not how it works. And any task that touches more than one microservice will involve a lot of overhead with teams coordinating.
While I do feel like one team should hold ownership of a service, they should also be working on others and be open to contributions - like the open source model.
Finally, going from a monolith to 10 services sounds like a bad. I'd get some metrics first, see what component of the monolith would benefit the most (in the overall application performance) from being extracted and (for example) rewritten in a more specialized language.
If you can't prove with numbers that you need to migrate to a microservices architecture (or: split up your application), then don't do it. If it's not about performance, you've got an organizational problem, and trying to solve it with a technical solution is not fixing the problem, only adding more.
"10 teams of 3 each owning their own little slice of the pie sounds like an organizational nightmare; mostly, you can't keep each team fully occupied with just that one service, that's not how it works. And any task that touches more than one microservice will involve a lot of overhead with teams coordinating."
I guess that's where the critical challenge lies. You'd better be damn sure you know your business domain better than the business itself! So you can lay down the right boundaries, contracts & responsibilities for your services.
Once your service boundaries are laid down, they're very hard to change
It takes just one cross-cutting requirement change to tank your architecture and turn it into a distributed ball of mud!
Which has to stand as a damning indictment of the one-service-per-team model, surely?
Something so inflexible can't survive contact with reality (for very long).
At work we run 20-something microservices with a team of 14 engineers, and there's no siloing. If we need to add a feature that touches three services then the devs just touch the three services and orchestrate the deployments correctly. Devs wander between services depending on the needs of the project/product, not based on an arbitrary division.
If you are doing http/json between microservices then you are definitely holding it wrong.
Do yourself a favor and use protobuf/grpc. It exists specifically for this purpose, specifically because what you're doing is bad for your own health.
Or Avro, or Thrift, or whatever. Same thing. Since Google took forever to open source grpc, every time their engineers left to modernize some other tech company, Facebook or Twitter or whatever, they'd reimplement proto/stubby at their new gig. Because it's literally the only way to solve this problem.
So use whatever incarnation you like.. you have options. But json/http isn't one of them. The problem goes way deeper than serialization efficiency.
(edit: d'oh! Replied to the wrong comment. Aw well, the advice is still sound.)
I once worked at a company where a team of 3 produced way more than 25 microservices. But the trick was, they were all running off the same binary, just with slightly different configurations. Doing it that way gave the ops team the ability to isolate different business processes that relied on that functionality, in order to limit the scale of outages. Canary releases, too.
It's 3 developers in charge of 25 different services all talking to each other over REST that sounds awful to me. What's that even getting you? Maybe if you're the kind of person who thinks that double-checking HTTP status codes and validating JSON is actually fun...
I've done that a couple of times. It's a good pattern!
I worked on an e-commerce site a decade ago where the process types were:
1. Customer-facing web app
2. CMS for merchandising staff
3. Scheduled jobs worker
4. Feed handler for inventory updates
5. Distributed lock manager
6. Distributed cache manager
We had two binary artifacts - one for the CMS, one for everything else - and they were all built from a single codebase. The CMS was different because we compiled in masses of third-party framework code for the CMS.
Each process type ran with different config which enabled and configured the relevant subsystems on as needed. I'm not sure to what extent we even really needed to do that: the scheduled jobs and inventory feed workers could safely have run the customer app as well, as long as the front-end proxies never routed traffic to them.
It really depends upon what those 25 different services are. If they are trivial to separate sure. Like an image resizing microservice, an email sending microservice, and so on. I mean go wild, these are trivial. Coincidentally, when people like to talk about how easy microservices are, they love to talk about these trivial examples.
What isn't trivial is when someone decides to make an Order microservice and an Account microservice when there's a business rule where both accounts and orders can only co-exist. Good fucking luck with 3 developers, I'm pretty sure with a team of 3 in charge of 23 other microservices you aren't exhaustively testing inter-microservice race conditions.
Or better yet: a massive front end SPA talking to an array of stupid microservices that are not much more than tables with web accessible endpoints for CRUD and probably an authentication service thrown in for good measure. All of the business logic on the JS side of things. The worst of both worlds, now you have a monolith and a microservices based architecture with the advantages of neither.
I work on an application like that. Not 3:25 but not that far off either. I'm quite content with the situation.
The apps all handle a bespoke data connection, converting it into a standard model which they submit to our message broker. From then on our services are much larger and smaller in number. It's very write-once-run-forever, some of these have not been touched since their inception years ago, resulting in a decreased complexity and maintenance cost.
The trick is not having REST calls all over yours services. You're just building a distributed monolith at that point.
Last startup I've been part of, we've been having a bit of fun building our server architecture in Swift and so far the One-Binary-Many-Services model has been working out pretty well. You can run it all on single machine, you can have debug hooks, that make it seem like a Monolith, or scale it out if need be. When it comes down to it the Authentication Service really doesn't need to know about the Image Upload Service, and splitting it is all about defining good interfaces. Just need to put some effort in to keep your development environment sane.
I'm kind of dealing with your awful scenario right now. It is pretty bad. What happened was the department used to be a lot, lot larger, and people tended to only have to deal with 7 or 8 of them at a time (it was still excessive, I often had difficulty debugging and keeping things straight for tasks) but after two years layoffs, other employees quitting, and executives pulling our employees into other departments, we're a tiny shell of what we used to be, and we still have to manage all of those microservices, and it's so difficult.
I've been daydreaming about monoliths and will be asking at interviews for my next job hoping to find more simplified systems. I came from the game industry originally, where you only have one project for the game and one more for the webservice if it had one, and maybe a few others for tools that help support the game.
This wasn't actually that. All of them did the same job, just that one did it for widgets for the widget-handling team, and another did it for whatsits for the whatsit-handling team, and another did it for both widgets and whatsits for the reporting system, etc. etc.
The definition of “micro” appears to be hugely variable! If you’d asked me I’d say that sure, my last team definitely built microservices. A team of around 10 engineers built and maintained something like 3 services for a product launch, each with a very different purpose, and over time we added a couple more. Three people maintaining 25 services sounds absolutely bonkers to me.
If your monolith goes unwieldy you have a problem with your code structure which microservices won't solve. As we all know, you need well isolated, modular code with well defined boundaries. You can achieve this just as well in a monolith (and you can also achieve totally spaghetti code between microservices).
Microservices is a deployment choice. It's the choice to talk between the isolated parts with RPC's instead of local function calls.
So are there no reasons to have multiple services? No there are reasons, but since it's about deployments, the reasons are related to deployment factors. E.g. if you have a subsystem that needs to run in a different environment, or a subsystem that has different performance/scalability requirements etc.
That's just "services", though, and it's been the way that people have been building software for a very long time. I can attest to have done this in 2007 at a large website, which was at least 7 years before the "microservices" hype picked up (https://trends.google.com/trends/explore?date=all&q=microser...). When people say "microservices" they're referring to the model of many more services than what you describe, and the associated infrastructure to manage them.
I also think designing in the microservices mindset (i.e. loose coupling, separable, dependency free architecture) is something which can be done on a continuum, and there's not a strict dichotomy between The Monolith and Microservices(tm).
Even if you're working on an early prototype which fits into a handful of source files, it can be useful to organize your application in terms of parallel, independent pieces long before it becomes necessary to enforce that separation on an infrastructure/dev-ops level.
You start with microservices when you realize that including the Elasticsearch API in your jar causes dependency conflicts that are not easy to resolve.
While there are cases where I think microservices make it easier to scale an application across multiple hosts, I don't understand the organizational benefits compared to just using modules/packages within a monolith. IMO a team that makes an organizational mess of a monolith and makes it grow unwieldy will likely repeat that mistake with a microservice oriented design.
The main thing, however, is many people think that, by breaking up their monolith into services, that they now have microservices. No, you don't. You have a distributed monolith.
Can you deploy services independently? No? You don't have microservices. Can you change one microservice data storage and deploy it just fine? If you are changing a table schema and you now have to deploy multiple services, they are not microservices.
So, you take a monolith, break it up, add a message broker, centralized logging, maybe deploy them on K8s, and then you achieve... nothing at all. At least, nothing that will help the business. Just more complexity and a lot more stuff that need to be managed and can go wrong.
And probably a much bigger footprint. Every stupid hello world app now wants 8GB of memory and its own DB for itself. So you added costs too. And accomplished nothing a CI/CD pipeline plus sane development and deployment practices wouldn't have achieved.
It is also sometimes used in lieu of team collaboration. Now everyone can code their own thing in their own language without talking to anyone else. Except collaboration is still needed, so you are accruing tech debt that you know nothing about. You can break interfaces and assumptions, where your monolith wouldn't even compile. And now no-one understands how the system works anymore.
Now, if you are designing a system using microservices properly, then it can be a dream to work on, and manage in production. But that requires good teamwork on each team and good collaboration between teams. You also need a different mindset.
Do you have a recommended way of handling transaction boundaries that span multiple services. Everyone always likes to outline how happy path works and when it comes to real world it basically comes down to well now you have eventually consistent distributed system that there is no general valid way to unroll a change to multiple services if one of the calls fails.
If several databases/services are so tied together that they need to be updated/rolled back at the same time, then they belong to the same service/database.
There will be times when you trade data consistency for performance/scalability, i.e. in a scenario where you are breaking user actions away from the main user service, if a user is deleted but deleting from user actions failed, you don't roll back the user delete. Either just let the invalid data sit in user actions database, or do separate clean-ups periodically.
"In order to be reliable, a service must atomically update its database and publish a message/event."
so in other words to some transaction log so we just reinvented a distributed transactional database system with crapload of custom programming overhead. Whatever solution is described they all pretty much boil down to an equivalent of distributed eventually consistent database or distributed acid database but with a large number of database functionality pushed into the application.
You can break up the monolith, use a message broker or even let the services communicate via http but you do not need K8s. Its pointless unless you want to orchestrate multiple vms/images and your infra scales to that of more than 10-15 servers/containers etc.
When I started programming professionally it was the era of "Object Oriented Design" will save us all. I worked on an e-commerce site that had a class hierarchy 18 levels deep just to render a product on a page. No one knew what all those levels were for, but it sure was complicated and slow as hell. The current obsession with microservices feels the same in many ways.
There appear to be exactly two reasons to use microservices:
1. Your company needs APIs to define responsibility over specific functionality. Usually happens when teams get big.
2. You have a set of functions that need specific hardware to scale. GPUs, huge memory, high performance local disk, etc. It might not make sense to scale as a monolith then.
One thing you sure don't get is performance. You're going to take an in-process shared-memory function call and turn it into a serialized network call and it'll be _faster_? That's crazy talk.
So why are we doing it?
1. Because we follow the lead of large tech companies because they have great engineers, but unfortunately they have very different problems then we do.
2. The average number of years of experience in the industry is pretty low. I've seen two of these kinds of cycles now and we just keep making the same mistakes over and over.
Anyway, I'm not sure who I'm writing this comment for, I guess myself! And please don't take this as criticism, I've made these exact mistakes before too. I just wish we as an industry had a deeper understanding of what's been done before and why it didn't work.
A great many of our problems in tech are the result of "...we follow the lead of large tech companies...but unfortunately they have very different problems than we do."
Imagine if we built single-family houses based on what made sense for skyscrapers. Or if we built subcompact cars based on a shrunk-down version of semi-tractor trailers. They would not be efficient, or even effective.
But, if your aspiration is to get a job at a skyscraper-builder, then it MIGHT be what makes sense to do. "Have you used appropriate-only-for-largest-websites technology X?" "Why yes I have." The same incentives probably apply to the tech management, btw. We have an incentives problem.
Its not about the age of the engineers but maturity. Some just dont care about quality of their work because they get paid either way. Look at Silicon Valley, "ageism" is real there. They need young devs with ideas and huge skill to bring them to life, to stay ahead of the competition. Most companies dont understand that and blindly try to copy that often because their management is not competent enough.
Reasons for current situation are plenty. World and ppl are complicated.
LOL :D You mean cheap devs who will work for half the price and will never raise their concerns and will gladly accept the stupidest and most menial tasks.
You can optimise per use case. In the monolith everything has to work for every use case. In a service you might not care to write, you might not care if your writes are super async. This means you can start to take liberties with the back-end (e.g. denormalising where necessary) and you have room to breathe.
I'd like to point out that microservices are not always as cheap as you may think. In the AWS/Lambda case, what will probably bite you is the API Gateway costs. Sure they give you 1,000,000 calls for free, but it's $3.50 per million after that. That can get very expensive, very quickly. See this hacker news post from a couple years ago. The author's complaint is still valid: "The API gateway seems quite expensive to me. I guess it has its use cases and mine doesn't fit into it. I run a free API www.macvendors.com that handles around 225 million requests per month. It's super simple and has no authentiction or anything, but I'm also able to run it on a $20/m VPS. Looks like API gateway would be $750+data. Bummer because the ecosystem around it looks great. You certainly pay for it though!"
Worth saying: Now that ALBs support Lambda as a backend, reaching for APIG w/ a lambda proxy makes less sense, unless you're actually using a lot of the value-adds (like request validation/parsing and authn). Most setups of APIG+Lambda I've seen don't do this, and prefer to just Proxy it; use an ALB instead.
ALB pricing is a little strange thanks to the $5.76/mo/LCU cost and the differentiation between new connections and active connections. The days are LONG GONE when AWS just charged you for "how much you use", and many of their new products (Dynamo, Aurora Serverless, ALB) are moving toward a crazy "compute unit" architecture five abstraction layers behind units that make sense.
But it should be cheaper; back of the napkin math, 225M req/month is about 100RPS averaged, which can be met with maybe 5 LCUs on an ALB. So total cost would be somewhere in the ballpark of $60/month, plus the cost of lambda which would probably be around $100/month.
Is it cheaper than a VPS? Hell no. Serverless never is. But is it worth it? Depends on your business.
Right, there are a few use cases for Lambda that make lots of sense, and then some that don't. If you're not extracting any operational benefits or costs (think a request that needs to run 5m per hour) from the managed portion of Lambda then it's probably not for you.
The ALB point is very strong. APIGW can add lots of value with request response manipulation and the headaches of managing your own VPS, but you need to make sure that you don't just need a bare bones path -> lambda mapping, which is where the ALB can shine.
The $750/month is well worth it to organizations with billions in revenue, wishing to protect user data. Better to route all traffic through an API gateway, and exposing all of your micro services on the public internet.
Everyone has to communicate through the API gateway. Then, you get a single point where things are easily auditable.
It has a lot of benefits that apply to business use cases. Your free API may not have as strict requirements.
I’ve never used API gateway outside of quick prototype tests to access lambda. $750 per month doesn’t sound like a lot of money if you have 225 million requests per month. A free API is probably an exception but I do realize why that would too expensive for your use case.
Recent encounter: 70+ microservices for a minor ecommerce application. Total madness and while I'm all for approaching things in a modular way if you really want to mimic Erlang/BEAM/OTP just switch platforms rather than to re-invent the wheel. In Erlang it would make perfect sense to have a small component be a service all by itself with a supervision tree to ensure that all components are up and running.
Apps like WhatsApp/Insta have relatively small feature-sets. I doubt you can pin their success on their tech-stack. It's more like they aggressively avoided feature creep.
I'm equally curious about flow-based programming. Treating things as filters has worked well for the Unix command line. There's no reason we can't have processes do sort, filter, merge, lookup, and such in well-defined ways and chain them together as needed for many kinds of applications. Although "network" and "port" mean something particular within FBP it doesn't at all mean the processes can't be on different machines and talking via TCP/IP.
I don't know Erlang and I'm only a couple of months in to learning Elixir in my (limited) free time. Grated, it's not purely functional, but... I've been reflecting on this exact equivalence, as you put it, and am happy to hear that my inclination about the Actor model isn't unfounded.
In Erlang it makes perfect sense to have a single counter variable be a gen_server by itself and be part of a supervision hierarchy. Deploying and versioning that counter separately will still correctly get laughed at.
I feel like this argument of Monolith vs Microservice is really a discussion about premature optimization. There is nothing wrong with starting out with a monolith with great design. Limiting the responsibility of the monolith is the key I believe to a maintainable piece of software. Should you need something or your business needs grow to be outside of that defined responsibility creating a new service should be discussed.
For example I have a service that hosts / streams videos. I have 1 service that handles all the metadata of the video. Handle users, discussions etc... one could even think of this as a monolith. Then video encoding piece started interfering with the metadata stuff so I decided it might be smart to separate the video encoding into its own service since it had different scaling requirements from the metadata server.
In that specific case it made a lot of sense to have 2 services I can justify it with the following reasons.
- Resource isolation is important to the performance of the application.
- having the ability to scale the encoder workers out horizontally makes sense.
So now it makes sense I’m managing 2 services.
There should be a lot of thought and reasoning behind doing engineering work. I think following trends is great for fashion products like jeans / shirts etc... but not for engineering.
If you are starting a project doing microservices chances are you are optimizing prematurely. That’s just my 2cent.
> It feels like we’re starting to pass the peak of the hype cycle of microservices
I feel like any article I see on microservices bemoans how terrible/unnecessary they are. If anything, we're in the monolith phase of the hype cycle =)
If you're moving to microservices primarily because you want serving path performance and reliability, you're doing it wrong. The reasons for microservices are organizational politics (and, if you're an individual or small company, you shouldn't have much politics), ease of builds, ease of deployments, and CI/CD.
It's like almost everything else a matter of balance. A monolith that cross-connects all kinds of stuff in ways will become an unmaintainable mess over time, a ridiculously large number of microservices will move the mess to the communications layer and will not solve the problem.
A couple of independent vertically integrated microservices (or simply: services, screw fashion) is all most companies and applications will ever need, the few that expand beyond that will likely need more architectural work before they can be safely deployed at scale.
Even at a very large company, it's probably too common to spin out some logic into its own microservice. Often times owning your own microservice is a great argument for a promo, even if the code could just as well be in an existing service. Companies would claim to discourage that tendency, but it's a false claim.
So that's an additional reason for microservice frameworks: a welfare safety net for software engineers.
There are also two purely engineering considerations: scalability and crash isolation.
Scalability -- for when your processes no longer fit on a single node, and you need to split into multiple services handling a subset of the load. This is rare, given that vendors will happily sell you a server with double-digit terabytes of ram.
Crash isolation -- for when you have some components with very complex failure recovery, where "just die and recover from a clean slate" is a good error handling policy. This approach can make debugging easy, and may make sense in a distributed system where you need to handle nodes going away at any time anyways, but it's not a decision to take lightly, especially since there will be constant pressure to "just handle that error, and don't exit in this case", which kills a lot of the simplicity that you gain.
Microservices are not good for scalability. You want data parallelism, like 20 webservers that all do the same thing. Not splitting your app into 20 microservices all doing something else.
Yes. If you would balk at multiple SWE years worth of work to move to microservices because of all the opportunity costs for your business, microservices aren't for you.
I've come to the conclusion that microservices work for large organizations where division of labor is important but for small development teams it actually makes things worse.
What once was just a function call now becomes an API call. And now you need to manage multiple CI/CD builds and scripts.
It adds a tremendous amount of overhead and there is less time spent delivering core value.
Serverless architectures and app platforms seems to correct a lot of this overhead and frustration while still providing most of the benefits of microservices.
I've never found the division of labor argument all that compelling. Using multiple services compels clean divisions and impenetrable abstractions, but shouldn't it be possible to achieve that within a single program? A language with strong support for information hiding should be able to enforce the restrictive interfaces that microservices compel, but without the complexity and overhead of going over the network.
If that's not possible, I'd take that as a sign that we need new languages and tools.
I'm on a small team working with microservices. I have different complaints than yours. The main issue I run into with microservices is I lose the benefit of my Go compiler. I don't like in dynamic languages because of all the runtime errors I run into. With microservices, even using a statically typed language becomes a nightmare of runtime errors.
If I change the type on a struct that i'm marshaling and unmarshaling between services, I can break my whole pipeline if I forget to update the type on each microservice. This feels like something that should be easy to catch with a compiler.
If your services need a shared, implicit understanding of types, you're not respecting the microservice boundaries. Each microservice needs to offer a contract describing what inputs it accepts and immediately reject a request that doesn't meet that contract. Then type mismatches becomes very obvious during development when you start getting 400s when testing against the DEV endpoint. Don't pass around opaque structs.
If you use code gen like gRPC or thrift, it is easy for a compiler to catch. Even if you're using microservices with a shared DTO library, its easy for the compiler to catch.
Specifically relevant to the discussion is this passage:
> However, if an application reaches a certain scale or the team building it reaches a certain scale, it will eventually outgrow monolithic architecture. This occurred at Shopify in 2016 and was evident by the constantly increasing challenge of building and testing new features. Specifically, a couple of things served as tripwires for us.
> The application was extremely fragile with new code having unexpected repercussions. Making a seemingly innocuous change could trigger a cascade of unrelated test failures. For example, if the code that calculates our shipping rate called into the code that calculates tax rates, then making changes to how we calculate tax rates could affect the outcome of shipping rate calculations, but it might not be obvious why. This was a result of high coupling and a lack of boundaries, which also resulted in tests that were difficult to write, and very slow to run on CI.
> Developing in Shopify required a lot of context to make seemingly simple changes. When new Shopifolk onboarded and got to know the codebase, the amount of information they needed to take in before becoming effective was massive. For example, a new developer who joined the shipping team should only need to understand the implementation of the shipping business logic before they can start building. However, the reality was that they would also need to understand how orders are created, how we process payments, and much more since everything was so intertwined. That’s too much knowledge for an individual to have to hold in their head just to ship their first feature. Complex monolithic applications result in steep learning curves.
> All of the issues we experienced were a direct result of a lack of boundaries between distinct functionality in our code. It was clear that we needed to decrease the coupling between different domains, but the question was how
I've tried a new approach at hackathons where I build a Rails monolith that calls serverless cloud functions. So collaborators can write cloud functions in their language of choice to implement functionality and the Rails monolith integrates their code into the main app. I wonder how this approach would fare for a medium sized codebase.
shopify's problem can be fixed without microservices by writing modular code. The monolith should be structured as a set of libraries. I find it so strange, the way these microservice debates always assume that any codebase running in a single process is necessarily spaghetti-structured. The microservice architecture seems to mainly function as a way to impose discipline on programmers who lack self-discipline.
Well seeing as Italian-Cuisine-Driven Development is so crushingly prevelant, it seems most average developers within one standard deviation of ability on the normal distribution lack either discipline, know-how, or time. Or any combination of the three. So it's hard for software not to be spaghetti unfortunately. That's the natural state it wants to be in over time.
Microservices won't change any of those variables. Nor will it change the normal distribution of them. So one thing is for sure. No matter what the architecture of paradigm is being used, we can on average expect average quality software. Most people on average don't gush about how amazingly clean their architecture is or how well defined their bounded contexts are. They tend to talk about spaghetti. I infer from that that on average it's spaghetti and theres a chance it might not be.
> All of the issues we experienced were a direct result of a lack of boundaries between distinct functionality in our code
This is the key lesson to learn: if you are struggling to have clear separation of responsibilities, you are going to have a bad time with either approach. To the extent that a replacement system ends up being better it's probably due to having been more conscious about that issue.
Microservices at least force people to draw a line in the sand between subsystems/services. How effective or useful the lines you draw are, that's up to the skill of the engineers building your stuff.
I'm not saying microservices are better, but people should really take more serious considerations between the boundaries between subsystems. Because it's so easy to create exceptions, and things end up infinitely more complex in the grand scheme of things.
Clear, well-defined boundaries matter. It's the only way a developer can focus on a small part of a problem, and become an expert at working on that subsystem without needing greater context.
These conversations always get scattered because people don't post the experience they have that forms their view.
Me: I've only worked on HUGE systems (government stuff and per-second, multi-currency, multi-region telephone billing) and on systems for employees with less than 100 people.
My take: Monolith or two systems if at all possible.
This is good: A Rails app that burps out HTML and some light JS.
This is also good: A Rails app that burps out JSON and an Ember app to sop it up and convert it to something useable. Maybe Ember Fastboot, if performance warrants the additional complexity.
This is hellish: Fifteen different services half of which talk to the same set of databases. Most of which are logging inconsistently, and none of which have baked in UUIDs into headers or anything else that could help trace problems through the app.
This is also hellish: A giant fucking mono-repo[0] with so many lines of code nobody can even build the thing locally anymore. You need to write your own version control software just to wrestle with the beast. You spend literally days to remove one inadvertently placed comma.
Sometimes you have to go to hell though. Which way depends on the problem and the team.
[0] Kinda sorta, maybe the iOS app is in something else. Oh and there's also the "open source" work, like protobuffs that "works" but has unreleased patches that actually fix the problems at scale, but are "too complicated" to work into the open source project.
I think for the very large tech companies that have hundreds to thousands of engineers, microservices can make sense as a way to delegate a resource (or a set of resources) to a small group of engineers. The issue is that a lot of smaller companies/engineers want to do things the way these large companies do without understanding why they're actually doing it. The onboarding costs as this post mentions is huge. An engineer at a small company likely needs to know how the entire app works and spreading that over many services can add to the cognitive load of engineering. The average web app just doesn't really benefit from the resource segregation imo.
>I’m sure our specs were good enough that APIs are clean and service failure is isolated and won’t impact others.
Surely if you're building microservices, this line of thinking would be a failure to stick to the design? If your failures aren't isolated and your APIs aren't well-made, you're just building a monolith with request glue instead of code glue.
I appreciate the point is more that this methodology is difficult to follow through on, but integration tests are a holdover - you can test at endpoints: you should be testing at endpoints! That's the benefit.
I've had this feeling in some place that 'SOA' is a bit of a dirty word because it connotes a certain style of systems architect, or working like you do in Java or enterprise-scale PHP.
Many monolithic apps would benefit from a refactoring towards that rather than distributing a call stack across the network. The microservices can come later on if there's a need for it. If nothing else, it'll present a clearer picture of how things fit together when you start enforcing boundaries.
SOA also brings up memories of soul destroying ESB's.
I would love a resurgence of discussions about services and how to best build and govern those without always resulting in a focus on the micro versions.
How are people building a modern IT landscape consisting of different services and system?
The problem with ESBs in my opinion is one of tight coupling; all of these distributed systems that know about and depend upon each other. The solution to this is to loosen coupling whilst formalising interfaces/contracts/schemas, with a design that enables versioning and mandates graceful evolution. The modern version of ESB is an event based (note: this does not necessarily and probably does not mean event sourced) architecture built upon a distributed, append-only log that is ideally fed directly from a database transaction log.
ESB is new to me but just casting a glance on it makes me think... Kafka.
You can't avoid the centralisation unless you want infinite repetition.
The human spine is composed of multiple vertebrae and forms a consistent network with the brain and CNS and the rest of your body. The spine itself is the centralism, no matter how much you separate the bones into vertebra.
A service bus is basically putting all of your eggs into one wire. Or so it seems... it's so easy to strawman yourself to microservices.
But conversely, how much of the purported benefits of microservices are really the benefits from having well-defined contracts between components? Are microservices mostly a forcing function for good architectural practices, that could be applied equally in a monolithic design (with internal component boundaries) with enough discipline?
I think that's a fair starting point, but once you start enforcing better boundaries, you are actually able to access the microservice benefits (i.e. independent failure, component-variant scaling, independent update deployments, etc) because there's no risk of failure from a boundary break - and those benefits are definitely inaccessible to even the best-designed monolith.
It's when an organization grows and the software grows and the monolith starts to get unwieldy that it makes sense to go to microservices. It's then that the advantage of microservices both at the engineering and organizational level really helps.
A team of three engineers orchestrating 25 microservices sounds insane to me. A team of of thirty turning one monolith into 10 microservices and splitting into 10 teams of three, each responsible for maintaining one service, is the scenario you want for microservices.