fundamental problem is cargo cult developers trying to copy massive companies architectures as a startup. They fail to realize those companies only moved to microservices because they had no other option. Lots of startups hurting themselves by blindly following these practices without knowing why they were created in the first place. Some of it is also done by people who know it isn't good for the company, but they want to pad their resume and leave before the consequences are seen.
same thing applies to Leetcode style interviews, no startup should be using them honestly. They are for established companies that have so many quality applicants they can afford to filter out great candidates
re: "copy massive companies"; I don't recall seeing a single "microservice" at Google. Granted, in the 10 years I was there I mostly worked on non-cloud type stuff. But still.
Google has perfected the art of the giant, distributed, horizontally scalable mostly-relational database. They have services, yes. But in large part... they generally all talk to F1 or similar.
Microservices with each thing having its own schema and/or DB seems to me to be a phenomenally stupid idea which will simply lead to abusing your relational model and doing dumb things like your own custom bespoke server-side joins with your own custom bespoke two phase commit or other transaction logic.
Before Google I worked in one shop that did 'microservices' and the release process was a complicated nightmare that would have been better solved by a complicated combinatorial graph optimization library. There were cross service RPCish calls made all over the place to piece data together that in a more monolithic system would be resolved by a simple fast relational join. I shudder to remember it.
Meanwhile, I come from the perspective that a shared database that everyone talks to is a shared infrastructure point. And... I have seen those cause more problems than makes sense.
My bet is I'm also just an old man in this discussion. Such that I really think we can't underline enough how particular the discussion will be to every organization.
If you have shared data with relationships in the data, that is shared infrastructure whether it is physically manifested in one "place" or not.
If you choose to parcel that information out into multiple pieces of infrastructure to avoid a single point of failure, you've now moved the nature of your problem from maintaining the shared database to maintaining the process of managing that data in multiple places.
You'll be doing client-side or service-side joins all over in order to produce a unified piece of data. But note you still haven't removed the dependency on the shared state. You've simply moved the dependency. And in the process made it more complicated.
This might have the advantage of making it "not the problem" of the ops/sysadmin/SRE. But it has the problem of making it "the problem" for either the customer (in the case of shitty data in the client) or the developer (in the case of having to fix bugs.) And I can guarantee you that the developers working on "mycoolmicroservice" are way shittier at understanding data serialization and coherence and transaction management than the people who wrote your database engine.
The ACID relational model was developed for a reason. The set of compromises and problems we are dealing with in infrastructure has been known since the 60s when Date&Codd et al proposed it. The right thing to do for engineering & the infrastructure level is to make the pieces of shared database super reliable and performant at scale, not to try to handwave them away by pushing the problem elsewhere.
I mean, you aren't necessarily wrong. But, I will again lean on "depends on fundamental organization factors of where you work."
In large, there is no reason for people in your HR department to have access to data in your sales department. Even if they have relationships between them. (For payroll/commission/whatever.)
If you are small enough that having wide access to all of the data is fine, then it is fine.
And, ACID isn't some magic bullet that will prevent things from needing an audit process to make sure data is accurate. In particular, if you have an inventory system, your actual physical inventory will trump whatever your database says.
I’m older than i was when people started telling me as a tech founder that we needed micro services to be cool. At 25 i felt it didn’t deliver any real customer value and made my life harder, at 35 it seems like the people that do this are not people i want to hire anyway.
I started sorta the opposite and thought that nano(!) services (think one AWS Lambda per API call) were the best approach, and now I look at that younger wisdom with a parental smile..
I agree that we need to at least name nanoservices -- microservices that are "too small". Like surely we can all agree that if your current microservices were 100x smaller, so that each handled exactly one property of whatever they're meant to track, it'd be a nightmare. So there must be a lower limit, "we want to go this small and no smaller."
I think we also need to name something about coupling. "Coupling" is a really fluid term as used in the microservice world. "Our microservices are not strongly coupled." "Really, so could I take this one, roll it back by 6 months in response to an emergency, and that one would still work?" "err... no, that one is a frontend, it consumes an API provided by the former, if you roll back the API by 6 months the frontend will break." Well, I am sorry, my definition of "strong coupling" is "can I make a change over here without something over there breaking", for example rolling back something by 6 months. (Maybe we found out that this service's codebase had unauthorized entries from some developer 5 months ago and we want to step through every single damn thing that developer wrote, one by one, to make sure it's not leaking everyone's data. IDK. Make up your own scenario.)
Nano actually did (and does) make sense from access control perspective - if a service has permissions to do one thing and one thing only, it is much harder to escalate from. But I'm not sure if these benefits outweigh the potential complexity.
Your definition of coupling seems a bit too strong to be useful. By that definition, just about nobody has an uncoupled API, because if anyone uses it (even outside your company) then you can’t really just revert six months of changes without creating a lot of problems for your dependents. If those changes just happen to be not user facing (eg an internal migration or optimizations) then you might be ok, but that is a characteristic of the changes, not of the coupling of a service and it’s dependents.
IMO it’s more valuable to have strong contracts that allow for changes and backwards compatible usage, so that services that take a dependency can incrementally adopt new features.
That definition of strong coupling is in fact standard. Like if you ask people why they don't want strong coupling they tell you exactly that when you change a strongly coupled thing you induce bugs far away from the thing you changed, and that sucks.
Now you might want this strong coupling between front-end and back-end and that's OK—just version them together! (Always version strongly coupled things together. You should not be guessing about what versions are compatible with what other versions based on some sort of timestamp, instead just have a hash and invest a half-week of DevOps work to detecting whether you need to deploy it or not. Indeed, the idea of versioning a front-end separate from a back-end is somewhat of an abdication of domain-driven design, you are splitting one bounded context into two parts over what programming language they are written in—literally an implementation detail rather than a domain concern.)
Other patterns which give flexibility in this sense include:
- Subscription to events. An event is carefully defined as saying “This happened over here,” and receivers have to decide what that means to them. There's no reason the front-end can't send these to a back-end component, indeed that was the MVC definition of a controller.
- Netflix's “I’ll take anything you got” requests. The key here is saying, “I will display whatever I can display, but I'm not expecting I can display everything.”
- HATEOAS, which can function as a sort of dynamic capability discovery. “Tell me how to query you” and when the back-end downgrades the front-end automatically stops asking for the new functionality because it can see it no longer knows how.
- HTTP headers. I think people will probably think that I am being facetious here, what do HTTP headers have to do with anything. But actually the core of this protocol that we use, the true heart and soul of it, was always about content negotiation. Comsumers are always supposed to be stating upfront their capabilities, they are allowed a preflight OPTIONS request to interrogate the server’s capabilities before they reveal their own, servers always try to respond with something within those capabilities or else there are standard error codes to indicate that they can't. We literally live on top of a content negotiation protocol and most folks don't do content negotiation with it. But you can.
The key to most of these is encapsulation, the actual API request, whatever it is, it does not change its form over that 6 month period. In 12 months we will still be requesting all of these messages from these pub/sub topics, in 12 months’ time our HATEOAS entry point will still be such-and-so. Kind of any agreed starting point can work as a basis, the problem is purely that folks want to be able to revise the protocol with each backend release, which is fine but it forces coupling.
There's nothing wrong with strong coupling, if you are honest about it and version the things together so that you can test them together, and understand that if you are going to split the responsibilities between different teams than they will need to have regular meetings to communicate. That's fine, it's a valid choice. I don't see why people who are committing to microservices think that making these choices is okay, as long as you lie about what they are. That's not me saying that the choices are not okay, it's me saying that the self-deception is not okay.
I think strong versioning in event-driven arch is a must, to avoid strong coupling. Otherwise, it becomes even worse than "normal" call-driven service arch, because it's already plenty hard to find all of the receivers, and if they don't use strong versioning then it's so easy to break them all with one change in the sender.
Yeah I would tend to agree! I think there is freedom to do something like semver where you have breaking.nonbreaking.bugfix, which might be less “strong”... But in a world with time travel you tend to get surprised by rollbacks which is why they are always my go-to example. “I only added a field, that's non-breaking” well maybe, but the time reversed thing is deleting a field, are you going to make sure that's safe too?
And I think there's a case to be made for cleaning up handlers for old versions after a certain time in prod of course.
As fuel for thought, consider that when python was being upgraded from 2 to 3, shipping lots and lots and lots of new features, and breaking several old ones, there were many libraries which supported both versions.
Some of them may have functioned by only programming in the mutually agreed upon the subset, but given that that subset did not really include strings, that was clearly the exception rather than the norm. Instead people must have found a way to use the new features if they were available, but fall back to old ways of doing things if they weren't.
Some of those mechanisms we're internal to the language, “from __future__ import with_statement”. So how can my JSON-over-HTTP API, within that JSON, tell you what else is possible with the data you fetched?
Ugh sorry for weird autocorrect errors, I had to abandon for a mealtime haha... I was going to add that there are also mechanisms outside the language, for instance that you might just detect if you can do something by catching the exception if it doesn't work... If it's syntax like JS arrow functions this might require eval() or so. The API version is just to expect the failure codes that would have been present before you added the new feature, usually 404, and handle them elegantly. If it's a follow-up request that RFC-“must” be done for consistency, maybe your app needs a big shared event bucket where those can always go, so that when “POST /foo-service/foos/:fooId/bars/:barId/fix" fails, you can fall back to posting to the deferred request buffer something like
Just catch the exception, record the problem for later resolution, use a very generic template for storage that doesn't have to change every month...
Think creatively and you can come up with lots of ways to build robust software which has new features. Remember the time reversal of adding a feature is removing it, so you have to fix this anyway if you want to be able to clean up your codebase and remove old API endpoints.
I almost went down that road once, and got very good advice similar to the first couple of steps (plan, do your homework) in the original post here: "before you start making implementation choices to try to force certain things, try to really understand the architecture you want and why you want it, and ask yourself if it's a technological problem you're hitting or just a system design one"
Yep have seen this firsthand. Beware the evangelist who shows up bearing fancy new architectural patterns and who can't be bothered to understand nor discuss details of the existing system.
I have seen the resume padder on many occasions, and have had to clean up after they left. It's amazing to see a team become visibly happier with their day job when you move to sensible foundations based on sound reasoning.
The teams that were the most efficient, and happiest with their day to day were the ones which picked frameworks with good documentation, a plethora of online resources, and ignored the latest shiny toy on the market.
What's amazing is how often people have to see it to believe it.
One of my running jokes/not jokes is that one day I'm going to hire a personal trainer, but then instead of having them train me, I'm just going to ask them a million questions about how they convince people who have never been 'in shape' how good it's going to feel when they are.
Wouldn’t it be an engineer’s hell working at a place that employs based on resumes from resume driven developers?
You learn the latest shiny tech, then change jobs to a new company using your sparkling resume, self-selecting for companies that like the beautiful sparkles in your resume, then you get to work with other like-minded sparkle tech lovers.
At what companies do resume padders end up in?
Quote from elsewhere in thread: “Whatever is the new thing, jump on it hard and fast like your life depends on it.”
It's hard to find non-faang company not filled with cv padders (i mean who always votes for unnecessary but shiny frameworks and an additional architecture complexity).
People are motivated with money more than anything else, and a cv which looks good opens access to that money. So incentive to pad cv is very high, even at the expense of the system they build and their teams and companies.
>> fundamental problem is cargo cult developers trying to copy massive companies architectures as a startup. They fail to realize those companies only moved to microservices because they had no other option.
This.
Microservices add complexity; therefore, they should be avoided unless necessary. That's Engineering 101 folks.
same thing applies to Leetcode style interviews, no startup should be using them honestly. They are for established companies that have so many quality applicants they can afford to filter out great candidates