I am one of the biggest proponents of microservices. I helped build the platform at Netflix, I've literally traveled around the world extolling the virtues of microservices.
But I also advise a lot of startups, and you know what I tell them nearly every time?
Build a monolith.
It's so much easier to start out with one codebase and one database, and that will scale for a while. Especially if you use a key/value store like DynamoDB (although you will lose relational functionality that can be helpful at the start). And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services?
And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!). The one that may need to scale independently, or that you want to be able to deploy separately.
Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency.
"A while" is underselling it. As long as you have people who are half-decent with SQL, "just put Postgres on a big db server" will get you to 50 million row tables before you have to start thinking about even hiring a real DBA.
50M is something that can easily be done with just a dev who understands that sql is more than select, insert and update with the manual, google and chatgpt.
You can get really damn far with a fat postgres box.
The problem with these discussions is that you can always get a lot out of any given architecture, structure, db approach or whatnot. Can. There's just a lot of daylight between "can" and "likely will."
Ultimately, everything has its limitations and tradeoffs. If we respect them, it's generally a smooth ride. Problem is that we rarely do... within a company (startup or otherwise) under real conditions. There's also a dynamic where we build until the point where something stops us. Tech debt, complexity, over-engineering, under-engineering, feature bloat or antagonism between early decisions and current goals.
There's a self-regulating aspect to this. If architecture is spot on, perfect for the task at hand we can move faster to reach the point where it no longer is.
My point is that postgres is, compared to almost everything else, easy to get from "can" to "likely will" with just somebody with a brain, a manual and google. In absolute terms it of course depends, but the point is relative.
A brain, a manual and google is usually a pretty good way generally to make solid decisions, respect the limits of your chosen stack.
What happens when there is >1 brain involved... or when brainless, manualless decisions eventually get made... or two pivots from now...
I'm not disagreeing with your approach. I agree with it, especially as starting point. I'm cautioning that resilience against complexity isn't about how easy it is to make good decisions when you understand the spec, read the manual and calmly proceed. Complexity and fragility accumulate when one or all of these are absent. How easily you can (and thus inevitably will) make a mess... not how easily you can keep it clean.
IRL situations with regular rdbs, a very common trend seems to be long term drift between schema and spec. The flexibility and approachability of postgres often enables a lot of kludge eventually.
Data stores have this dichotomy between "look how easy" and "is limiting factor" that speaks to difficulties we don't know how to articulate or isolate.
Of course if there are lots of bozos in the startup then even getting to and maintaining 50 million rows is going to be very difficult.
But if you have a small group of folks that are at least as competent as the folks at WhatsApp pre-acquistion, then there really shouldn't be any doubt whatsoever.
> a small group of folks that are at least as competent as the folks at WhatsApp pre-acquistion
That's a success case... beware survivorship bias.
> if there are lots of bozos in the startup
No arguing that quality engineers are fundamental to quality engineering. That said... by this standard, there's no point in having this entire discussion. Every good db/store out there is good. They all work very well if used as they should be, with due respect to tradeoffs. Yet, almost everyone has db problems. Almost every one of these problems occur well within the technical limits of postgres or whatnot.
"It shouldn't be a problem" when it usually is irl is tunnel vision. There is an empirical reality disagreeing with you. Walking into it with "this shouldn't be a problem unless everyone is a moron" is bad strategy. If you can't think of reasons why architecture can and will become a problem, then just assume that you (or some of you, some of the time) are morons, and try to make it moron proof.
I've been on both the sysadmin, development, and hiring sides, and with data models at scales of 50M+ records, devs who "understand that sql is more than select [...]" are rare in my experience, as they're a cross between db admins and developers.
Administrating (in particular, query planning and production operations) databases with tables sizes with magnitude of 10M and more records is challenging, and requires a skill set that is very different from pure development.
One won't get "really damn far with a fat postgres box", unless they're doing very simple SELECTs, which is not the case with modern web apps.
> Administrating (in particular, query planning and production operations) databases with tables sizes with magnitude of 10M and more records is challenging, and requires a skill set that is very different from pure development.
Is it more or less challenging than the alternatives? Is it less challenging enough to add a new tech to your stack, add the required knowledge to the team, etc?
I mean, knowing "enough-to-perform-CRUD" SQL is table stakes for developing on the back-end, but knowing $CURRENT-FLAVOUR-NOSQL (of which there are multiple products, all with such substantial differences that there is no knowledge transfer between using them) isn't, so there's going to be ramp-up time for every dev, and then every dev that is added to the team.
I'm not disputing your argument, I'm just pointing out that, sometimes, it's easier and faster to upskill your PostgreSQL developer to "scale the DB" than it is to teach them how to properly use, maintain, architect and code for DynamoDB and others.
It's not just $CURRENT-FLAVOUR-NOSQL. It's also doing custom transactions and locking on top and/or thinking in terms of eventual consistency. It's so, so much more complex than just SELECT FOR UPDATE/BEGIN TRANSACTION.
It's not even funny how many software engineers just don't know that, SQL is crazy fast and performant, if you have basic understanding about it - I was once refactoring (or, rather, getting rid of) a microservice that was just a JSON blob storage on top of Postgres, without having any schema for blobs, with 100s of 1000s of them, no indices, and main complaint was - it's slow.
Unpopular opinion: if you're skimping on ops/DBA resources (as you may need to do in a startup), then MySQL is a better default. By all means use postgres if your use case demands it, but personally I find the ops story for MySQL takes less engineering overhead.
Yes, and most successful companies who started in the last ~20 years started ( and many continue ) with a monolith and a MySQL database.
Only the mega-cap ones started to pursue other options mostly due to their type of business and bucket-loads of "free" VC money with explicit orders to burn it and get "unicorn" status - which involves hiring thousands of developers in record time and the whole thing turns into a zoo. Which is an organizational problem, mostly not a tech one.
Other than the ones we pretend are the whole Universe, there are thousands and thousands of medium to big companies with billions of revenue who started their product with a monolith and a MySQL database and many still do just that.
I agree with this "unpopular opinion". Worked with both MySQL and postgres based mid-scale apps of several thousands of users. Postgres is so deeply lauded here at HN yet requires two more magnitudes of operations work to keep it up and running. Vacuuming sucks hard.
That is a very specific use case, and might only be a small subset of your actual data. If you don't have these specific requirements (eg. CRUD apps), you can save yourself a lot of unnecessary headaches by defaulting to MySQL.
My main point is attempting to counter the narrative popular on HN that postgres should be an automatic default. For sure there are many aspects in which postgres is superior, I absolutely do not debate that, especially when it comes to developer experience. But there is much more to it than that when it comes to delivering business value. That's where ops and DBA concerns start to matter, and IMO MySQL is so far ahead in this regard that it outweighs all the other hideous warts of working with it, when you consider the bigger picture of the business as a whole.
The problem isn't storing or inserting 50M rows, it querying 50M rows in non trivial ways. And the difference in performance between doing that 'right' and 'wrong' is orders of magnitude.
Eh, intelligent table design should knock most of that out. If you've got an 8 page query implementing a naïve solution to the knapsack problem (I've seen this in the wild) several mistakes have been made.
I've scaled 300M rows in just one of many similarly sized tables to 1M users... in 2007... on a single box with spinning rust in it. Heck, my laptop could handle 100x the production load.
It amazes me that my comment (while admittedly flippant) got voted down.
It really is true that your phone can update a 50M row table about 10K times per second!
That people are incredulous of this is in itself a stunning admission that developers these days don't have the faintest idea what computers can or cannot actually do.
Just run the numbers: 50M rows with a generous 1 KB per row is 50 GB. My iPhone has 1TB of flash storage that has a random access latency of something like 50 microseconds, which equates to 200K IOPS. An ordinary NVMe laptop SSD can now do 2M. Writing even 10K random locations every second is well within mobile device capability, with 50% headroom to "scale". At 1 KB per row, this is just 10 MB/s, which is hilariously low compared to the device peak throughput of easily a few GB/s.
It's not that good usually, e.g. PostgreSQL writes data in pages (8 KB by default), and changing 10K random rows in the 50M rows table can be quite close to the worst case of 1 changed page per changed row, so 8x of your estimate. Also need to multiply x2 to account for WAL writes. Also indexes. It's not hard to hit a throughput limit, especially with HDDs or networked storage. Although local SSDs are crazy fast indeed.
Agreed: 80MB/s for the random 8K page updates. However, transaction logs in modern databases are committed to disk in batches, and each log entry is smaller than a page size. So a nice round number would be 100 MB/s for both.[1]
For comparison, that's about 1 gigabit per second in the era of 200 Gbps networking becoming common. That's a small fraction of SSD write throughput of any modern device, mobile or not. Nobody in their right mind would use HDD storage if scaling was in any way a concern.
[1] Indexes add some overhead to this, obviously, but tend to be smaller than the underlying tables.
At several places, the biggest challenge to engineering velocity and the ability to innovate is that models are joined in interesting ways that prevent them from being decoupled. A user model tied to the package model preventing independent scaling of either. Every module reaching into all the tables. Joins, joins, and joins. Vast scans. The queries become inefficient and even the mighty postgres slows down at 2k rps. It is just too tempting to reach into another team's datastore. Now the two teams are in lockstep and can't alter their own datastore because others have assumptions on how it is stored and they access it directly.
At SendGrid, we had an few tables to deal with IP management. Over time, the like 5 tables were in use by 15+ services owned by different teams with competing priorities. When we finally had to scale the database, it took over three quarters of working with other teams while we supported both legacy and the new hotness.
It is hard to see early on where whole teams can sprout up and have domain ownership and to know which areas will be common/platform-like for other teams. But as soon as two services share a table, you should see a train coming at you.
There is severe hindsight bias in play here. How many tables weren't problematic despite all those joins and just chugged along scaling fine? How many startups were bogged down by unnecessary complexity or trying to keep a semblance of consistency instead of shipping features and just didn't reach the scale when performance becomes a problem?
Joins are good, actually, and normalisation should be the default. It's much easier to keep things consistent by default rather than trying to make them somewhat consistent afterwards. When something becomes a problem there is a bunch of tools (from denormalisation or materialised views to specialised datastores) to deal with that, but before that it's just premature optimisation.
And then the order table will need to set user ids as a foreign key and you will have two sources of truth lol
Will be fun when you decide that you want to have organizations to be linked to orders instead of users.
All this to say, data management and having more than one source of truth complicates your world a lot. The responsibility for writing the data has to be solved at the appropriate layer: only one entity writing it.
There is a way to splitting data. The service which owns the data always does the writes and others who need reads on that data can store replicas. ofc the complication then will be in replicating data, but this will enable services to massively scale and eliminate SPOFs
The other part of this pattern you mention (replicating data) seems solved in many cases by the data warehouse patterns of the last few years. Stuff it all in Snowflake or BigQuery then readers can query as they see fit. Query engines like Trino can paper over data storage heterogeneity / breakdowns in centralization. I'm not a fan of the "lakehouse" terminology, but it is the thing.
There are downsides to coupling more loosely with data consumers, but it keeps service owners moving without wasting time vending data.
> Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing.
No matter how hard you modularize and instrument your service, that won't give you regional deployments, which you absolutely need to have ti be able to drive down customer-facing latencies from many hundreds of milliseconds to a few tens of milliseconds.
How do you pull that off with a monolith when you have a global userbase?
Do the next simplest thing: shard geographically. If you really need to handle users transiting geos, that's still dramatically simpler than meegroserbusses.
Wise words. They boil down to a very simple truth that was as accurate half a century ago as it is today:
Make things as simple as possible, and as complex as necessary.
I can always make something more complex than it is now. As you say, we can take out things from a monolith and make them into a service. It can be hard to do so, sure. But nowhere near as hard, as trying to get complexity OUT of a system once it's in. Everyone who ever tried to revert a bunch of microservices back into a Monolith knows exactly what I am talking about. It usually amounts to the same work as a ground-up rebuild.
> It's so much easier to start out with one codebase and one database, and that will scale for a while.
Lots of businesses don't even need to scale anyway; Netflix is a "high customer, low revenue per customer" type of business, but there's lots of "low customer, high revenue per customer" businesses too, perhaps even more than the first one. These are often the type of products where you could quite literally run production on your laptop if you wanted to.
At the last place I worked they built all this microservice bonanza for ... 300 customers... If they ever got a few thousand customers it would be quite successful, and tens of thousands would be hugely successful. What usually happens is that they don't spend that "25% of your engineering time just maintaining the platform", so the platform was impossible to run locally, and production was hanging together with duct tape.
(Aside: in a recent interview I was asked to design a system for "500 million concurrent users" – this is for a business that's mostly in the low-customer/high-revenue type. I still don't know if the test was to point out that 500M is an utterly bonkers number – about a 10% of the internet connected people on the planet – or that they really thought this was somehow a vaguely realistic number for a startup. I answered with "build the simplest thing possible, a monolith, and focus on features and making sure you're building something people want". I didn't get hired, so I guess they were somehow under the misapprehension that you need to design things for 500M users right from the start?)
You’ve got a hell of a resume. And been accidentally incredibly convincing in getting many people into many early messes.
Jokes aside, thanks for at least coming around to advise startups sanely.
I’d personally never advocate for microservices until at the scale of Netflix or Amazon or Reddit, and even then only with in-house expertise at your level. Otherwise it’s a nightmare.
Thanks for everything, especially your contributions of sanity.
Aww, thanks for the kind words. I apologize if I caused you any harm with my talks. I did in fact start out saying everyone should use microservices, but I pulled back as I saw how damaging that can be to a small startup, or even a large enterprise that doesn't actually need it.
I wrote this 8 years ago: Microservices vs. "air-gapped" modules https://www.linkedin.com/pulse/maintainable-software-archite.... You can achieve the same end as a microservice using a module, by simply having lint rules that prevent you from importing other application level modules. This way the only possible comms interface is to pass events with a payload of primitive typed parameters.
An interesting idea. Sort of a good half way. But one issue I see is that you can still have a shared data store. That means you can accidentally (or intentionally) use the database to pass back-channel messages —- have one module store data and another read it.
That could lead to hard to find bugs. Did this ever come up for you?
The best "microservice" based system I've ever worked on was full ten years before the term had been invented. It had multiple services that read and wrote to the same database, but each used a common dynamically linked library that was the common interface to access that database (no naked SQL was allowed -- you had to #include <foo> and then do Foo* foo = Foo::findById("xxx"); etc. Worked remarkably well. The above common library was very carefully maintained and any addition had to be reviewed and approved. The approach seemed to work well.
That's the services model that Facebook uses and parts of Amazon use. A single data store surrounded by a bunch of small services with a prescribed data access layer.
That's really more of a monolith though. One of the key tenants of microservices is data separation -- each service has its own data store.
I think systems like this are prone to buckling under product pressure. If it takes a few minutes to override a lint rule it's bound to happen sooner or later. With a microservice system the amount of effort is genuinely higher to add cross-dependencies.
I think if we also "air-gap" (loosely) the dependencies, we get a typical monorepo in, say, JS or Golang? That is, a module in a monorepo is a special case of your airgapped modules?
Not exactly sure what you mean, but in my model, the modules all get built into a single runtime artifact/executable. In the case of monorepos as I understand them, the "mono"ness is in the version control system, but each code base builds to its own artifact or executable, which have to communicate with each other over a network interface.
I agree - microservices are a technical solution to a people problem. Stevey's seminal "Platforms Rant" (https://gist.github.com/chitchcock/1281611) touches on this - the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way.
Splitting into services siloed each team and allowed them to move independently and by side effect, faster. Not due to inherent properties of (micro) services but because one goes faster by removing the things that slow you down.
As a startup, you do not have this problem. You likely will _never_ have this problem as your default future state is 'dead'.
Do the simplest thing that can possibly work - build a monolith using tools you know that give you the ability to predictably, rapidly iterate on the product.
After you hit some semblance of product/market fit and need to scale, you can do that. Scaling is a solved problem. Premature scaling is not.
[1. This is a joke. Kubernetes wasn't even a thought in Google's eye at this point in history]*
> the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way.
Not quite. Amazon's problem was that they were experiencing too much impedance between teams, and some teams were even siloing themselves to the extent they were creating problems for everyone around them. Bezos' diktat was intended to break through a bunch of petty office politics bullshit that was dragging down the company and preventing teams from delivering their work. The diktat boiled down to basically ordering everyone to grant access to the service they owned and provided to external teams that need it, no exception, and would be held liable if they failed to provide it, intentionally or not
Most people don't understand the point of microservices. They look at the idea and are attracted by the power of modular interfaces, customizable scalability and independent deployment, without really thinking of the additional engineering overhead that are required to unleash these capabilities.
I've seen ex-Netflix software engineers taking jobs elsewhere and proposing microservices for systems that would receive little to no benefit from them. In practice, the implementation of microservices in these contexts become a costly solution looking for a problem.
Just chipping in with thoughts on DynamoDB, (although I have worked on much smaller scale systems)
I am a long term dev, done lots of SQL, but for the past few years I have been using DynamoDB, and I am using it for my new startup (So I rate it).
Cons
- You have to be very aware of your query patterns, and not having ad-hoc queries is a pain.
Plus sides
- With on demand billing, its free if you aren't using it
- Built correctly, it will scale
- No Schema upgrades (This one is massive for me)
On the last point, I really do appreciate not having to worry about keeping schemas upto date across all devs and environments.
We use quite a simple pattern of a table per entity, as opposed to single table design, because it allows us to just use the API at the highest level of abstraction, where you just write your objects to dynamoDB.
(You can still do lower level requests to poke values and such like)
At Amazon, relational databases are banned unless you get an explicit exemption from senior leadership. This is the primary reason why. Too many cases of schema upgrades causing outages.
The problem with DDB or other NoSQL applications, like you say, is how much you need to consider your query patterns. The last major project I worked on using DDB, we spent a couple of days just thinking through our query patterns so we could come up with the right database design and data structures. (We still believe it was the right choice, though.)
>Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency.
My take is, microservices multiply your operational problems so you need a really solid platform/infrastructure/developer experience team otherwise you'll introduce a bunch of new headaches. These include things like CI/CD (building, storing artifacts, testing artifacts, promoting artifacts between environments), observability (metrics, logs, distributed tracing, error monitoring), distributed systems quirks like cascading failures, release/change management and communication, service discovery/routing, automated infra provisioning for things like datastores. Unless you have a really good handle on all these pieces (which I think most startups don't), you end up in an operational nightmare spending tons of time trying to keep the system going.
Of course, the next step is throwing on 3rd party products to try to solve this which adds even more complexity. Throw in Datadog for observability, Istio service mesh for networking/traffic, ArgoCD/kustomize/Helm/Kubernetes to manage all this infra, etc etc
How would you split the total overhead between monolith and monorepo?
(sorry this ran away with me)
A dumb example is that if I start with my single codebase running on a single server, I am likely to have a single git repo (foo).
Then I have a genius idea and put all the email handling code into foo.mail and soon I have foo.web and foo.payments.
All is fine as long as I am just checking out HEAD each time. The code running in the runtime is still one big set of code.
If I get creative and put in load balancers it's still a monolith.
But if I split out the web servers from the email servers, then I kind of start to see microservices appear.
I am trying not to be pedantic, but I am truly interested in experienced views on where the pain really starts to appear.
At this point (server to server comms), I should look at mTLS, and centralised logging and all the good stuff to manage microservices.
But how much pain was there before?
What if I was a large company and so hired a dev or two per repo (you know to get that 9 women one month effect). Co-ordinating multiple devs over different repos, even with one monolithic runtime, is painful (experience tells me).
So I am interested in where the break points are, and whether there are easier paths up the mountain?
Splitting out the webserver (assuming it is the main entrypoint for users) seems more like an infrastructure choice than application architecture and having an independent email-sending-service looks more like replacing a third-party offer (like turboSMTP) with an in-house service.
I do not think that this is what people mean by microservices.
I get that. I was trying to describe a small path from monolith to microservice without inventing yet another student and teacher database. But the even the step you mention matters. going to a third party service is very similar to reaching out to your own internal service. Make it a "customer" service.
Is the pain in microservices (or APIs and mTLS) or is the pain in managing a team that now has to do customer stuff only ? or email stuff only?
Common path to pain right there — services based on nouns and entities rather than verbs and business processes.
Having a single "customer" service means anything involving customer data has to talk to that single service. If that service has a problem, every other service stops working. You basically have a tightly coupled distributed monolith, so network boundaries instead of function calls between modules.
Think instead of splitting that data to separate services around business processes like billing, shipping, outbound marketing, whatever. Each service owns the slice of customer data required for its processes and can run fully autonomously.
At Netflix we had both noun and verb services. We had the "play movie" service, which was very much verb, as well as the listing service for example. But we also had the subscriber service, which was the source of truth for any account data, and the catalog service, which held all the info about the movies.
Yes, subscriber was a Tier 0 service and required a strong reliability and scaling story. They had more rigorous deployment setup and were usually over-scaled for traffic. But they also vended a library that services used to connect to them. That library had intelligent caching and fallbacks, so that even if the service was down, in a lot of cases the system could keep running. And the catalog was the same.
Microservices is very intertwined with how your engineering teams are built. We had a team that worked on subscriber data, one that worked on catalog data, one that worked on making movies play, one that built listings. So we had services to match.
Thank you. I sometimes feel microservices are an evolution of Conways law. How a company sees its internal organisation is how it will want to arrange its microservices and vice versa.
I remember trying to build out a "barium bullet" through very complex intertwined systems - following known customer data and looking for their footprints in all linked systems. It gave some insights on how to redesign systems - like someone else said here, removing complexity is ridiculously hard.
That seems like a great approach for Netflix and investing in a resilient, redundant Tier 0 service like that was likely worth it there
Most orgs don't operate on that scale though and resources would likely be better spent elsewhere
> Microservices is very intertwined with how your engineering teams are built.
I wish more people understood this. Microservices are much more about scaling your organization than about scaling technology. At a certain size, teams can't collaborate effectively enough and splitting out different services for each team makes a lot of sense
…until you need to market to people who didn't get a shipping last month, delay shipping until at least one recharge attempt, or ship before charging for customers you're reasonably sure will pay anyway.
Business changes, and putting rigid firewalls in place without seeing an established pattern of change and without established need for that is a bad idea.
One global database buys you a lot (easy data access, easy transaction consistency, simple locks, even transactional notifications), and you can take it very far performance wise, throwing that away just in case you might need more performance later seems unwise.
> …until you need to market to people who didn't get a shipping last month, delay shipping until at least one recharge attempt, or ship before charging for customers you're reasonably sure will pay anyway.
These are all very simple things to handle when services encode processes instead of entities
> Business changes
Yes, and then you throw away the code for that outdated process and encode the new one.
> One global database buys you a lot
Absolutely, this is the correct choice for probably 95+% of companies. I will always argue in favour of a monolith and single database.
If someone is dead set on microservices though, they'll avoid a world of pain by actually designing them properly
I honestly believe that doing small services is long term better approach than a monolith. But as you I strongly believe it's not where you want to start.
But I do not agree on is more time consuming than the other. It's just time spent on different matters. For the sake of money spent, I have not seen any data on how a monolith outperforms small services. For maintaining the platform, I think 25% sounds a bit high.
Money spent should be measured in many different aspects. One is definitely productivity. And I have seen more stale monoliths than I have seen stale small services. Whether it's better or not is not up to me to judge. I just know what I prefer.
Having been exposed to small service architecture where it has been working really well and very poor, there are few things that stand out.
- Conquer and divide (your monolith over time).
- Responsibility boundaries are easier to cope with for developers in a small service since they often don't have generic and yagni abstractions.
- Less code to comprehend for a small service and the cognitive load decreases.
- Build time, test time, deployment time.
- Should lean up against a direct business measurement and value.
- Group chatting services into one.
I have yet to see a monolith that over time is not really deteriorating, but I haven't worked with SO or Shopify. We could also argue that Cobol and Fortran is still of good use but time has also changed leaving that style of system development exposed as old and dusty.
But like any software development occurrence, it takes responsibility, mandate and proper leadership to get things done in a decent manner. So if you start by making 3 services that has to chat to find a user profile you probably don't know what you're doing. And 25 people (the Threads image) is not a small team IMO.
The issue isn't blindly adopting microservices or a monolith as a religion or bikeshedding thereabouts.
Architectural and ops concerns lead to dividing an overall service into sufficient and necessary units of abstraction. To efficiently run more than 1 service leads to standardization of automation of the concerns of application platform infrastructure:
- stateful data backup, protection recovery
- configuration management
- OS security
- authentication, authorization, audit, encryption, and identity management
- monitoring
- analytics
- rate limiting
- A/B, etc. feature tests
- sharded deployment
Chopping up 1 service into many more services doesn't make the above concerns go away. Neither does collapsing many into 1.
The internal pieces (and often interfaces between business units) need to be broken down into decoupled units of abstraction. If people want to call that "microservices" or a "monolith", it's kind of irrelevant. Containers of abstraction should serve a purpose rather than hinder it.
> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!).
Wasn't this the rule of thumb for microservices from the very start? That the main driving force is to ensure separate teams work independently on stand-alone projects they own exclusively? Scaling tends to only become an issue when the project grows considerably.
I don't think so. One of the very first things that people learn when onboarding into microservices is the microservices tax, whose corollary is don't onboard onto microservices if there isn't a clear significant win that justifies all the tradeoffs.
Moreover, the main selling point of microservices is organizational. That's also right in the first lesson on microservices. If you're not working in a large team who owns a large project and wants to lower managerial overhead by splitting up the team, you need to try very hard to find a justification to peel off specific responsibilities from a service into an independent microservice. That's microservices 101.
Those who jump onto the microservicss bandwagon without having a concrete justification have only themselves to blame, and not any particular architectural style. They are making the piss poor decisions and blaming abstract concepts for their own mistakes.
> break out an independently scalable part of the system into a microservice … The one that may need to scale independently, or that you want to be able to deploy separately.
How often would you estimate that security concerns are a valid reason to isolate functionality into a microservice? Separation of concerns, reduced attack surface and blast radius, etc
Wise. Every time I’ve started a project trying to make an amazing platform with everything segregated with micro services etc. I’ve spent so much more time on the _platform_ than on the actual product.
Now I go with SQLite and some basic python script and pivot from there. Ironically that’s how I used to do it before the micro service fad.
It depends on your programming language and how many libraries you are importing. If you use Python you can get startup times less than 200ms. Also, if your app is active, most startups won't be cold (Lambda reuses the VM a bunch of times before recycling it if it's staying active). You can also add a health check API and just poke that once a minute to keep it warm if you want (at the cost of increased invocations).
But yes, if you're doing something that requires a lot of setup, then this pattern isn't for you. But it turns out most web apps are simple enough it can be done this way, especially if you're already using a pattern where the frontend is doing a lot of the heavy lifting and your API is mostly just marshaling data from the client to the database with some security and small logic in between.
Absolutely agree, I've seen far too many companies spend far too much time working out the infrastructural relationships of microservices. Spending very little time working on the actual applications needs itself.
The problem is not with the devs but with the investors I believe. They're looking for unicorns, unicorns attract million of customers, so they need to work at scale, hence microservices.
Well, in the previous cycle it was clear that the investors were looking for buzzwords and marketing hype because they often don't understand and frankly dont care - they just wanted to make you look good enough to sell to the next guy, as they were all assuming you were unprofitable in and unprofitable out.
They all just want a story that you are going to be the next google so they can sell that story to the next guy.
Will that hold in the non zero interest rate world? The AI hype doesn't make me think the world has changed that much.
As an example, you'd write your entire API as a flask app, and then deploy that app to Lambda. Then send all requests to that one Lambda. As long as your startup time is quick (and your datastore is elsewhere, like in DynamoDB) then it will work great for quite a while. Lambda will basically run your app enough times to handle all the requests.
You have to be careful when you design it such that you don't rely on subsequent requests coming to the same machine, but you can also design it so that if they do come to the same machine it will work, using a tiered cache.
I work on a massive monolith that has about 800 contributors and its just as complex to add something as simple as a user’s birthday, just not all the complexity is technological. It requires “organizational alignment” since you’re touching everyone’s code.
There will be endless iterations on design and review. Sign off required by at least 2 architects. It will get added to multiple planning iterations. The actual code will take an afternoon or less. We’ll have to ensure we hit out 90% test coverage during code review but because of all the tests, it’ll be too big for one PR so it will need to be broken up into multiple PRs probably landing over multiple weekly releases. To facilitate that, we’ll put it behind a feature flag (of which there are currently 13,000). Once it hits production, and the dashboards/monitoring are put in place, it will get enabled and disabled over and over again as we’re not totally sure why our birthdate feature broke the metering service, but we think that’s the root cause—need to do a few weeks of analysis.
Then, finally, in a year the engineer who was tasked with it will get a good performance rating, maybe be up for a promo! Just in time for him to jump into another project that’s failing horribly headed into its 3rd year in development: Allowing the user to set their timezone.
This comment shows IMO that the real issue here is not really microservices or not microservices, but what the article calls "The apostles of the Church of Complexity".
Neither microservices or monolith are a golden hammer, silver bullet or whatever (nor the opposite). They are tools each with their tradeoffs, which combine with the many context-dependent tradeoffs of each organization. They are not the problem.
The problem is a) engineer's fascination with complexity, and confusing "simple" with "hack", b) how organizations keep cargo-culting tools, architectural patterns, and so forth. Applying architectures, design patterns, whatever naively, based on the belief that usage alone will deliver benefits. It doesn't.
I don’t think complexity is usually intentional though. Really, writing complex code is easy. It’s writing simple code that’s hard.
I’m in the process of wrapping up a reactor of one of our systems that I wrote a few years ago. Largely the refactor has been an exercise in just simplifying everything. The only reason I can effectively do this now is because we have a much better idea of our use case and how the entire system fits together. At the time, the complicity seemed necessary, though I’m not quite sure why. I think I just didn’t really understand what exactly this was really going to do when we were building it.
I think this probably applies to companies growing quickly as well. Beyond core product, there’s a lot of things you can do, but what benefit does it bring? How does it fit into overall strategy? It’s easier to build something if you just carve off your own little area to ignore the rest of the system and assume you’ll eventually understand better how of make it integrate nicely or add value.
Some people who advocate for huge monorepos, like Google has, tend to forget that Google has whole teams building tools just to wrangle the huge singular codebase for refactoring and testing.
It was a mess before, but the mess wasn’t visible. Now the code is all visible to the IDE, it all compiles together, and tests together. You can see the mess and fix it. It’s possible to see enough of it to be able to change it and when you do the build stays green because you can make atomic changes.
It’s less about why monorepos are good, and more about why on Earth would you split a product up over hard repo / tooling boundaries. Perhaps a Big Brain could get those boundaries right. I can’t,
so when I need to move the boundary I would rather it was done by editing code inside a repo than coordinating changes between micro products.
I’m obviously describing one scenario. What are the counter examples for where hard polyrepo boundaries are helpful? When your team are all IC1s?
The trouble is, you're often left unwinding a half-dozen layers of abstraction, and in many cases experiencing what looks like some sort of quantum code interaction, where changes in one part of the codebase seem to impact completely unrelated bits of code which now fail tests.
The polyrepo + microservices approach helps enforce boundaries. It also makes rapid iteration easier with more limited tooling, because my code can no longer blow up someone else's code -- a problem which can be solved with dev toolchains, but those toolchains take a non-trivial amount of resources to support.
I do agree, when you need to move boundaries, or need to refactor across multiple packages at once, a monorepo + monolith is great. (A monorepo + microservices can introduce many of the same problems as a polyrepo + microservices, though.)
> The trouble is, you're often left unwinding a half-dozen layers of abstraction, and in many cases experiencing what looks like some sort of quantum code interaction, where changes in one part of the codebase seem to impact completely unrelated bits of code which now fail tests.
This perfectly sums up all of the microservice apps that I’ve had the pleasure of working on, with the exception of issues being caught by tests. Typically you get either silent production data corruption or a seemingly unrelated error in the UI x weeks down the line.
People seem to think Google’s monorepo is amazing, but I was often blocked from deploying anything because some team I never heard of submitted a changelist that broke the world. We should be using known-good dependencies rather than alpha testing the bleeding edge.
At $work i see lots of projects leaving the monorepo due to that very mess.
The problem is that services outside have other CI and maintenance issues that wouldn’t happen in the monorepo. But monorepo got so messy that it got unbearable.
The proper solution would be to have a dev teams dedicated to tooling for it, the org has the resources, but it’s just not investing enough in that area.
It sounds like this company doesn't have a competitive need to ship code changes, but when it does (i.e. a challenger appears or the moat disappears) it'll be trouble.
I've tried--with varying levels of success--to really take to heart that software engineering is a never ending battle against complexity. You have to do it all the time and it has to be your paramount value, otherwise stuff like this happens. I don't think there's an architecture or ideology that ends the war; this is just the nature of the job.
Having 13,000 feature flags is just insane. I'm so happy that at my previous job I insisted that we use "release" flags for every new feature were there wasn't an explicit requirement to be able to enable it for some customers and disable it for other ones. We made it so that you couldn't enable one of these release flags in production, and developers were required to remove the code for the flag before final tests and release. It requires a little more work for every feature which requires a flag, but it really pays off in terms of reduced code base complexity in just a few months.
It is indeed insane, but there is some value to leaving coarse feature flags in production. I've seen outages resolved by flipping a feature flag that was accidentally left in production months after deployment, where without it we would've had to resort to more drastic or hackish measures to restore service. (Eg. If one query is overloading the database, you can disable the feature that runs the query and investigate the real problem at your leisure while the database stabilizes.)
That only works in moderation, though - with 13k flags, there's no way the particular combination that you're about to release directly on production has been properly tested.
You (correctly) assumed that it's a monorepo, by the way. But yeah, you're being reductive overall. It's not a "microservices vs monolith" debate. I was using this example to show that people can fuck up any model.
OMG this sounds painful ;) both in processes and state of the codebase
I can ship something in a couple of hours on a monolith where 3000+ engs work and deploy every day, but to be fair that's not what I've seen in any other company I worked at.
I think this is a great point, and shouldn't be just hand-waved away like you hit some bizarre edge case of Monoliths.
Monoliths can have crippling downsides -- just different flavours of downsides from Microservices. What you gain in network latency and DRYness, you can lose in Autonomy and Breadth of codebase.
Microservices vs Monoliths - just like everything else - is a question of tradeoffs, and making informed choices about when to apply them, and how to mitigate their downsides.
The slightly more nuanced point in the OP's article is that adopting any engineering practice and blindly following as though your identity is linked to it, is a bad move.
Microservices are a solution to a social problem, not a technical one.
A team of N engineers requires N² coordination. Large teams get mired in endless meetings, email, design reviews. Small teams are more effective, but struggle to maintain large systems.
Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
Yes, microservices add complexity and overhead, but this approach enables a large organization to build and iterate on large systems quickly.
> Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
This does not happen at all. When you break a system into subsystems, all the previous connections that get remapped to new connections between subsystems still need to happen, in order to solve the fundamental problem that the system solves — except now instead of just making the connection directly, there has to be a "cross-functional" meeting between teams and a complicated communication layer between the systems. And if somehow you find a breakdown that requires minimal connections between subsystems, then those connections wouldn't have existed in the original system either, and the N² problem doesn't exist.
It's all fun & games until product wants to add a feature that doesn't map cleanly to your micro servies architecture. Then you end up hard coding you services into a macrolith. Good times.
If that's the experience then you're doing services wrong. Each service should have its own datastore and a single API. The interface between services should be a single connection.
There should be maybe one meeting where the caller defines what they need the service to return to them.
The problem with this is that you have to be really damned careful how you split things up. If your separate data stores end up having to be joined together later on because some new business feature requires them to be cross-referenced, you're painted into one of two corners:
1. Merge the two services (and their data stores) into one and cause havoc downstream of either service
2. Burn through your network latency/throughput budget trying to reinvent a DB join across RPC boundaries (and god forbid if you can't batch multiple lookups in a single API call!)
Exactly. Software developers will never learn that the separation of concerns is a myth. In reality, UI, business logic and data are deeply entangled. Hence, moving these things apart makes everything worse.
I do not like appeal to authority either. But based on my experience, most of these issues were resolved in one to two meetings (I said maybe one, after all).
The team that needs data goes to the team that has that data and says "can you make an API that vends the data that I need?". Then they make it. And then maybe there is another meeting with a follow up, or perhaps an email asking for the data to be presented slightly differently.
But if you come to a team with a good use case and some examples, it shouldn't be too hard for them to create an API that vends the data appropriately.
This of course is all predicated on a good microservices architecture where you already have systems in place to handle API discovery and communication between services and proper data separation.
your "argument" was literally just telling the person that they were naive for thinking things could ever be handled by one meeting.
turns out, they inarguably have a whole lot of experience at high levels in the biggest companies in the world. Either they're straight up lying, or it is in fact possible to resolve things with a single meeting. Why wouldn't that be the case?
I can tell you many other wisdoms: code should be tested, code shouldn't contain bugs, everything should be done before deadline, etc.
The thing is in reality it's not that easy.
> Either they're straight up lying, or it is in fact possible to resolve things with a single meeting.
Something can be done in a single meeting but not everything. But you gave me only two alternatives without anything between so there is another useful link for you https://en.m.wikipedia.org/wiki/False_dilemma
oh sweet summer child; have you not heard of the fallacy fallacy?
snark aside, just because a logical fallacy exists in an argument, it does not mean the argument is untrue. My experience echoes jedburg's - services should maintain their own datastore and provide api access. This allows them to be decoupled and allows the service to alter their datastore as needed. When multiple services leverage the same underlying datastore, things can get overly coupled. This becomes increasingly important as organizations grow because the coupling means more overhead communicating and aligning around changes.
Jedburg is an expert because he has seen this happen in multiple companies. I have too, but I don't have the speaking/consulting breadth of experience that they have. Credentialing and appeals to authority work because we are squishy humans and we can't verify everything so we lean in on others. That does not mean that authorities are always right, but when their experience and advice mirrors other's experience and advice, there is something to it.
And for the record, in public companies (two of them), I have exactly seen this play out: a well defined api that does not allow others to muddle in their datastore leading to _single_ meetings where new functionality is discussed and agreed upon and later implemented. I have also experienced the pain of multiple teams sharing a datastore when the datastore now needs to be altered which led to dozens of meetings across several teams and a rollout plan that was measured in months and quarters due to coordination needs. I can firmly say that the latter is a much harder (and expensive!) problem to solve in each case I've experienced.
> Of all the things wrong with micro-services this isn't one of them.
It is, exacerbated by the over-use.
> The "complicated" communication layer is always either REST/JSON or GRPC.
Going over network, which is slower, unreliable, poorly typed. It's orders of magnitude more difficult to refactor a published REST API in comparison to e.g. Java interface within a monolith.
Microservices are generally far from easy to debug. In the best case scenario you have the whole stack locally and can put breakpoints, add logging immediately. But that happens rarely, debug port is often blocked (security), you can't easily modify the code to add some diagnostics without a complex CI dance etc.
But I've never worked on a monolith project that I could run entirely locally.
And this idea that APIs are slow, unreliable and unable be to be strongly typed is nonsense. This is 2023. We have plenty of tooling and techniques to make this robust.
The (monolith) product I currently work on has about 200 engineers working on it (backend + frontend + devops). I'd say it's of medium size, certainly not small. And yes, we can run it locally easily. The monolith starts up in like 20 seconds which I consider quite acceptable.
There are few narrowly defined responsibilities which are handled by dedicated services, and it is often more awkward to get them working locally. But because they are so specialized, you need them only rarely.
> And this idea that APIs are slow, unreliable and unable be to be strongly typed is nonsense.
Network APIs are inherently slower and more unreliable in comparison to an in-process method call. Some typing solutions are there, but they are way more awkward (and in some ways weaker) than statically typed monolith interfaces.
For a monolith it's having to run the entire platform locally to fix one bug.
That includes databases, caching, auth, ML models, mock API endpoints etc all pre-populated with sample data then the actual application which for complex, JVM based ones can often fail to launch if there isn't enough memory to pre-allocate.
Many systems I have worked on all of that would simply not fit on a 16GB MBP.
I've never seen teams organized around microservices like that. What I've seen, again and again, is one huge team where "everyone is responsible for all the microservices" (meaning, no-one is responsible for anything).
On a theory level I would agree with you - I've just never seen that happen in practice.
I'm not that much of a supporter of microservices, but my experience is the opposite: in every company I've worked for that used microservices, each team had their own set of microservices they were responsible for.
It should be like that but in practice sometimes other team just too busy to implement required feature. I can say my PM that we have to wait for a month but in reality I will have to implement it myself.
The complexities of software development could be solved with this one weird trick - if only programmers remembered FOLDERS. What an absurd thing to suggest.
Sure teams in large organizations allow other teams to randomly create folders? Also I've rarely seen an internal code base that is easy to grasp. You can create your own microservice and set up an API in 1/5th of the time it takes to understand a foreign code base and make a small adjustment.
I'm honestly not even super convinced that small teams struggle to maintain large systems. I've been on a team that was only 7 good engineers maintaining a 3.5 million line project that had both a web UI and thick client. It supported 2 different databases and had a horizontally scalable job runner.
At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done. There was just so much less time spent keeping everyone aligned. So many fewer meetings, sign-offs, reviews, plannings, retrospectives, management meetings, etc. Developers had a lot more agency, so they just got stuff done. Technical debt repayment became 50% of our time, as we easily knocked out features in the other half of the time. We kept ruthlessly cutting complexity, so it got faster to add new features.
I'm sure some projects just need more bodies, but I think there's an upper bound to how much complexity can be added to a system in a given unit of time. Adding developers over a threshold will result in the same amount of features per week, just everyone does a little less and spends a little more time on communication.
Repeat up to thousands of developers where adding a single field takes months.
>At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done.
Years ago I did two back to back contracts for two different pharmaceutical companies. They were both about the same size but one had an IT group that was ten times the size of the other. You can guess which project was late and painful.
The problem microservices try (tried?) to solve isn't about namespaces, it is about too tight coupling between code. Whether that tightly coupled code sits in a subdirectory or in a different repo doesn't matter.
It can be benefitial to maintain well defined interfaces at boundaries between certain parts of your code. It can also produce a lot of work and add complexity. But beyond a certain scale adding systemic boundaries and honoring them isn't something you should avoid.
Devs who do microservices just tend to go too far too early.
>The problem microservices try (tried?) to solve isn't about namespaces, it is about too tight coupling between code.
You can use folders to decouple code by making code private within that folder. Only publicize parts of the code with your languages version of exporting.
Or you can make it private behind an entire new repo and behind an entire service. Only publicize parts of the code through api interfaces like http and make everything 10x harder for the sake of decoupling code.
Many languages misses that abstraction to make things private in a folder (namespace) hierarchy to outside consumers. This is something I like to see solution to (integrated with your IDE of choice)
Usually the most important designation of the separate system is that it uses a separate database. You only have immediate consistency within one service.
At the point where you have to write an API call instead of simply joining to a table, you have introduced orders of magnitude more complexity, failure points, testing challenges, race conditions, coherency issues, code duplication etc into your system.
I don't mind stateless microservices too much. This forms more of a hub and spoke model (many services talking to 1 database). But the minute you bring separate state into the equation it's a chaos factory.
Joining a table is nice until that table is owned by a separate team. Becomes either gridlock or constant breakage, performance can get precarious, and testing involves populating a whole fake DB vs just mocking a few RPCs.
I really like relational DBs. I'll make a system rely heavily on complex joins. But I'd rather call some other team's API than share DBs with them.
Nah but there are real instances where you have to use a different database for performance profiles.
The biggest one in web is regular entity databases vs. timeseries databases designed for analytics.
For this case entity data is just synced to to time series database every so often. Not super chaotic. It does make sense to divide services along those lines.
Though I would argue a folder still works for the web app part.
If you have centralized data then you have broken the abstraction and are in for a world of hurt. You shouldn't need centralized data, just a service front door where that service is the master of that part of the data.
Each service is authoritative on its own data, and it works neatly. Any medium to large size system is going to have natural places to separate that, or else will buckle under complexity or even hardware constraints if you try to keep it all in one DB. I've seen it repeatedly.
Nah this doesn't work out. Because requirements are so chaotic and modular design isn't an axiomatic science things will for sure go wrong with the design both because you got the design wrong and because the requirements shifted to a point where another design was better.
You encapsulated data under one service and you find that it's actually utilized much more in another service. This happens a lot.
The longer you can centralize your storage the better and easier everything is.
Except I've seen this work at dozens of companies?
This thread is very painful, feels like CS101 coming in to tell us why we don't need relational databases and that we can just shove it all inside mongo.
It's not the thread it's most likely you. I don't know if people tell you but dealing with you is likely to be extremely painful if you say rude garbage like this to peoples faces.
Multiple databases can "work" but you get the problem I described above.
You know when you go to a company and eventually you see idiot practices or technical debt? Unfortunately this belongs to that category. You seeing it at dozens of companies doesn't validate anything except for the fact that bad patterns are common.
It also validates the existence of people who often do things poorly and don't know it (aka you).
Multiple DBs does work, and large orgs do it this way with good results. But I'm not here to insult people for disagreeing. It's just software, nothing to get heated over.
No worries. You didn't call the thread painful. That kind of comment causes heat. It doesn't belong here.
Of course it "works". But you can do better.
Splitting data into several dbs when one db suffices is called "extra work". Good results can arise from "extra work". But the extra work wasn't necessary.
You have to think like this: Sure multiple dbs work. But would 1 db in it's place work just as well?
The top commenter here from Netflix one of the people who promoted the whole microservices thing is telling startups to go monolith for a reason.
Our large org has been familiar with the 1 DB approach for over a decade. At some point the need for microservices became clear, and years after that, they actually switched. Even our subteam is feeling the need to divide up our service because its scope has kept expanding.
Startups shouldn't start with microservices. I've worked for (or led) a few different ones, and I've always created a monolith while avoiding other kinds of premature scaling (sometimes even "folder, bro" is overkill). Doesn't mean I'd run things that way with 200 people owning systems with evolving requirements for 10+ years, or even three separate dev teams of 5 people.
Right. But you likely split the db for performance not just for decoupling data and code. Microservices is an option for scaling. My argument is more against it as a way to decouple and organize logic.
Performance wasn't the issue; mono DB was actually fast enough. Multiple teams' services sharing tables became effectively a fragile API between them with hidden dependencies. Small new business requirements were causing too many breaking changes, and the org relied too heavily on a few experts who understood it all. Plus, schema changes were behind a ton of red tape since they affected everyone.
Also if by "decoupling data and code" you mean decoupling data from code, that's not the goal. It's to decouple one team's data from another team's data, and to properly abstract it.
The only reason our mono-DB even lasted this long was because previously, very few requirements were changing over time. So when people say their big org is monolithic, I can believe it. It works in some scenarios, but probably not most.
I know, but our DBMS doesn't support schemas. Instead, it's easy to spin up more "databases" that all share one instance but have separate ACLs, quota, backups, etc. Postgres has something similar but with less isolation. It doesn't really matter how exactly you slice it, point is you aren't sharing tables.
Exactly. And having more than one source of truth for your data complicates a lot of things.
Where I work we run a massive MySQL database that has a pretty simple sharding rule, so I don't believe that you need to split the database even at scale.
It's not primarily about the size of the DB, it's about what's in it. If your massive MySQL DB is only used by one team, then that's fine. If separate teams' services are speaking through shared tables on it, that becomes a mess of an API easily.
We have thousands of developers on it. I’m not saying it’s a silver bullet, but it works and it works well. Also, the service writing on it is only one (monolith).
Not really. You can't really reduce the blast radius of crashes or bad deployments. You need to have the discipline of a good CI/CD instead of siloed but decoupled workflows.
Just keeping things neat doesn't go nearly as far as a separate process on separate machines. Monolith might be better but I don't think it's a situation where you can have it all.
This argument always confuses me. It depends on what you’re doing but if you’re doing web services, as most here are, the crash is limited to the request being served.
The blast radius is a single request, right?
In every likelihood it _is_ a separate process on a separate machine.
If schema migrations take non-zero time and you want zero-downtime deployments, then the both service versions need to be compatible with either the new schema or the old schema, regardless of service size.
The software should be compatible with both the old and new schema until all of your database servers have moved over to the new schema. All rollouts are going to be somewhat staggered even if you go straight to 100%.
The same problem exists in microservoce land, only you now have a load of separate teams managing their own database, perhaps with different solutions.
I think you misunderstood what I said, which is you still have databases and ergo need to manage migrations and such, only that this now typically falls to the specific teams.
That said, loads of folks do microservices with a shared dB.
Even for a web service, imagine a small feature is causing the app to crash to OOM, or too many file handles or something that will take out the whole process and not just the one request.
If you have long running requests (something beside small rest calls like large data transfers) then any being served by that same process are taken down.
Yes, only it takes a while to figure out the right level of abstraction for your organization. It's difficult to start right out of the box with the right type of modularization.
FT.com did a recorded seminar session on their microservices architecture and one of the benefits they extolled is if someone wanted to improve on a feature they could just make it all over again and replace the old microservice with a new one. No need to look at the last persons code, just blow it away like it never existed.
I gathered their site is actually a black box filled with hundreds of black boxes of microservices. All a mystery, they either work or they don't and if they don't they fail gracefully quickly.
Microservices are not a solution to the problem you describe. Nothing in your problem description requires the micro part.
Microservices are about splitting the application into very fine-grained sub-applications. It's not about modularity in general, it's about making things as modular as possible (up to one function per service). That's why they are micro. Otherwise we'd just call them "services" and nobody would have any problem with that.
>Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
Assuming coupling is reasonable. If you have a "distributed monolith", you still end up with all the meetings because every microservice change risks breaking interfaces other people are using.
In the context of coupling, I'd argue the same applies to monoliths. Multiple teams can successfully work on a monolith given architecture where they're not constantly stepping in each other's toes (each team works mostly in their own modules/classes/packages)
Exactly, I missed this completely in the article. In my experience, microservices attempt to solve organizational problems and not technical problems. There are technical downsides to microservices that may be outweighed by organizational benefits. With monoliths you might have a large amount of hidden opportunity cost that technically never surface.
I do agree that this is less of a problem for startups so it doesn't make sense to start with a complex microservice architecture. But in large organizations, especially corporates, it absolutely makes sense.
Ummm, extremely large teams develop monolith (single binary) things called OSes or databases etc. The modularity for scaling developers cross-communication comes from.... modules! duh! Aka libraries.
Yep, I work in a large org that used to be a monolith (which means single DB really). Was a mess for the reasons you'd expect. Even our subteam of 10 needs to split things up more.
"Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination."
The coordination is still there because a microservice team does not live in a vacuum. They build services based on demand from other teams that typically build web apps, mobile apps, sometimes server-to-server.
Hence, the "independent" team now becomes a roadblock for higher order features.
I get the point about creating boundaries to document the dependencies - however if your language supports packages and private keywords you can do that without having microservices.
And once you've split up your app into lots of independent microservices who owns the arrows between the microservice boxes? ( The actual app ).
- Current CTO/VPs built/helped build original monolith
- Nobody wants to tell the CTO that their code is shit (and/or is from a different era and needs a complete overhaul), unrelated to that fact it's a monolith. CTOs are too busy doing marketing/getting funding to make a decision on microservices vs monotolith, so the newly-hired architects gets to call the shots
- Everyone cheers on microservices because it fits within the story of a fast growing, serious, technical company and nobody wants to be that lone dissenting opinion/criticise the CTO.
Nobody is seriously and truthfully recommending microservices because they believe them to be the best trade off and superior choice. It's because they like their job, they like hiring people, and it fits within the narrative.
And it just so happens during the massive overhaul that you get to rewrite a ton of code and improve it, while just calling it a migration to microservices
So it's a way of not hurting the feelings of the CTO, going along with the crowd and a way of rewriting a ton of old bad code with an excuse supported by almost everyone
Nobody wants to tell the CTO that their code is shit
It's a pattern because it's a factual inevitability. Whether you're an individual lead engineer or a CTO/Founder, eventually you always look back and conclude things could have been done better and watch in pleasure/horror as you reap the benefits/drawbacks of how you laid down patterns and processes that the team dutifully followed.
Even with good patterns and processes, the shifting landscape of requirements and priorities and business objectives can lead an application into the weeds.
Initially, you may think you're building a single, focused organ like a liver. A few months in you realize that liver needs to power a specialized toaster for English muffins. Eventually, your core offering becomes a conveyor belt that takes toasted rye bread to the buttering rack, and now you need the toaster that only fits two round English muffins per minute to produce 10,000 square pieces of rye per minute while still relying on the liver for power.
(God, how I love confirmation bias when it's my own bias.)
I've been saying this for years, the microservices insanity it's just an excuse for mediocre engineers to be in demand. It is fueled by mediocrity but it is also what keeps so many tech companies going.
There are simply not enough competent engineers who master UNIX and who can build beautiful minimalist systems like StackOverflow's or a bunch of others' mentioned in this article. Therefore microservices as a smoke screen for mediocrity is here to stay, it's not going away any time soon, especially considering that cloud providers like AWS promote themselves via all possible channels and encourage the decision makers to take that route anyway.
I don't know why but no one hires or listens to the architect who recommends sticking to safe monolithic-esque systems. If you're not talking cloud and microservices and the newest unproven frameworks, you're an old fogey who needs to get with the times. Even though those types of systems are rarely the most efficient, powerful or safe systems
I work for a company that builds software to be used internally by other business. We have like 200 people tops using it simultaneously with no usage spikes, a perfect environment for regular web servers as we'll never have unexpected scaling issues, yet everyone is dying to go to the cloud. Why? I think it's just because our management, programmers and even customers are convinced by cloud provider marketing that the cloud is cool
Architects that propose pragmatic and boring solutions are usually not hired. It's the emperor's new clothes, companies like the idea of an architect who comes with some revolutionary new concept that will lead to a breakthrough that finally will make everybody rich.
If they don't understand the architect's idea, so much the better because then it must be really state of the art.
While there are definitely developers who are enthusiastic about microservices for all the wrong reasons (e.g. it looks good on the resume) I think it's more about how companies deal with complexity.
Companies don't just ship their org chart, they ship all their dysfunction and historical baggage. A beautiful, well-architected platform from 5 years ago, with an efficient, thought-out data model and API and well-written and tested code, might be a total mess after 5 years of constant pivots from the CEO, last minute customer requests the sales team have pushed through, product managers doing their hydrant-meets-dog act of adding features nobody needed or asked for, and never enough people and time to do as good a job as the developers would like.
One day you wake up with a big pile of tech debt and fixing bugs and adding features takes way longer than it should, and microservices are that siren call that promises a solution that doesn't involve burning the whole thing to the ground and starting over.
In my experience, a mediocre engineer can build a decent monolith, especially if they're using an opinionated framework (like Ruby on Rails). On the other hand, building microservices is much more complex as there are more moving parts and more failure modes, so the mediocre engineer - and even many non mediocre ones - is much more likely to create problems.
We are wasting 20+ people's time because we were forced to rush into something that resulted in 5-6 different microservices where a monolith written by 6 competent engineers over 6 months would have sufficed.
What you describe regarding microservices and AWS is also a thing internally.............
I feel this, working on a small team moving things to microservices. My primary problem is observability. It's become a huge chore figuring what, exactly, is going wrong in production when something goes wrong. It's not enough to tail the logs of some distributed application, I need to tail the logs of several distributed applications where there messages are interspersed with one another. I suppose when we get some way to visualize these traces - tooling - it'll be okay. But, small team, limited human bandwidth, and we don't have this tooling in place yet.
The monolith, in contrast, had NewRelic integrated years ago. There were performance problems with this monolith which have been mostly solved through indexes and a couple of materialized views. Trivial to figure out what is going wrong. The code may be old and full of race conditions, but solving problems isn't difficult.
I dread dealing with multiple separate database instances each backing their own microservice when it comes time to upgrade those databases instances. I was hoping for a single database instance with multiple databases, but that particular architecture isn't on the menu. :\
Take a look at the OTEL (Open Telemetry) tooling and libraries. Or Grafana stack/offering with Prometheus, Tempo and Loki. Centralized logging and service calls/code execution tracing is not exactly new. It is often an afterthought.. and then you get yourself is this kinds of unpleasant situations.
And since you didn't implement correct tooling from the start, your team is even smaller and more limited... because you have little to zero idea on what your services are up to.
As per db instances... you upgrade them one by one. Unless there's some really bad bugs present (security or otherwise) there's no rush in upgrading stuff just because.
Are you not using cloud? Because the cloud providers provide centralized logging, so all you need to do is pass a request id between services and include that I’d in log entries, and you can trace requests across services.
I find it weirdly theraputic to read things like this, and reminisce about loudly proclaiming the same a decade ago while being shushed as a non-believer...
Please excuse me if the paradigm of "microservices" has left a bad taste in my mouth, but I have real-world experience with the repercussions of whole-heartedly embracing the latest tech-dejour without completely understanding the tradeoffs...
Many years ago I was hired at stumbleupon around the time the leading compsci doctor decided to take a working and profitable monolithic php app and turn it into a scala/java microservices architecture... in fact part of the newhire process was a weird one-on-one with said mad-compscientist where he extolled the many merits of microservices and skillfully dodged questions like "why would you build a distributed service that just adds a list of numbers?" with a bunch of "you wouldn't understand why it's so much better..." type hand-waving. Fast-forward to 30+ new hires and 4+ long years of intense development and the no-longer profitable company was left with a new slower, buggier, impossible to debug distributed hellscape... as the main designer/architect of it all decided it was a great time to take a "sabbatical"... it wasn't long after that the Nth round of investor money ran out and we were all looking for work.
This article is off the rails (to borrow the authors amtrak metaphor).
The author posits that if you make pile of crap microservices then all you had to do instead was make a monolith and magically it’ll all be fine.
You can put the same kind of design and engineering work in that results in a pile of crap microservices but if you target a monolith instead then you’ll be golden.
The author enjoys stroking his own ego as he goes (the comment about full stack js devs for example). And yet, there’s very little here in terms of actual engineering. Want some measurements or some data to back up the waffle? Well the author says tough! You just get a diatribe instead.
Keeping the cost to change a system low by managing complexity is a fine goal, but that’s not what’s being proposed here. This article could have been better if it recognised this. This article could have been better if it gave some data - hell even anecdata, a single motivating example, would have been a start.
On my team, I’ll take on a bright enthusiastic front end dev who decided they want to spread their wings and grow into a full stack dev over someone who believes they already know everything and has no growing left to do.
Agreed the dichotomy presented is so reductive that it makes you question the author’s credibility. Any architecture that isn’t well maintained will become crushing over time.
I have been fighting this in my current position, with some success with some teams and far less success with others (who are now fighting with the distributed monolith they have created). Ironically the ex-faanmg folks are all pro-monolith.
I think most people fail to realize microservices are what you get from an iterative process. You pull out of the monolith the things you need to scale eventually.
Microservices need plenty of basic infrastructure that you don't want to mantain unless you need to. Most companies don't have an adequate platform to support the pattern anyway.
I feel that. At my shop, we have given up on debugging altogether, because the tooling simply doesn't work.
To run one service on our local machine would require us to adapt the other 9 to run against this one. And god forbid you build a "feature" for the "customer." Now you have to, of course, touch at least 2 services at the same time to move more data.
A breakpoint on one end is a timeout on the other.
So every developer is deploying release builds (no hot reload) on a local VM, inserting console.logs, System.PrintLines() and _loggers and then reading the disparate log files. Needless to say, I'm jumping ship.
Sounds like the third paragraph could at least be somewhat improved on by switching everything to structured logging and being able to point them all to a local aggregator.
Moving everything to distributed tracing would be even better, but there’s a larger investment as the tracing metadata then has to be unified across the board to properly track requests.
Would likely help more with ops than with dev but should help nonetheless, even just getting proper spanning information can provide a lot of insight into a clusterfuck.
> A breakpoint on one end is a timeout on the other.
We use mirrord together with a staging environment to run specific services without the need to bring up the rest of the stack locally (we have ~5 services so it wouldn't even be an issue).
A post like this pops up every few months arguing one way or the other.
I have lived in absolute nightmare monolith systems with so much cyclic dependency that you couldn't move without breaking something. So monolith isn't the answer. I have also seen many instances where people have completely missed the underlying principles of microservices and have ended up with an equal nightmare.
Neither is a hero or panacea, what matters is to understand what you are building and why and get the interfaces correct. Then understand the principles of different architectures before moving forward.
If you fail on this, you get predictably a nightmare. There is plenty of nightmare code sitting on this planet everywhere we look. My recommendation isn't to vilify either architecture but to understand their strengths and weaknesses.
"You don’t need hundreds of engineers to build a great product. I wrote a longer piece about this, but essentially, despite the general stage of startup we audited being pretty similar, the engineering team sizes varied a lot. Surprisingly, sometimes the most impressive products with the broadest scope of features were built by the smaller teams. And it was these same “small but mighty” teams that, years later, are crushing their markets.
Simple Outperformed Smart. As a self-admitted elitist, it pains me to say this, but it’s true: the startups we audited that are now doing the best usually had an almost brazenly ‘Keep It Simple’ approach to engineering. Cleverness for cleverness sake was abhorred. On the flip side, the companies where we were like ”woah, these folks are smart as hell” for the most part kind of faded. Generally, the major foot-gun (which I talk about more in a previous post on foot-guns) that got a lot of places in trouble was the premature move to microservices, architectures that relied on distributed computing, and messaging-heavy designs."
It is completely insane that some places want 20-30 people for a CRUD website. Yes, there is domain complexity there (that’s what makes the job actually interesting, unlike the tech we have to use) but that much?
Really the issue is we continually subvert engineering needs for business wants.
Until that is no longer the norm the fractal of WTF will grow deeper.
I am literally working on a project where the backend team consists of three coders, and the backend currently consists 33 microservices. Common tasks for this platform involves calling around 7+ microservices.
Thankfully I am working on the frontend side of things, but this is going to be hell when when we launch.
One of our projects has ~15 backend engineers and ~30 microservices. It also requires a separate platform team. A k8s cluster with several nodes. It has incidents like every week because the whole thing is like a house of cards. A small feature requires changes in ~5 microservices and a lot of coordination between the teams. They always need to think about things like circuit breakers, timeouts, APIs (schema synchronization, versioning), event queues, distributed transactions, sagas, distributed locking, k8s configs, Git repository management, distributed tracing etc. etc.
On the other hand, there's another project, which is a monolith. Around ~15 engineers. Incidents happen very rarely. The infrastructure is simple: just one beefy server without k8s (but there's a standby replica of course), so the team often does maintenance of the server on their own instead of a separate SRE/production team. The monolith is modular so it enjoys same advantages as microservices when it comes to module isolation etc. Highload functionality is moved to separate Go services only when needed (currently 2 or 3 satellite microservices). All changes happen in the same codebase so features are released faster and it's easier to reason about.
I'm currently on the second project and whenever I attend architecture reviews of the first project, I'm shocked how overcomplicated the first project has become. A simple addition of a new object property can end up in 5-10 engineers arguing for hours whose microservices are affected and how they should best communicate.
This isn't even a joke. Technology in software development has been progressing horizontally. Nothing is improving but abstractions and patterns are changing constantly.
Engineers need something to talk about in their OKRs. One strategy is to refactor things into microservices.
Sounds like you're making it hard for yourselves on purpose. May I ask why you (the team) chose this over other technology and what was the decision process at the start the project?
> Thankfully I am working on the frontend side of things
Honestly, given the opinion of frontend on this forum, I'm surprised that you're thankful! I am 100% certain everyone here would take any kind of backend work over the hype-driven mess that is frontend development.
I moved from backend to frontend years ago, when "hype-driven mess" was a much more accurate description of the field than it is today. There's simply many more interesting challenges in web application front-ends than backends, especially if you need spacial design- things where canvas or webgl excel.
Comparatively, there are far fewer backend positions available that are more than basic CRUD, even if it those CRUD operations are gussied up with a bunch of needless complexity juggling sources of truth across a bunch of different services.
There's plenty of drudgery out there, and some interesting work in every field, but on the whole I personally have found frontend product development- product configurators for engineering / manufacturing, network visualization, visual design tool development, and data driven interfaces to be more fulfilling work.
To each their own, but maybe you might want to be a little more aware of the people that you work with when you feel like saying something condescending.
Edit: FWIW, you'd have to be blind to not see hype driven development in backend, especially in HN. "Rewrite it in rust" is the meme that continues to keep on giving.
A dozen micro services per developer on rickety infrastructure serving data for a pixel generator built on top of 5 years worth of JS hype cycles. Why are we doing this to ourselves? It is just so stupid.
> I am 100% certain everyone here would take any kind of backend work over the hype-driven mess that is frontend development
The back-end is becoming an even more hype-driven mess these days. What with k8s and dozens of micro-services and controllers and telemetry-everything. And people don't even listen nowadays if you try to propose something simpler.
The front-end is at-least constrained by browser limitations and the client device. One can only do hanky-panky upto to a point. But at the backend, "throw another cloud machine" at it has become the de-facto solution for most performance issues.
I was involved in a merger with the platform I architected on one side, and a platform that does the same thing on the other side. We were told the company wanted one platform.
I do everything I can to follow KISS. These are important enterprise systems that drive the entire company. I went with a front-end framework in TypeScript, C# APIs, and a relational SQL DB as the gist of it. It turned out great ... performant, reliable and the users were reporting positive feedback (this one was a recent upgrade from an older system).
The other side had the same number of users, but had over 100 PHP microservices, Docker containers, queuing frameworks, and all sorts of additional technologies in the mix. It had 5 times the number of engineers on it, but was at the end of the day was functionally equivilant and had the same volume of traffic.
When comparing the up-time, it was clear the microservices architecture was the main source of pain. They weren't even comparable. We are always talking about whether we're at "5 nines", and the other system was having major outages every other week.
You could argue that it was a poorly architected system, and that microservices weren't the root cause, but that really wasn't the case. The entire system was based around the microservices, and they weren't working as promised. What originally was seen as separation of logic and concerns and everything else that microservices offer eventually over time became a tangled, inter-dependant system of services that were very high in bandwidth, slow in performance, difficult to maintain and bad in reliability.
This isn't the first time I've seen this. I've worked with microservice architectures in other companies, and I haven't yet seen it "done right", or at least how I understand it. It always comes across as great in principle but bad in practice.
One of the things this article alluded to (but I didn't see it explicitly stated) as to why microservices is a default go-to architectural choice:
Because then a bunch of engineers can stick it on their résumé and try to get a job at a big company
I worked in a funded startup where we built a service-oriented architecture out of a few key Laravel monoliths. This was entirely manageable for a team of less than 10 engineers and scalable for the context.
Because we built it as a distributed system, each of those core services was ready to handle outages. So it was extremely easy to deploy
What's more, it barely cost us $20k PER YEAR to run it all
But now I can't say I've worked with microservices so I'll likely not stand a chance at any of those interviews
I have two huge gripes that I've run into when dealing with microservices:
1. They're actually nanoservices. The article touched on this. A single function shouldn't be its own service unless it's something that requires a lot of processing, like a service that converts files from one type to another. But something like handling user registration? You don't need separate services to handle registration, login, and the forgot password flow.
2. The services are poorly named. At a previous job, we had services with names like "Kafka", "Avatar", and "Sophos" and none made sense for what they did, especially Kafka, which I expected to maybe be a wrapper around Apache Kafka? But no. It should have been called the User Preference Service, because it was simply a service for users to get/set their preferences.
I don't quite understand the mentioning of node at the start of the article as a cause of this. You can write a monolith in node just as well as you can write a web of microservices. That said I agree with the sentiment of this article, it's very frustrating to deal with in any company. Especially when these microservices aren't versioned under the same repo and/or are hosted at different providers causing unnecessary latency.
They argue that Node tricks frontenders into thinking they can do server-side, then says they 'huff and puff' when this is pointed out to them. It's an incredibly patronizing argument to make.
One piece that I removed from the draft was about how I wanted to join in on the Node fun. In 2014, when I was writing very simple async Python Tornado code with "await", I saw the mess that was Node asynchronous development, and I was truly bewildered by why it was considered revolutionary and cool.
How would you sell Node to other old guys? Especially compared to Go.
I try to favor simple, efficient software. I used to dislike PHP because it was so messy and inefficient, until newer frameworks like Django made it look like it was slim and snappy.
Maybe, in this race to the bottom, Node stands out in a way I hadn't seen, so I'm genuinely interested.
> How would you sell Node to other old guys? Especially compared to Go.
JavaScript has a huge business case in the front-end. Going another direction is possible, but you're really fighting against the grain to do so.
Go node/bun/deno and you can build back-end and front-end with one language. This has pretty significant advantages. Particularly if your goal is small teams (or solo) with "full stack" style devs. There are fewer siloed specialists required in the org.
Language concepts, syntax and patterns are a significant part of the cognitive load needed to develop.
If you're in a siloed org with specialists anyway, there is minimal draw.
I've done things like have a backend I brought into the front end to mock an immature system. Shared code between the back end and front end for offline-first systems. You just can't do that kind of stuff when you don't share a language between front-end/back-end.
I’m currently at a place using microservices, in a monorepo, where business logic is executed in consuming services, and some business logic is in a monolithic graphql layer, that only frontend clients call. Tries all the tricks, fails at all of them.
Still, it’s a profitable business. The dirty truth is that the current crop of engineers are the ones paying with their blood, sweat and tears - and I’m not sure about you, but that’s not what I signed up for as a software engineer, so we should demand better.
Starting with a monolith is nearly always great advice. However, monoliths tend towards spaghetti because it's too easy essentially to draw new lines on the architecture diagram by importing from anywhere.
To scale a monolith codebase without devolving into spaghetti you need to have a well defined layered structure for modules. The other aspect is being able to hide internal code to prevent it being imported by other modules.
I did a write-up a while ago about how we do this on my current project [1], and published the Maven enforcer rule and ArchUnit test example as an open source project [2].
I wish language designers would spend more time on designing higher abstractions, like modules, rather than only low level stuff like borrow checkers and async.
As other gifts of computers field: death by a thousand tables in DB, death by a thousand class in Java, death by a thousand tickets in Jira/Redmine/Bugtracker/Gitlab..., death by a thousand "go to" in Basic...
Microservices are oversold. CIOs jump on the bandwagon because they read a couple of articles on line and think magic happens. The last company, a spin off from a larger company in the financial sector, I worked for before retiremnent decided to port their existing monolith to a cloud-based, microservice platform. Did they need it? No. The total customer count was projected to be 25 million or fewer with an average active user count in the low 1000s, about 1800 when I left. But cloud/microservices were "sexier" to the ownership group whose plan was to sell it in a couple of years. Thus far they haven't succeeded and the microservice guru that succeeded me has been fired for failing to delivery a working product after three years. If a company isn't going to be operating at internet scale, e.g., NetFlix, Facebook, Amazon, there are usually better options than going all in on microservices. Then again, I've worked for companies that had no idea how many simultaneous users they had.
Been working for 6 month at a company with ~150 engineers that uses Microservices.
As a simple software engineer I love it, It allowed me to easily get right in to parts of the system and conceptually understand the boundaries between different systems and business unites.
But it sure adds complexity - complexity that I fortunately do not need to deal with... Yet.
Just use Elixir/Phoenix and get all the scaling benefits of Microservices running seamlessly across datacenters while maintaining the sanity running a single app on your localhost that is fast to test, easy to reason about and deploy!
Having worked in several places that adopted Microservices with nowhere near the Netflix or Uber scale just because it's trendy I can attest to the mess and glacial pace of both learning and development for new team members. If you want to move fast and make something people want, don't pretend you need "webscale", you don't. You're building an apartment block for a few people not a city for millions.
The beauty of Erlang/OTP is that it's designed to scale to millions without any extra effort from the individual contributors or fancy DevOps.
i don't see how erlang / otp or elixir / phoenix is relevant to solve the problem most companies are facing when hitting their first "web scale" problem : bottleneck at the DB level.
Agreed. The DB is the bottleneck way before the code architecture. But that is a "solved" problem from a deployment perspective. All the Cloud providers have managed Postgres that scales to petabytes and millions of transactions per second. And the part being discussed in the article is the Microservices. Where Elixir/Phoenix shines at delivering the benefits of Microservices i.e. scaling a specific function in your codebase to multiple processors/instances while still being easy+fast to run on a single Dev machine without hundreds of Docker processes.
Erlang/OTP (and therefore all BEAM Languages such as Elixir) have a supervision tree which means that each module in a monolithic application can be run independently in as many processors/machines as required. Essentially you get Kubernetes + Docker for free (virtually zero overhead) inside the Erlang VM. See: https://blog.appsignal.com/2021/08/23/using-supervisors-to-o...
> First, a whole army of developers writing Javascript for the browser
> started self-identifying as “full-stack”, diving into server development
> and asynchronous code.
This is gatekeeping--we're all ultimately self-identifying as software engineers. But more importantly it's not correct. Many of the original engineers working with Node had non-UI backgrounds. Node.js made asynchronous programming easy and promised that you could use the same language on the back-end and front-end, which seemed like good bang-for-buck for people that wanted to focus on products.
Also, not to point the blame somewhere else, but wasn't it Golang engineers that went really hard on the microservices koolaid?
I also know people that spend 40k euro a month to run the database of their rails app on the biggest instance they can find on aws. The traffic is seasonal and the workload is mostly one way payment traffic. But hey, they are still making money so good for them. I've personally been exploring a more flexible architecture for another project using a combination of cloudflare workers, storage and k/v store. The hardest thing about it currently is that you have to engineer everything yourself to be specific to your problem. Either way, both monolith and microservices have their challenges, I hope to find something that can avoid both but it feels like a stab in the dark.
Didn't I mention the Citadel pattern, "trunk and branches"? Neither did I suggest that somehow starting with a monolith will force you to write better code.
But it will definitely be easier to debug and refactor by those who come after.
Tell that my team mates putting all stuff into libraries because "we made that already" and building a huge distributed monolith. We combined sucessfully the worst of both worlds!
Why this is getting blamed on JavaScript / Node.js is beyond me. Microservices were/are all the rage in Java/Scala land, and seemed to initially be pushed by (Dev)Ops engineers' desire to roll their own Kubernetes cluster (or whatever came before). I've been on multiple such project and I always loathed the additional complexity (especially combined with actors/event sourcing and Cassandra). Node.js was limited to some frontend tooling that only ran locally or in CI.
I think the author meant "Web developers", not specifically "JavaScript developers". There's just a very big overlap between the two.
So, Java and friends fits perfectly into this pattern. DevOps (saying as someone who used to be "Ops" w/o the "Dev") is also largely a Web phenomena. If you were exposed to Web development some time in your career, but then moved away from that, Kubernetes will definitely trigger PTSD in you. It's Web developers writing system software, and they don't really know how to do that.
Let's not throw the baby out with the bathwater. Distributed horizontal scaling is a necessity at any non-trivial scale. Yes, mindless cargo cult excessiveness of anything is bad. The real crime here is forgetting how to organize large codebases effectively. We used to have beautiful software frameworks that allowed things to be developed in neat little modules with interface bindings that can be turned into in-proc method invocation, or IPC invocations or RPC invocations through compile time switches to create one or more deployable binaries with neat little infrastructure for service discovery and routing, end to end tracing and profiling etc. The modules were independently verifiable for correctness and performance. Modules had semantic contracts clearly defined and their implementations could be changed transparently. We had sophisticated distributed persistence layers with MVCC, zero-cost snapshots etc. with remote addressable storage 20 years ago. Instead of scaling these patterns for web-scale with the necessary engineering discipline, we took a lot of cheap shortcuts only to pay for them 10 times over in the form of wasted dev time and hardware.
Having taken part in building the Wix microservices architecture, I have to say that I understand and accept the critique of the article. Microservices is not a magic architecture, it solves some problems with the price of others.
When we stared Wix, we stared as a monolith - at 2006. In 2008 we split this monolith into two services due to different SLA - building websites and serving websites.
In 2011 we stared splitting the monoliths into micro services to solve software delivery pipeline - ability to move fast and scale with number of developers. Today we have about 20,000 deployments of new software every year, with over 1000 developers.
At Wix we are constantly investing a lot to maintain the microservices architecture and make it productive for our developers. We are making tools and utilities to return DRY into microservices architecture (think CDC support, GDPR support, Monitoring support, Scaling, Access Control, etc.).
My takeaways -
* Microservices do not come for free. You have to invest in making it work for you
* When building a startup - build a monolith. Break it down later when you understand what you need.
* We as an industry do not understand how to build micro services. There are a lot of fundamental aspects of microservices that are not commonly known / understood or ready as of the shelf solutions.
The small org I work for has two distinct AWS accounts and nobody knows whats in each one, let alone whats in the regions for each one. Everyone is an IAM user that a third party IT company creates after emailing helpdesk a few business days prior.
both AWS Accounts have a bunch of lambdas randomly doing necessary things in a variety of our development environments, but mostly production
a single AWS account is fully capable of catering to multiple development environments
This post reads like a cry for help. I don't really know what they're arguing for. Sure the broad-strokes 'microservices bad monolith good' is something I can get behind in some cases, but the slightly unhinged tirades against Node developers and the 'bloated farce' of the modern tech industry might as well be satire for all the value it contributes to debate.
service oriented architecture is a disaster even in large companies. without a monolithic database and normalized schemas you always end up with a bespoke ad-hoc never-consistent data store. data and results are unverifiable and continually incorrect and the performance is abysmal.
a really large company can waste hundreds of millions of dollars papering over the inherent deficiencies of the architecture but it is an exercise in building additional stories on a house where the ground floor is made out of cardboard that happens to be on fire. soa was created purely for business organization needs. any technical justifications are post hoc rationalization. from a technical perspective it is pure trash.
a much better architecture is to keep services but have them all built on top of a single monolithic db. at scale the monolithic db can be a facade and then you disaggregate the database into horizontally scalable services so that you can scale your monolithic db facade to whatever you need.
At AWS on my service we have lots of microservices (and I think we need them at our scale), but they also share a bunch of the same code. So one problem is reasoning about when your random java library gets merged into some component and it takes a while to wind its way through pipelines and then subtely breaks something unexpected. Take a lot of engineers to deal with this architecture and I certainly wouldn't recommend it for most folks. In fact, one of the slightly hilarious things is that we started out with microservices and end up combining them together because - guess what - when you are building systems level software you really need to make inline calls instead of network calls (without timeouts, retries, circuit breakers, and all the BS).
As a manager I really miss monorepos, postgres, and all the other simple stack stuff that makes developers super productive and IMO the work much more fun.
IMHO, blaming the unexperienced Javascript developers is unfair, because in my experience mostly the senior Java developers were the ones jumping on the hypercomplexity infrastructure bandwagon and embracing microservices as the logical next step after wrapping design patterns with design patterns in their code bases.
Well, the idea of microservices would be to build some degree of flexibility into the setup, so that you can handle problems you don't have now, but you might have later.
I understand that the Monolith - if done right - is a viable option as well.
As a consultant, our customers rarely know what they want, and they can't really describe their vision, no matter how many workshops we throw at them. Microservices give us the opportunity to say "We might not know this now, but with this architecture, we won't be locked into a certain pattern or meet certain technical limitations by a monolithic approach".
It seems that what you are really paying for with microservices is flexibility and maneuverability "down the line".
Solving only the problems you have "here and now" with the simplest solution will also lead to technical debt.
I don't think I agree with your thoughts. Microservice setups are generally less flexible than monolithic setups. Particularly if you get the boundaries wrong it's notoriously difficult to refactor multiple microservices, particularly if you need full uptime.
For monoliths you can refactor the whole thing with editor support, and then deploy a new version.
So microservices will cost more, give you less flexibility and maneuverability, at more upfront cost. It seems like the worst of both worlds.
In a multivendor scenario, it could make sense to split a big monolith in say 2 or 3 components, but I have seen dozen of microservices (one of which just managed 3 read-only tables...).
But if your 6-guys team ends up deploying 6+ microservices, you are doomed, in my huble opinion.
We have exactly two microservices atm: one sends emails, one registers users and creates a backend (one backend per customer with us). Should I hire a team just to adhere to your golden rule? I wrote them myself, no support from anyone needed, they are low maintenance.
Do architecture first, have a modular monolith, if you need microservices, use 'em. The rest if this thread is just a waste of time and a bunch of strong opinions.
I just hate this whole thread. Full of weird assumptions. People here don't know enough about the scenarios to hate on either solution. Do what's right for your project and don't listen to the internet. Easy as that.
If you use a language on the BEAM virtual machine, like the article hints at (WhatsApp - Erlang), you get the best of both worlds. Every process is ostensibly a Microservice. So not quite the same a a Rails or Java/Spring monolith.
The name microservice is a fanatical view of how a service should work. That is the problem. If you say you made a service, then it is not a "micro"-service, so you are lacking "experience" with microservices :) Therefor we must build microservices to have jobs.
In reality, you need services that are engineered properly. They can be monolith or not, can share databases or not, can live on the same server or not... depends on the situation. But, as soon as you say the work "micro"service, you are doomed because it won't be a microservice if it does not adhere to millions of articles on the internet saying how to it should behave...
I don't think microservices are as common as often suggested. In my experience with small and medium sized companies, if they go SoA, its typically just "services", almost always sharing the same DB, and pretty much never "microservices". Usually it's something like: a service for the webapp, maybe an api service, a couple services for something like a job queue or sockets, possibly with the a service for compute bound tasks like media processing or reports. I've also worked at places with 50+ services working together in production, but they're rare (at small and medium sized companies).
A good maintainable software solution is always build out of relatively small and simple components with very important: clear inputs and outputs without too much sideeffect behavior. It says nothing about if its a monolith or microservice. I have seen a lot of microservice architectures so complex with 100s of layers and configurations that nobody in the company understood how it really worked. I have also seen the very same thing in large monoliths. Too big. Too complex. The discussion should be about the software component architecture. Not about ifs a monolith or microservice.
Microservices are often used as an excuse to be able to write part of the system (=microservice) in own technology stack and not in the one the rest of the system is already written in...
> Scale is measured in millions of instances: On 2022/12/21, the microservice topology contained 18,500 active services and over 12 million service instances.
They do go on to say that microservice as a concept is poorly defined. Are you suggesting that they mostly have 18500 of what we’d normally consider monoliths?
I thought Google at least was a bunch of micro services which is why they needed something like Borg and eventually open sourced a version of it called K8s.
Total BS.
Google has tons of services.
They are lots of them but they don't have a `fileRead` service and `fileWrite` service. Rather, they have a `gfs` service that can read/write/modify etc. everything related to files.
I can agree with this sentiment now after working at a handful of companies and many interpretations of micro-services oriented architecture.
The one thing I don’t miss about monoliths, especially at low budget/low rent firms, is that the project often exchanges so many hands/owners/teams between bottom bid contractors and “rockstars/ninjas”. This often leads to the usual god classes, half assed abstractions, massive utility classes, and half assed code refactors.
If you have an expert in design / architecture and know the domain problem / scope / boundaries, go for microservice. Otherwise, monolith.
A good microservice is better than a good monolith since the boundaries are clearer. However a bad microservice are many times worse than bad monolith since on top of the wrong scope, intertwined boundaries and other problem from bad design that exists in monolith, now you're faced with additional infra challenges.
The worst WTF system I have ever seen in my 30+ years career was a microservices system.
The best most maintainable systems I have ever worked on were all 1 million+ line monoliths. Split into N libraries with clean API’s and separate teams working on them.
You have a problem so you pick microservices. Now you have N*(N-1) problems.
LOL. This is how a former out of academy right into startup CTO position colleague tried to argue his choice of approach to enforce. Those were dire times, luckily I found my new place fast, just in time to avoid the spectacular implosion.
There’s a tendency to throw the baby out with the bath water here. There’s something to be said of well partitioned services optimised for cache locality. For most start ups that may mean a small handful of services. Scale is somewhat automatically built in as you can scale up the parts of your ecosystem that have particularly hot paths.
Well architectured software I’ve ever encountered adapts to need and usage over time and tends to always look unique (correspondingly to the uniqueness of the problem domain). Going ‘all in’ on any one architecture strikes me as ideological.
I'd say JavaScript and Node are responsible for a lot of evil in the world, and they definitely had a role to play in microservices popularity.
Nobody was really doing microservices until there was Kubernetes. I mean, maybe some big shots who had a system similar to Kubernetes, but Kubernetes enabled newcomers with small teams to be like the "big guys". So, Kubernetes was the driver, not Node. But... Kubernetes was built by Web developers (unlike Docker for example). Coming from systems background it's painfully obvious that people who built Kubernetes didn't know how to system. And there are a lot of indicators that they, essentially, were building a system to support unwieldy Web site deployments.
Kubernetes was the system built by DevOps, who, unsurprisingly are the result of Web development. The whole idea of DevOps was called into existence by growing popularity of services. Instead of selling software, the market started selling service subscription: from financial perspective, subscription is hugely preferable to one-time sale.
This drove a lot of products that didn't need to be services into the service space (think Google Docs, followed by Office 360, Adobe Creative Suit and so on). Frequent internal releases fuel the competition between similar services, but also necessitated DevOps. And this is how we also got "shift-left", continuous deployment, X-as-a-service etc.
Web, by itself is not the reason these things appeared. It facilitated their appearance. The driver was the financial bonuses of subscription. Web was the convenient mechanism. But, this also means that Web developers were those who made it happen. They were the first and the largest group of programmers who moved into DevOps. They were the people who created Kubernetes and a lot of programs in its immediate environment. Not surprisingly, they also used the tools common in Web for their work.
Had Kubernetes being started by system people, Go would've been an unlikely choice for the language (C would be the most likely candidate, followed by C++ or maybe even Python). YAML wouldn't have been the choice of configuration language (it's very common in system world to roll your own, if INI doesn't work). It's because Web developers built Kubernetes it's networking, storage, user permission model... are all super naive and wrong -- they were designed by people who used to consume them, not by people who used to develop them, and they simply didn't know how they work on a more fundamental level.
A lot of complexity of Kubernetes (and by extension of the microservice world) can be explained by the lack of experience of people who built it with how system software is usually built. And, yes, Web programmers are the one responsible for it. Although, perhaps, Node specifically had little to do with it.
There's always something hot to hate on. I'm getting flashbacks to the days where everyone was going "nosql is trash" because they all cargo culted to mongodb back in the day and then tried to do olap on it.
Heck, this isn't even the first time SOA has been in the frame of hate. Member SOAP? Member how everyone more or less jumps between doing everything on the server vs everything on the client every year? We've been having that battle ever since networking or things clients have been a thing, the "sportsification" of that dichotomy is older than I am.
I like the other person's take in this thread about how you can largely get the semantic benefits of micro services by making a well crafted monolith. I honestly agree, but I think the followup is "aren't people who poorly break up service boundaries going to go do regardless if the interface is a network endpoint or a function call/dependency injected class?"
> "aren't people who poorly break up service boundaries going to go do regardless if the interface is a network endpoint or a function call/dependency injected class?"
Today there are several tools we can use that enforces boundaries in a monolith (Spring Modulith for example). If the project uses one of these tools, it is harder to accidentally cross boundaries and you get many of the same benefits as you get with a micro service.
The big advantage is that if you find out that you made a mistake, your only dependencies are within the same service and is easier to refactor. In a micro service oriented architecture, changes might impact several services and teams that needs to be coordinated. I'm not saying that refactoring a monolith can't be time consuming, but you have at least a better control of the flow of data between modules.
An interesting angle would be how to get out of this mess.
Here's one: In politics we see national and supranational governments take on the job of making large decisions. Style guides and best practices on whether to use microservices or not are, at best, made at a company level.
Would it be interesting if these architecture decisions were made, or at least kept in check by a body that is larger than a company?
Perhaps we could have some laws that dictate that you must (at least partially) understand how software works before you can buy it. Especially for government contracts.
The second you think you are smarter and more informed than the people on the ground is the second you revealed yourself to be a fool.
Because having spent decades working for government and enterprises I can assure you that we aren't all stupid and need laws to protect us from ourselves. Instead we are often placed in really challenging and unique circumstances that drive the architecture and design of what gets built.
For example micro-services often works well in places where you have distributed teams that for security, governance or logistical reasons need to work independently and can't collaborate effectively with a monolith architecture. Or where certain components e.g. authentication, payments need a hard separation from the rest of the platform.
Obviously not all in government are stupid, I was not implying this. I was rather suggesting that some are, and that something might be done to avoid problems with that.
Also, microservices can be great, but when used in the wrong context, obviously are not. We are not discussing that either.
Edit: perhaps you may have misunderstood my comment, and that may be due to my clumsy way of stating it. I was actually suggesting that companies learn from governments.
Picturing a real-world situation here in my recent past...
We start with the pinnacle of engineering, a SPA, could be built in React, Vue, whichever.
It loads, which on slower smartphones takes a long time as they suck in running JS. When it's booted, there's nothing there. It has to reach out for data.
And so it does, to the microservice. Which might start doing something in 20ms or 200ms, it's all very intermittent due to excessive virtualization. It can't just go and return the data, no...it has to take the authentication token that was passed in and authenticate it with yet another service, which is vendor managed and even slower. Then there's rate limiting, logging, all types of overhead. Eventually, it returns 3 product rows.
Sadly though, for each of the product items we have to make an individual call to get related data, part of it coming from yet another microservice. All of these calls having their own startup time, authentication and overhead.
So for what one might call a classic "join", we're into multi-second territory. Which is fatal territory. Not to mention the shit state of data, as it's all no-sql garbage. Mind you, this is about 1/10th of what users are supposed to see on the initial dashboard.
We're talking one tiny part of a small- to midsized "app". It all costs a fortune and it produces garbage that users hate.
Old dinosaur me used to code up a CRUD app like this by the dozens. Using LAMP, .NET, whichever. Performance would be near-instant, I wouldn't allow any query response time to go over 10ms.
Every part of our stack is "best-practise" and enterprise-grade. And it just doesn't fucking work. It's 10 times more expensive and 100 times slower whilst the very idea of such an app is that it's supposed to be more interactive/responsive compared to the traditional stacks.
Luckily though, these services are multi-client. Except they're not, because then the mobile app team needs different data, and it's all just bolted on to the "generic" service. Services teams independence is a problem, not a solution. Plan a UI feature and good luck getting it orchestrated across teams each having their own roadmap.
And let's end with comedy gold where Lamda developers now need to calculate the running costs of their logic line by line. Whoops, slightly inefficient loop, that'll be 30$ extra per month on the bill. Now "scale" that problem to dozens of such developers building hundreds of services.
What a joke we've become. Distributed computing is an error, not an architecture. The point of computing is to put compute and data as closely together as possible. As the brightest minds on the planet deliver in stunning hardware advances which us software developers piss away.
But hey, the God of Complexity pays well and those footing the bill have no idea what we're doing but it does sound impressive.
Microservices is mostly an RDD (Resume Driven Development) practice I've found. Really, as the article says correctly and hilariously, unless you're a greenfield or exceptionally large, you end up with a fleet of shitty golf carts.
My personal anecdote is, watching from the sidelines with face cupped in hands, watching a monolith being broken into hundreds of microservices at the cost of decreasing performance, increasing complexity and 90% of the traffic still hitting the monolith. Of course there's no time to do anything for the customers any more because everyone is updating software versions in all the microservices and promoting that "it'll all be fine if we just move one more thing to a microservice". The architectural vision will be complete in 2130 at this rate.
I want out from this demented ego driven fashion show.
> Microservices is mostly an RDD (Resume Driven Development) practice I've found
I agree. The worst I've seen is with contractors/consultants, because the incentives are particularly bad.
They often stay with the project only for a limited time and "grokked customer's business/domain logic" is not something you can put on your resume and the typical 6 months are too short for that anyway. But there's enough time to introduce a schema registry or other piece of tech which can look pretty good on your CV. Jump onto a different customer, rinse and repeat. I call this "technology-oriented mindset" vs. "goal oriented mindset".
I have avoided, by luck alone, microservices, except the occasional small service where it makes sense (think AWS Lambda to covert doc to pdf for spikey loads) and I am glad. It means I don’t get far in interviews at small companies that bizarrely want microservices experience (as opposed to problem solving experience!)
> I want out from this demented ego driven fashion show.
The author of TFA kinda hits on this in the first paragraph:
> absurdity of the state of the current tech culture. We laugh, and
yet bringing this up in a serious conversation is tantamount to
professional heresy, rendering you borderline un-hirable
As soon as I read that, I felt I knew what the rest of the essay would
be about, and I was right. It's about culture.
A culture of insecurity, wannabeism, fads, in-groups cliques,
so-called "best practices", megalomaniacal levels of scaling,
over-reach..... let's all admit shall we (at least privately) , that
"tech" has an _ugly_ culture, that we've lost those early joys of
solving problems. We've replaced that with a culture of making
problems so we can look good by "solving" them. Just realising that
would bring enormous economic benefits.
In wider human culture, tech is looked down on, for good reasons. At
worst it is a proxy battleground for personal whims and ideologies in
which one group of people set out to impose upon another larger group,
their particular idea of how the world should 'work'. Code becomes our
poor man's mathematics, apparently adding rigour and substance to what
is arbitrary and manifold. The helpless dependency of those who must
use what we create, makes that a very abusive relationship.
It's a shame that Gerald Weinberg's concept of Egoless programming
came at a time before the "boom", and so couldn't address the
devastating effects of ego-driven development on the wider culture.
It's more about the internal dynamics of the development team.
Mirrors my experience, except some teams also decided to implement the new services in different languages...
After a downsizing, we are now left with 15 services, 5 different programming languages and complicated kubernetes setup. We have less than 10 developers that has to maintain this. In addition we also have a separate flow for managing data and BI.
Kubernetes is a bit of the new OpenStack. It's like an all-you-can-eat buffet of complexity concealed behind interfaces.
Either some startups go rampage with it and deploy needlessly complex stacks because the tooling makes it easy, hence a headlong rush into more hidden complexity to solve problems. Or they use it for the hype and merely deploy a couple monoliths with it, which Compose / Terraform would do just as well without the maintenance burden.
We broke our monolith ~2015. 8 years later we still have the original monolith plus the original mircoservices are now monoliths too which need splitting. Upgrades were a nightmare for first 5 years, but we solved it with nitpicking code reviews where non-trival changes are blocked forever until everyone's ready.
At least we solved the problem of decreasing performance the old fashioned way: by throwing more RAM and CPU power at the problem.
When proposing a system design - have two. One using a monolith and second using microservices. Put the first in production and the second in your CV and everybody goes home happy.
I have seen managers propose microservices to their own bosses just to show their own bosses that their team will embark on something Netflix does - so we should do too.
It's nice to see the trend unwinding, microservices are egregiously overused as a silver bullet for any problem (not just scaling).
There are probably dozens (/s) of companies that would benefit from going all-in with microservices. Some companies would benefit from having a few microservices; but most would be more than successful running a monolith.
The most significant problems with microservices arise when they're adopted too early and when data separation is not properly handled, resulting of even the simplest requests ended up querying 3-5+ microservices.
Latest anecdata point: doing exactly that (unwinding microservices) in a team of 20 engineers (15 swe/5 sre) and 400+ repositories (which isn't a "wrong", but having so many different places to update, for example, Python version is nuts). The whole project is nuts.
And persuading management was not an issue, it was enough to show current AWS bills and potential savings from moving from 50+ microservices to smaller numbers of consolidated services and deprecating/refactoring some features. The real challenge was persuading the engineering team to prioritize and work on this issue - which is still ongoing.
The author would disagree with you; I think they think it's more like:
Node engineers: "lEt'S bUiLd SoMe MiCrOsErViCeS!!!!! YAY I'M CODING!!"
Chad UNIX architect who won't code in any language that doesn't give you the tools to blow your own foot off just printing a string to stdout: "No let's keep adding functionality to this one codebase until it's as complex as a suite of microservices."
But I also advise a lot of startups, and you know what I tell them nearly every time?
Build a monolith.
It's so much easier to start out with one codebase and one database, and that will scale for a while. Especially if you use a key/value store like DynamoDB (although you will lose relational functionality that can be helpful at the start). And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services?
And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!). The one that may need to scale independently, or that you want to be able to deploy separately.
Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency.