I completely agree with the idea that scalability is often overrated. While it's certainly important for large organizations and enterprises, for smaller companies and side projects, focusing too much on scalability can actually hinder growth and creativity.
It's often more beneficial to focus on solving a problem effectively and efficiently, rather than trying to scale it before it's even proven to be successful. By starting small and iterating based on customer feedback, you can create a better product that solves a real problem, which will naturally attract more users and drive growth.
That being said, scalability should definitely not be ignored altogether. As a project grows, it's important to plan for future scalability, but this should only be done once the product has proven its worth and a solid foundation has been established.
It's kind of along the lines of "premature optimization is the root of all evil." A true quote, that doesn't mean optimizing isn't important! The whole trick is in being able to tell what's core and what's premature.
Another way of looking at it -- scalability is pretty much the entire value proposition of software. It allows people to do the same, or more, amount of work without having to linearly scale out the number of people working on something. When you're first starting out, demonstrating even being able to _do_ the work is the first hurdle, but past that you're definitely going to need to show you can scale that out efficiently.
> Another way of looking at it -- scalability is pretty much the entire value proposition of software
That sounds like a good way to look. People are Turing-complete, although that doesn't mean that they are very good at scale without instructions, and "programming" clusters of people has its own set of quirks. :)
> It allows people to do the same, or more, amount of work without having to linearly scale out the number of people working on something.
But this is not the entire story, but only the throughput part of it. And there are entire classes of software where at least part of the value lies in latency. That's, for example, most of Internet as we know it, from the hypertext browsing to videoconferencing, and most software we would call "embedded", from watches to spaceships.
> But this is not the entire story, but only the throughput part of it. And there are entire classes of software where at least part of the value lies in latency.
That being said, scalability should definitely not be ignored altogether.
Yeah. You have to think about what's possible to scale.
You're planning to scale to 1,000,000 or 1,000,000,000 users eventually? OK. Well, what's your service? Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Or, is it a SaaS where nobody needs to access anybody else's data? OK, then you can probably simply scale vertically for a while up to $SOME_MILESTONE, and then scale horizontally via some fairly simple sharding or maybe even just database partitioning or something. No need to do that now.
> You're planning to scale to 1,000,000 or 1,000,000,000 users eventually? OK. Well, what's your service? Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Thinking this way actually leads to one of the most common failure modes. Specifically the day-dreamer death where the principals become so enamored with a fantasy of what the end-state could look like that they lose focus on the next step to improve their odds of success. This is under-reported because day-dreamers are so common that when they try a startup and fail, it's not really an interesting pattern to report on. Such founders will tend to cite specific reasons that sound better in context (eg. wrong market, product flaws, sales funnel flaws, etc). But as someone who's been in the web startup space 25 years, I've seen it over and over again where founders fail to fully engage with the hard problems right in front of them in favor of working on what's fun, easy or tied to some dopamine hit of imagined future success.
In practice, you should not spend one second thinking about the requirements of 1B users until you have at least 10M. You will need to completely rewrite everything multiple times along the way to that scale anyway, and more importantly, you won't know the product requirements to actually get there until you achieve the earlier milestones and get the necessary user feedback about what's working at each size of user base.
There's a misunderstanding. I may have miscommunicated.
I'm advocating for understanding your business, not for prematurely building things.
If you're in the extremely rare class of businesses trying to build something on the scale of FB or YouTube where that sort of massive scaling is intrinsic to your mission then yes, you need to start thinking about those problems early.
For everybody else, yeah, absolutely focus on product and getting to 100 or 1000 or 100K or whatever users first.
Precisely. It'd be like trying to make an F22 fighter jet right off the bat when you're still in the era of biplanes.
I think people often underestimate the strata of learning, and the essential feedback that gets you to the next level. Or at least that's how it works for mere mortals like myself :)
I'm advocating for thinking about if you'll need an F22 fighter jet someday.
If the answer truly is "yes", that's something you'll need to figure out early or at least plan for.
For the other 99.99% of situations where the answer is "no" or "almost certainly not" then absolutely, you should not be wasting time building an F22 or even thinking about it.
> Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Even Facebook didn’t start trying to figure that out until they had to. Famously you needed an invite for a very long time. I’d imagine infrastructure limitations factored into this.
You’d be surprised how many successful companies push off dealing with scaling problems until they absolutely have to.
> Famously you needed an invite for a very long time.
And then it opened to needing a college/university email address. Then it opened to everyone.
I got in with my college email address, and at the time it was considered a feature, not a limitation: a combination of higher quality since people couldn't just make new spam accounts whenever, plus a bit of a status symbol compared to just being on MySpace.
That's exactly right: it depends so much on the context and (expected) access patterns. Sometimes you really do need to plan ahead, but even that's more about not painting yourself into architectural corners rather than specific tech.
What I see also is not just scalability over-engineering, but the same in the solution domain. When you don't really understand your customers needs sometimes there's a desire to create something with ultimate flexibility.
In the days of Xp we used to say YAGNI when people would be overly general. I'm trying to bring that back into fashion.
IFIFY: focusing too much on scalability WILL actually hinder growth and creativity.
It will hinder growth and creativity because you are solving the scaling problem, which is a hard problem. Especially if you're early on and don't really have a product or customers.
I've seen multiple projects tank because people working on it were solving problems they didn't have and wouldn't have for the next 2-3 years at least. But they only had a few months of funding.
> (...) scalability is often overrated. (...) focusing too much on scalability can actually hinder growth and creativity.
I find this blend of comment unproductive and intellectually dishonest. "Often" and "too much" are weasel words that remove any objective meaning from the verbs they qualify, to the point where stating the exact opposite is also valid.
The truth of the matter is that scalability is only not important when you don't need it, but it's already disastrously too late if you failed to scale when you need to.
The problem is if the commenter says “doing X works” instead of doing “doing X often works” there will be a HN avalanche of “yabut so-and-so didn’t do X” and allegations of survivorship bias.
> (...) there will be a HN avalanche of “yabut so-and-so didn’t do X” and allegations of survivorship bias.
Isn't it also survivorship bias to claim that scalability is overrated because a non-scalable system didn't broke in a particular case of a local deployment with residual demand?
Meanwhile hikes in demand hitting an unscalable system can and do break businesses, and when it happens is already too late to rearchitect things.
It's often more beneficial to focus on solving a problem effectively and efficiently, rather than trying to scale it before it's even proven to be successful. By starting small and iterating based on customer feedback, you can create a better product that solves a real problem, which will naturally attract more users and drive growth.
That being said, scalability should definitely not be ignored altogether. As a project grows, it's important to plan for future scalability, but this should only be done once the product has proven its worth and a solid foundation has been established.