More and more, I'm realizing this applies more broadly than just for code. Abstraction is a form of optimization and shouldn't be done before the space has been properly explored to know what abstractions should be built. Standardization is a form of optimization and shouldn't be proposed until there's a body of evidence to support what's being standardized.
Failure to validate a product before building it? Premature optimization.
Build infrastructure without understanding the use case? Premature optimization.
Build tools before using them for your end product/project? Premature optimization.
This advice comes in different forms: "Progress over perfection", "Iteration quickly", "Move fast and break things", "Don't let perfection be the enemy of good enough", etc. but I find the umbrella statement of not prematurely optimizing to encompass them all.
> Build infrastructure without understanding the use case? Premature optimization.
Hence my issues with micro services and Kubernetes/containerisation (by default.)
I've always hated the fact people simply jump onto these technologies and methodologies as if they're automatically the right solution because everyone is talking about them. What they don't understand is that they're optimisations.
You build a monolith and put it on one machine to begin with. No load balancer. Just a single EC2 instance with snapshots. As the customer count grows and demand increasing you scale it out...
Now you're on two EC2 instances and might want to consider using RDS. You have an ALB and you're using ACM to offload TLS certificate management. More customers come along and the monolith begins to slow down, so you optimise the application this time...
Now you have the most successful/popular parts of your application split out into separate components but still using the same database. You're still just running Docker on a few EC2 instances though. You don't need orchestration yet. But now your customers start demanding more features and changes on a more frequent basis. Also your customer count is rising more and more. You're now ready to scale out and re-architect things again...
Now you've got 80-85% of your monolith split out into separate components, in Docker images, and you're using Kubernetes to orchestrate the whole thing because you need to iterate and deploy parts of the software on a near daily basis.
Taking it slow and keeping things simple in the beginning allows you to focus (from a systems perspective) on stability and security, which are much easier when you have a monolith and two EC2 instances. As you need to iterate faster and more often you increase the complexity of the network to meet the needs of the development team. It becomes much harder to secure and manage, but the trade off is worth it.
That's how you optimise your infrastructure over time.
The only situation in which I would contradict my self on this point is if you're developing a product that you know will need micro services and K8s to begin with AND everyone on the team has extensive experience implementing an application in that manner.
Why people start at the K8 end state is twofold, fashion and a lot of younger people don't look at highways and see they started out as either single lane tarmac or dirt roads that were remade in place.
There's a lot more talk these days about refactoring complex systems. But its really a job for senior engineers or young folk who have time for it (Like the new grads who complain about being dumped on legacy systems :) ).
When you're jumping from framework to framework and shipping features, sitting back and considering things is really tricky, and the inertia to build from scratch but also have 20 years of refactored improvements delivered makes jumping on K8's and the magic that brings seem like a no brainer.
I really, really like your metaphor of the dirt roads remade in place. It’s fascinating to look down at the sidewalk in NYC and know that it was farmland only a few centuries ago. We know the aphorisms: Rome wasn’t built in a day, but we don’t always think them through.
At work, when a new person joins our team, it’s fascinating (in a non-cynical way) to hear “why are we using a dirt road here instead of a highway? Wouldn’t that be more efficient?” It’s fun to imagine the implications of such a project, and it’s a great opportunity to give perspective on the traffic flow of customers, developers, operations.
Could we build a highway there? Yes. But usually, it’s a better allocation of our finite resources to do some maintenance on the existing highway. Sometimes, we actually need to take highways apart and replace them with something more informal, since their maintenance costs more than they’re worth.
Something tangential this reminds me of is the concept of “Desire Lines”: https://en.m.wikipedia.org/wiki/Desire_path. The work of an SWE can be summed up as the identification and “improvement” of desire lines. If you build a highway, but it’s not on a desire line, you’ve just wasted a lot of energy!
With many apps, you can completely bypass kubernetes entirely and just use something like elastic beanstalk or heroku for scaling (just by adding more nodes behind a load balancer).
I invested several days setting up my product's backend on Elastic Beanstalk. After several weeks, I learned that Elastic Beanstalk is not reliable.
Then I spent several days learning Kubernetes and gave up when I learned that it is overly complex and mostly undocumented.
Then I wasted a week using Packer to build EC2 AMIs and deploy with terraform and run with supervisord. I learned that Packer is slow and Linux package repositories are unreliable. Both of these things could prevent me from deploying an emergency patch. Also, there are no reliable tools to clean up old AMIs generated by Packer.
Then I spent several weeks learning Docker and migrating my product's backend to that. Docker's APIs and tooling are hard to use but at least they're reliable. And I can run the same binaries in production and on my dev machine.
Throughout this process, I frequently looked at Heroku and then remembered how much they charge. I just can't bring myself to spend $500/mo for their ops automation. Perhaps I'm being penny-wise and pound-foolish.
I find it strange that you believe that Elastic Beanstalk is not reliable... That hasn't been my experience at all.
My company has been using it to host our production infrastructure for over a year and a half now, and it has been performing excellently. Our site reliability over the past year has been somewhere around the 99.99% range (and few outages we did have were not related to Elastic Beanstalk in any way).
We have our site hosted in multiple redundant physical locations (AWS regions), with a load balancer routing requests between multiple nodes.
We do about 10-20 deploys to production on a daily basis, with zero downtime. Any time there is a deployment, the code is automatically deployed in rolling batches, and the nodes aren't added back to the load balancer until they pass their health checks.
With many apps, you can completely bypass kubernetes entirely and just use something like elastic beanstalk or heroku for scaling (just by adding nodes).
You're right, but it's worth reading the rest of the paragraph:
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." -- Knuth
Knuth was trying to say you shouldn't guess. If you think you will get a performance gain, but you're not sure where or how the performance is going to suffer, hold back. But if you know you're doing something that is normally bad and you can fix it, it's OK to get on with it.
Your final paragraph makes me think we're both on the same wavelength here, but I think it's worth remembering that Knuth was not saying "never optimize until you have to"
All fine and well, just consider that Donald Knuth was talking about shaving off some cycles by carefully selecting machine instruction.
Donald Knuth, the same guy that got upset about being forced to use 64-bit pointers in a binary compiled for x86-64 even in programs where he would've been just fine with a 4GB address space.
He didn't exactly mean to say "omg wtf computers are so fast just use create-react-app", but that seems to be the general reception.
I'm at a big slow non-tech corp and as much as I would prefer "move fast and break things," I can't see it working here. "Move fast" becomes "hurry up and wait" as you end up being dependent on another group's input (10-20 people total get involved, 3-10 meetings) to move forward on something. You end up trying to over-plan because it lets you schedule things to happen in parallel. Everything has to be a huge depth first search because being on the wrong track can cost months and even be fatal. By the time you figure out it won't work, you're way behind schedule and restarting would take another 10+ meetings to coordinate. You have to exhaustively evaluate every path beforehand because it's so slow to right the ship, at an institutional level.
“Move fast and break things.” wasn’t advice given to every individual at every company. It won’t work if you’re the only person doing it. It’s the mantra from the top down for the whole company.
> Failure to validate a product before building it? Premature optimization.
I largely agree with your comment, minus this. Unless everything after the initial idea for a product is an "optimisation" - which is quite a claim - building something shouldn't be lumped together with optimising it.
My point is that it seems like you're attributing more to this idea than is fair.
Its a stretch of the original meaning, but a good one I think.
Building a product before a good indication it is the right thing to build is prematurely optimizing a business.
At any level of a project, it is important to do things in the right order, to avoid wasting unrecoverable time and effort on work that never needed to be done.
I would also add to that: adopting tools before you need them. Premature optimisation.
Too often I see people adopting tools just because it's the latest fashion or because they've seen their friends use them. But do you actually need them? The need should come first. For two reasons. First, you might simply not need that tool, so why not keep your life more simple and make it easier for developers to work on your project? Second, if all you have is a hammer, everything looks like a nail. Grow your project first, decide what tooling you need later. Otherwise you'll be constrained by the limitations of your tools. Some projects will need completely custom tooling, many won't, but you won't know until you get your teeth into the damn project.
This is good when you have infinite time. The converse is putting an end date on all of your tasks.
Building a body of evidence to support a standard, but never building the standard because there's always more to do and everything has decayed into a mess of different forks by the time the deadline appears.
Spending time to investigate the use case and requirements change. You're further down the road with no infrastructure to show for it, a changing deadline will take you out.
Building tools as you need them works fine if you have an infinite project.
Planning work with a limited time-frame means you are required to 'prematurely optimize' everything. With perfect understanding you don't hit those problems. You're trading a lack of understanding that comes from revelation and analysis with spending time on process to compensate. Neither understanding or spending time are perfect, it's unsolvable, they are separate categories.
I was thinking “premature action ...” covers it, but it doesn’t properly match the stuff in your last paragraph.
The way I usually put it is “level and division of effort should match and be informed by the level of knowledge”, but that also doesn’t cover “move fast...” properly, because some people take that to mean “I don’t know enough, so I should move slowLy” whereas often it should be “I don’t know enough so I should experiment quickly”
More and more, I'm realizing this applies more broadly than just for code. Abstraction is a form of optimization and shouldn't be done before the space has been properly explored to know what abstractions should be built. Standardization is a form of optimization and shouldn't be proposed until there's a body of evidence to support what's being standardized.
Failure to validate a product before building it? Premature optimization.
Build infrastructure without understanding the use case? Premature optimization.
Build tools before using them for your end product/project? Premature optimization.
This advice comes in different forms: "Progress over perfection", "Iteration quickly", "Move fast and break things", "Don't let perfection be the enemy of good enough", etc. but I find the umbrella statement of not prematurely optimizing to encompass them all.