"Premature optimization" is a dangerous concept. It tends to imply you can
defer all consideration of performance until you have the behaviour and UI
But, it turns out if the UI, core APIs or formats are hard or impossible to
implement quickly, it is hard to unwind them.
Now I'd have a more nuanced view of what "avoid premature optimization"
means: something more like "avoid premature micro-optimization."
The key question is: how expensive will it be to come back and change to a
faster implementation later on, and how much does it save you to write a
simple or naive one now.
If there's one function or routine that is written in a straightforward way,
going back to rewrite it should cost only whatever time it takes you to
rewrite it, and then to test, debug and release that change. So it will
often make sense to defer that work until you’re sure the requirements for
the code really satisfy the user, your understanding of the problem is
correct, there are no unknown bugs or edge cases, etc.
However, there are some things that can make changing the code much harder
and we ran into several of them...
If you get to the point you can't optimise because the fundamentals are too baked-in to realistically change - cut your losses that day and start pasting the usable parts in to a new project.
It's important to differentiate between architectural optimisation (which should be optimised from the outset) and code optimisation.
I do believe there is an argument for minimal code complexity (optimisation) in early phases - even if that leads to some latency - it will assist refactoring and onboarding new developers - later in the project team can revisit code that can benefit from being more complex.
At a deeper level, the problem here is cargo culting of rules or heuristics.
Another example is the proscription of "security through obscurity". It's perfectly valid as a component of a larger, more comprehensive security strategy. But you're gonna get hurt if it's the only component.
One particular example is using non-standard database ports. I have to fight all the time.
"It's pointless. You make it harder for users (well, not really, the one-time setup of an application maybe), and it's not any more secure because it can be port scanned."
Personally I disagree because I think port scans are the kind of thing you should, you know, pick up through other software. Their response to this is that nobody tracks internal port scanning, and that it's always external.
I give up by that point. Because you can't complain about insider threats but then decide not to bother looking at anything internal.
And then for two equally useful mental models, prefer the more concise and accessible one. Kind of like the concept of "parsimony" as used in the physical sciences.
In software, one begins to see where "best practices" come from and how they can be valuable, but also where such heuristics break down.
Getting philosophical, I believe this is a lifelong search -- we will never reach full understanding. Perfect truth is irreducible, so the best we can hope for is an strong approximation.
Which is exactly what the quote says. If performance is going to matter, you absolutely have to get your overall architecture right so after-the-fact tuning with a profiler works.
If you get your architecture wrong, you're in for a world of hurt and might have to start over.
I'm doubtful that if bzr was just a bit faster it would have taken git's place. hg was arguably better than git and it still lost.
If git took 2x the time to do everything it does today - it still would have won.
"Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified."
Above the famous quote:
"The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by penny- wise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal; and I believe the same viewpoint should prevail in software engineering. Of course I wouldn't bother making such optimizations on a one-shot job, but when it's a question of preparing quality programs, I don't want to restrict myself to tools that deny me such efficiencies."
All this from "Structured Programming with Goto Statements", which is an advocacy piece for optimization. And as I've written before, we as an industry typically squander many orders of magnitude of performance. An iPhone has significantly more CPU horsepower than a Cray 1 supercomputer, yet we actually think it's OK that programs have problems when their data-sets increase to over a hundred small entries (notes/tasks/etc.).
Anyway, I write a lot more about this in my upcoming book: "iOS and macOS Performance Tuning: Cocoa, Cocoa Touch, Objective-C, and Swift"
Admittedly, this was probably related to misconfiguration and database issues (i.e. having zero oversight or administrative maintenance of the underlying MS SQL Server). That specific local minimum might not apply to the context of the article (optimization in code and systems design).
For instance, I find test cases that spin up in-memory, in-process Spark extremely slow, but the spin up is quite fast overall in the context of a job that processes gigabytes of data per task.
I don't think that it required all the verbiage about what Knuth said or what he really meant. Make the argument you want to make. Whether it's compatible with knuth or not is rather irrelevant to whether it's correct.
On a developer's machine with local SSD, a small database and a really small working set, even large inefficiencies can become invisible because the computer is just that fast. However, add in some network latency between the application and the database, some unpredictability about the timing of requests, and a much larger data set so that you won't hit data in memory 100% of the time, and suddenly code that was working fine grinds to a halt as it crosses a threshold where it can no longer process requests faster than they come in.
Unfortunately, with modern libraries helping "abstract away" underlying architecture like databases and the network, code like this is very easy to write without even realising that there might be a problem until it hits you in production.
I think that modern library abstractions are a tip of the iceberg. There is entire copy-and-paste class of developer who doesn't understand what's going on under the hood with most anything they're doing. We're operating on a stack of a abstractions a dozen layers deep, and people rarely take the time to look more than one layer down.
These folks are competent developers who can get things done. Even big things. But they are either incapable of envisioning how the entire system will interact, or they lack the appropriate functional details of how the abstractions work "behind the curtain" to be able to form an accurate mental model. As a result, they just wire things up without understanding how everything will interact.
The Mythical Man Month talked about the "surgeon" role on a team. The programmer that's now known as the "10x" developer. That's what's needed for any project that will eventually need to scale, or you're almost certainly going to have major inefficiencies that make everyone's lives difficult for years. I'm actually considering marketing my consulting services as being that person for startups -- the architect you really need to design a solid foundation, so that you don't accidentally create a house of cards that will fall over the first time you hit the front page of HN.
Not to say that anyone can foresee all interactions in a complex system. You still have to profile and test carefully. But I do consulting and have seen systems that had, to me, obvious bottlenecks that were poorly handled. One recommendation I made brought hosting costs for a small company down from $10k/month (and spiraling up out of control) to less than $500/month. And their product wasn't even that popular; it was not nearly at a Reddit, HN, or Quora level of usage. They peaked at about 2000 requests per second. I was handling more than that on a $10/month VM for my own backend (different load, and different requirements, but theirs could have been handled by a pair of app servers and a paid of database servers, two of each entirely for redundancy).
I wish more software provided appropriate metrics for performance analysis in live systems. I've seen people blaming the CPU or the disk or the database or the network for issues, but when you actually look at the graphs, the CPU is mostly idle, IO is almost nonexistent and the database sees a few dozen selects per second at worst, but somehow the problem is not in the software... And it's really difficult to get people to believe otherwise when all you have is data from around the software itself, not from inside it.
I never see the network blamed, it's usually ignored. I've seen way to many instances of slow database queries being replaced by slow cache reads when the problem wasn't slow queries but 1000's of fast ones.
Their "multiple queries" and "data updates" benchmarks seem to be more of a test of the database and the specific database bindings than anything like production differences in database access. If you look at the data, the database is clearly being pushed to its limits, and is the constraining factor for all of those tests.
It's fine to test the bindings for speed, but they should be clear that's what they're doing. Instead they claim to be testing the speed of the frameworks ("Web Framework Benchmarks" is the topmost title on the page). I think the binding speed is rarely the constraining factor in production, and that if you had a properly scaled database backend you'd find the same languages on top of that comparison that you find on top of the "PlainText" and "Single Query" benchmarks.
In a production environment you typically have a database that is running on different hardware, sometimes in shards, and that is scaled to handle your access patterns. In that environment all queries have higher latency than a database on localhost, and you can push a single application server (where the "framework" runs) a lot farther when you're using, say, Go or Node.js, which can more efficiently pipeline requests, than you can when you're using a thread-per-connection language like Ruby or PHP.
Finally, the TechEmpower "Fortunes" benchmark is nearly 100% a benchmark of the default templating approach in each framework; Go has template libraries that are literally 10x faster than the one in the standard library and could be among the top performers on that page as well.
I guess the moral of the story is that I should contribute to the discussions around TechEmpower benchmarking... :)
The problems are that people dont know how to optimize things or are blocked by imposed architecture.
I've no words to express how much I agree with this. Developers who ship bloated software that chugs painfully even on high end computers, with the defense that "the developers time is more valuable than the CPU's", the developers who prefer to use brutally slower, but easier to use tools, they need to be reminded that this is their job. Your objective shouldn't be to make your own job easier. It should be to deliver the best product you can, and the best experience to the users. Not make them suffer long loading times, slow applications, unresponsiveness, and massive size bloat so you can write your shitty app in your breezy to use language.
If a performance choice affects the overall architecture, a public API, the user experience, or the success of your product in the market, optimizing now is not "premature". Now is exactly the time to get it right because if you don't, there might not be a later time to optimize. It is only premature if it taking the slow path doesn't matter much today and the technical debt won't accrue quickly such that it really matters in the future.
If you can safely optimize it tomorrow, without it being significantly more costly to do so than it would be today, it's probably premature. Otherwise, it's probably not.
Likewise, "optimization" doesn't just mean any code where you think about performance at all. You should always be aware of the performance of your code, just like you're aware of its maintainability, robustness, and test coverage. Performance is a fundamental aspect of the craft.
Optimization is really about improving the performance of the code at the expense of other valuable things. Optimized code either takes significantly longer to develop (burning time you could have spent on other features or robustness), or is significantly harder to maintain (usually because the optimization relies on certain assumptions that you can no longer be flexible about) than non-optimized code.
If the faster version doesn't take much longer to develop and doesn't lead to a more calcified codebase, then doing that isn't optimization. Not doing is just silly.
* I might think I know what's important/critical-path and be wrong (it's happened)
* Users might be willing to live with slowness in some areas I didn't expect and demand performance in others
* More performant solutions might come to light later
* Particularly before product-market fit, features and their optimizations might be discarded
* New development might change the critical path
OTOH, I've found it's often beneficial to organize data and processing in a way that can take advantage of parallelism like multi-threading and vectorization. Often that's the kind of thing that's harder to do later.
What is so special about efficiency? Efficiency is just one of the concerns the software development orgs are facing, together with cost, utility, usability, deployability, maintainability and so on. The tradeoff between these (and the ability to choose this tradeoff correctly early in the cycle) often defines the success of a project or an organization. Focusing on one while disregarding others can be lethal.
YMMV. Watch your controls.
You could argue that the fallacy of premature optimization has fallen many a line of business app. But if you're an engineer and you aspire for greatness, microoptimize. That's the rarer, harder-earned, harder-to-learn skill.
2. Stuff you're running is usually not running in a vacuum but as a part of a bigger system, up front micro optimizing without measuring is again blind guessing which ends up being wrong and a waste of time.
Micro optimization is a relic of the past, something that embedded/console/driver developers working on specific hardware need to care about - for the rest this stuff it's just a pointless guessing game - I've seen a few old C developers stuck in their ways try to do this to look smart and when you actually go to measure it ended up being wrong because their assumptions are outdated. Spending your time on high level system optimization seems much more worth while (picking the right algorithms, exposing the right kind of interfaces for efficient consumption, etc.).
In other words, things that great engineers need to care about :)
I'm not an embedded developer, but to reiterate, I'm just advocating for a focus on performance if you aspire to be a great engineer. This is different than being a great businessperson, or a great software developer kit / library designer.
Really ? I feel like most cutting edge engineering is done in building distributed systems, ML, high performance computing, etc. Micro optimization is where it was last century.
But, when you have to ship 50k IOT devices and they have to be under $20. Your bosses will squeeze you for every ounce of optimization.
You're working on couple M $ project total ? Meanwhile a guy shaving off a % off performance in a FB data center saves them millions a year in power consumption alone.
I took the IOT one as illustration. But even the current project, isn't that big, and they do want properly done job.
Where we apply those optimizations is, of course, driven by profiling, so those statements aren't entirely in conflict.
What I'm saying is the only way to do optimization correctly is by doing the things you and everyone else is doing when it matters - measure before and after - the rest is just assumptions that rarely pan out the way you expect them to - at least that's my experience with optimization when I was still doing game dev. You can really do micro optimization up-front
* Machine learning
* Computer graphics
...but not much else.
The Facebook app is a nightmare of terrible design that has something like 100Mb of generated code to do something that should take a couple megabytes at most. And they are a large organization with R&D by any definition.
With respect to the backend of their main application, they created things like Cassandra to meet their unique database needs.
But in terms of R&D, you could argue that Oculus is their major public R&D project. That project requires lots of performance optimization. Indeed, that is their greatest obstacle.
Maybe you could call that computer graphics, but that's missing the point.
Making the sensors and signal processors work on battery-powered hardware (the end goal) is really challenging. And then consider all the backend services that will serve these peripherals.
This Premature Optimization Koan is really just a side effect of enumerative thinking. Like if you think about the problem hard enough, you'll be able to enumerate all of its aspects and select an appropriate solution.
That could mean enumerating all the possible applications of high performance software, or it could mean enumerating all of the strategic goals of a piece of business software and identifying that performance isn't one of them.
In the former case, you're going to feel like you've enumerated enough (e.g., three examples) and then come short of the actual space of high-performance problems.
In the latter case, in my personal experience developing a line of business application, it turns out that performance eventually starts to matter. It sort of becomes the hardest problem, like people have said elsewhere.
I'm sure at Facebook they share your concern. They probably work extremely hard to reduce the huge amount of code their iOS client is built from. I'm advocating that people train to improve performance, not to ignore the problem or dismiss it as "premature optimization," at least if they aspire to be engineers.
As a user, no they don't. On android at least, the facebook app is absolutely horrid. If anything, it's way over engineered.
Oculus falls under games for me, which is unusual to find in an enterprise company, and frankly probably isn't being run much like the rest of Facebook anyway, so the point is rather moot.
>That could mean enumerating all the possible applications of high performance software, or it could mean enumerating all of the strategic goals of a piece of business software and identifying that performance isn't one of them.
I'm a big fan of optimization, and yet I've done exactly the latter and come to the conclusion that a particular admin backend that would never have more than, say, a couple dozen users EVER would be fine to be written in Ruby. It was an internal tool, and the available engineers were efficient in Rails, so I gave a green light for them to write the admin tools in Rails. There wasn't even a debate. I just said yes, go.
But when you're writing a service that should have a solid front-end that can scale, you need to design scalability into the architecture. It seems to rarely happen.
>With respect to the backend of their main application, they created things like Cassandra to meet their unique database needs.
Cassandra is good and well optimized. Point.
>In the former case, you're going to feel like you've enumerated enough (e.g., three examples) and then come short of the actual space of high-performance problems.
Those are the things that came to mind. No, I'm sure I didn't hit all possible high-performance issues. But that's probably the 80th percentile.
And at least 95% of "enterprise" problems are not treated with the respect they should be toward optimizing. That I have from several personal sources.
>They put in a lot of engineering effort to make the best client for the appropriate device.
They put in 100x as much engineering effort as they should have needed to to create a solid mobile app, and the result is a monstrosity of garbage.
Several indie developers have produced replacement Facebook apps that are far better, faster, and would actually be more robust if Facebook weren't actively trying to prevent them from working. So you can't tell me you need a hundred developers writing generated code to create two apps (the main Facebook app and Facebook Messenger; if you want to read messages, the main app has that DISABLED with a nag screen to download Messenger if you want to read messages) are actually even remotely competent (as a group) if some developer in their bedroom can crank out a better app in their spare time.
I've also written an entire app from the ground up that had about 80% of the Facebook app's functionality, written against some of the worst backend APIs you can imagine (a mix of non-standardized, inconsistent XMLRPC, JSON, and other APIs) and it took me, writing code entirely by myself, about 5 weeks. To a finished, polished product. And that's at only about 30 hours a week.
>In the latter case, in my personal experience developing a line of business application, it turns out that performance eventually starts to matter. It sort of becomes the hardest problem, like people have said elsewhere.
I don't dispute that they should pay attention to performance. I just dispute that enterprise developers (excluding exceptional situations like Cassandra) actually optimize architecturally. Sometimes they can't for structural reasons, since ideal optimization needs to happen across teams and domains, which can be difficult in an environment where each team controls a domain.
Both Knuth and Hoare would never think that carefully choosing the optimal or near-optimal architecture or algorithm is premature.
Unless the number of things you are processing is small and it doesn't matter what you do, you may as well do the easiest thing. Or, if the optimal thing is super hairy, are you sure it will be worth it?
They are just arguing for a bit of common sense: identify the most important performance issues, and make sure you work on those. There are all kinds of optimizations that people bring up that are way down the list in terms of impact.
I like Discord's compromise.