One more datapoint here. Good performance helps avoiding scaling problems. One example is an API I recently wrote. I was initially in Python and performance was OK I guess. I had to take care of horizontal scaling, how to manage the infrastructure, a bit of distributed scaling and all the other goodies that come along. By rewriting it in Rust and caring about performance I was able to run it on a single machine (which is ok for that product). That saved me quite some hustle. And my average response time went from 300ms to 29ms.
If the app would scale a lot I would still need to go into all that trouble but at this early stage the benefits of really simple infrastructure is immense.
Rewriting your C++ app in Python can help it scale from 1000 servers to 100,000 servers without an increase in total throughput (although latency may increase.)
It can also scale your power usage up from 1KW to 1000KW.
If you have a datacenter that is 99% idle, you may be able to maximize utilization by switching to Python.
Well then further excuses come flying on why he is wrong again, and how performance doesn't matter to them. I have a legacy code base that I until recently worked with that domain specific Business Intelligence product built with SOLID principles. 20% of the processes CPU usage was for the GC, whenever you made a dump of the process it was 99% fragmentation of memory and pointer chasing through object trees every time. The customers complain about performance because they aren't getting their data in a timely manner and not getting the savings they were promised with the product. It is not a game but it does math on a bunch of data and there is a requirement to do it in a timely manner, which it is failing. Stop coming with excuses that it is niche, if you are doing anything useful to the data then you might want to pick up on Caseys advice.
And what exactly is that advice? Most programs are not game engines operating upon millions of entities with the same basic logic.
The usual application where GC is most relevant is connected to 10s of services over some network protocol, and often has to load each business object/entity it operates on into memory to check permissions, sometimes even issue another network call, etc. Loading that entity from a list of pointers is absolutely no bottleneck in such a setting, at all - the way you might speed up the app is doing the work on the database itself, which is already written in some low-level language, efficiently.
Your anecdote is just a badly written application, I have plenty examples of very performant ones written in managed languages.
The advice is that consider memory layout when doing repetitive operations on your data.
> The usual application where GC is most relevant is connected to 10s of services over some network protocol, and often has to load each business object/entity it operates on into memory to check permissions, sometimes even issue another network call, etc. Loading that entity from a list of pointers is absolutely no bottleneck in such a setting, at all - the way you might speed up the app is doing the work on the database itself, which is already written in some low-level language, efficiently.
No my example isn't a complaint on it using GC language. It is that it is loading and unloading business entities multiple times and doesn't take into account that operations could be done linearly much faster instead of hunting a field in object graph to accumulate in various ways. That is how e.g. numpy works if you make sure that the array you do your operation on is of the same type.
> Your anecdote is just a badly written application, I have plenty examples of very performant ones written in managed languages.
Yes following the ideas of what is considered SOLID principals. The objections isn't managed vs. unmanaged it is going full OO and not giving a shit about how computers work.
OO encourages objects graphs scattering your data unpredictably in memory and modern CPU:s likes to have it's data in continuous memory area where it can do prefetching and prediction on where your data resides.
That's not really true anymore. RAM is so big these days that most databases can fit into it and be blazingly fast. It's all the classic shit that slows people down. Loops in loops in loops in loop with nested if's. Not understanding the correct way to structure IF statements (Common case first etc).
The point I was making is that a trip to disk is generally not occurring, and that having a soup of pointers in RAM isn't as bad having badly laid out memory on disk that needs to be paged in. Most performance being lost, isn't from memory layout patterns and not getting full pages. It's not even that much from bad branch prediction. Most of the bad performance these days is the stacks of extra app layers that exists for little benefit.
And I see comments saying that Casey is doing some strawman thing here and yet here you are with claim "1. No need".
I'm just coming here with personal experience and my own anecdote that memory layout does matter even on modern CPU:s and it isn't niche. And because it was designed with OO mindset there wasn't any quick fix available.
Starts off with premise of why performance matters and why you should care. Proceeds to only top 10 biggest tech companies as examples. Are you working in a FAANG? If not, this article does nothing to convince me that I should care about performance (even though I kinda do).
He's a games programmer. He's using examples outside his industry to illustrate that his points aren't specific to games (excuse 4). What industry would you like him to talk about?
To be fair, performance is usually part of the “does it work in real life” question, it’s just that doing sensible things is probably good enough.
The crud app at your startup probably doesn’t need to worry about cutting optimizing every last millisecond off of page load or scaling across some k8s cluster or whatever. But I’ve stopped using (or never seriously tried) otherwise good services because of flow-stopping performance problems i.e. multi-second page loads, stuttering pages, battery drain, etc.
i'm not the one you need to convince, i'm just saying what marketing collateral i need to convince the skeptical: specific, detailed stories from those sectors, ideally at least somewhat verifiable (not a pseudonymous comment your wife's cousin read on hn)
I have worked for +20 years for small companies. A "small" here in colombia is smaller than in US.
You could say that are the targets for "not care for performance"... and they ARE. *A LOT*.
One of my main competitive advantages? Speed. I work on niches were my app has 2/3 reports and my competitors +300, and the users use my app instead. Like, for example, a query instead of using +20 minutes work in 6 sec(1).
Mind you: That was before I do Rust, and start pay more care about performance. Long ago!
(1) Today I tried to hit the <1sec. Still could be better. But like this post say, many other stuff also need to be better so have the balance across the broad is challenging.
I don't think this is true. There are a lot of applications and services that I avoid because they are so slow that they are painful. I don't think you should be worrying about ensuring that your API is always <10ms but good general performance patterns (especially considering the scaling as you write code, limit yourself to O(n) unless you can be very sure that `n` will always be small) can keep you within interactive performance without needing to worry too much about it.
Absolutely hilarious and ridiculous to claim that Facebook cares about iOS app performance. In reality they have a few dozen people who care making heroic technical fixes to mitigate the damage of thousands of people adding mountains of mediocre crap to the apps.
If you care about performance from the beginning, you would never even get to the point where you’re saying “oh crap, our messenger app can barely run, let’s rewrite it in C”.
The point is that performance is so important to customer satisfaction and the bottom line that even a horrendous company like facebook would begrudgingly spend time and money to improve performance. If anything your comment reinforces his point.
The point is that performance is so far down the list of things to prioritize that [company of choice] made it to [revenue] without having to care at all about it. It wasn't worth them focusing on until they had already acquired a very large marketshare. Only when already large and successful did the scope and complexity of their system impact performance enough to bother focusing on it, at which point they begrudgingly did.
----
Performance is simply a requirement for the product, some products require very good performance, some don't.
Where you fall is a discussion to be had, because like ALL product requirements, it comes with a cost to develop and maintain.
I don't drive an F1 car to the grocery store. I don't take my minivan to the track.
The article above is bad - it treats a conversation about product requirements as an antagonistic space, where voices that may prioritize a feature other than performance aren't making judgements about how to allocate limited resources, but "excuses"... Worse - he cherry picks the most extreme take of those value judgements for his examples as easy targets to attack.
“ Where you fall is a discussion to be had, because like ALL product requirements, it comes with a cost to develop and maintain.
I don't drive an F1 car to the grocery store. I don't take my minivan to the track”
No. The authors point is that you don’t have to drive an F1 car to the grocery store. All you have to do is stop slashing your tires, dumping sugar in your gas tank and driving a Kia.
The amount of mental gymnastics you people go through to keep your delicate little worldview from crumbling is truly a sight to behold. Thank god we have people like Casey in this world to balance out the bell curve.
I somehow don't think I'm the one with the delicate little worldview in this conversation.
I think the delicate little worldview is the folks who consistently refuse to accept that performance just doesn't matter that much.
When performance does matter, it's usually because the value provided by the service is SUPER FUCKING LOW.
like say... twitter, or facebook. Which both are so worthless they can't reliably get users to pay for the service at all, and instead are selling ads to users and need to optimize the time they spend on the site.
Basically - if the difference between a customer choosing your product or not is latency measured in ms/seconds... you sell a fucking shite product.
Late to the party but this is the same point of disagreement I had with the video, though overall it was fine. Why do these performance efforts and rewrites at the companies he listed take years? How complex is the uber eats app anyway? Well, it's because most engineers at those (and other) companies do not care or know about performance, and when it actually matters to the business, have to be dragged kicking and screaming to a slightly better state by the minority who do. It won't last long (hence the continued twitter examples).
There's often a lot of low hanging performance fruits when writing code.
Need to loop over some order lines and find distinct article numbers? Use a hash-based set with O(1) access, not just a list which will have O(n). If not you'll end up writing an O(n^2) routine for no good reason, which will work swimmingly on you 10 line test order and cause grief in production.
I don't think a lot about performance most of the time, just enough to try to avoid silly stuff.
O(1) hash vs O(N) array loop is a bit more complicated - for small N linear search can be faster - no need to compute a hash function, if data stored directly in the array (instead of pointers) then there will be less CPU cache misses. Exact threshold can be determined in a benchmark but the key here here is that N should be bounded - if a user can manipulate N to be huge and you have O(N^2) somewhere in your code it is not only performance but also a potential security problem - a DoS vector.
IMHO the key to not writing silly code is to have a good mental model of how your code works and how it fits into a bigger picture which includes OS, hardware, network and other services (like a DB). This model should be updated from time to time based on benchmarks and metrics from production services but once you have this knowledge it takes almost no additional time to apply this in practice and even if you'll be taking shortcuts to save development time you'll be better aware of this tradeoff.
In my experience it is better to default to the lowest complexity algorithm. Even if linear search is faster for small N your app probably performs acceptably for small N either way. At some point being fast enough to be interactive is good enough and there isn't much point making your interactions take 2ms rather than 10ms.
However n^2 algorithms will bite you. They may be outside of your everyday working set, and even dogfooding you may not have larger datasets than all of your users. At some point one user will hit a dimension that you "expected to always be small" and get bad performance.
If one of these O(1)N algorithms ends up in the hot path you can always optimize it when profiling points it out. But O(N)N algorithms will leap off the cold path and become hot when some user starts using the "wrong" data patterns and the developers may never be aware, or at least not aware until it is too late.
Interestingly, some of the folks who harp about performance would see your "reads better" as a problem. I'm completely in agreement with the idea that, until the code is proven to be a performance problem, readability and clarity matters most. But the performance-first mavens would say the choice should be determined by Big-O performance and memory usage, then by readability.
Yeah I was thinking in a CRUD setting, where I live, where you won't be doing the operation too often. In that case the inefficiency for small N is a non-issue vs the pain of a O(N^2).
Even better; let the database handle it if you have one, calling tons of extra data down the pipe is very slow compared to even slow iteration, and even the slow iteration is often fast if you only pull relevant data
"With so much evidence refuting the five excuses, hopefully it is clear that they are ridiculous. They are completely invalid reasons for the average developer, in the common case, to dismiss concerns about software performance. They should not be taken seriously in a professional software development context."
I believe the average developer should care somewhat about performance, and depending on their industry they might need to care a lot, but I'm not so convinced for the average case.
The average developer is not working on FAANG-sized codebases. Also, I'd imagine any large systems built up over the years that are refactored would likely see great performance gains. That's just the nature of long-term software.
You know, reading these comments is absolutely hilarious. It's because of this and that, hundreds of excuses that dance around the simple truth: you're an incompetent programmer. That's it. No, you're not capable of making something twice as fast if only you cared or had the time. If you haven't done it as a constant exercise, you can't.
> It's because of this and that, hundreds of excuses that dance around the simple truth: you're an incompetent programmer.
Classic arrogance and naivety from a Casey follower. Name call all you want, you can't hand wave the reality that the business determines the requirements, and in my industry they don't care about performance until it's a noticeable problem. Oh and the requirements they gave you are solving problem X when they really want to solve problem Y, so your optimal solution to problem X needs to be deleted.
I write correct, readable code as performant as it can be in the time I'm allotted. Call me incompetent, but it's what I'm hired to do.
All this bickering and harsh feelings are stemming from the author's inability to understand that different industries have different priorities.
> in my industry they don't care about performance until it's a noticeable problem
Yeah, I've done several dozen 10x or more performance improvements on our codebase. It's not always trivial, but most of the time it's not super-hard either.
In fact just today I did a 10x speedup of a query. After a couple of hours analyzing the issue, the fix was relatively simple: populate some temp tables before running the main query. A bit more complex than just running the query, but not terribly so.
Why hadn't we done that before? Because a customer suddenly got 1000x the volume of the previously largest user of that module, and so performance was suddenly not acceptable. It's 5 years since we introduced the module...
I don't think the structure of the "business requirements" argument is correct, and I will try to explain why.
To a first approximation, the reason modern software is slow isn't due to failure to optimize this algorithm or that code path, but rather the entire pancake stack of slow defaults — frameworks, runtimes, architectures, design patterns, etc. — where, before you even sit down and write a line of "business logic" code, or code that does something, you're already living inside a slow framework written in a slow language on top of a slow runtime inside a container with a server-client architecture where every RPC call is a JSON blob POSTed over HTTP or something. This is considered industry standard.
The "business requirements" guy is basically saying, I have to ship this thing by friday, I'm just going to pick the industry standard tools that let me write a few lines of code to do the thing I need to do. Ok, but that's the tradeoff he's making. He's deciding to pick up extremely slow tools for the sake of meeting his immediate deadline. That decision is producing unacceptable results.
It's not enough just to say people have different priorities. Selecting an appropriate point on multivariate system of tradeoffs is part of the skill of being a programmer. And if there's no point on the curve that delivers acceptable results in all categories — if, given a certain set of tools, it's not possible to ship quickly and deliver acceptable performance — then it should be an impetus for the programmer, the craftsman, to find better tools, improve his skills, push the "production curve" outward, until he can meet all the requirements.
For instance, a large percentage of modern programmers don't really know how to program from first principles, and tell the computer to do precisely and only the thing it needs to do. Essentially they only know how to glue tools together. Then in their head they're like, well gee, given that skillset, I could either (1) spend a bunch of time optimizing "hot spots," writing crazy algorithms, heroically trying to fight through all that slowness... or I could just (2) deliver the business logic and call it a day. Then they call this "prioritizing business requirements." No, there's a third, alternative, better option, which is to use better tools, which might initially be harder and more time consuming and less ergonomic to use, and then learning to get good with those tools, putting in the practice, recognizing patterns, thinking faster over time, coding faster... all of this is part of what mastering the trade of programming is about.
At the end of the day, there is just an ethic of self improvement and craftsmanship that is totally missing from programming today, and it surfaces whenever this debate comes up.
The problem with this line of reasoning, eg "if you write slow code you're incompetent", is that it applies to everything that programmers do - if you write slow code you're incompetent, if you write buggy code you're incompetent, if you write undocumented code you're incompetent, if you write untested code you're incompetent, if you write code slowly you're incompetent, if you write code that doesn't fulfil all the requirements perfectly you're incompetent, and so on for each and every measure someone dreams up to measure how good code is.
You'll very quickly find there are no competent developers.
The flaw in this point is, there are really only a handful of measures that actually matter: writing performant code, shipping quickly, delivering business requirements (really this is an official sounding way of saying "doing the actual thing the program needs to do"), and eliminating bugs.
The other things are just proxies for the real measures, that people made up, and in fact are often harmful to the main goal. Like "documenting code" and "writing tests" a lot of the time are just cargo culting to make people feel like they're being responsible and following "best practices" without actually improving the measures that matter. I think that the other unlisted metrics in your "every measure someone dreams up..." are likely to fall under this category.
There isn't an infinite number of possible measures like you're suggesting, there's a finite number and a rather small number at that. You can definitely be really good or really bad at quickly shipping performant bug-free code that does its job. The problem in this debate is that one side is completely ignoring one of these measures, and trying to claim that it's because they have to prioritize the other ones, and that this is just an inevitable tradeoff, rather than that we lack the skill as an industry to do all of these things at an acceptable level. Being a good programmer may involve more axes than being a good chess player, but I think the claim that there are so many axes that it negates the existence of programming competence reductios to absurdum pretty quickly.
100%. Just claim that documentation and tests are cargo cults, take up previous cpu and memory like it’s 1985, and pretend that your shite, undocumented, unmaintainable, untestable, shift-right code is perfect.
Working with your type sucks. Confidently incorrect all the fucking time.
I mean, sure, this comment sounds nice on a web forum, but some of us have 0 hesitance bucketing others on a scale of competence. Looking at the state of software I interact with on a daily basis, I really don't care about being nice anymore.
The general problem though is that "and that doesn't matter" isn't backed up by anything but gut feeling.
When FB dove into the numbers, they found they could save 50% in hardware costs. That _does_ matter.
It may be the case that your company has a 20k server rack to support 20M of sales and so it doesn't matter. But unless you've actually looked up the cost you shouldn't be making the claim because it's unbacked; it's just a bad faith arguement.
The cost to rewrite your system to be twice as fast is about the same if you have 20k of hardware costs or 20M, but one saves 10k and the other 10M. Only one offsets the cost of the programmers
The argument is not that performance optimization does not matter at all, but rather than low-level performance optimization, like rewriting compilers, writing your own custom storage system, or rewriting it in unreadable speed-optimized C++ is worth it.
You don't need to do a deep analysis to tell if optimization is needed or not, it is enough to assume best-case improvements and compare it to your FTE + overhead cost.
Do you have many of your servers run CPU-bound C++ apps you wrote? If not, don't bother eliminating class hierarchies, optimize your core logic instead.
Do your webapps spend most time waiting on database and microservices? See if you can eliminate or cache those, the wins are going to be much bigger than rewriting it in Rust.
Is your GraphQL compiler too slow? Unless you are FAANG with hundreds of thosands of developers and hundreds of people to spare, you will get much more bang-for-the-buck with some smart caching or just getting a bigger machine for CI jobs.
According to levels.fyi, the average senior SW engineer in San Francisco is $312k/year. With overhead, the actual cost is likely $500k/yer or so. There is _a lot_ of servers you can buy for that money before you can justify maintaining your own custom version of the existing software solution.
> The argument is not that performance optimization does not matter at all, but rather than low-level performance optimization, like rewriting compilers, writing your own custom storage system, or rewriting it in unreadable speed-optimized C++ is worth it.
People use these excuses for all kinds of performance arguments besides low-level/etc.
> You don't need to do a deep analysis to tell if optimization is needed or not, it is enough to assume best-case improvements and compare it to your FTE + overhead cost.
Sure, my point is people haven't even done rough math of FTE + overhead cost or even best-case improvements while still making those claims.
---
Less w.r.t. the article and more w.r.t. li4ick's comment. I've found numerous 10x gains in performance just by swapping an O(N^2) with an O(N) one (typically converting code using a List to using a Set instead). That doesn't cost 312k and if the original author was more concerned with performance they wouldn't've done things that way.
In some sectors software quality truly does not matter beyond a certain point (and the bar is really low). The customer very rarely has the resources or the time to try all the alternatives, especially if switching afterwards is expensive so if your marketing is good and your product works and has the needed features, you get sales, and if it's not, you don't.
Is it terrible that the app takes minutes to add a few thousand numbers? Of course it is. But it does not matter, the customer is used to software being terrible so he won't waste the time and money to switch to another software that is probably also terrible.
I have no idea how you would take what I wrote as an insult. It's insulting to me that people are paid 400k a year and still think that what I wrote is somehow NOT the baseline. It's not even a debate. I'm gonna take my own advice from the past and block HN for another 3 years and get back to work. Good luck.
Ironically, this article sets up a strawman of people who claim performance NEVER matters, and then rants long windedly in a tone suggesting it always matters.
My general principle: don’t give uni-directional advice when optimizing a u-shaped loss function.
I usually find that those failing to advocate the nuanced position “you can spend too much AND too little time on perf, here is how to prioritize” are not adding useful information to the conversation.
The truth is, most startups don’t need to worry much about perf. It’s a feature that your customers don’t usually ask for at first. At the other end of the scale, giant companies invest huge sums in taming performance. And your own situation will have more parameters than just that one simplified spectrum.
The issue is that there is no need for software to be incredibly slow by default. This used not to be the case. Back in the day you just wrote straightforward code in C (or C++) and you got good performance for free.
It makes sense that people want to use a programming language that allows for rapid prototyping if they want to get V1 out of the door. That's fine. But that doesn't explain why languages like Python and Ruby are incredibly slow in the first place. These are 20 year old languages that people have invested a ton of effort into. And despite a full decade of trying to make the languages faster they're still slow to point of near unusability.
These languages, along with PHP and Javascript, were designed by people who paid zero attention to the performance consequences of the design decisions they made. Today, after industry having spend untold billions (yes, with a B) on trying to get the performance to an acceptable level getting anything written in these interpreted languages to run at an acceptable speed is painful.
We are collectively wasting untold hours because our programming environments are bad, the languages are bad, the libraries we use are bad, and the standards are bad. This is not to say it's all hopeless, but we could be doing a lot better if we didn't focus so much on "moving fast and breaking things" and pursued quality instead.
Casey does say that almost always you don't need to "optimize", but rather just need to think about the execution of your code on hardware. This video isn't quite as nuanced, but he does talk about this in other ones.
> this article sets up a strawman of people who claim performance NEVER matters
That's just ONE out of the five excuses he mentioned. And just because you haven't met people who claim performance never matters doesn't mean they don't exist.
> Yadda yadda
You're basically agreeing with Casey but make it sound as if he's saying the opposite from what you're saying.
He isn't saying that you should spend time on optimising your code, rather that you should write performant code from the beginning by not giving the cpu more work than it needs to do to perform the task, typically by avoiding OO paradigms, followings things like SOLID, etc. Then if needed, real optimisation can handle the hotspots in your code and is typically something that only a few of an orgs best engineers can do properly, but if everything is slow then its an insurmountable task for these few engineers.
Also note that “Facebook reduces the storage it needs to store user photos by 50%” is something that contributes, like, nine-figure dollar savings to the bottom line. Facebook does all sorts of crazy stuff to improve performance–I don’t think many people who work there understand how the javascript bundler works, for example. It’s a lot more than something like WebPack.
I like the positive message he leaves us with: the barrier to entry for good performance is as low as it's ever been. You can get good performance without needing hand-coded assembly SIMD vector optimizations.
So the biggest tech companies in the world, now that they are huge, are making their programs faster. OK. To me, that says maybe the reason they beat their competition to becoming huge, was that they wrote a lot of features quickly and didn't nitpick about performance. Maybe the ones that wrote efficient high performance code from the start stayed small or were out-competed.
When I was starting out using the web, one of the reasons for using Google over competition was that it was fast. The home page loaded fairly quickly, and the results were instantaneous. Google spent a lot of time optimizing for performance during its early years and they were explicit about it.
The problem with this argument is that there are many contradicting sample points. Like chrome.
On one hand, Google created the JS-Engine V8 which finally made the web fast. But on the other hand, did nothing to fix the RAM issues that came with that until recent years.
So I see it as "Let's do performance only when it's an advantage for us". Which all of the OP examples falls to. OP want to conclude from that that performance=advantage so everyone need to work on it all the time (because who will ignore advantages?), but all we see is if(performance=advantage){refactor}. Which does not support his conclusion.
Example cases are startups, monopoles, high cost (like refactoring bank code) etc. which have other silos of advantage.
I mean I'm not parroting anything, but OK I will also agree that this isn't an especially mind blowing point that I was making. Probably a lot of other people would make that same point too!
He didn't refute "start faster, refactor later" at all, with any of his examples. So him saying he did, does not matter. "Evidence" as he says, is needed.
“For example, one argument would be that the evidence I’ve presented here is consistent with a strategy of quickly shipping “version one” with poor performance, then starting work on a high-performance “version two” to replace it. That would be completely consistent with the evidence we see.
But even if that turns out to be true, it still means programmers have to care about performance! It just means they need to learn two modes of programming: “throw-away”, and “performant”. There would still be no excuse for dismissing performance as a critical skill, because you always know the throw-away version has to be replaced with a more performant version in short order.
That kind of argument is great. We should have it. What we should not have are excuses — claims there is no argument to be had, and that performance somehow won’t matter anywhere in a product lifecycle, so developers simply don’t have to learn about it.”
The author doesn't really debunk any of what he calls "excuses" when he jumps immediately to Facebook, then Twitter, Uber, Netflix, and Microsoft. Those are companies with huge scale problems. He even says as much, near the end, claiming "the evidence against these excuses comes from some of the largest and most financially valuable companies in the world". Nobody is arguing that performance doesn't matter at those scales.
Here's the author's five points, and how at least one of the examples he gives proves the reasons.
No need. These companies operate on the leading edge of hardware performance, on purpose. They can't just go out and buy faster hardware, it doesn't exist. Google even builds their own, just to optimize for their uses.
Too small. Again, at the scale of Facebook or Netflix, a 5% performance gain translates to an enormous advantage, which leads directly to the next point.
Not worth it. Here again, we're talking about saving millions of dollars but only because the systems are so enormous.
Niche. Facebook, Twitter, Netflix, and Uber's performance needs are a niche of their own.
Hostpot. Here we can get to a specific example the author quotes. "Cutting back on cookies required a few engineering tricks but was pretty straightforward; over six months we reduced the average cookie bytes per request by 42% (before gzip). To reduce HTML and CSS, our engineers developed a new library of reusable components (built on top of XHP) that would form the building blocks of all our pages."
So their Facebook does have a hotspot, it just happens that it's a very large spot on a colossal size system.
Finally, the author says, "If you look at readily-available, easy to interpret evidence, you can see that they are completely invalid excuses, and cannot possibly be good reasons to shut down an argument about performance."
Performance matters, but I’m not sure people are saying that it doesn’t. I’m not a performance “first” type of programmer by any means, but I always keep it in the back of my mind, what I also tend to keep in the back of my mind is cost.
I recently “inherited” a couple of back-end services when a developer left our company. It turned out that the code was terrible and that they haven’t used, any, or our helper tools. Since we use Typescript everywhere ignoring our quite opinionated and a little fascist linter rules is almost impossible, but the developer in question had the authority to turn it off, which they did and in doing so shot themselves completely in the foot. The back-end services were developed in JavaScript more than Typescript and since both our linters and usual teat pipelines were disabled, and since it’s software that has been developed over almost a years worth of changes, it was just horrible. We’re talking loops comparing values, that were probably there once but are now just sorting things as undefined === undefined kinds of terrible.
The performance was also atrocious. Basically what the service did was gather info on a couple of thousand projects and link them with tens of thousands documents in Sharepoint, but because it was build wrong, it wasn’t pulling the correct documents and it was taking 5-10 minutes each run time. It’s now running at around 10 seconds for its complete run time. Which is a massive performance improvement, and it’ll be even better once I finish building the cashing. So you might think I’m inclined to agree with the article, but I didn’t rewrite it because of its poor performance, I rewrote it because it didn’t work correctly and the performance gains were simply a happy “coincidence”.
This is because the performance didn’t really matter. Yes, it was costing us at most $77 a over our 3 year Azure contract, but the time I spent rebuilding it cost the company almost exactly $1500. Those $1500 were well spent because it wasn’t working, but would they have been well spent in terms of performance? Not really. That being said, it wouldn’t take a lot of those services to become expensive, so it’s not like the author is really wrong either. It’s just that I’m confused with whom he is arguing.
In my career I've meet people who genuinely don't care about performance. For them code either works or doesn't. Back in the days when everyone used bare hardware if a developer would push a change which suddenly requires 10x more hardware (or makes 10x more requests to other's team service) it would break the system and force to do something about it - either order more hardware (takes time) or fix the software (usually faster). Nowadays cloud would auto-scale and such change would go unnoticed, the only difference will be the next month's cloud bill.
The points he lays out at the beginning of his post are basically Martin Fowler's (and thus 90% of enterprise engineers') position in Refactoring. Fowler was probably responding to engineers making code inscrutable in the name of (dubious claims of) performance, and Muratori here is responding to the ensuing overreaction.
> [bad developer burns a year worth of salary building something that doesn't work]
The cost you cite for the rewrite implies a week or two worth of work. Why did this developer spend an entire year on it?
Really bad developers are really bad, yes. Double work is very expensive.
> I didn’t rewrite it because of its poor performance
> ... massive performance improvement, and it’ll be even better once I finish building the cashing.
So even with the massive performance improvement, you still need to improve the performance even more? It sounds like you would of had to rewrite parts of the system even if it was decently written to begin with.
> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 MBs off the total machine code size and bought us enough runway to keep development going.
Wow. You know you have made bad choices when you need a desperation play like that.
I kind of wish Casey would stop making these videos. I'm being completely serious when I say that this kind of knowledge and mindset gives you pretty much a superpower in the industry. Everywhere I go people think I am amazing, even though I just know a few things about performance and low level programming (like, enough to be amazing for "mere mortals", but someone at RAD would snicker at my knowledge). If the industry doesn't care, it's simply a competitive advantage for you as an individual.
I know what you mean; I’ve made a lot of money by cleaning up other people’s messes. Still, I’m pretty sure I’d prefer a world where software was just better overall and there were fewer messes.
I had a two year stint where I ran around doing "database optimisation" projects that were a week of billable time at consulting rates, and my entire "toolkit" was to simply flip three boolean flags from bad to good values.
Now in the public cloud my gimmick is to switch VMs over to current-generation AMD EPYC models, and then collect my pay-check for the "performance tuning".
The fruit is hanging so low that I'm trampling over it. There's no actual fruit picking going on.
I never got enough of this type myself, they were just random gigs between other types of project work.
IMHO, the IT industry as a whole has internalised wastefulness and accepted poor performance because there's an expectation that "CPU upgrades will fix it". Over five decades that was true, it's just that few organisations have adapted to the new reality of only ~20% uplifts in single-thread performance over the last decade or so.
In principle, there ought to be many more optimisation projects, but unfortunately they're still rare.
Scale / data set size tends to expose performance issues. But that's a good problem to have.
Yes, it would be great if all developers could write performant code, but let's face it - there's only so many hours in a day and days in a week. Developers already struggle to keep up with all the required knowledge. It's not that people don't want to be competent. We're building more and more complex things while expanding the number of people employed building software which means average skill level is probably slightly decreasing.
Part of the point of Casey's videos (Previous one https://news.ycombinator.com/item?id=34966137) is that it _doesn't_ take more time because very often the simple thing is the fast thing.
So if most people unlearned the _more complex_ way of programming, they might end up with simpler _and_ faster code.
True, but it's normal in many professions that lots of people working in it are not passionate and industry just deals with it. We can't expect that average software developer is going to be passionate and spending their weekends honing every-increasing set of skills: technologies, platforms, tools, etc.
The entire startup ecosystem is based on extracting from the economy people who are motivated, do their work efficiently and learn high skills, and giving them millionaire-or-bankrupt stakes to ensure they’re committed.
The expected value from start-up shares is so low, you're bit of idiot if you're degrading your life for it.
Actual founder? Sure. Early employees get a raw deal. Honestly it better if you earn the most from salary, and invest the extra cash in startups. You'll get more equity.
It's often not about learning more, the techniques and methods in Casey Muratori's streams are often very well understood and even taught in school.
Dismissing people as just being there for the paycheck - that's you and every other developer.
The number of people who produce useful OSS projects because they just like the project and aren't filling out resume points to hope bigtech senpai notices them is astronomically small, and usually the sort of political project nobody wants to touch.
I think its usually that their competence is judged from above based on their speed of development as that's the only metric people above them can understand. So quality is ignored and performance is an aspect of quality - many things can be made to perform ok for the demo and the first 100 users.
Do you want to reduce your compute costs, or improve the user's perceived response time? Those are quite different problems. They're approached in very different ways.
I agree performance is important and we should aim to improve it if we can, but the cost x benefit of these improvements must be evaluated. It's easy to believe we're making huge gains when there's actually little gain added overall.
Two examples:
1. I've seen people mentioning that following good programming practices make the code slower, and by removing them you can have improvements around 40%. That sounds like a great number, until you realize the real bottleneck are other things (e.g. database queries, network latency, etc). When you calculate the overall improvement for the request, the gains are negligible.
2. There are some frameworks that market themselves as crazy fast: "If you use us your app will boot almost instantaneously!". Looks cool, until you realize that a good pipeline will gradually rollout a new version and this will take time. Usually it comes with monitoring the new version for a while and then after it's deemed healthy we switch versions completely. Now instead of waiting a few minutes + 10 seconds, you will wait only a few minutes, which doesn't make much difference.
Performance gains will come with tradeoffs and, before committing to that, it's a good idea to evaluate what are the real benefits of doing the changes we're planning to do.
It bothers me that all the examples provided shows that companies care about performance ("Company $X made their app 20% faster"), but none of them talks about the impact of caring about performance ("Company $X made their app 20% faster, and saw a 200% increase in revenue as a result")
I wonder what Casey thinks about using Javascript frameworks like Electron or Mobile Angular UI for everything, including desktop apps and mobile apps.
I’ve never seen anyone make the argument that performance is not important. I have seen people make the argument that performance is less imprtant than some other property (maintainability, extensibility, legibility etc. etc.) given certain aims and constraints.
If you say that maintainability, extensibility, legibility etc. etc. is more important than performance then you are, in fact, saying that performance is NOT important.
But those are mostly just the excuses that Casey is ranting about.
There's nothing inherent in writing fast code that makes is not maintainable, not extensible, not legible not etc., not etc.
In fact, the things that make code less maintainable and less legible are often the things that make is slower.
Things like replacing new Foo() with FooFactory and replacing FooFactory with FooFactoryProvider and then encoding logic in XML files.
Those are things that people do way more often that writing SIMD intrinsic.
And there are those who way overemphasize supposed "legibility" over performance.
In JavaScript land you get legion claiming that the world will end if they rewrite unnecessarily slow map(), filter(), forEach() etc. with a regular for loop.
That's literally one of the excuses he debunks. Case in point: it is such a large factor on customer satisfaction and the bottom line that large companies such as Facebook would spend significant sums of time and money improving performance.
Yes, clearly in that case performance is important but is performance always the most important thing?
I read the article and didn't find it convincing, I would argue that it's clearly not always the most important thing. This isn't an "excuse" it's a calculation that teams make, the author feels that people make the tradeoff at the wrong point but instead of making that argument he frames decisions not to prioritise performance as "excuses" which is bullshit. There's always more performance optimisations one can make and there always comes a point where it just doesn't make sense to do so for all manner of reasons.
A trivial example: I have a script which downloads a few thousand GIS Shape files and converts them into geoJSON. It runs automatically once a month, usually whilst I sleep. A run takes about 5 minutes at the moment but there are a couple of things I could do to make it run in a fraction of that time but then the script would be two or three times longer and more complex, and I'd have to spend a couple of hours writing and testing code, (there'd also be some edge cases that I'd need to account for which the current setup allows me to ignore). I judge that to be a waste of time which would make anyone who has to take ownership of this script in the future's life more difficult. So that's my "excuse" and I'm sticking to it.
Except he hasn't debunked anything. He's pointed out the specific niches where performance matters and one of the 5 "excuses" he raises is countered. Some software counters some of the "excuses" some of the time. Not all software counters all of the "excuses" all of the time.
If anything, the article points out the accuracy and value. of the five metrics for evaluating performance needs over other business needs.
Obviously performance matters, it's just a question of opportunity cost. The article just takes some absolute statements about performance not mattering and argues against them, which is obvious and pointless. They ignore that the benefit/cost ratio for Facebook's performance improvements is vastly different to that of the average software company with tens of employees and some SaaS CRUD app. Their own example of Uber's app rewrite due to their previous architecture not being good enough is an example against their point - they got to where they were with imperfect architecture, then later fixed it once the opportunity cost tradeoff made sense.
Did you read the same article I did? He explicitly states exactly that:
> For example, one argument would be that the evidence I’ve presented here is consistent with a strategy of quickly shipping “version one” with poor performance, then starting work on a high-performance “version two” to replace it. That would be completely consistent with the evidence we see.
> But even if that turns out to be true, it still means programmers have to care about performance! It just means they need to learn two modes of programming: “throw-away”, and “performant”. There would still be no excuse for dismissing performance as a critical skill, because you always know the throw-away version has to be replaced with a more performant version in short order.
> That kind of argument is great. We should have it. What we should not have are excuses — claims there is no argument to be had, and that performance somehow won’t matter anywhere in a product lifecycle, so developers simply don’t have to learn about it.
> But even if that turns out to be true, it still means programmers have to care about performance!
Who is he even talking to at this point? He's arguing that performance has more than 0% relevance to software, which no-one on disagrees with.
>claims there is no argument to be had, and that performance somehow won’t matter anywhere in a product lifecycle
No-one says performance never matters. People disagree on what performance merits optimization. A 5s to 1s page load improvement is massive, a .5s to .1s improvement starts hitting diminishing returns in user experience.
Yes, there are definitely people who argue that performance never matters.
And the research Facebook (and others) have done clearly shows that a half–second loading time on a webpage loses them a lot of money. So much money that they were willing to have dozens to hundreds of engineers work for years to fix it across all of their apps and their web servers and apis. Presumably it costs a small–time business a similar percentage, but if that’s not enough money to justify paying a developer to fix the problem then by all means spend the money elsewhere.
>Yes, there are definitely people who argue that performance never matters.
Never seen one, post an example.
>half–second loading time on a webpage loses them a lot of money.
I didn't say milliseconds don't matter, I said it was diminishing returns. The size of those returns depend on the business. For Facebook it matters because users have other options to use their attention on such as Youtube and other social media. A B2B SaaS such as some CI/CD tool can care less about it because their users have lock in, they can't just switch tab to a different CI/CD tool and start using it immediately. So the threshold of acceptable performance depends entirely on the use case, which the author keeps ignoring to fight strawman absolutist arguments instead.
He's bringing out the men made of straw because there's a culture in many places, including my work, of YAGNI.
We did not properly federate many of our services and aystems, it is costing us tons of person hours to work around or deal with as our org chart grows. However, it's exactly the kind of thing that unlike Facebook - it is difficult to get metrics on how many dollars it will save or create, so it doesn't get fixed and people still wonder why anyone would care enough to fix it.
The problem is he has presented strawman arguments. If you are genuinely arguing with someone who believes performance never matters, stop arguing, you won't change that person's mind.
Generally you will be arguing with a person who is considering the opportunity cost, but has considered it so many times and performance always lost, so they start saying (but not meaning literally) never.
To this person you need to make the argument, "This time is different because..." and avoid the strawman arguments from the article.
generally i'm arguing with a novice programmer who thinks they're an expert, but doesn't know anything about performance, but thinks performance isn't ever important
that said, it's just as common to be arguing with a novice programmer who thinks they're an expert, but doesn't know anything about performance, but is convinced of the critical overriding importance of the difference between n lg n and n squared when n is 5
Haha, yes I have had similar experience with people focusing on asymptomatics. I think you are correct, these arguments could be good for a junior, but definitely not an obstinate one.
I once was brought into a team that fervently bought into the "hotspot" argument, blustering ahead under the notion that performance was tomorrow's problem where someone would spend a day with a profiler and it would all be fixed.
In reality their project was death by a thousand...neigh million or billions...of cuts. Poor technology choices. Poor algorithm choices. Incompetent usage (e.g. terrible LINQ usage everywhere, constantly). This was the sort of project where profiling was almost impossible because any profiling tool barfed up and gave up at every tier.
Profiling the database was an exercise in futility. Profiling the middle tier was a flame graph that was endless peaks. Profiling the front-end literally crashed the browser. I ended up having to modify Chromium source to be able to accurately get a bead on how disastrously the Angular app was built.
This is common. If performance doesn't matter to a team, it will never be something that can be easily fixed. Maybe you can throw a huge amount of money at the problem and scale up and out to a ridiculous degree for a tiny user base, but making an inefficient platform efficient is seldom easy.
Is there no performance gain that’s small enough to be not worth it? If there is, then we’re just haggling over where the line is. If there truly isn’t, well, enjoy that! Because you won’t be shipping.
The author fails to make the business case for their arguments. Performance tuning costs money. Don't spend money with no expected ROI.
Yes at Facebook with end user facing software it is crucial to get performance right. If you're running payroll at a 3 person company it doesn't matter if the software is inefficient. Most of the time it's Excel, and that is not the most efficient way to do those calculations. But it's not worth investing in a better solution until processing payroll is a problem.
If the app would scale a lot I would still need to go into all that trouble but at this early stage the benefits of really simple infrastructure is immense.