I'd be so curious to get a window into the profiling, back-room discussions, etc. that happens at orgs like Slack and Teams. What's going on here? What all did Teams just improve? What are the barriers they're still running into?
I don't want to be that person who says "mrahhh a new CS graduate could have built Teams in 2002 and it would have been faster!" But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting!
I've built high-complexity, high-performance web apps, and while there are challenges (the smooth infinite scroll without any placeholders in the video was actually pretty slick), there are other things like 900ms to switch tabs and 9.1s to boot that are just really hard to believe (and those are the improved numbers!)
Some of it's probably back-end; Slack for sure has back-end performance problems, where the UI is snappy but then sits there and loads the conversation for several seconds (or has synchronization problems, or loads an old version of history, or...). And I can understand having some performance challenge here- these apps deal with huge amounts of data per-workspace. But also... it's very simple data. Mainly a series of messages, displayed in chronological order. That's one of the most straightforward possible things to optimize.
I'm just so fascinated. The HN commentary around this topic is usually lazy, but there is a genuine mystery here.
Most engineers are unfamiliar with how to improve their software’s performance, unfortunately. This has nothing to do with any individual technology stack or product, really, it’s more of a general idea of how you can figure out how things are slow and make them faster. It’s just not something most people ever learn. The way performance blog posts are written on the internet doesn’t really help, either, because most of them are tired arguments about specific things being slow or one-offs that don’t apply elsewhere. Learning how to use a profiler, or have an intuitive sense for how long something “should” take, is critical here, and it takes time to develop.
For example, speaking completely independently from the problem here but putting on my performance hat, I would expect that if we take all things as a given (“no major architectural changes allowed”) Teams should not take 10 seconds to start up to show UI. We know that the tech stack they’re using isn’t fundamentally that bad. This suggests that there is probably a handful of things they are doing that are low-hanging fruit but haven’t been identified yet. Once they find those and bring down the time to something more reasonable like a second things will get much harder for them to fix.
Teams is one of Microsoft's headlining apps right now, and it's been such a PR nightmare that they're explicitly advertising a major performance improvement, complete with a slick video (when was the last time you saw an everyday app do that?)
So I highly doubt the people driving this initiative are just "unfamiliar with how to improve their software's performance". Something else is going on here: maybe they were saddled with something so bad it could only be fixed so much, maybe there's some technical hurdle we're just not appreciating, maybe it's an organizational problem (this is my guess). But Slack has a similar problem, so I doubt it's the first possibility (the 1.0 of the app just happened to be botched). There's something going on here
Here's my wild uninformed guess: Teams has lots of features/integrations, and they're constantly expanding because it's in the limelight right now, so they've got too many engineers on it working on different features at the same time and not talking to each other enough. The system is now too sprawling to be considered (or optimized) as a whole, which means there's tons of redundancy and toes-stepping going on. Shipping the org chart, etc.
I suspect (obvious disclaimer, this is speculation, but I've seen this happen many times elsewhere) is that the Teams team (heh) was asked to improve their performance for this sprint and they poked around a little bit to find some of the easiest wins, which is what we're seeing here. At most places, presenting that you've improved launch speed by over 2x is a major success, and their PR team decided it was worth advertising. The people who worked on this should be proud of the work they did but there is still clearly a long way to go to improve the app's performance.
> Something else is going on here: maybe they were saddled with something so bad it could only be fixed so much, maybe there's some technical hurdle we're just not appreciating
They started with Electron, and are now moving to WebView2. When the project started Edge was still not Chromium based (it was the IE fork) so they couldn't even use that.
One thing that id did to hit their target framerate of 60fps on (IIRC) Quake 3 is have a policy that boiled down to:
"We always run at 60 fps, because if we finish the game, then try to optimize our way to 60, we'll never hit it."
>> This suggests that there is probably a handful of things they are doing that are low-hanging fruit ...
The Quake annecdote suggests the opposite. If I had to guess (from personal experience with web development), it's probably a million-billion little things that, when added up, make your whole project a slow, gelatinous pile.
That's not to say there's not also low-hanging fruit that might make switching between conversations take less than a second, but .. it should take .. like .. a short amount of time, like 16.6ms, including the network trip.
"It's a bit of understatement to say @WebKit
project cares about performance or efficiency of the software. We're quite obsessed with it, and we throw a lot of engineering resources at it, not to regress, and to improve upon."
Let's be realistic, it would be pretty hard to hit 16.6ms. Not impossible, but very complex with little benefit (at this point you need to care about such low-level things as the way you schedule frames to be displayed by system compositor, for example - and even how the compositor itself is doing its work too!). Something like 200ms, however, should be perfectly achievable with little effort and just as snappy in practice for this kind of app :)
(note that a game running at 60 FPS does not mean that you get 16.6ms mouse-to-screen latency at all)
If we can't hit 60fps with a chat app, one of the simplest applications imaginable by man, what the hell can we hit 60 with? It's drawing a couple thousand triangles (maybe) with a couple hundred textures (I guess). Game engines fart a couple thousand triangles when they're done drawing 50x+ that. Oh, and they do about a gazillion other operations .. at 60fps+
The slow part of switching between conversations is fetching the data which, on a fiber connection in a city, is < 5ms round trip. Hell, I live in the boonies in Canada and my round trip to Seattle is only like 20ms. You could also cache the first 20 messages to every conversation (disk space is cheap) to avoid the network.
As for the notion that "you need to care about such low-level things ...", yes, you do. To write fast software you just deal with it and move on.
Being able to hit 60fps is just a single prerequisite for low latency, but it's far from enough - and most importantly, it's not what I was talking about at all. The application may be able to render itself consistently even in 2ms just fine, but effect of your action won't appear on the screen until much later unless you make your whole stack (including the OS) surprisingly complex (which is worth doing in case of competitive video games running in fullscreen, not for applications).
Seems like you're simply not aware of complexities of the input to display path. Please make a simple windowed OpenGL application displaying a single triangle in a fraction of millisecond that changes color when you press a key and measure its latency, you may be surprised ;)
Did you know that merely enabling FPS limiter on Steam Deck imposes additional latency just because there's currently no way to do it better when using XWayland? That's what "low-level" meant in my comment.
Okay, I see what you're getting at now, and yes, there are a few frames of latency between the time the app is done drawing and the time pixels are in front of your eyeballs, but that is _completely_ missing my point, and the point of the tweet.
If we're splitting hairs about how we define how long an application takes to update, we've already won.
In reality, we're talking about an app that should take ~0.01s to perform an operation (update a rectangular region with text), that's taking 1.0s instead. And that's THE OPTIMIZED VERSION. In the real world, we've already lost, which is what Casey was pointing out in his tweet.
The only measure the user cares about when it comes to snappiness while switching between conversations is input-to-display latency, and the whole point of my comment was to be realistic about it. A chat app won't hit 16.6ms latency, but it doesn't have to. 100-200ms is realistic and perfectly fine, and if you don't manage to hit it, something is seriously wrong with your application.
And yes, of course, something is seriously wrong with a lot of applications these days, otherwise we wouldn't be having this conversation at all.
In 16.6ms a modern processor will execute about 128 million instructions. We need to locate about a screenful of text and/or cat pictures. We need to perform layout and then render it. The amount of work being done is not very much, or very difficult. The resources available are luxurious.
What part of the problem is it that makes it an unrealistic target?
Because most screens are running at 16.6ms refresh rate, composition is vsynced and windows are double-buffered at multiple stages in the whole display pipeline to avoid tearing and other kinds of artifacts. You're naturally going to have 2-3 frames of latency unless you try really hard to avoid it with advanced frame scheduling that won't be worth employing (or maybe even feasible to employ) for an office application. You usually optimize application rendering for smoothness (no frame drops, no artifacts, consistent 60 FPS) rather than latency, and improving one usually hurts the other. And that's ignoring the input path, which brings its own latencies too.
Ah, ok so you are considering the latency rather than the throughput. I was thinking that being able to perform the update within 16.6ms should not be too hard - there is a sibling comment about SSD latency but the working set should fit in RAM.
Sure, so there will be latency, maybe 4-5 frames with input lag. But this is normally bearable in a game so I'm guessing that it would work fine for an office app. It would be difficult to remove, but 80ms response time from an application that can always hit its 16.6ms deadline should be workable.
This seems like needless pedantry. The computer should be able to easily do it in 16ms or less. Add on the 16ms or whatever of extra latency for the mouse and display, and it's fine.
> Most engineers are unfamiliar with how to improve their software’s performance, unfortunately.
Then they aren't "engineers" yet. Engineers can figure out how to use a profiler and how to improve performance of their software. Calling themselves "engineers" while being unable to do so should be a signal to everyone in their circle that they're incompetent hacks. They should at least be honest and give themselves an appropriate title, like "apprentice" or "intern" or "novice" or "student".
Heinlein has a quote in one of his works that goes like this:
> A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyse a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly.
I think all software engineers should be taught to be able to handle a profiler, poke around in a debugger, write a short correctness proof, and consider how an attacker might abuse the APIs they expose. I know a lot of people who are working to make this dream a reality right now. Until this happens, though, gatekeeping the definition of an engineer is inappropriate and does nothing to help improve the situation, because people are generally resistant to learning these things if you call them "incompetent hacks".
software profiler : software engineer :: ruler : <pretty much every other> engineer
Profilers are simple tools that every software engineer should be familiar with. They are literally just a way to measure performance of a software system. The basic ones do this by tracking time, or counting the number of times a block of code are entered. People have recreated them a million times (probably more) because they're so damned easy to create.
I'll take my hit for this one: If someone were to come up to me and say, "I'm an engineer, but I don't know how to measure the software I make." I'd think they were a hack, yes. If they instead were honest and said, "I'm a student studying to become an engineer ..." or "I'm a novice trying to become better..." then at least they're honest individuals and not liars or hacks. If that's gatekeeping, then so be it. They are not engineers yet if they can't use the software equivalent of a ruler.
> If that's gatekeeping, then so be it. They are not engineers yet if they can't use the software equivalent of a ruler.
It's not gatekeeping.
Traditional engineering fields have somewhat standardized expectations around what a new grad should know. Same as law or Medicine/Nursing. When someone calls himself a Software Engineer, I too have an expectation of what they should know.
It's part of what made the scene where Katherine Johnson explains Euler's method as a little known math trick to a group of bewildered NASA engineers so funny. Everyone in that room would have been familiar with that concept as it's the first numerical method encountered in a differential equation class [0].
I would love some sort of standardized curriculum for Software Engineering. No exam, just a list of pre-requisite and concepts.
[0] The real work she did would have probably required a separate two hour documentary so I understand the filmmaker's decision here.
It's fair to expect a professional cook to be able to cook well. Guarding their feelings for their incompetent cooking serves no purpose, since they become useless to their firm, and customers will want to avoid what they cook.
The same expectation should hold for professional engineers. If they are incompetent hacks, tell them they are. Guarding their feelings above describing reality to them only lets their illusion of competence to continue, and is what causes poor software, more incompetent incompetent engineers, and troubled users.
They are not incompetent hacks. Stop calling them that! They were failed by a system that did not tell that they should learn how to do these things, and are struggling to catch up. You can point out areas of improvement without being mean.
Businesses and the commercial world have no obligation to anyone, much less to incompetent people.
It is the incompetent hacks who have failed themselves, by not being aware, and by not adapting and improving accordingly. The 'system' did not fail them.
The real world is not a college or school or university, where things are fed to you on your time. Employees must learn on their own time and other people's time. If they cannot keep up, then let's be real: they are incompetent.
By blaming the system, you take away the accountability and responsibility of incompetent people. If many better programmers were able to catch up, why can't they? We can't make excuses for incompetence.
I work in the commercial world. It is critical that we help our engineers learn the skills they need to be successful, because we spend a lot of money on them. We value our employees because of their capability to learn things when necessary.
>>because people are generally resistant to learning these things if you call them "incompetent hacks".
Therefore we should hide the truth?
Sorry, it doesn't matter what field you are in, if you can only build stuff but you lack at least a solid familiarity with the tools and techniques of your trade that enable you to improve the performance of your product/service, you are simply not yet fully competent.
Period. Full stop.
If you can write code that compiles and runs, and even do it at large scale, that's nice. But have some humility and recognize that while this is definitely far better than nothing, it is NOT the end, it is merely the beginning, and you are not yet anything special.
Same applies for any field. It is really great go from nothing to being able to build functioning stuff, but it is only the beginning.
Sadly, many people get one year of experience 20 times over, and then get all full of themselves thinking this is the same as 20 years of of digging deep into the technology. Saving them the hard assessment does no one any good.
> Heinlein has a quote in one of his works that goes like this:
>> A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyse a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly.
“For as soon as the distribution of labour comes into being, each man has a particular, exclusive sphere of activity, which is forced upon him and from which he cannot escape. He is a hunter, a fisherman, a herdsman, or a critical critic, and must remain so if he does not want to lose his means of livelihood; while in communist society, where nobody has one exclusive sphere of activity but each can become accomplished in any branch he wishes, society regulates the general production and thus makes it possible for me to do one thing today and another tomorrow, to hunt in the morning, fish in the afternoon, rear cattle in the evening, criticise after dinner, just as I have a mind, without ever becoming hunter, fisherman, herdsman or critic.” - Karl Marx
> I know a lot of people who are working to make this dream a reality right now.
What are they doing?
> Until this happens, though, gatekeeping the definition of an engineer is inappropriate and does nothing to help improve the situation, because people are generally resistant to learning these things if you call them "incompetent hacks".
So they should instead be coddled?
How does it not help to simply say: Here's something everyone from a decent engineering program should know on graduation?
That's not what I said. I do not wish to have an "Electron sucks" argument here, but I am open to a "actually Electron apps should take 10 seconds to start up" viewpoint if you wish to share.
For me, the only possible explanation is that Microsoft had nobody with actual experience left. The entire Teams team seems to be cheap junior developers together with CoPilot.
It's not just that Teams is failing at solved problems, it's also that they aren't using Microsoft's own SDKs and APIs themselves anymore. Windows has good fast UI frameworks and all that cool stuff like SuperBoost which can make app startup appear instantly. Why is nobody using it?
And there is no useful integration to speak of. Why do I need to re-upload an OneDrive file into Teams? The exact same file is already stored on the exact same server. Does nobody there know about the magic of SHA512?
And how come if the internet has a hiccup while I upload a file, I have to re-upload the whole damn thing? In 1980 we already had resumable uploads via FTP.
Oh and who's the vendor creating one of the best performance tracking and optimization tools? Microsoft themselves. Did nobody ever dare to run this in a proper Visual Studio?
I'll add another absurdity to your list: why doesn't Teams use the Office dictionary (which has customizable functionality) instead of making an undocumented hard-coded call to the Google Chrome repository and using their dictionary (which is not augmented with a customizable dictionary)?
It is possible but then, for example, Windows has always had some degree of amateurish stuff in it. Rumour was, so take this with a grain of salt, that the CD loader in Windows 95/98 was developed by a high school intern over a summer holiday. Would make sense how it was stall out the entire system while when you put a CD in until it has been verified.
> I don't want to be that person who says "mrahhh a new CS graduate could have built Teams in 2002 and it would have been faster!" But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting!
> it's very simple data. Mainly a series of messages, displayed in chronological order. That's one of the most straightforward possible things to optimize.
When a company grows and add features that customers need, the data model has to grow with it. Queries necessarily become more complex.
I used to work at a big video sharing site. I wasn't there in the beginning, but I imagine the initial profile pages had code like
SELECT * FROM user_videos WHERE user_id = ?
Then some users said "hey, we want some of our videos to be private".
So then the simple query became
SELECT * FROM user_videos WHERE user_id = ? AND privacy = ?
Then they said "ok, but I want people who are mutual friends to be able to see some just-for-friends videos".
And then the query suddenly became much more convoluted. We started loading all of a users' videos in memory, then in-memory filtering, returning only the videos the viewer had access to (slow for users with thousands of videos).
Then some users came along with 30,000 videos and we had to abandon the in-memory filtering because we were getting OOMs, so we developed a hybrid system that could run queries either in-memory or in the DB depending on volume.
This is a trivial example (there are much more painful examples of data model bloat) but it illustrates the point: if you think it should be easy, you probably have an innacurate mental model of what needs to happen behind the scenes.
We're talking about Teams here, the modern UI for what used to be IRC chat. 10k users on a single server used to be a solved problem for this type of application.
Yes, that's the real elephant in the room. Slack and Teams both packaged up IRC into something that looks nicer, but runs 1000x slower and now sell enterprise licenses for this garbage.
> We're talking about Teams here, the modern UI for what used to be IRC chat.
That's like saying that Chrome is a modern UI for what used to be Lynx. Those tools offer very different features, even if they share a rudimentary set of behaviours.
While … sure, that's all true enough, I still routinely watch the UI load.
E.g., clicking the "emoji" window in Slack. I'll permit the custom emojis in the LRU emoji list to delay, but even the stock Unicode "text" emoji take palpable time to "load".
Maybe. I have more experience on the UI side, so my intuition may be off here, but I still have trouble believing that these kinds of basic slicing-and-dicing-records problems aren't solved already. Especially when we're talking separate workspaces per-org, which could each get their own DB or shard; you can't hold all those messages in client-side memory at once, but surely at the DB level it's still much less data to process than what eg. Twitter deals with. Databases are fast, and this seems like a very vanilla database-shaped problem
Twitter is very read-heavy. Think about the average timeline — Tweets don't change (they recently introduced "editing", but it's not an in-place edit). That means many things can be heavily cached, and most people on Twitter aren't writing Tweets, so the write volume is relatively low compared to the read volume.
Now think about the average chat app like Slack or Teams. Those have a very different read-write ratio. Some things can be cached for long periods of time, but many other entities become invalidated quickly, and are subject to intricate permissions policies.
I don't know how true this is. Most people in my corner of twitter tweet several times a day, some of them have rapid-fire conversations in the replies. And that cache-busts (or at least appends to) a whole lot of people's feeds, just like it would people's chat history (while in both cases, almost all writes are appends)
But idk, I'll stop speculating here because I'm not basing it on much
I think the boring but true answer is simply that no one cares.
Users don't care because Teams still basically functions and the performance problems are at most a minor inconvenience. As a result, organizations don't care because Teams is cheap and does everything they need it to. As a result, Microsoft doesn't care because dedicating dev resources to something which neither saves them money nor attracts more customers is a waste.
Microsoft employs many smart people and I'm sure Teams' performance could be improved by orders of magnitude if they wanted to. But why would they want to?
(And to reinforce my point, when the performance _did_ get so bad that people apparently actually started caring, Microsoft not only improved it but made a video advertising it. It's just that there's no incentive to optimize any further than is necessary.)
Anxiously, I would almost say "No one important cares"?
In my experience, the further up the org chart you are, the less likely you are to use Slack directly, instead deferring to assistants or at least only posting announcements periodically.
Ditto goes for Internal IT who I've commonly found never use these chat platforms instead deferring (correctly) to ticketing systems.
Those two groups are both the ones who would probably advocate for adopting these platforms yet they never use it in any regular sense so they don't feel the sharp edges
I know of mid-size organizations where everyone up to the very top complains at length about Teams being slow and buggy. But they still use it, because it's cheap and doesn't actively hinder work getting done. I don't think it's simply a matter of bean counters being out of touch.
Talk is cheap - actions are what reveal true preferences. The fact that Teams is so widely used despite it being slow garbage indicates that people actually don't care (enough to overcome whatever benefits they see in Teams).
Sorry, I imagine you posted this before my edit. I don't think this marketing video contradicts my point at all. For Microsoft to commit a large amount of dev time to rewriting Teams and improving its performance, there must have been an actual business reason. That's not something you do because some nerds on HN think your app is too slow.
Presumably, Teams was so unbelievably slow that it crossed over to being an actual problem for customers and not just a mild nuisance. So Microsoft improved performance just enough to stop affecting sales. Is the performance _good_? No, it's still comically slow. But there's simply no incentive to improve the performance further.
I'd like to see a real study done on this, but if I were to hazard a guess, I think it boils down to: organization size, feature prioritization, development process, and finally lack of knowledge or care. It's a symptom of the way software gets written in the modern day, and isn't unique to Slack or Teams at all. Companies that don't do this tend to care obsessively about it, but it's not common at all IME.
When you work in orgs with hundreds or thousands of developers working at a breakneck pace churning out feature after feature for years, on a perpetual "sprint", your architecture tends to resemble a multi-layered bowl of spaghetti. A single button click may result in hundreds of database calls (maybe concurrent, maybe cached), or at a minimum invoke dozens of javascript packages in a Rube Goldberg-esque fashion to perform an action which should take a modern computer a single digit number of milliseconds, but instead results in 10 second load times. Sometimes people just completely forget about entire code paths, and then it becomes dangerous to delete since the persons(s) who wrote it either no longer work there, or got reorged and moved on.
There are absolutely people who care about software performance, but it isn't a priority because it usually doesn't make you more money, won't get you or your boss promoted (unless it saves money), and is difficult to maintain in the long-term given the pace of change in large codebases. When you do tackle performance problems, it's because you were absolutely forced to due to necessity. In certain domains it is also more important e.g. games, embedded systems, ML applications, etc.
One other thought, is that many developers today cannot build software from first principals, as they never learned how. Many only know how to build within the context of a browser environment, and think a "minimal" app is https://create-react-app.dev/ . That's not just me grumbling, it's how our industry works. You forget the layers below you, and build new ones, each generation forgetting most of what the prior knew. The result is giant bloated applications, and a normalization of slowness in software.
I still have hope for small teams of dedicated devs. Fast, coherent software gets written by tiny teams or single individuals.
Microsoft used to be able to create really great desktop software. I thought Windows 98, XP, and 7 were incredibly slick and well done. Old versions of Office ran well on CPUs slower than those found in modern toasters.
I too am curious of what sort of... dynamics are going on internally. I blame Agile/Scrum lol.
> But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting!
Probably a few thousand features you have never heard of and will never see or use. And the vast majority of their customers will never see or use. But each one has one or two clients for whom someone decided it was vital at the time to implement that feature to please that client.
You can probably imagine one small feature that only one or two clients use that adds a very tiny, almost imperceptible amount of loading time. Now just multiply that by several thousand.
Easily solved by introducing Teams Lite and Teams Professional. Maybe Teams Lite Plus and Teams Professional Lite. Eventually everyone can get exactly what they need.
I mean isn't this the actual problem? That teams are incentivized to pump out features and not delivery quality software? I remember a time in my career where I had to spend a good 2 months coming up with ways to reduce our build time in like 2014/2015. I was lucky to work at a company where most of leadership were engineers with engineering backgrounds but I can only imagine how bad my environment would have been if the room was full of MBAs.
Seems like when you hire bean counters and MBAs, the incentive/value structure of your team/org/company changes for the worse.
This is what you get when your developer team has only ever known programming through the lens of terrible tools and practices. They’ve probably never worked on good software before.
they replaced electron with a MS Edge Web View which is still running V8. I wouldn't say they rewrote it "from scratch"—that would be more like rewriting it in C#
Making software is not challenging. A CS graduate from 2002 could build Teams. And it would be faster! You can hire dozens of engineers based on whiteboard problems, throw them into a disjointed cacophonous open-plan work environment, give them arbitrary deadlines and 10 layers of middle management, and somehow, somehow, software will still be built!
There's no reason to streamline the architecture, or the workflow of the team. If the software is going to be made regardless, why bother? Just do whatever feels normal.
Contrast this with the engineers who built the Saturn V. I'm sure a lot of brainpower went into thinking about the overall architecture of the rocket, and how the teams would be structured, and how they would communicate. Tons of process.
Anyway, as a result of this lack of organization, software teams usually self-structure into a network of arbitrary mini-siloes. The way this works at FAANGs is you'll have the Foobar Core team, the Foobar Presence team, the Foobar Video team, the Foobar Audio team, the Foobar Notifications team, etc. etc. It's like microservices for teams. And they barely communicate (unless two of these teams are lucky enough to share the same corner of the same open-plan floorspace!)
Anyway, these mini-siloes ensure that there are lots of seams between modules, and no one can optimize across boundaries easily, without stepping on the toes of someone else's team. And unless customers are actually switching to other products, there's no political willpower to do that toe-stepping in the name of saving the product.
Reminds me of how GTA Online was crawling on initial load because a single thread was parsing a 10MB JSON file.
Maybe modern software is like that. Just a hundred small inefficiencies that don't look like much when benchmarked on an individual part of the stack but combine into a monster.
Because most of them don't know the difference between web dev and software engineering. Hence the joke of claiming that a 10-second startup time for what is essentially a web app is touted as advertising-worthy.
As software features evolve over years, they gradually incur more and more incidental complexity.
Architecture suffers, code health, and performance suffers. Silly things are done, but it's hard to see they're silly because it's hard to see the entire picture
Could you build it better from scratch rather then organically? Of course. And often that's part of the solution to these situations.
tl;dr Building software is hard, and its difficulty does not scale linearly with complexity. (And yes, Teams/Slack are complex, even if they "shouldn't be")
> I don't want to be that person who says "mrahhh a new CS graduate could have built Teams in 2002 and it would have been faster!" But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting!
One way to test this...MSFT could open the Teams API and some 16 year old could try building a client.
I remember reading how MS Word formatting is such an utter PITA because of required backward compatibility going back to the first version(s) that were just done terribly. I wonder if it might be this way in all their apps
Teams is an entire web server written in an interpreted language compiled on the fly using a "Just in Time" optimiser (node.js + V8) -- coupled to a web browser that is basically an entire operating system.
You're not "starting Teams.exe", you are deploying a server and booting an operating system.
Seen in those terms, 9 seconds is actually pretty good.
Seen in more... sane terms, showing 1 KB of text after 9 seconds is about a billion CPU instructions needed per character.
But that's not really the the slow part. JavaScript with V8 is one of the fastest interpreted languages. A bare Electron app starts pretty fast. Google wants to have the fastest browser, so they invest a lot into Chromium (which Electron uses).
Even React isn't all that slow. If you look at benchmarks you may be impressed by how quickly it can do some operations. It's consistently a bit slower than other frameworks, but not that far off. There are (big) React apps that are quite fast, too.
So writing performant applications with these technologies isn't impossible.
Yet a lot of apps built with them are slow. The fast ones are rare.
I don't know what it is yet (happy to be pointed somewhere), but I'm fairly convinced that these technologies have common footguns that encourage programming things in slow ways. I can imagine it being related to React's coarse-grained and easy mismanage reactivity and wonder how much of an improvement Solid is in comparison (it is also functional, but is built to always recompute as little as possible rather than saying "whatever, it's cheap", which React does).
I don't like Javascript but you're mistaken on the performance part, even I have to give the JIT people props for how fast they've made the language in general.
Depending on specific use-case, today's JavaScript code is only 2–7 times slower than corresponding C++. That's pretty damn fast. Faster than some compiled languages. Fast enough that if you wanted to obsess over performance you can build the fastest backend framework in Node.js[0][1]
> showing 1 KB of text after 9 seconds is about a billion CPU instructions needed per character
Correct me if I'm wrong, but on a 3ghz CPU (3bln cycles per second), there are 27 billion cycles per character for 1kb of text. Take into account that on any non-stone-aged CPU you've got probably 4-16 cores, and can do ~4-8 SIMD ops per cycle (at full clock rate), so at the limit we're actually talking .. something like _3.4 TOPS PER BYTE_ ..
A typical modern CPU can retire 2-3 instructions per clock, per core. Somewhat more due to hyper-threading, so maybe 5 per clock per core. So a 3 GHz, 8-core processor for 9 seconds is 5i x 3Ghz x 8c x 9s = 1 trillion instructions. Divide by 1KB to get roughly 1 billion instructions per character.
SIMD is a little unfair, because virtually none of what an app like Teams does is amenable to vectorisation. If we allow SIMD, then we have to allow the GPU, and then we're talking about an entire different order of magnitude.
A decent last-gen GPU has something like 30 teraflops, so about 265 billion instructions per displayed character!
Well, and then there's my work laptop which seems to enjoy clocking in at 800 MHz instead of the rated 3.4GHz because 60°C is too warm or so?
But yeah, it is incredible when stuff used to load on single core 500 MHz CPUs.
Or that I had a full graphical desktop on a 20 MHz CPU once. And it really wasn't that different? Launcher of apps at the bottom, even, despite being a completely different OS…
> You're not "starting Teams.exe", you are deploying a server and booting an operating system.
Sounds about right and, actually, is quite a feat.
I just wonder how they went from "build a chat app with some filesharing built in" to "include everything and the kitchen sink and oh, in your own universe" kinda thing...
Because then they could be absolutely sure that the people they hired to build their application didn't need to know how to build the things needed to build their application, or build the things needed to build those things, etc. Looking at electron et al apps purely based on performance is like comparing the cost of a restaurant meal to the cost of cheap groceries without acknowledging the benefits. If you're a software developer, the benefit probably doesn't seem worth it. If you're a project manager hiring developers to be responsive to new feature requests from the product managers, interface changes from the UX team, branding changes from the identity team, adapt to infrastructure changes from the devs designing the services, etc etc etc then having a battle-tested-on-most-platforms batteries-included platform makes a lot of sense... even if it comes with 60 kinds of batteries when you only need 5. Especially when for the most part, only a tiny percentage of people actually give a shit about whether something takes 50ms to load vs 3s. The large operations that take 30s to load occur rarely enough (startup, for example) that most users just don't care.
Casey (author of tweet) currently has a great course out called the Performance-Aware Programming Series where he goes over the fundamentals and first principles of how computers execute programs, and what actually needs to happen for the computer to complete the actual task it's trying to do: https://www.computerenhance.com/p/table-of-contents
It's striking to try and imagine the relevance of the course to the average web programmer. Like a React developer is so far removed from these considerations that I'm not sure where he'd even begin to try and apply these lessons. That's crazy to me, that this stuff is what the computer is actually doing, and modern programmers' model of software development doesn't even have the concepts or language to relate to these ideas.
Wow, that there's a video on haversine is uniquely interesting to me. We used that to compute "all points within a circle", which is basically a loop over your candidate points, compute the distance from the circle's center, and then discard those too far away. (The candidates are generated by some conservative approximation, such as by geohashes or R-trees. And there's all sorts of room in that first approximation for optimizing too.)
There's all sorts of room for optimization. Those familiar with 3D programming probably know the "use distance², not distance" to avoid needing to sqrt(), and similar tricks apply (you don't need to work in miles: e.g., you can convert the target distance to arc distance, and use that, which is what haversine speaks natively). We were also trying to get Go to inline some of the wrapper calls (so that we could work in more sane units, like lat/lng, and have a wrapper convert to, IIRC, radians) … but alas, Go only inline leaf functions, and the trig calls haversine requires basically meant we were never a leaf.
Also hit the uniquely human problem of: if you compute the query, you can spend infinitely long doing it: the right circle is fine if centered over Wyoming, but quite a problem if centered between NYC & Philly. (So as to overlay both.) If you don't detect the case, you'll time out, and users are unhappy. If you detect it and abort, users don't get data … and are unhappy. There's no way to win.
… then the "higher level" stuff knocks you down three pegs for trying to eek a few milliseconds out of haversine when you find out the protobuf library encodes the response slower than JSON, despite being binary. (But protobuf was written in Python, vs. the JSON library is in C. There's a C protobuf lib … but its own unittest segfaulted at the time. I hope it has been since fixed, but ugh.)
Also looked at our point type in Python. As you can imagine we alloc'd a few million of those, and the simple definition in Python of,
class Point:
__slots__ = ('x', 'y')
def __init__(self, x, y):
self.x = x
self y = y
contains three heap allocations, and not small ones either, for what can be done for 16 B on the stack in some languages.
(And then we'd done some rather dumb things … which I see repeated a lot, not just at this job, like SQL databases with an ENUM column declared as VARCHAR, and the same string repeated forever and ever through the DB. All you needed was a SMALLINT, an ENUM if its supported…
Maybe not directly related to "performance" but I just need to say this somewhere...
The back button in the new Bing DOES. NOT. WORK. and it's absolutely wild to me that this isn't talked about more. "Don't break the back button" is a trope in web development.
I genuinely thought Bing was a better search engine than Google and tried switching my default (this was just before all the ChatGPT stuff). After a week I just couldn't handle it and had to revert back to Google
YouTube lately has this issue where the back button just doesn't work at all. I click to take me back to search results and nothing happens; I click again and then I see whatever I had opened prior to YouTube.
Also Google Shopping doesn't let me change my search terms: I type something new, press enter, and nothing happens.
And Google Search has been showing the wrong dates for Reddit threads, showing older posts than the time range I selected.
All of these issues have shown up only in the past month or two (maybe massive layoffs lead to massive mistakes?).
I wager that one issue is the generation gap now. I grew up with software on MS-DOS, Windows 95/NT and Unix (SGI IRIX). Shit was just fast compared to today.
Technically I blame mainly platform bloat and everything being a webview pretending to be native app these days.
But from a human perspective, younger developers never experienced how snappy desktop stuff was back then. So what seems 'ok' or 'fast enough' to them is just how sluggy stuff is, in average, nowadays.
>> If you were making a parody video, you would imagine it would look exactly like this video - like joking about how people would be proud of these timings. But IT'S REAL!
I think they used a pretty slow computer for demonstration purposes.
Cynicism about modern software aside, I’m glad to see Teams, an app that many of us are forced to use by our employers, improve performance. It was desperately needed.
I don’t think this type of performance is some kind of horrid modern software epidemic, it’s just Teams being a garbage application that was thrown together to try and answer to Slack.
Teams is also an incredibly successful product, moving far beyond Slack in terms of usage, so it's an interesting case study. My guess is that speed just hasn't been one of the top priorities, and they have instead prioritized cramming lots of features in there, both to copy Slack's functionality and to add integrations into other Microsoft products that help make the sale when they're pitching Teams to customers.
It is certainly aesthetically displeasing for such a relatively simple product to be slow, but it often makes business sense that it's not a top priority.
The problem with this argument is that software is not just a little bit slow, it is insanely slow. Like if you break down what Teams actually does, and count how many CPU cycles that should take with some quick napkin math, Teams is slower than that by at least a thousand times. Obviously this kind of first-principles thinking is idealistic, but the result should not be underperforming that by several orders of magnitude.
The real problem is just the general practice of how software is developed now. If we lived in a world where software development was sane, it would be possible to prioritize speed of development and cut some corners to ship faster, and get a result that is somewhat slower but not ten whole seconds to start up slow.
My girlfriend works for a large consulting firm. Her mid-to-high-end work laptop runs the standard smattering of PM applications you’d expect (e.g. Teams).
I watch her screen sometimes to see the differences in our workflows. It is incredible the high latencies and low speeds she has come to view as acceptable in application performance. 2-3 seconds to toggle a menu that’s toggled 30 times in a day; 4 seconds for a small file list update that is performed regularly; tabs in Edge frequently hanging; on and on this goes.
Just sad, really, especially when we have a half gig connection going both ways. The work being performed (and the features designed to do
it) should not be seen as resource heavy… because it isn’t. It’s the same exact bs that was done on computers in corporate settings 20 years ago. We’ve added collaborative editing and auto save as defaults - that’s the extent of the difference.
I wonder if this has to do with the Steve Jobs/Apple-ification of software, followed by a bastardization of those principles. It feels like during the "computers as the intersection of art and technology" era, it was understood that while the goal was to create something beautifully designed, the way to get there was through feats of technical ingenuity, and that design included a large functionality component. Then computers got super duper fast, and it became possible for web software to keep the "art" part (being visually gorgeous) but not the "technology" part (being slow as shit).
That seems harsh. Apple famously hardly uses the web stack and prefers their own platform. Their software is also pretty high performance in general and starts quickly, partly as a consequence. Software would have focused on branding no matter what, that was already a trend in the 1990s with WinAmp, every Office version having a unique look etc.
Just wait until you try Ventura. Every submenu in the systems settings menu has gone from being perfectly fast, to taking around two seconds to load (at least on my 2019 MBP)..
As a dev team we use Slack and I think it's way better than Teams, that I still have to use because all others non dev colleagues use Teams, it's free with M365 and good enough for a few messages and a video call recording.
Average human lifespan = 2.3 billion seconds. With 280 million teams users, a human life's worth of time is needlessly consumed every 8 seconds that Teams spends yanking its own chain.
I don't think it's an absurd metric at all. I think it's terribly unfortunate.
Software optimization on the whole would be a lot better off if developers took a lot more time to think about the scale of the products they work on and if users refused to put up with software and systems that don't respect users' time and agency.
If someone wants to calculate the lifetimes lost to Windows Update, I'd be interested in knowing how it compares to historical wars.
The amount of micro-stuttering, scroll lag, input lag, etc of modern software is at the level where in the past you would have assumed your computer was broken if it performed this way.
it's pretty obvious if you game. if you don't but code with efficient tools (and we have a lot: vscode, jetbrains, terminal) it's still easily worth it
I had to work with a 2.1 GB JSON file today. It was really striking that, on a machine with 4 GB free RAM, I could find literally nothing that could work with the whole thing in memory as JSON, including every GUI editor I looked at, Node, jq, etc.
less works fine on files with enormous numbers of lines, but not lines that are extremely long. Open up a 1G file with lines that are all 10K long and you'll see just how slow the pager can get!
ipython, plus a bit of magic to json.loads to get it to intern the object keys? (Assuming you have enough repetition in the keys to result in enough memory savings to get you within your budget.)
I had to do this once for a large piece of JSON that ballooned quite large when loaded into RAM. But in my case I could wait until after json.loads to do the interning; but I think you can do it on the fly with object_hook or object_pairs_hook.
Alternatively, Rust+Serde & doing as bare-bones a computation to get it working, again with an eye out to whether the decoded form will fit. (Hopefully, since Serde'ing into a struct should make the keys 0 bytes, essentially, but there would be some memory lost for pointers and the like.)
I believe emacs chunks large files and then lazily loads them to enable this. I remember having to mess around with a specific mode to get it to work in the past, but I think it's included in base emacs now.
In this case it was a single giant object I was trying to extract a specific list of keys (with unknown value lengths) from. Every method I found with jq died with out of memory errors.
Most developers don’t have a clue about performance. A friend of mine (an average developer by his own account) joined a company that had just lost a $10 million contract because of low performance software. He has a background in games development. So he raised an eyebrow, ran a profiler to figure out what the bottleneck was, and fixed the problem. In less than a week. Now he is treated like a super star by management and other developers. So knowing how to make software fast can be a real benefit.
Performance just isn't prioritized. If you spend a week optimizing making something faster, you just get hit with the ill-quoted premature optimization line and shamed for not working on higher priority tickets. Fine, I'll just leave it slow then.
My defense is usually that it's ok to have a little pride of craftsmanship, and if you let people scratch that itch they'll be happier and more content in their jobs, making it a long-term cost saver.
I can't remember which studio (maybe naughty dog), but I remember a story from the 90s of a studio who forced their developers to work on minimum spec workstations during development, with the theory that it would ferret out costly performance problems early.
I keep a couple old laptops kicking around for perf testing on and, can confirm, I routinely find perf hiccups that when added together would create a huge amount of debugging work.
While I like doing dev with a threadripper & beefy GPU, I certainly support the idea of having crappy, old test machines around.
I mean, honestly, I think all devs should. Except maybe the people who work on such cutting edge stuff that they already need every bit of optimization they can get, even on their $2k+ pc.
Though, maybe not that specific processor. Maybe more like a severely underclocked 8 core, or some really old 4 core. Having only a one core to develop with is gonna seriously hinder the development of multicore optimizations.
Because Teams (and Slack) are glorified IRC for the text portions and yet we have experiences like Teams[1]. You're seeing pent-up frustration about poor performance that we all deal with every single day, on most applications and almost all websites other than those that keep things simple (like HN).
As the speed of computers increases, software speed remains pretty much the same.
Teams is an Electron app, I think, so the whole thing needs a huge baseline in order to work.
Being proud of a chat app now only using 500MB instead of 1000MB is odd.
Obviously they do, someone prioritized making teams 2x as fast, and many other apps seem to be faster too.
The problem is they've only cared for like, the last 3 years, and everyone was enslaved by that "premature optimization is the root of all evil" quote for the last several decades, while blaming "complexity" for slowness.
The people in charge of signing the checks do care but consider Office/Outlook to be non-negotiable, and then Teams is 'free', so it's seen as the lesser of two evils.
Is it too cynical to think that Microsoft deliberately made their Teams app sluggish in order to persuade people to buy new PCs during the pandemic, but when that didn't work out as well as they hoped, they reluctantly optimized the app?
Whenever a product does something truly baffling, management and money are involved. The management forced the engineers to do something stupid, in order to sell something to someone. The details vary.
I'm for sure getting older cus I remember the times we used to make fun of ppl for being neckbeard performance nerds, and now I find myself firmly in that camp.
I don't want to be that person who says "mrahhh a new CS graduate could have built Teams in 2002 and it would have been faster!" But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting!
I've built high-complexity, high-performance web apps, and while there are challenges (the smooth infinite scroll without any placeholders in the video was actually pretty slick), there are other things like 900ms to switch tabs and 9.1s to boot that are just really hard to believe (and those are the improved numbers!)
Some of it's probably back-end; Slack for sure has back-end performance problems, where the UI is snappy but then sits there and loads the conversation for several seconds (or has synchronization problems, or loads an old version of history, or...). And I can understand having some performance challenge here- these apps deal with huge amounts of data per-workspace. But also... it's very simple data. Mainly a series of messages, displayed in chronological order. That's one of the most straightforward possible things to optimize.
I'm just so fascinated. The HN commentary around this topic is usually lazy, but there is a genuine mystery here.
Edit: A light overview of the new version was posted below (https://news.ycombinator.com/item?id=35352521) but I'm linking it again here for visibility: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...