I built hud after getting burned by blocking in async Rust services during one too many 2am debugging sessions. Blocking on Tokio worker threads can quietly tank throughput and explode p99 latency, all without panics or anything obvious in the logs.
Most profiling tools technically work, but they expect you to reason clearly, correlate timelines, and keep a full mental model in your head. That’s fine at 2pm. At 2am, it’s hopeless. hud’s goal is to reduce cognitive load as much as possible: show something visual you can understand almost immediately.
The UI is modeled after a trans-Pacific night cockpit: dark, dense, and built to be readable when you’re exhausted.
Under the hood, hud uses eBPF to track scheduling latency—how long worker threads are runnable but not running. This correlates well with blocking (though it’s not a direct measurement). You can attach to a live process with no code changes and get a live TUI that highlights latency hotspots grouped by stack trace.
The usual suspects so far: std::fs, CPU-heavy crypto (bcrypt, argon2), compression (flate2, zstd), DNS via ToSocketAddrs, and mutexes held during expensive work.
Tokio-specific (worker threads identified by name). Linux 5.8+, root, and debug symbols required.
Very open to feedback—especially around false positives or flawed assumptions. Happy to answer questions.
I built hud after getting burned by blocking in async Rust services during one too many 2am debugging sessions. Blocking on Tokio worker threads can quietly tank throughput and explode p99 latency, all without panics or anything obvious in the logs.
Most profiling tools technically work, but they expect you to reason clearly, correlate timelines, and keep a full mental model in your head. That’s fine at 2pm. At 2am, it’s hopeless. hud’s goal is to reduce cognitive load as much as possible: show something visual you can understand almost immediately.
The UI is modeled after a trans-Pacific night cockpit: dark, dense, and built to be readable when you’re exhausted.
Under the hood, hud uses eBPF to track scheduling latency—how long worker threads are runnable but not running. This correlates well with blocking (though it’s not a direct measurement). You can attach to a live process with no code changes and get a live TUI that highlights latency hotspots grouped by stack trace.
The usual suspects so far: std::fs, CPU-heavy crypto (bcrypt, argon2), compression (flate2, zstd), DNS via ToSocketAddrs, and mutexes held during expensive work.
Tokio-specific (worker threads identified by name). Linux 5.8+, root, and debug symbols required.
Very open to feedback—especially around false positives or flawed assumptions. Happy to answer questions.
Interesting framing. The article makes a compelling case that we're seeing the hangover from 14 years of ZIRP-fueled hiring rather than an AI apocalypse.
But I'm curious what people think the equilibrium looks like. If the "two-tier system" (core revenue teams + disposable experimental teams) becomes the norm, what does that mean for the future of SWE as a career?
A few scenarios I keep turning over:
1. Bifurcation - A small elite of "10x engineers" command premium comp while the majority compete for increasingly commoditized roles
2. Craftsmanship revival - Companies learn that the "disposable workforce" model ships garbage, and there's renewed appreciation for experienced engineers who stick around
3. Consulting/contractor becomes default - Full-time employment becomes rare; most devs work project-to-project like other creative industries
The article argues AI isn't the cause, but it seems like it could accelerate whatever trend is already in motion. If companies are already treating engineers as interchangeable inventory, AI tooling gives them cover to reduce headcount further.
For those of you 10+ years into your careers: are you optimistic about staying in IC roles long-term, or does management/entrepreneurship feel like the only sustainable path?
As someone with many animator friends, this sounds very bleak. Their work processes are very similar to software engineering, with the difference that their hiring process is much quicker (they just show a bunch of reels and shots they made for previous films or cartoons, and they’re hired). The sad part is that they have almost no labor rights, and competition is incredibly high, which means pay is very low and turnover is high. All these years, I’ve been setting my expectations that one day my field may become like that.
The difference is animation is part of the entertainment industry and software engineering is part of every industry. I don't really think it will ever be like animators are today, but I wouldn't be surprised if wages fall.
#1 is inevitable; the Pareto distribution is everywhere. I read a paper on how it naturally shows up when people can act on preferences, but can't remember the paper (and it was in the context of why cities show a Pareto distribution)
#2 there will always be craftsmanship companies, but they will always be small companies, or a small team within a big organization. Craftsmanship doesn't scale; engineering does.
#3 contracting won't become the default. Many places have experimental or well-contained projects, or not enough ongoing work for full-time, but anywhere that custom-built software is important to the business will always need changes and maintenance. The problem with using contractors is that after their contract is over, they go find another contract, so they may not be available when you would like to re-use them, and then you've got to start over with a new one.
"Consulting/contractor becomes default - Full-time employment becomes rare; most devs work project-to-project like other creative industries"
I'm in the tech industry and have been doing this for 12+ years now. In the beginning, it was because I wanted to live overseas for a few years, without a break in my career.
Now, it's about survival. I buy my own health insurance (me and my family) in the marketplace every year (so I'm not tied to an employer), work with multiple clients (I never really have to worry about getting laid off), and make much more than a FTE.
While all my friends in tech are getting laid off or constantly in fear of getting laid off, I don't have to worry.
I also find that because I touch so many different technologies, I have to turn down work. I turned down a company last year, that wanted me in-house and one this year that would have been too demanding on my schedule.
I was consulting like this too, but my wife also had a full time corporate job. It all kind of ended with her team having layoffs around the same time that most of my clients figured out that they could use AI to do most of my work at a tiny fraction of my rate. I had built up a lot of experience working with oil and gas software, and I thought that was pretty solid, but it's hard to compete with $20 a month. Anyway, after all that, I found a stable job with good health insurance.
Have been doing this for ~3 years as a way to bring into the industry. Has more or less worked (to that end), but the art of finding + maintaining a pipeline of potential clients while working contracts full-time to sustain a family is.. difficult. (And now that I'm qualified to get a real job in the industry, many raise eyebrows at the plethora of short-term contracts).
I'm happy to hear it's been working out for you, though. How do you manage/succeed?
Almost always they start by having connections that hire them (old colleagues, former friends, etc.), building out those connections (conference talks, doing really good work, writing high quality blogs), and then if you're lucky, some word of mouth.
Good points - admittedly, I didn’t put enough effort into building connections through different pipelines back when I was contracting. Upwork and a few personal connections were my sole sources.
It just felt really difficult to do both the engineering work while trying to do customer development at the same time.
The fact that OP has been able to do this for so long, while supporting a family, piqued my interest.
I am 10+ years into my career. I don’t think mgmt / entrepreneurship feels like the only sustainable path. But I believe I may become a manager of a 5-10 Claudes.
I always saw it as bimodal (even before LLMs) - you have folks working in regular shops making average comp, and you have the big tech elites earning high base + equity comp.
The toxicology is pretty damning: codeine without morphine in stomach contents means someone crushed up Tylenol-3 and gave it to a 12-day-old baby. That's not a metabolic quirk—it's
homicide.
Koren's "ultra-rapid metabolizer" theory provided cover, and his research went on to clear 17 other caregivers in similar deaths. How many of those were actually murdered infants?
2035: Apple takes 30% of my Patreon, Google matched it through their "Competitive Parity Agreement," and the EU fined them both €2 billion which they paid in 45 minutes of revenue then raised fees to 32% to cover legal costs.
The real innovation was convincing us this was inevitable.
Surprised London holds this position given the cost of living. Housing alone eats such a massive chunk of salary that I'd expect talent to gravitate toward cities where their equity/salary goes further. How do early-stage startups compete for engineers when rent is £2k+ for a one-bedroom?
>> Surprised London holds this position given the cost of living. Housing alone eats such a massive chunk of salary that I'd expect talent to gravitate toward cities where their equity/salary goes further. How do early-stage startups compete for engineers when rent is £2k+ for a one-bedroom?
SF's challenge is that the business distract is split across 2 areas (SF, SV) 50mi apart, with extremely sparse public transit in SV. Everything is doable, just be prepared for $100 Uber bills as you go between meetings.
In NY the business district is thankfully mostly centralized. However, poor commuter train service outside of Manhattan makes everything more expensive as there is insatiable appetite in central areas to avoid the commuter trains.
Yeah, I've lived in London, New York and San Francisco for work and the first had the lowest cost of living in absolute terms. Local developer salaries are, however, shockingly low outside of finance and consulting to US eyes.
The public transportation point is definitely key: London is just so unspeakably large spatially and it's all more or less well-connected that there isn't the same scarcity of commutable apartments as in NYC/SF. It wasn't uncommon for older colleagues to even commute in from Kent or elsewhere in the English countryside -- and often their morning train wasn't much longer than my own.
The issue is that wages in London are much lower for tech than in SF or NY. If you want to make good money in London, you have to work in finance, and preferably in the investment business.
London is a very expensive city, and the median income is actually very low compared to it. There is a high concentration of people who are very well off, but everyone else is struggling.
High rents encourages startups to be founded by those who haven't coupled up and who choose to live together.
A company of 3 single founders in their mid twenties that rent a two bed flat and then live and breath nothing but their startup collapsing on the couch each night can make two $50k angel cheques go a long way.
Edit: SEIS allows a friends and family seed round if up to £250k in London that the government will rebate 78% of in taxes (50% immediately and 28% if the company eventually goes bust).
Most live in flat-shares, which honestly I prefer to living alone anyway. You can easily find central London flat shares for £1000 a month which are walking distance to various central locations, and extremely quick by bus/bike/tube.
I've lived in London for 8 years now as a student (undergaduate masters then PhD) and have lived on around £20k a year (post tax) and it's completely fine tbh. I earn more through extra work but save pretty much all of it.
Is there a computational advantage to constructing φ geometrically versus algebraically? In rendering or CAD, would you actually trace the circle/triangle intersections, or just compute (1 + sqrt(5)) / 2 directly?
I’m curious if the geometric approach has any edge-case benefits—like better numerical stability—or if it’s purely for elegance.
Fair enough—I wasn’t imagining tiny compass-wielders. I was thinking more about whether the structure of a geometric construction might map to something computationally useful, like exact arithmetic systems (CGAL-style) that preserve geometric relationships and avoid floating-point degeneracies.
But for a constant like φ, you’re right—(1 + sqrt(5)) / 2 is trivial and stable. No clever construction needed.
What’s the latency on a hostcall? A PCIe round-trip for something like File::open is fine—that’s slow I/O anyway. But if a println! from GPU code blocks on a host round-trip every time, that completely changes how you’d use it for debugging.
Is there device-side buffering, or does each write actually wait for the host?
At ~$100B in AI funding, the question isn’t capital—it’s physical constraints.
Data centers need power (H100s are ~700W each), and recent capacity additions were mostly pre-allocated. Chip supply is also constrained by CoWoS packaging, not fab capacity, and expansions take years.
If power, packaging, and GPUs are fixed in the near term, does $100B mostly drive inflation in AI infrastructure prices rather than materially more deployed compute? Are we seeing the real cost of a usable GPU cluster rise faster than actual capacity?
Has anyone modeled what $100B actually buys in deployable compute over the next 2–3 years given these constraints—and whether that figure is shrinking as more capital piles in?
Oh it is driving inflation alright, inflation for everyone else because OpenAI are buying things like a years supply of DRAM with money they don't even have yet (surely that should be an investigation?).
The 160k → 90k LOC reduction is nice, but the parallel rollout is the more interesting part. Running Rust alongside the C++ version and using differential fuzzing to check equivalence is a lot more realistic than “rewrite and pray.” You get incremental validation with the old system as a fallback. Curious how long they ran both before cutting over.
Binary size is a real concern on the client side. On servers the Rust stdlib overhead usually doesn’t matter, but when you’re shipping to billions of mobile devices, every KB counts. Good to see they invested in build tooling instead of just accepting the bloat.
Did they say anywhere what they did? Rebuilding the stdlib as part of your build can shrink it a lot depending on how much of it you use, but that is still nightly only. Maybe they went no_std or created their own?
They didn't but keep in mind that the app is currently 170MiB. The standard library shouldn't have added more than a few hundred kilobytes. They already likely pay similar costs for c++, but it's more worthwhile as they have a lot more c++ code total.
Also note that if you statically link to the rust std library, lto will excise the majority of it anyways, no need to rebuild it.
The default hello world stripped with one codegen unit and panic=abort was 342kB both nightly and stable. Adding lto dropped it 42kB in stable and 40kB in nightly. Adding build-std and only building core did not reduce it any further in size.
That is what I was talking about. It didn't reduce it at all over just lto. If I'd set optimization to z it probably would have gotten some back, but that starts impacting performance
Most profiling tools technically work, but they expect you to reason clearly, correlate timelines, and keep a full mental model in your head. That’s fine at 2pm. At 2am, it’s hopeless. hud’s goal is to reduce cognitive load as much as possible: show something visual you can understand almost immediately.
The UI is modeled after a trans-Pacific night cockpit: dark, dense, and built to be readable when you’re exhausted.
Under the hood, hud uses eBPF to track scheduling latency—how long worker threads are runnable but not running. This correlates well with blocking (though it’s not a direct measurement). You can attach to a live process with no code changes and get a live TUI that highlights latency hotspots grouped by stack trace.
The usual suspects so far: std::fs, CPU-heavy crypto (bcrypt, argon2), compression (flate2, zstd), DNS via ToSocketAddrs, and mutexes held during expensive work.
Tokio-specific (worker threads identified by name). Linux 5.8+, root, and debug symbols required.
Very open to feedback—especially around false positives or flawed assumptions. Happy to answer questions.
https://github.com/cong-or/hud