This is the second unikernel article this week I think, and the second that really undersells the benefits of having a full OS. Let's just swap out a couple of words:
> With a unikernel, the [observability] surface is much smaller. If the functionality isn't in the application (the operating system), then the [SRE] is screwed. There's no next hop.
Tools such as `strace`, eBPF, `lsof`, `ss`, etc. have tremendous value. The article mentions some nice things for observability ("structured logging, OTel") but so much gets thrown away.
Not just that. Unikernel sounds ok if all your linux server does is run one service that doesn't need anything but tcp/ip, and whose users don't need extra resources. But being able to deploy a new binary and other files through any means (ssh, pipelines, git), having localized log files, etc., are more than a convenience. Sure, I could log on another server, run a db on another server, and deploy html/css/js from yet another server, but would that make life easier? And this is just a really simple case. If you have multiple services, you'll need multiple servers with a uni-kernel approach, or build some monster thing. It's horses for courses.
Welcome to cloud computing, where software is cattle, not pets.
There isn't a Linux server taking care of a couple of services.
There are services deployed as Kubernetes pods, or serverless language runtimes across a cluster abstraction, running directly on top of type 1 hypervisors, or minimal kernel images enough to power distroless container images.
maybe we _should_ be thinking about building proper distributed operating systems instead of loose collections of services. we really missed an opportunity to rethink the OS interface for 'the cloud'. or rather we kind of wasted alot time and maybe we should get our shit together and do it properly. no matter how much we chant 'cattle not pets' everyone is still installing individual nodes and logging in and puttering around.
Agreed! There were many promising distributed operating systems, most got stuck at the research phase however. But the cloud pretty much killed them all.
Not that many people keep large distributed workloads anymore, and SLURM-like schedulers on top of regular operating systems were deemed good enough for supercomputers.
not always. Asci-red had a distributed version of OSF/1. this wasn't a great design because it was built on distributed shared memory, so it had fault tolerance and performance problems. the MTA had a shared memory monolithic BSD, but that was almost a forced choice.
The Cray T3D had a really nice distributed services model (really kind of a distributed exokernel) that would be worth emulating
on some machines there was an interactive ganged run facility like 'mpirun'. I think even this distinction where there are 'login' or 'service' nodes and anonymous runners would be at least a little structure. I guess you could argue that k8s is this, but it kind of falls short and seems to require a lot of continuous propping up.
I worked on a single system image project that didn't ever get shipped. it turns out that a syscall proxy layer isn't that difficult to implement, and that leaves you with an undisturbed user space.
anyways, there are a whole host of design options that don't look like k8s or slurm. and having spent some time recently on non-hyperscaler clusters, that bottom level of bringup (provisioning, networking management, reporting, fault tolerance, distributed filesystems) is really a nightmare. I think we lost a lot of traction there by raising generations exclusively in someone else's operational context.
Yep. We say cattle not pets, but even cattle often need personal attention for one reason or another. It doesn’t mean you’ll never need that option in your back pocket
if you didn't have a multi-user operating system running on every node that basically demands this, because of its complexity and fragility, and you had the necessary introspection to look at memory usage and open sockets, and disk bandwidth, and everything else you might need to mange those processes, then the utility of your escape hatch - which also is complicated and fragile, becomes negative.
Let me know when your local OS does multi-machine scheduling across heterogeneous hardware and application constraints, networking, observability, service accounts and a finer grained permissions system, in a way that’s far less confusing and piecemeal.
Also, this argument only works if you just broaden the definition of “OS” to “anything that runs that isn’t your apps”.
In general, the more specific tool you have, the more tailored to the goal approach you may have, with all features optimized for that goal. If you need those tools from the full OS, you might have their specialized versions in a unikernel.
Take this with a grain of salt. I haven't run unikernels in prod, so it's all a figment of my imagination.
> Tools such as `strace`, eBPF, `lsof`, `ss`, etc. have tremendous value.
But doesn't their value lie in getting you access to the core of the OS that you wouldn't be able to touch otherwise? What if your network stack was a library you could instrument just like you do with the rest of your app code?
Also, I don't need a tool to tell me I'm being punished by the IO or CPU scheduler if I am the only one using the CPU and IO :)
> But doesn't their value lie in getting you access to the core of the OS that you wouldn't be able to touch otherwise? What if your network stack was a library you could instrument just like you do with the rest of your app code?
Yes and no. The traditional separate kernel exposes one set of interfaces that these tools use. A library version might expose a different set of interfaces that accomplish the same thing. Your application could then use those to provide similar observability. But maybe not—that consideration was mostly absent from this article. And with new tools, you need to retrain your developers, SREs, LLM agents, etc. If your new tools are that much better, it might be worth it. But these articles have fallen far short of making that claim.
> Also, I don't need a tool to tell me I'm being punished by the IO or CPU scheduler if I am the only one using the CPU and IO :)
Disagree. Embedded programs still have IO and CPU scheduling problems, including priority inversions. You don't need different programs to have CPU scheduling problems, just different threads, or different tasks with an async scheduler. Any time you have concurrency you need to consider scheduler observability.
It's also not like you are stuck with a full OS when you are running Linux. You can remove features from the kernel you don't need, and you can configure a bare-bones busybox to be your userspace, or just run your application directly as the init process.
In a type 1 hypervisor running a JVM on top, you would use Java Flight Recorder.
The tools exist, and in a container world no one should be running those tools directly anyway, it is all about rootless immutable containers, ideally without OS layers for deployment.
If your app, is the only one running, and the whole kernel is a library, you can simply have said library instrumented with as much observability as you need, directly and you now no longer need to worry about which processes are using cpu and IO because…it’s all you.
> Tools such as `strace`, eBPF, `lsof`, `ss`, etc. have tremendous value. The article mentions some nice things for observability ("structured logging, OTel") but so much gets thrown away.
Trivial to implement. I have a full custom (from scratch written) debugging/benchmarking implementation written both for Linux and two of my custom kernels, both of which are REALLY unique when it comes to architecture. My custom debugger also supports numerous inline scripting languages.
I'd love to read more about your debuggable/benchmarkable/observable unikernel architecture and interfaces. Much more interesting than these "unikernels are the future" articles which make little reassurance they've replaced what they've dropped.
I wonder if they ever considered a table-driven approach for `#[derive(Debug)]`, as `facet` [1] does. That would have been my first instinct for something this formulaic where binary size and compilation time matter more than execution speed. But my impression is facet hasn't quite realized its promise on those fronts, so maybe the table-driven approach in std was similarly tried and rejected.
You're never going to get promoted with that attitude!
I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.
Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.
I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
> Now they're run based on the data, and the data says they have enough loyal-to-a-fault customers like the author that chronically understaffing is more profitable than providing a good experience.
If they are indeed saying understaffing doesn't matter because their current customers don't care, their loyalty numbers will only go up because of that sampling bias. Relatively upscale fast casual places (e.g. Chipotle) will replace McDonalds for the customers they lose. The circle of service industry life.
The "world" chart shows an increase from 19% renewal to 34%. Did they cherry-pick that? (Also, "European Union" is more than one country.)
> I don't doubt that it's economical for individuals when the govt is subsidizing it.
Does that distinguish renewables from fossil fuels? Haven't governments been essentially subsidizing fossil fuels (not least by allowing environmental externalities to be ignored) for as long as they've been in use?
Because a lot of countries in the world, especially the EU, are subsidizing solar. That doesn't make it economical.
Externalities ignored from fossil fuels, yes. That's not a subsidy though. I'm not saying they should ignore it, but if they do, they aren't the ones who pay for it.
That's not a good point though, a lot of countries, including the EU, also subsidize fossil fuels.
Solar is cheaper, but requires more room and time to spin up (think datacenters, where you can put a turbine within a month) and storage or backups for windless nights.
Agree with your broader point, but out of curiosity:
> It cost her nearly $10k to have a _used_ aluminum ramp installed for her house.
Why so expensive? I did a quick search, and the only way I could get a (new) aluminum ramp up to $10k was by selecting the 30-foot platform model, [1] which I'd guess would be excessive in most cases. I'm not counting installation fees but it looks like something I could do myself in an hour, and I'm not that handy.
The author is seeing that it's easier for one team to get backing (VC money, course credit, design awards, TED talk applause, etc.) for "solving" an accessibility problem than for us to enforce that entire industries follow good accessibility practices in new work and retrofit existing work. In theory industry-wide enforcement is the sort of thing legislation like the ADA solves, but obviously that hasn't been completely effective. Not sure how to fix it.
There was a subthread recently complaining about password managers and their UX problems. These aren't aimed specifically at "disabled" users, but I think it's a similar situation. We can't reliably enforce good standards for sign-in (whether that's just setting the autofill attribute correctly and choosing the correct event handlers for Javascript validators or something far more ambitious), so we expect a password manager to just deal with the web as it is. IMHO they work remarkably well given that constraint, but it's not optimal overall.
Yeah, it kinda feels like a banal thing to hate on. "They're only doing this because it's easier to create a prototype invention than to redo all design standards and enforcement in the country!" I mean, yeah? What exactly is the implied alternative for the sort of individuals targeted by the blog post?
The video of the prototype invention has the effect of making people think they don't need to be concerned about ramps and doorknobs and ADA enforcement because some teen in a garage has just solved this problem forever. People who need accommodations also require the political support of able-bodied people in order to actually make change happen, and his argument is that these videos dissipate that support.
> his argument is that these videos dissipate that support.
That seems like a major stretch. Really feels like letting the perfect be the enemy of the good here, like saying that a video about improved airbags means that nobody will care about safer road design anymore.
If it weren't for all the supermassive cars rolling around, purchased because they're ostensibly safer in a crash for the person driving them (while markedly decreasing the safety of the other driver) then maybe I could agree with you. But with so many people intentionally electing to value their individual safety at cost of the safety of others, I actually think that it makes for a great reason for them to not think about road safety! Who cares if the road is unsafe if I'm driving a tank?
The expressed (not implied) alternative in the post is for those people to stop pretending they care, and start actually caring. Stop trying to save the world. The world doesn't need an invention to save it. Just put that energy into caring about others.
People do care, but there are so many problems in the world, and only so much caring to go around.
It's extremely cynical to view it as "people pretend to care", rather than looking around and realizing there's a ton of important problems out there that the caring is distributed over. People care about disabilities, but they also care about war, about public schooling, about the environment, about feminism, about animal rights, about transparency in government, about road safety, about child well-being, about unemployment, and so on. All of these are important things, and of course there are dozens more that I haven't listed.
And inventions are sometimes actually useful. Sure, most of those prototypes will go nowhere, but every once in a while something that really helps is invented. And you can't predict in advance only the ones that will actually go on to help.
Some people actually care, but they can't always win the battle at their own companies when there are other priorities, much less go fix all the other websites, pour concrete at other businesses, etc. Is the solution for everyone to care? If, so how do you accomplish that? ("Americans with Disability Act" v2?)
The world doesn't need an invention to save it. Just put that energy into caring about others.
This is an excellent summary of the sort of anti-tech thinking that I strongly disagree with. Caring about people with Alzheimer's is well and good, but what we actually want are vaccines and cures.
The dream of a vaccine or cure is all well and good, but when my grandfather has Alzheimer's today and there's no properly functioning social supports, processes, etc to get him into a home and it takes months and months of phone calls and panicked searches when he wanders off to improve the situation at all, I think maybe there really is a dearth of "caring about the problem and not just the solution"
There's a general problem on the web (and to a lesser extent computing in general) that UX is just widely terrible. I think part of this is because developers tend to work at the very edge of their competence, and getting something working at all is a major feat, with no room left over for making it work well. And part of it is developers or designers deciding they know better than everyone else. A lot of this was a solved problem back in the 80s: the platform provides a standard UI toolkit and a set of standard idioms, and all software uses them. But this lesson is ignored because developers and designers don't like it when their stuff looks and works just like everything else. Making buttons that look like everyone else's buttons means you don't set yourself apart, so we end up with custom buttons that look different and don't work as well. And, of course, are less accessible, because accessibility requires integration into the overall system that doesn't exist with a custom solution.
I was a Mac fiend back in the "beleaguered" days, and standard widgets and idioms were a big deal in that community. There was broad consensus that apps should all look and work the same, as much as that was possible. A button should be a button and a text field should be a text field no matter what program it was in. That all went out the window with the web, for no particularly good reason, and it's infecting non-web platforms at an alarming rate.
100% agree about the UX problems. Most engineers would be happy to fix those issues, but of course, companies always want the new features...because that's what customers want and what drives more $$$.
Sadly, it's true that the % of new sales generated by improving a11y beyond the bare minimum isn't going to be terribly large. That's why it's important to have regulations that mandate decent support, because the monetary incentive that drives corporate behavior just isn't there.
I feel like that hasn't been true for a long time. Companies no longer compete on featuresets. I mean take any tool you've used for 10-15 years, the trend is often less features, not more. And the irony is that its the products from the biggest most well compensated companies in the world falling into this pattern of removing features and making things slower and less usable over time. Compare humble old VLC, no shortage of features, to what google routinely does to their products.
Quality, functionality, standing on its own merits, no longer is the aim of business. It is about capturing the mindshare through propaganda a.k.a. advertising. It is why these shitty companies can continue to exist on the top while serving continually inferior tooling: they've flooded the mindshare such that even if some superior product came about, it would never gain traction.
> I'm pretty sure doing janky things to the AC you don't own is a lease violation.
Why? Not all "janky" things are the same:
* Sloppy changes to the internal 240V wiring: yeah probably a violation of the lease, and legitimately likely to burn the place down, with associated liability.
* Temporarily mounting to the exterior this mechanism that turns the knob shaft, with no changes to the wiring, no holes, no nothing. I can't think what grounds they'd have for considering this a lease violation or a reason to keep any of the security deposit, as long as you pop the original knob back on when you move out. And as inigyou pointed out, how would they even know?
> The rest of this blog post is a walkthrough of a software engineer’s approach to home automation and building custom hardware, and let me tell you: both the final product, and the journey to get there, are hella jank.
IMHO it's not that bad? With the disclaimer that I'm also a software engineer who dabbles very very lightly with electronics and has no mechanical engineering expertise. But the overall approach of doing this by stepper motor seems quite defensible given the security deposit constraint. The only part that seems really janky to me is that it's not physically held in place that well as noted in this paragraph:
> Ultimately, the biggest issue is that binder clips and cardboard aren’t quite as robust of a mounting mechanism as I’d hoped they’d be… and over time, the stepper motor tends to “sag” a bit, resulting in a bit too much friction between the motor, the coupler, and the underlying knob, causing the mechanism to stall out until someone manually goes and reseats it…
Involving MQTT and HA can sometimes get frustrating too; having your AC not working properly if there's a problem on a Linux machine can suck. If this becomes a problem in practice, iirc ESPHome also has a thermostat component, and temperature sensors are really cheap and easy to wire in, so you could do it all on the microcontroller pretty easily in a low-code way. But if this part is working, no need to mess with it.
That's what I did. Just an esp32 relay board with a tiny screen and a knob. Then I put more esp32 sensors in each room to get individual temps, with HA logic to select which room the thermostat pays attention to. If HA dies for some reason it falls back to a local control and I'm back to the same situation as before, a thermostat that only pays attention to the temperature in the hallway it's installed in. All the boards were flashed with ESPHome.
Nice. I don't have any ESPHome thermostats right now, but I have an ESPHome fridge/freezer temperature monitor. I adopted the same idea: it should integrate with HA when HA is available, but it should be useful even when HA is down. So in addition to exporting the raw temperature to HA, the alert threshold is kept on the device rather than in HA so it can locally compute the alert state. This triggers a directly wired piezo buzzer, in addition to an HA phone notification.
You do have to set the "api: reboot_timeout: 0" if you don't want it to reboot itself every 15 minutes while HA is down. I persisted the alert threshold to flash though so even if it does reboot it comes right back up and works fine.
It does feel hella jank, but it's pretty obvious from the tone of the text that the author is fully aware of that. He went through every available option in order from reasonable to jank and this is the first one that worked.
There's nothing wrong with jank as long as it's not going to burn your house down the way some of his other rejected ideas would have.
> Honestly, back when the 737 MAX debacle happened, a lot of consumers claimed that they would stop flying aircrafts if they ran into 737 MAXs. And I don't think it happened in enough numbers - or even enough to make news. Sales went through the roof, everything kept working.
Is this kind of consumer revolt even really possible?
If you feel strongly enough that you refuse to fly altogether, then of course you can avoid flying on a 737 MAX. But I think most people did not feel the risk was that high. They just want to select "guarantee no 737 MAX" when booking a flight, and as far as I can tell that option doesn't exist.
Even if the flight is not a 737 MAX when you book, they can and sometimes do change aircraft, and as far as I know there's no option to get your money back when they do. If you show up and see it's a 737 MAX...you either get on or you lose your money, and have to find some other way to get where you're going, right?
In 2013 there were approximately 24,000 737 flights per day[1] - likely more today. If narrowed to just the MAX variants, it's still thousands per day.
Two, albeit high profile, crashes out of all the daily MAX volume is simply not something to worry about - let alone influence your booking choices.
You're applying everyday casual risk analysis to the highly-regulated environment of commercial air transport, where the MAX crashes absolutely were out of the norm and well beyond accepted levels.
Bear in mind when the crashes occurred there were fewer than 100 MAX in service.
Until you are on the plane. Sorry, but reasonably people and countries expect zero crashes and any single crash is worth worrying about because shareholder money should not trump a single human life.
> With a unikernel, the [observability] surface is much smaller. If the functionality isn't in the application (the operating system), then the [SRE] is screwed. There's no next hop.
Tools such as `strace`, eBPF, `lsof`, `ss`, etc. have tremendous value. The article mentions some nice things for observability ("structured logging, OTel") but so much gets thrown away.
reply