As a reader... I can't tell if there's a fundamental design flaw with io_uring, or just growing pains because the implementation is newish. Like how speculative execution had fundamental vulnerabilities which had to do with timing your code, not a specific implementation.
The title makes it look like a design problem with io_uring, Google's reaction makes it feel like a design issue. But the article itself sounds like growing pains.
It's complex. No, there's nothing so fundamental about io-uring that it could not be made safe. But also, there are significant factors that are leading to it being ripe for exploitation.
1. Security has been sort of an afterthought. io_uring was being built out as a complex, powerful capability and only much later were things like auditing, selinux hooks, and ebpf even considered.
2. io_uring is inherently complex code. It's highly optimized, concurrent, and new. It does a lot of work to avoid things like copying across the kernel boundary. This code is hard to get right under the best of circumstances.
3. io_uring is very different. This is basically a new system call interface that behaves in a different way from the other interface, we're playing catch up in security.
So in theory this could all be fixed but it's going to take a lot of work and, importantly, a shift in priorities - io_uring can't just be developed as the "gotta go fast" interface, it needs to be designed around safety.
> Speculative execution is not fixed in modern CPUs?
Very generally speaking, not really. It's a huge class and the pace of discovery of new vectors has continued nearly unabated since Spectre. See https://en.wikipedia.org/wiki/Transient_execution_CPU_vulner... The effectiveness and cost of various remediations--software, microcode, hardware--varies depending on the particular vector, so some vectors can be considered fixed, while other vectors, such as some of those relating to SMT, might never be fully fixed without effectively giving up some of the benefits of SMT.
Some of the more robust theoretical remedies, like cache coloring and kernel scheduler-based process affinity schemes, still haven't seen much investment or attention, so I would expect the state of things to remain relatively crappy. It just may seem less crappy because the industry has become inured to the threat, and also perhaps because there's usually much lower hanging fruit for attackers (i.e. traditional exploits like with io_uring), so these vectors are rarely used in the wild.
Spectre marks the beginning of a new bug class. It's more like "to write secure program on processors with speculative execution you must not do A, B and C". New processors fixed known bugs in the implementation of the silicon itself, but some of them can only be fixed in software.
Spectre as a general vulnerability class is NOT fixed. And you can write perfectly vulnerable-to-Spectre code in Rust, or anything you like. As long as there are "virtual" privilege separations under the same hardware privilege level, which is extremely common nowadays (hey, it's called sandbox). It would be vulnerable on ANY processor with speculative execution, including past, current and future Intel, AMD processors, M1, M2 and you name it. No need for JavaScript in browsers. Please just read the page, it's not browser specific.
In this case the only thing one can fix is, surprise, the vulnerable code!
Oof. I hope this doesn't get wedge io_uring into a place where people can't use it because it's disabled, so they don't, and the whole improvement gets stuck and wasted.
> While io_uring brings performance benefits, and promptly reacts to security
> issues with comprehensive security fixes (like backporting the 5.15 version
> to the 5.10 stable tree), it is a fairly new part of the kernel. As such,
> io_uring continues to be actively developed, but it is still affected by
> severe vulnerabilities and also provides strong exploitation primitives. For
> these reasons, we currently consider it safe only for use by trusted
> components.
As often in the kernel (and probably everywhere): there are people caring for performance and there are people caring for security, and a lot of the time there's not that much overlap.
Ensuring that io_uring is covered by the Linux Security Modules (LSM) framework, that provides hooks for things like AppArmor or SELinux (and a variety of others, some even can even get stacked), was only done after it was already quite popular.
So FWICT, even if io_uring is existing since a while, 1) the security experts vetting it are looking only closer since relatively recently and 2) it's architecture is quite different from other kernel user space interfaces, which certainly needs some adaptions on applying the established security mechanisms of the kernel, especially as the project still wants to stay high performance.
IMO, this doesn't mean io_uring is inherently unsafe, or has no use, but that one certainly needs to evaluate if great async submission model is worth the potential security implications. E.g., in an isolate environment, where no untrusted third-party apps or people have access, it can be fine, while especially in environments that Google runs it isn't worth the trouble for now.
But, I think that the features which io_uring brings are too nice to have for quite some (I/O or syscall heavy) workloads even in those environments, so there will be an incentive it to the integration into the existing Linux security stack and mechanisms, and evolve the latter to be also a good fit for this rather different interface.
> integration into the existing Linux security stack
It's less about integration into the security stack. It's just that the implementation of io_uring is hard to get right, and at 2023 there seems to be an endless stream of race condition bugs. Quite like user namespaces ten years ago. It doesn't matter how good your security design is. Hackers just find a memory corruption bug and get around everything.
To be fair it isn't endless, finding an io_uring bug which can be exploited to cause memory corruption and ultimately kernel code execution in 5.15 tree is measurably harder now. But io_uring moves very fast and it's an entirely different story for mainline kernels.
> It's less about integration into the security stack.
Yes, that's why I wrote:
>> it's architecture is quite different from other kernel user space interfaces, which certainly needs some adaptions on applying the established security mechanisms of the kernel
But user namespaces are actually a very good examples, they also broke with a lot of things that were often used as axiom, so there really are some parallels to io_uring from that POV.
> Hackers just find a memory corruption bug and get around everything.
If the Rust experiment pans out well, it could make a lot of such bugs straight out impossible, at least if devs aren't bending backwards to lower the risk to roughly the same as C now has in the best case. At least at $work this pans out quite nicely.
While I like Rust and get paid to work in it, and am happy it's now possible to work in the kernel in it, I am skeptical about the promise of Rust to solve issues like this, honestly; at least the for short term...
Because I think once you get down to the kind of code being written here it's going to be chock full of unsafe {} and UnsafeCell and various snakey error-prone tricks to make the borrow checker work in the context of the entirely different world that is the kernel, closer to the metal. And so realistically most of the same issues that impact C code will happen here.
I haven't looked at the io_uring security issues in general but I've used io_uring (from Rust) and I can see how it could go wrong. And I can see that if one was writing a facility like this in Rust it could end up subject to many of the same issues, because in the end io_uring is holding references to user instructions and executing them in a pretty privileged context. Many things could go wrong here: how one manages ownership, what the boundaries between this thing and the rest of the kernel look like, etc. There's no magic bullet that Rust provides there, other than perhaps an expectation of a certain kind of borrowing discipline?
Hopefully Google shutting this down in their GKE environment doesn't spread beyond into AWS and Azure hosted instances, etc. because I think there's plenty of people who will be impacted quite negatively. io_uring has incredible promise, and the kinds of companies that it will hurt are the fairly innovative ones pushing the envelope for database or server tech generally (including a former employer of mine!).
Yes, it is no different than OPSEC with heavy procedures in place which take the actor's mobility at the sake of security. A universal security property.
Shame but understandable. io_uring (like netlink) is essentially a completely new interface to kernel-land things so until security boundary stuff is sorted out this is probably the best move.
I'd like to see a source for this. Even if they do run older kennels, it doesn't mean they're missing patches. Google has significant involvement in security and enough engineers to roll their own systems. I don't expect they'd be missing any relevant patches, much less for years. (I know this was an older patch in this case, but they did mention other mitigations, so it may have been a considered choice not to backport)
Given how large they are as a target, we'd also hear about them being exploited all the time if that was true.
Google actively patches all its kernels and keeps them pretty up to date. The problem isn't the vulnerabilities Google or the public knows about, its the ones it doesn't know about, and the rate of discovery is too high for io_uring to be used in certain situations.
"As such, io_uring continues to be actively developed, but it is still affected by severe vulnerabilities and also provides strong exploitation primitives."
With respect to Android, I was under the impression that (in part because key members of the Linux community are allergic to the notion of a stable device driver ABI) the specific release of linux used on a particular model of phone was typically frozen at some point before the time the phone shipped, and subsequently only patched as needed.
The mere fact that the source is available doesn't mean that someone will take the (potentially unbounded) time to port it to a future release and test it sufficiently.
> [Google is] running kernels from three years ago that are missing security patches from two years ago.
This is ridiculous. No, they aren't. No one is. Kernels on production Linux devices[1] are cut from an LTS branch and get prompt and expertly audited security backports of everything anyone knows about. Now, sure, that process does break down occasionally; but this is an area being shepherded by some of the best engineering effort in the world. In fact I'd go so far as to say a regularly updated LTS kernel is probably the software product that is most responsive to security issues.
[1] All the ones I know about anyway. Surely there are some gadgets being less prudent, but Android and ChromeOS are absolutely not on the list.
At least for my workstation, it doesn't seem so? The kernel is only 2 month ago, and I tend to postpone my OS update to the very last moment unless it's forced. I'm pretty sure that the production even got much more frequent updates.
There's viewings, showings, hearings, shootings, cuttings, smearings, belongings, holdings... It's not like the pattern doesn't exist even if it doesn't hold for all verbs.
IMO the original title was way better since it doesn't bury the lede. For better or worse, very few programmers care about "Learnings from kCTF". A lot of programmers care about io_uring being disabled.
On HN we use the original title unless it's actively misleading, or it's linkbait --- obviously, this title is anything but that. Titles are community property: it's not the prerogative of the submitter to determine the most interesting thing in the story and retitle it accordingly.
> As the Linux kernel is a key component not just for Google, but for the Internet, we started heavily investing in this area. We extended the VRP's scope and maximum reward in 2021 (to $50k), then again in February 2022 (to $91k), and finally in August 2022 (to $133k).
So an award roughly equal to four months' salary for an average Silicon Valley engineer is considered "heavily investing" in security, when it comes to potentially industry-shattering exploits?
I imagine that anyone who knows where to look can easily find dozens of interested buyers willing to pay a lot more than that, from intelligence agencies and their contractors to crime syndicates.
Who is lawfully paying out more for bounties on open source projects they didn't even start?
You also seem to discount the monetary value of not becoming a criminal by selling exploits unlawfully. Do you want to be the person that reported a vulnerability to increase security in the world or to give more power to spy agencies or crime syndicates?
> Who is lawfully paying out more for bounties on open source projects they didn't even start?
That’s kind of begging the question. Google invests a non trivial amount employing Linux kernel developers. How do these bounty payouts compare in terms of the amount they pay for their own engineers and security researchers? This also isn’t a donation. This is rewards for work Google finds valuable enough that they’ve changed several products based on this work. They also benefit immensely from all the free labor other companies and volunteers out into the kernel.
I think OP raised an extremely fair critique of the amount being too small.
If it's too small, what is the right amount, and who should pay it?
I believe the world governments should create an international organization funded by per country quotas like the IMF for example to pay for bounties, but until then, the fact a private company does this sets the market price for exploits, and saying "it's too low" seems meaningless if there's no proof anyone is willing to pay more.
I'm sure if it were $1m per bounty some other HN user would say it should be more as well. That's why you define this by how much someone _actually_ pays out, because talk is cheap.
> and saying "it's too low" seems meaningless if there's no proof anyone is willing to pay more.
The black market is a good proxy for estimating the true market value of an exploit. There’s a coupon to be applied in terms of the legal costs associated with selling on the black market, but still comparable.
The other way to look at it is when are there diminishing returns. From the chart they posted, they’re very far away from that.
Finally the other way to look at it is compared with other kinds of bounties they run for OSS.
Anyway, of course they should be commended for having any bounty. They’re certainly not required by legislation. But maybe instead of having an international governmental or NGO with funding, you could require that some percentage of a SW company’s revenue is diverted to bug bounties for SW depending on how critical the SW is to the ISV’s business. That way it’s a cost directly borne by the ISV using the software to internalize the security risk into a P&L line item for the software being used (and not just OSS - all software from an ISV should be forced into bug bounties like that). And the vendor has a vested interest in confirming the VULN whereas a NGO could have all kinds of soft corruption to incentive flowing payments to friends. That being said, I’m not sure the status quo would really improve in such any alternate scheme.
Finding black market prices has been done in the past. The issue is that people often do not have a clear understanding of what a “product” actually looks like on the black market and what the value of the “coupon” is.
If you are very confident that the buyer is legit and not evil, then you’ll be fine.
Imagine though, that this information you willing sold to a shady party was used in a big hack. Say national security or an oil pipeline or something.
Do you think your defense will hold up in court? How many years will you be in court? Even if you are deemed not guilty, will your life ever be the same?
You’re gonna risk all of that because you thought you could squeeze a $200k out of your bug?
Why "black market"? Surely any intelligence agency worth their salt has a public, open, completely legal program where you can just contact them and offer what you have, no? They'd be stupid not to.
Selling to governments is not trivial, at all. You can't just post these things on eBay and hope for the best. Like all things, you need hard fought relationships to be able to even talk to anyone, let alone actually make a sale.
> You can't just post these things on eBay and hope for the best.
Of course not. You start a "security consulting company" and then apply to be a contractor. There are hundreds, if not thousands, of companies like that. And as an individual researcher, you can either join such a company, or freelance for one.
It's indeed not trivial, but I imagine that anyone with enough experience in the business would know how to pull it off.
I'm pretty sure it's "lawful" to sell exploits to intelligence agencies of your own country. It wouldn't make any sense for it not to be, considering that the same people who control those agencies also control what's legal and what isn't.
Can you link me to where I can submit bounties to a spy agency, and any material or personal anecdote about them paying more than Google for Linux kernel exploits? Because so far this all sounds like speculation on your part.
> Can you link me to where I can submit bounties to a spy agency
No. I have no idea. But it would clearly be in the interest of such institutions to have such programs, so common sense says they do. I'm sure that people who are active in this business would know more about it.
I’m not active in this business, but I know people who are. It’s not as easy as you think it is. You cannot just go to a government and say “hey, here’s my bug, give me a million dollars”. First off, your bug probably isn’t worth a million dollars, at least if you’re just converting bug bounty submissions directly. They’re looking for reliable exploits, and payouts will depend on how long they can continue to field the chain (if it gets patched, you might be on the hook for another exploit!) and whether you’re selling it exclusively to them. Payouts will typically happen on a schedule.
Unless you’re a very known quantity they’re not going to talk to you directly, anyways. You’re going to go through a broker who will take a cut. My understanding (though I don’t know much about it) is that say, the NSA, has their own internal teams and they don’t need to buy anything anyways. The other federal agencies will contract out to a handful of firms that generally have salaried employees. You can work there, but now you’re seeing very little of the “profits” that come from the sale. Maybe your exploit will get some shoddy forensics glued into it and end up being sold as “data recovery” or something for a local police department, who probably doesn’t have much money to spare anyway.
How do you define "selling exploits unlawfully?" I'm not aware of any law that says only Google is allowed to purchase information disclosing exploits in open source software.
- this isn't employment income, that's a separate category with higher taxes
- you want the payment sent to your LLC or similar, not directly to you. See above
- the requirement to file a 1099 is on the payer, not the worker
- many transactions don't require filing a 1099, most notably all payments made using a credit card. If you're an independent contractor in the US and accept credit cards, you'll get paid 5 minutes after sending an email instead of 30 days later after it goes through layers of approval
> I imagine that anyone who knows where to look can easily find dozens of interested buyers willing to pay a lot more than that, from intelligence agencies and their contractors to crime syndicates.
If that were the case, why did even a single person report a vulnerability to Google? Why didn't these muppets just sell it to somebody else for ten times as much? I would suggest tha it's because what you claim isn't correct. It's not actually "easy", and the payoffs won't be "a lot more".
Now, it's pretty hard for anyone to actually refute your argument since literally the only evidence you're providing is your imagination. The one thing we do know is that tens of serious vulnerabilities were reported to Google via this program. Those are a concrete existence proof of this VRP being a sufficient incentive for bugs to be reported via this legit channel. Surely that outweighs your complete lack of evidence.
It's a lot more than the $0 or negative reward that most companies offer. There are many examples of security researchers being rewarded with threats and police reports.
Funny how when a person/company does a small good deed, they get criticized for not doing more but when they don't do anything and implicitly offer $0 they don't get criticized.
You don't include reputation and publicity benefits into the overall profit formula. If you sell stuff on grey market, you'll have to keep your lips tight over that deal. It isn't going to your resume, so if one day you'll decide to join a security division in some corp, you'll have nothing to show them. Whereas a public achievement like this may be a door opener for many places. Same thing with things like pwn2own or security conferences. Why burn knowledge there? Because it promotes you.
> I imagine that anyone who knows where to look can easily find dozens of interested buyers willing to pay a lot more than that, from intelligence agencies and their contractors to crime syndicates.
This is unfortunately false. The running price for Linux kernel exploit isn't that high.
My criticism doesn't concern Google's ethics, it concerns their practicality.
Common sense says that with bounties like that, they won't attract the most serious exploits, which isn't an ethical problem, but can absolutely be a security problem.
They only have to beat the prevailing rate on zerodium to get it first. It's not like every exploit is some magically usable thing that can exfil all the googs. People really overrate a bunch of things.
There was a crypto company running with all spectre:meltdown mitigation off. And no one took the money until the company went bust and an insider tried to run with it.
Just wonder if the the reward for security bounty would increase such as 1 - 10 million dollars to match the black market, do you think it would attract more serious security researchers?
Some people also have moral values and selling these would conflict with their personality.
Not saying that the payout is high enough for skilled individuals to invest time.
"Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting."
Everything about io_uring has screamed security nightmare from day 1--the amount of kernel attack surface it opens up, the fact it put little thought into how it interacted with process and privilege boundaries and related overarching models (see, e.g., interactions with setuid and fork/clone), increased opportunity to exploit race conditions, etc. From a pure performance standpoint it's amazing, but it's circumventing a slew of interfaces that were, at least in part, designed to restrict process privilege--more importantly than merely designed, but refined and bug-fixed over decades.
The use of lockless buffers for high throughput IPC works better in microkernel designs. Linux being a monolithic kernel, io_uring is like one, big trap door for exploits.
The example in the article was fixed by backporting a patch from 5.15 to 5.10.
5.10 was released in October 2020. 5.15 was from November 2021.
They also changed the program so they only pay out for bugs in a recent kernel.
It seems more prudent to disable io_uring only on kernels that are say, over 2 years old.
Alternatively, they could upgrade their known-insecure kernels. I doubt the io_uring patch is the only missing one that was produced in the last three years.
That's not my read. I read that they've found a bunch of exploits in io_uring but are only willing to talk publicly about previously disclosed ones, like the one fixed by backporting to 5.10. Just because they used that as an example doesn't mean they're refusing to backport security fixes. It just means they don't want to talk about exploits they found in newer kernels, which is understandable.
• io_uring is quite new, and currently still having some severe security problems.
• Most kernel security bugs, they can mitigate effectively. But their mitigations aren’t fully effective on io_uring.
• Therefore, until the security problems die down or until they can effectively mitigate io_uring security bugs, they’re disabling it, because the security concerns outweigh the performance advantages.
> • Most kernel security bugs, they can sandbox effectively. But their sandboxing isn’t fully effective on io_uring.
This is definitely not true in general for their platform: I don’t believe they do any shared-kernel sandboxing of untrusted code at this point and their primary sandbox gVisor (used for stuff like App Engine, Cloud Functions, and Cloud Run) just doesn’t simulate io_uring at all.
“Sandboxing” was the wrong word to use; I was referring to what they call “mitigations”. Following through a few links, you can end up at https://github.com/thejh/linux/blob/slub-virtual/MITIGATION_... which describes what it all actually is. I was figuring that sandboxing was probably close enough as a description, and maybe more easily understood, but that wasn’t wise. I’ve changed it to use their wording.
