A memory safe linux kernel would be a fairly incredible thing. If you could snap your fingers and have it, the wins would be huge.
Consider that right now a docker container can't be relied upon to contain arbitrary malware, exactly because the Linux kernel has so many security issues and they're exposed to containers. The reason why a VM like Firecracker is so much safer is that it removes the kernel as the primary security boundary.
Imagine if containers were actually vm-level safe? The performance and operational simplicity of a container with the security of a VM.
I'm not saying this is practical, at this point the C version of Linux is here to stay for quite a while and I think, if anything, Fuschia is the most likely successor (and is unlikely to give us the memory safety that a Rust kernel would). But damn, if Linux had been built with safety in mind security would be a lot simpler. Being able to trust the kernel would be so nice.
edit: OK OK. Yeesh. I meant this to be a hypothetical, I got annoyed at so many of the replies, and this has spiraled. I'm signing off.
I apologize if I was rude! Not a fun start to the morning.
Memory safety isn’t why containers are considered insufficient as a security boundary. It’s exposing essentially the entire Linux feature surface, and the ability to easily interact with the host/other containers that makes them unsafe by themselves. What you’re saying about VMs vs containers makes no sense to me. VMs are used to sandbox containers. You still need to sandbox containers if your kernel is written in rust
Even just considering Linux security itself: there are so, so many ways OS security can break besides a slight (you’re going to have to use unsafe a whole lot) increase in memory safety
The culture around memory safe languages is a positive improvement for programmer zeitgeist. Man though the overreach all the way to "always safe forever" needs to be checked.
JS and Rust are memory safe languages with a culture of pulling in hundreds if not thousands of dependencies. So unfortunately, in terms of culture, at least those languages are not Pareto improvements.
Can you expound in this some? I am not fully grasping your point. Are you saying "building safe by default" is a bad thing or assuming "safe forever" is a bad thing. Or are you saying something entirely different?
I expect it's likely more of "memory safety in a language doesn't make it _safe_, it makes it less vulnerable". It removes _some_ issues, in the same way that a language with static types removes some ways a program can be wrong, it doesn't make it correct.
The problem is the word "safe," which is inherently ambiguous. Safe from what? A better term would be "correct," because at least that implies there is some spec to which the developer expects the program to conform (assuming the spec itself is "correct" and devoid of design flaws).
While I agree with your point in general, I don't think the choice of wording is "the problem" here. But I do agree your wording is more correct.
It is almost impossible to prove a program is "correct". But things like memory safe languages, static typing, and automated tests allow us to prove the ways in which it is not "incorrect". And the more places it is not incorrect, the more likely it is to be correct (even if we can't prove it).
I said "the idea of using memory safe languages is great!" And "using memory safe languages does not eliminate attack surface". (It's pre coffee here so I appreciate your probe)
I meant that it's over-reach to say it's completely trustworthy just bc it's written in a GC/borrow checked language.
The disagreement is that you wrote "imagine a memory safe kernel" but appear to have meant "imagine a kernel with zero vulnerabilities of any kind", and those things are not equivalent.
Clearly you can only do worse in Rust than you'd have with perfect C. But what's that?
The question is: what is the expected loss (time, bugs, exploits that lead to crashes or injury or death or financial catastrophe) with Rust vs other languages.
Unfortunately that's not the conversation we have. We instead have absolutism re managed memory, which does account for about half of known security flaws that have been discovered and patched. Removing half of bugs in one fell swoop sounds amazing.
It's not that we can't remove those bugs other ways. Maybe modern c++ can cut bugs in half at compile time too. But Rust seems nicer to play with and doesn't require much more than what it has out of the box. Also it's shiny and new.
Given that Rust is making its way into platforms that Cpp struggled into, it's potentially moot. I sincerely doubt Linux will accept Cpp, but we're on the cusp of Rust in Linux.
Rust has better diagnostics when everything does not go according to plan. Just like the automobile industry improved a lot the diagnostics for repairing cars the last decade, the same should happen in software.
Rust and Cpp both have the same characteristics when it comes to speed, they are both very fast, or executable size, they are both minimal size, or reasonable high level compared to say, C.
When it comes to program correctness and diagnostics, one is better than the other, and it is the one created the last 10 years. Linux kernel should definitely move past C, to Rust, Zig or something like that.
Rust has a few small but unavoidable runtime overheads compared to the best C (maybe they can be avoided with the right unsafe usage but that removed most of the benefits of rust). They are difficult to see the affect of without micro-benchmarking in my very limited experience, but C can avoid them
I am not saying that Rust is better, but not worse. Saying unsafe Rust doesn't count because it's unsafe but C gets to count feels like an unfair comparison to me: this did start out with assuming perfection on both sides.
Just the other day they were full kum ba yah over a holiday-time-released feature that's likely going to greatly increase the likelihood of building race conditions and deadlocks.
I know it's the curmudgeon NIMBY in me, and I love Rust and use it daily, but it's starting to feel like the worst of the JS crowd scribbled in grandpa's ANSI C books just to make him mad.
I am super happy with most features, but demand is demand and people demand runtimes. As a chimera it perfectly represents the modern world.
To be clear you're talking about async fn and impl trait in return position in traits? If so, how does that impact the likelihood one way or the other of race conditions or deadlocks?
Is this all information that should be communicated to a programmer, or is all this info something that should be automatically assessed by a compiler with clear responses automatically to all possible situations specified, black and white, in the code?
If so, if it is the responsibility of the compiler, why release a version of the language, stabilize a spec, where that's not the case?
> Memory safety is the primary issue with containers as a security boundary.
I don't know where you're getting your information, but this is NOT the consensus on the LMKL, among most Linux kernel people or at any serious large scale tech company.
If you wish to learn about this stuff, lwn.net has a good series of articles on the problems people are actually working on. Most of the problems are related to namespace confusion, privilege escalation through, e.g. block-level access to the filesystem, etc.
> Sometimes the issues are not memory safety ones! But many, most are.
Huge citation needed. In 15 years of security, 8 in Linux kernel security, I have seen maybe one practical exploit related to containers that boiled down to a C-level memory issue.
> That's on you, but I'd be happy to explain more to you if you have questions.
No, you're very confidently stating things that are at the very least debatable. This thread has people doing kernel security as a day job.
Respectfully, do you know what a container actually is? (I’m guessing you think it’s docker, which is a common misconception)
The kernel itself does very little to prevent containers from interacting with the host (yes, via syscalls) in a way that affects other containers or the host itself. Containers are not insecure/composed with VMs to protect against memory safety issues so much as to implement sandboxing preventing these syscalls from doing bad shit.
> Respectfully, do you know what a container actually is?
I am extremely familiar with containers, the linux kernel, and virtual machines. In particular from a security perspective.
> The kernel itself does very little to prevent containers from interacting with the host (yes, via syscalls) in a way that affects other containers or the host itself.
Namespaces, such as process namespaces, file namespaces, user namespaces, etc, will prevent a container from interacting with another container without even getting into the fact that you can leverage DAC to do so further.
This is getting a bit defensive. I think people are interpreting your post as saying all safety is guaranteed by using memory safety, but you rightly walk it back in comments to mean it addresses "primary" security problems.
It's just silly. I wrote an "imagine if we could trust the kernel as a boundary" and I get 100 posts about the same misconceptions. If people read into my post that I think a Rust kernel would solve all problems, perhaps I was overly simplistic with my language.
I don’t think people are nit picking. The issue of container vs VM security is unrelated to memory safety. So bringing them into the discussion here, even if just in a hypothetical, is odd. You could say the language might be misleading but Occam’s Razor would suggest they’re just conflating the two topics.
Frankly, their comments since (especially those around namespacing) doesn’t do much to convince me that understand the distinction either. So it’s not just the opening post.
The best think they could do in this situation is to ask detailed questions from those that reply, as to why people disagree. Rather than stating they do understand and everyone else misunderstands them, while continuing to post the same misconceptions but phrase differently. At least if they asked questions, there’s a chance they might learn something.
If you find this amazing, perhaps you should take a look at seL4, which has formal proofs of correctness, going all the way down to the generated assembly code still satisfying the requirements.
It also has a much better overall architecture, the best currently available: A third generation microkernel multiserver system.
It provides a protected (with proof of isolation) RTOS with hard realtime, proof of worst case timing as well as mixed criticality support. No other system can currently make such claims.
I wish L4 had taken off for general purpose computing. That and Plan9 are things I'd really like to try out but I don't have space to fit operating systems in amongst the other projects. They both strike me as having the Unix nature, either "everything is messages in userspace processes" or "everything is a file."
I don't think I've ever seen an argument for why "everything is a file" is a desirable thing. It seems like a kitchen where "everything is a bowl". Fine when you want to eat cereal, mix a cake, do some handwashing up, store some fruit; tolerable when you want to bake a cake in the oven. Intolerable when you've got a carrot in a bowl-shaped chopping board and you've got to peel and cut it using a bowl.
Why in principle should everything we do on computers behave and beshaped like a file?
the file is inconsequential. it could be any other universal abstraction, e.g. HTTP POST.
it's just something that every program running on a computer knows how to do, so why bother with special APIs you have to link against if you can just write to a file? (note you can still develop those layers if you wish, but you can also write a device driver in sh if you wish, because why not?)
I don't think the metaphor applies because kitchen utensils' forms are dictated are their purpose, but software interfaces are abstractions.
A fairer analogy would be if everything in the kitchen was bowl-shaped, but you could do bowl-like actions and get non-bowl behavior. Drop the carrot in the peelbowl and it is peeled, drop the carrot in the knifebowl and it is diced, drop the cubes in the stovebowl and they are cooked. Every manipulation is placing things in bowls. Every bowl is the same shape which means you can store them however is intuitive to you (instead of by shape).
The counter would be that different tasks are different and that's why everything isn't bowl shaped. Analogously, programming languages which are strongly opinionated (purely functional, array-based, strictly static) are far less generally popular than programming languages which are multi-paradigm and flexibly typed.
People have access to user-mode filesystems and FUSE, people could have made everything have a file-like interface by now if that was desirable - apparently it isn't. And because different tasks are different, it makes sense that they would fit different abstractions and interfaces, wouldn't it?
The function defines the shape of the kitchen utensil, yes. That's not a counter, those are physical objects. In software, we have far more flexibility as to what "shapes" are available.
Having a common interface for working with things is indeed why the flexibly-typed languages are popular. Everything is an object. You call methods on every object in the same way. You don't need to figure out how to interact with the network card, someone has written a library to make that into objects. You don't need to figure out how to interact with a file system, someone has figured out to make that into objects. Bringing up popularity of programming languages is an argument in favor of "'everything is a file' is a useful abstraction."
Either people will write FUSE plugins or they don't want consistent interfaces is a false dichotomy.
When you are interacting with programs, you use mostly keyboards and mice. Why not create a bespoke hardware interactions for each program so they can behave more like kitchen utensils? Deleting a file is a different action than creating one, so I should have to shake my computer like an Etch-a-Sketch to delete images and rub the trackpad like an eraser to remove text files.
> I don't think I've ever seen an argument for why "everything is a file" is a desirable thing.
A file system is a tree of named objects. These objects are seamlessly part of the OS and served by a program or kernel driver called a file server which can then be shared over a network. Security is then handled by file permissions so authentication is native through the system and not bolted on. It fits together very well and removes so much pointless code and mechanisms.
A great example is a old uni demo where a system was built to control X10 outlets and switches (early home automation gear). Each device was a file in a directory tree that represents a building with sub directories for floors and rooms - e.g. 'cat /mnt/admin-building/2fl/rm201/lights' would return 'on' or 'off' (maybe it's a dimmer and its 0-255, or an r,g,b, value or w/e, sky's the limit, just put the logic in the fs). To change the state of the lights you just echo off >/mnt/admin-building/2fl/rm201/lights.
Now you can make a script that shuts all the lights off in the building by walking directories and looking for "lights" then writing off to those files. Maybe it's a stage, all your lights are on DMX and you like the current settings so then you 'tar -c /mnt/auditorium/stage/lighting|gzip >student_orientation_lighting_preset.tar.gz' and do the reverse over-writing all the archived settings back to their respective files. You could even serve those files over smb to a windows machine and turn lights on and off using notepad or whatever. And the file data doesn't have to be text, it could be binary too. It's just that for some things like the state of lights, temperature or w/e can easily be stored and retrieved as human readable text.
That is the beauty and power of 9p - its removes protocol barriers and hands you named objects seamlessly integrated into your OS which you can read/write using regular every day tools. It's a shame so many people can't grasp it.
I like the example and the detailed explanation but "it works in some scenarios" is only enough to say that some things work well as files, it's not enough to convince that everything should be a file (or that everything should be the same, whatever the same is).
> "sky's the limit, just put the logic in the fs"
You can, but why is it better to do that? "just" put the logic in a Python script seems much simpler than putting it in a low level pretend filesystem driver, on the other side of an arbitrary interface you had to come up with to satisfy an ideology that everything should have to go through this interface, right?
Over Christmas I set some LED displays running, controlled by an RS-232 serial port. It would be possible to echo a line of text to a pretend filename and have it appear on the sign, but there is a command for whether the text is static or scrolling and one for whether the text appears on the top line or bottom line. That information has to be somewhere, either as different paths in the filesystem e.g. /mnt/sign/static/row/top vs /mnt/sign/scrolling/row/top or as formatted content inside the file, or both.
There's a command to change the colour of the subsequent text, which can be used anywhere in the message, many times over - so that can't easily go in the filesystem path so now there needs to be some intermediate structure which both the shell script and the filesystem driver speak, to be able to pass this command through the file interface. With a COM port in Windows, if one script opens it and another script tries to open it then there's a clear error that the port is in use. If a usermode filesystem driver held the com port open, what would happen if two scripts tried to write to these files? Does one get a mysterious "write failed" filesystem error to catch and debug instead of a proper relevant error message? Do they have to proxy error messages through /mnt/sign/meta/error ?
This is one of the most trivial things a computer can do, sending a short text, which is possible with echoing to a com port /dev/ttyS0, and the abstraction idea of making it a file is already creaking under the weight of the task, adding the burden of arbitrary path decisions and data formatting and task-specific escape codes and control sequences, and bodgy-error handling, while assisting with basically nothing - while you can find /mnt/* you get no help with that, no link to any documentation, no further discovery, no interactivity like you get from a command line binary program interface. Much much easier to put the sign's commands directly in a short Python script, isn't it? And other more complex tasks just get more and more intense, as the "plain text" files in /etc/ are an ad-hoc mix of formats and data types and parsing rules and symbolic links to /opt/ and other random places.
As I say in another comment, usermode filesystems never became the default or popular way to do things, why not?
Plan 9 IS NOT Linux or Unix. It is a pure VFS and only speaks 9p. A plan 9 file server serves a real file system. Think of the file server as an abstraction that closely resembles a micro service. It handles the multiplexing of multiple clients who might want/need to open the same file.
If you have a sign that takes formatted text via RS232 then you don't need the file server or the logic as the sign already handles this. You just need to write text to /dev/eia0 or whatever. Setting the com port settings is done via the com ports control file /dev/eia0ctl. This means you can use a shell script to set the com port settings by echoing the baud and all that into the ctl file.
However, what if you bought an LED panel that was just a bunch of LED's? It doesn't understand text or formatting so now you need to hook it to a CPU and write code. In this case, a file server running on a pi would be a great interface to handle that. The server would contain the logic to accept a formatted string of text and turn that into glowing LED's. Then any program can open that file to operate the display like a script, python program, or web server.
I think it's possible to run Genode[1] as a desktop on top of seL4 (Genode supports different kernels). However, I'm struggling to find a tutorial to get that up and running.
Eh, seL4 has a suite of tools that turn their pile of C and ASM into an obscure intermediate language that has some formally verifiable properties. IMO this is just shifting the compiler problem somewhere else, into a dark corner where no one is looking.
I highly doubt that it will ever have a practical use beyond teaching kids in the classroom that formal verification is fun, and maybe nerd-sniping some defense weirdos to win some obscene DOD contracts.
Some day I would love to read a report where some criminal got somewhere they shouldn't, and the fact that they landed on an seL4 system stopped them in their tracks. If something like that exists, let me know, but until then I'm putting my chips on technologies that are well known to be battle tested in the field. Maestro seems a lot more promising in that regard.
See here[0] one of Gernot Heiser's comments (part of the seL4 foundation) talking about how "there are seL4-based devices in regular use in several defence forces. And it's being built in to various products, including civilian, eg critical infrastructure protection".
There is also an interesting case study[1][2] where seL4 was shown to prevent malicious access to a drone. Using seL4 doesn't necessarily make an entire system safe but for high security applications you have to build from the ground up and having a formally proven kernel is the first step in doing that.
I have been fortunate enough to play a small role in developing some stuff to be used with seL4 and it's obvious that the team are passionate about what they've got and I wish them the best of luck
seL4 actually has an end-to-end proof, which proves that the final compiled binary matches up with the formal specification. There are not many places that bugs can be shifted---probably the largest one at this point is the CPU itself.
If you think about it, a bug in the proof will require a bug in the code.
This visibility is a good thing.
Overall, note that the critical code, such as the microkernel itself, is small (kLoC wise) to begin with, which minimizes the odds a bug will go undetected for long.
>(Eh, ) I highly doubt that it will ever have a practical use beyond teaching kids in the classroom that formal verification is fun, and maybe nerd-sniping some defense weirdos to win some obscene DOD contracts.
Uh, perhaps take a look at the seL4 foundation's members[0], who are using it in the wild in very serious scenarios.
You can learn more about them as well as ongoing development work in seL4 Summit[1].
Container vulnerabilities are rarely related to memory bugs. Most vulnerabilities in container deployments are due to logical bugs, misconfiguration, etc. C-level memory stuff is absolutely NOT the reason why virtualization is safer, and not something Rust would greatly
improve. On the opposite end of the spectrum, you have hardware vulnerabilities that Rust also wouldn't help you with.
Rust is a good language and I like using it, but there's a lot of magical thinking around the word "safe". Rust's definition of what "safe" means is fairly narrow, and while the things it fixes are big wins, the majority of CVEs I've seen in my career are not things that Rust would have prevented.
> Container vulnerabilities are rarely related to memory bugs.
The easiest way to escape a container is through exploitation of the Linux kernel via a memory safety issue.
> C-level memory stuff is absolutely NOT the reason why virtualization is safer
Yes it is. The point of a VM is that you can remove the kernel as a trust boundary because the kernel is not capable of enforcing that boundary because of memory safety issues.
> but there's a lot of magical thinking around the word "safe"
There's no magical thinking on my part. I'm quite familiar with exploitation of the Linux kernel, container security, and VM security.
> the majority of CVEs I've seen in my career are not things that Rust would have prevented.
I don't know what your point is here. Do you spend a lot of time in your career thinking about hardening your containers against kernel CVEs?
> I don't know what your point is here. Do you spend a lot of time in your career thinking about hardening your containers against kernel CVEs?
Yes, I literally led a team of people at a FAANG doing this.
You're saying the easiest way to escape a container is a vulnerability normally priced over 1 million USD. I'm saying the easiest way is through one of the million side channels.
OK, I apologize if I was coming off as glib or condescending. I will take your input into consideration.
I'm not looking to argue, I was just annoyed that I was getting so many of the same comments. It's too early for all of this negativity.
If you want to discuss this via an avenue that is not HN I would be open to it, I'm not looking to make enemies here, I'd rather have an earnest conversation with a colleague rather than jumping down their throats because they caught me in the middle of an annoying conversation.
Same, re-reading my replies I realize I phrased things in a stand-offish way. Sorry about that.
Thanks for being willing to take a step back. I think possibly we are talking about two different things. IME most instances of exploitation are due to much more rudimentary vulnerabilities.
My bias is that, while I did work on mitigations for stuff like Meltdown and Rowhammer, most "code level" memory vulnerabilities were easier to just patch, than to involve my team, so I probably under-estimate their number.
Regardless, if I were building computation-as-a-service, 4 types of vulnerability would make me worry about letting multiple containers share a machine:
1. Configuration bugs. It's really easy to give them access to a capability, a mount or some other resource they can use to escape.
2. Kernel bugs in the filesystems, scheduling, virtual memory management (which is different from the C memory model). It's a big surface. As you said, better use a VM.
3. The kernel has straight up vulnerabilities, often related to memory management (use after free, copy too much memory, etc.)
4. The underlying platform has bugs. Some cloud providers don't properly erase physical RAM. x86 doesn't always zero registers. Etc.
Most of my experience is in 1, a bit of 2 and mitigation work on 4.
The reason I think we're talking past each other a bit is that you're generating CVEs, while I mostly worked on mitigating and detecting/investigating attacks. In my mind, the attacks that are dirt cheap and I see every week are the biggest problem, but if we fix all of those, and the underlying platform gets better, I see that it'll boil down to trusting the kernel doesn't have vulnerabilities.
This discussion has been had a thousand times over back when people said "chroot is not security boundary". Now people say "containers are not a security boundary", but they mean essentially the same thing.
The thing is, chroots are pretty secure, if you know what you're doing. As long as you run each process as a dedicated uid, with readonly filesystems, without access to /proc or /dev, bar any kernel exploit you should be safe.
The know what you're doing part was where the problems arose. And that's why chroot was considered insecure in practice. People generally put whole Linux installations in chroots, complete with bind mounts or suid binaries. Either way could be a way to get open file handles outside your filesystem, which would make any namespaces a useless spectacle.
Containers are like that. I've seen people doing all sorts of crazy bind mounts, leaving the docker socket accessible, sharing filesystems, or running processes as root.
The kernel exploits are something else, they exist too, and something you at least in theory would patch after they get known. But the sidechannels are a hundred times more prevalent, in any containerized workload that I've seen.
Most kernel exploits are also related to device drivers or file systems, and are often written by third parties. Microkernels were said to contain those by running most of them as processes. That's a good idea, at least in theory. In practice it's tricky because you are dealing with buggy hardware that has DMA access. Any mismatch between a driver and a hardware state risks a system hang, data loss, or security exploit.
You two seem to have figured this out, but as far as I can tell, the disconnect here is that the vast majority of security issues related to the separation difference between VMs and containers isn't due to container "escapes" at all. It's due to the defaults of the application you're running assuming it's the only software on the system and it can run with any and all privileges. Lazy developers don't give you containers that work without running as privileged and demand from users to use that application after migrating from a primarily VM-based IT infrastructure to a primarily container-based one is too great to simply tell them no, and if it's free software, you have no ability to tell the developers to do anything differently.
Discussions on Hacker News understandably lean toward the concerns of application developers and especially greenfield projects run by startups who can take complete control of the full stack if they want to. But running applications using resources partially shared by other applications encompasses a hell of a lot of other scenarios. Think some bank or military department that has to self-host ADP, Rocket Chat, a Git server, M365, and whatever other hundreds of company-wide collaboration tooling the employees need. Do you do it on VMs or containers? If the application in question inherently assumes it is running on its own server as root, the answer to that question doesn't really depend on kernel CVEs potentially allowing for container escapes.
If we're just reasoning from first principles, applications in containers on the same host OS share more of a common attack surface than applications in VMs on the same physical host, and those share more than applications running on separate servers in the same rack, which in turn share more than servers in separate racks, which in turn share more than servers in separate data centers. The potential layers of separation can be nearly endless, but there is a natural hierachy on which containers will always sit below VMs, regardless of the kernel providing the containers.
Even putting that aside, if we're going to frame a choice here, these are not exactly kernels on equal footing. A kernel written in C that has existed for nearly four decades and is used on probably trillions of devices by everything from hobbyists to militaries to Fortune 50 companies to hospitals to physics labs is very likely to be safer on any realistic scale compared to a kernel written in Rust by one college student in his spare time that is tested on Qemu. The developer himself tells you don't use this in production.
I think the annoyance here is it often feels when reading Hacker News that a lot of users treat static typing and borrow checking like it's magic and automatically guarantees a security advantage. Imagine we lived in the Marvel Multiverse and vibranium was real. It might provide a substrate with which it is possible to create stronger structures than real metals, but does that mean you'd rather fly in an aircraft constructed by Riri Williams when she is 17 that she built in her parents' garage or would you rather trust Boeing and whatever plain-ass alloy with all its physical flaws they put into a 747? Maybe it's a bad analogy because vibranium pretty much is magic but there is no magic in the real world.
> if Linux had been built with safety in mind security would be a lot simpler
I'm replying simply because you're getting defensive with your edits, but you're missing a few important points, IMO.
First of all, the comment I quoted falls straight into the category of if only we knew back then what we know now.
What does it even mean "built with safety in mind" for a project like Linux?
No one could predict that Linux (which was born as a kernel) would run on billions of devices that people keep in their pockets and constantly use for everything, from booking a table at the restaurant to checking the weather, from chatting with other people to accessing their bank accounts. And that said banks would use it too.
Literally no one.
Computers were barely connected back then, internet wasn't even a thing outside of research centers and universities.
So, what kind of safety should he have planned for?
And to safeguard what from what and who from who?
Secondly, Linux was born as a collaborative effort to write something already old: a monolithic Unix like kernel, nothing fancy, nothing new, nothing experimental, just plain old established stuff for Linus to learn how that kernel thing worked.
The most important thing about it was to be a collaborative effort so he used a language that he and many others already knew.
Did Linus use something more suited for stronger safety guarantees, such as Ada (someone else already mentioned it), Linux wouldn't be the huge success it is now and we would not be having this conversation.
Lastly, the strongest Linux safety guarantee is IMO the GPL license, that conveniently all these Rust rewrites are turning into more permissive licenses. Which steers away from what Linux was, and still largely is, a community effort based on the work of thousands of volunteers.
> Lastly, the strongest Linux safety guarantee is IMO the GPL license, that conveniently all these Rust rewrites are turning into more permissive licenses. Which steers away from what Linux was, and still largely is, a community effort based on the work of thousands of volunteers.
There is nothing about permissive licenses which prevents the project from being such a community effort. In fact, most of the Rust ecosystem is a community effort just like you describe, while most projects have permissive licenses. There's no issue here.
GPL protected Linux when Linux started to become a threat to the most powerful tech company on Earth of the time, famous for their "embrace and extend" tactic.
Rust had no such problem, it was started by a company that every year takes hundreds of millions (81% of their revenues in 2022) from one of the most powerful tech companies on Earth.
If I am contributing to a community project, made by the people for the people, GPL protects my work from being stolen by corporations that pack it up and sell it for a price.
GPL means you work on something and in return you get something from someone else that improved on your work, you can't keep any published change private.
GPL also means that if someone forks the original source code and makes substantial changes, the subsequent work must be licensed under the GPL (what MS called the "viral nature"), nobody can re-license it, unless all the other contributors agree to it.
GPL software cannot be acquired to kill it, like Google or Apple do all the time, or severely cripple it after the acquisition.
It's a form of social contact that guarantees that no one gets screwed or exploited.
It grows on users' needs not on VC growth needs.
That's how Linux got big.
BSD was already 20 years old when Linux started and yet BSD is nowhere near the popularity of Linux.
I don't think it's that unfair, but I don't want to get into a whole thing about it, people get really upset about criticisms of the Linux kernel in my experience and I'm not looking to start my morning off with that conversation.
We can agree that C was definitely the language to be doing these things in and I don't blame Linus for choosing it.
My point wasn't to shit on Linux for its decisions, it was to think about a hypothetical world where safety built in from the start.
why not ada? Sure rust didn't exist when linux was first being built, but ada did and had a number of memory safety features. (not the same as rust's, but still better than C)
The "historical anecdote" is that somewhen around kernel 0.99, C++ was tried. And given up due to both compiler bugs and (worse) abysmal compilation performance.
For all I know about Ada, the latter has a reputation for "non-speed" on compile worse than C++ even. Nevermind that gnats (gcc ada integration) post-dates Linux by a few years.
Of course, it could simply be that Linus looked, got an allergic fit, and hence didn't bother ...
> Imagine if containers were actually vm-level safe? The performance and operational simplicity of a container with the security of a VM.
As far as I know, the order of magnitudes of container security flaws from memory safety is the same as security flaws coming from namespace logic issues, and you'll have to top that with hardware issues. I'm sorry but rust or not, there will never be a world where you can 100% trust running a malware.
> Fuschia [...] is unlikely to give us the memory safety that a Rust kernel would
Well being micro kernel make it easier to migrate bits by bits, and not care about ABI
Rust is not a magic bullet, it just reduces the attack surface by isolating the unsafe parts. Another way to reduce the attack surface would be to use a microkernel architecture, it has a cost though.
You're not really illustrating your point well with the link. If you look through the examples, they're mostly trivial and there's no clear way to eliminate them. Some reads/writes will interact with hardware and the software concepts of memory safety will never reach there because hardware does not operate at that level.
Check a few of the results. They range from single assembler line (interrupts or special registers), array buffer reads from hardware or special areas, and rare sections that have comments about the purpose of using unsafe in that place.
Those results really aren't "look how much unsafe code there is", but rather "look how few, well isolated sections there are that actually need to be marked unsafe". It's really not "a lot" - 86 cases across memory mapping, allocator, task switching, IO, filesystem and object loader is surprisingly few. (Actually even 86 is overestimated because for example inb is unsafe and blocks using it are unsafe so they're double-counted)
Practically speaking, even with `unsafe` the exploitability of rust programs is extremely difficult. With modern mitigation techniques it is required that you be able to chain multiple vulnerabilities and primitives together in order to actually reliably exploit software.
Bug density from `unsafe` is so low in Rust programs that it's just radically more difficult.
My company (not me, Chompie did the work, all credit to her for it) took a known bug, which was super high potential (write arbitrary data to the host's memory), and found it extremely difficult to exploit (we were unable to): https://chompie.rip/Blog+Posts/Attacking+Firecracker+-+AWS'+...
Ultimately there were guard pages where we wanted to write and it would have taken other vulnerabilities to actually get a working POC.
Exploitation of Rust programs is just flat out really, really hard.
Maestro has a lot of very old school hardware drivers: VGA, PS/2, IDE/ATA.
Newer hardware tends to look like just a couple of ringbuffers, and the drivers should need a lot less of these hacks. Here's an NVMe driver in Rust that intends to avoid unsafe fully: https://rust-for-linux.com/nvme-driver
> Consider that right now a docker container can't be relied upon to contain arbitrary malware, exactly because the Linux kernel has so many security issues and they're exposed to containers
If you don't run docker as root, it's fairly ok for normal software. Kernel memory safety is not the main issue with container escapes. Even with memory safety, you can have logical bugs that result in privilege escalation scenarios. Is docker itself in Rust?
Memory safety is not a magic bullet, the Linux kernel isn't exactly trivial to exploit either these days, although still not as hardened as windows (if you don't consider stuff like win32k.sys font parsing kernel space since NT is hybrid after all) in my humble opinion.
> Linux had been built with safety in mind security would be a lot simpler
I think it was, given the resources available in 1993. But if Trovalds caved in and allowed a mini-kernel or NT like hybrid design instead if hard-core monolithic unix, it would have been a game changer. In 1995, Ada was well accepted mainstream, it was memory safe and even Rust devs learned a lot from it. It just wasn't fun to use for the devs (on purpose, so devs were forced to do tedious stuff to prevent even non-memory bugs). But since it is developed by volunteers, they used what attracts the most volunteers.
The main benefit of Rust is not it's safety but its popularity. Ada has been running on missiles, missile defense, subways, aircraft, etc... for a long time and it even has a formally verified subset (SPARK).
In my opinion, even today Ada is a better suit technically for the kernel than Rust because it is time tested and version stable and it would open up the possibility easily formal-verifying parts of the kernel.
Given how widely used Linux is, it would require a massive backing fund to pay devs to write something not so fun like Ada though.
Chompie certainly worked hard (and is one of if not the most talented exploit devs I've met), but we're talking about a single exploit developer developing highly reliable exploits in a matter of weeks.
A single talented developer taking weeks sounds about right, that's what I meant by difficult but also you have vulns that never get a cve issued or exploit developed because of kernel specific hardening.
As for container escapes, there are tools like deepce:
I can't honestly say I've heard of real life container escapes by attackers or pentesters using kernel exploits. Although I am sure it happens and there are people who won't update the host's kernel to patch it.
Containers became popular because it doesn't make much sense to be running full blown virtual machines just to run simple single process services.
You can lock down the allowed kernel syscalls with seccomp and go further with confining the processes with apparmor. Docker has good enough defaults for these 2 security approaches.
Full fat VMs are not immune to malware infection (the impact still applies to the permitted attack surface). Might not be able to easily escape to host but the risk is still there.
> Consider that right now a docker container can't be relied upon to contain arbitrary malware, exactly because the Linux kernel has so many security issues and they're exposed to containers.
No, Docker container was never meant for that. Never use containers with untrustable binary. There is Vagrant and others for that.
"gVisor is an application kernel for containers. It limits the host kernel surface accessible to the application while still giving the application access to all the features it expects. Unlike most kernels, gVisor does not assume or require a fixed set of physical resources; instead, it leverages existing host kernel functionality and runs as a normal process. In other words, gVisor implements Linux by way of Linux."
I like Rust and work in it fulltime, and like its memory-safety aspects but I think it's a bit of a stretch to be able to claim memory safety guarantees of any kind when we're talking about low-level code like a kernel.
Because in reality, the kernel will have to do all sorts of "unsafe" things even just to provide for basic memory management services for itself and applications, or for interacting with hardware.
You can confine these bits to verified and well-tested parts of the code, but they're still there. And because we're human beings, they will inevitably have bugs that get exploited.
TLDR being written in Rust is an improvement but no guarantee of lack of memory safety issues. It's all how you hold the tool.
Yep. And tooling to secure C improved a lot in recent years. The Address-Sanitizer is a big improvement. I’m looking forward that C++ improves as language itself because it was already improved (smart-pointers, RAII, a lot of edge cases regarding sequencing) and they seem to be willing to modify the actual language. This opens a path for project to migrate from C to C++. A language inherits a lot from its introduction (strength/weak) but also changes a lot.
Every interaction with hardware (disk, USB, TCP/IP, graphics…) need to do execute unsafe code. And we have firmware. Firmware is probably a underestimate issue for a long time :(
Aside from errors caused by undetected undefined behavior all kinds of errors remain possible. Especially logic errors. Which are probably the biggest surface?
Honestly I struggle to see the point in rewriting C++ code with Java just for the sake of doing it. Probably improving test coverage for the C++ implementation would have been less work and didn’t created the security issue first.
That being said. I want to see an #unsafe and #safe in C++. I want some hard check that the code is executing only defined. And modern compilers can do it for Rust.
Same applies to machine-dependent/implementation defined code which isn’t undefined but also can be dangerous.
Which was/is a "safe" dialect of C; basically C extended with a bunch of the stuff that made it into Rust (algebraic datatypes, pattern matching, etc.) Though its model of safety is not the borrow checker model that Rust has.
Always felt to me like something like Cyclone would be the natural direction for OS development to head in, as it fits better with existing codebases and skillsets.
In any case, I'm happy to see this stuff happening in Rust.
I didn't know Firecracker existed, that's really awesome. Looks to be in Rust as well. I'll have to look at how this differs from the approach that Docker uses, my understanding is that Docker uses cgroups and some other built-in Linux features.
The Firecracker, io_uring, and ebpf exploitation posts.
Chompie was my employee and was the one who did the exploitation, though I'd like to think I was at least a helpful rubber duck, and I did also decide on which kernel features we would be exploiting, if I may pat myself on the back ever so gently.
They're using 'contain' to mean 'keep isolated'. If you put some malware in a docker container, you can't rely on docker to keep the rest of your system safe.
The issue of root vs rootless is unrelated to escaping the container. User namespaces lead to privescs because attackers who can enter a namespace and become the root within that namespace have access to kernel functionality that is far less hardened (because upstream has never considered root->kernel to be a privesc and, of course, most people focus on unprivileged user -> kernel privesc). The daemon running as root doesn't change anything there
> That this doesn't happen often shows the motivation of most companies using open source is ultimately ... just looking for free work.
Entering into contracts is really annoying. Companies don't just hand out checks, they have accountants who ask "what is this for, how is it being spent, how do we pay taxes on it", etc. "Pay this random developer that you have no pre-existing relationship with" is not as trivial as it sounds.
There’s also “how do we know they’ll be around three years from now for support?”. Working with a company feels more secure in that respect, especially if that company is large and reasonably old.
The “this” in “That this doesn't happen often” can also refer to “people who author GPLd code from making it clear that if a corporation wants a different licensing arrangement” not happening very often.
Yes, there are projects that explicitly mention it, but I think those are in the minority.
This is what keeps me from postgres for large amounts of data. I use postgres to store metadata or small data (ie: "users logged in currently", "customers", etc).
If you want large datasets (petabytes) you really need to look elsewhere to something with better compression support or that tiers its data off to S3.
It's possible that with bcachefs we're like a decade away from "good fs in mainline kernel with fs compression" but right now it's not a great situation.
> Firefox is the only major browser not built by a company that makes money from advertising and/or selling your personal data.
Firefox makes almost all of their money from Google, so...
> If that’s the case, web developers can easily write sites that work on all browsers
Not sure what the point is here but if everyone is using Chromium/Webkit obviously this issue goes away too.
And, oh, that's it. I thought this was going to have like... a list of reasons. This feels like a very odd post to be #1 on HN right now, it contains almost no information.
AI certainly isn’t a replacement for journalism, but that doesn’t mean journalism will continue to exist if no one pays for it. If everyone gets their news from chatGPT or the like there will be no investigative reporting. We’re already beginning to see this with most people reading the google/Facebook blurbs instead of clicking the link and giving ad money let alone paying.
But I've just explained that ChatGPT can't actually produce news articles. I can't ask ChatGPT what happened today, and if I could it would be because a journalist went out and told ChatGPT what happened.
So at first ChatGPT will copy journalists. Then journalists will stop working because nobody pays them. Then there will be no news. Some people may look at that situation and decide to start a new news business but that business will fail because ChatGPT will immediately rip it off. The end game is just no news other than volunteers.
You still literally have not explained how this works. ChatGPT could write a news article, but it's not going to actively discover new social phenomena or interview people on the street. Niche journalism will continue having demand for the sole reason that AI can't reliably surface new and interesting content.
So... again, how does a pre-trained transformer model scoop a journalist's investigation?
> Then journalists will stop working because nobody pays them.
How is that any different than the status-quo on the internet? The cost of quality information has been declining long before AI existed. Thousands of news publications have gone out of business or been bought out since the dawn of the internet, before ChatGPT was even a household name. Since you haven't really identified what makes AI unique in this situation, it feels like you're conflating the general declining demand for journalism with AI FOMO.
AI labs are working on and largely already have generative ai that can be actively updated. The generative ai scoops real journalists stories by watching their feed. This isn’t very different from the current status quo, it’s just a continuation of an already shitty situation for news organizations. If their revenue decreases even more than it already has they will cease to exist. Niche journalism barely has any demand today, it won’t take much more reduction in demand for it to not be worth the cost to produce. Just a few more people using big tech products as their news feed instead of the news organizations themselves is all it would take.
You can say that the people getting their news from the tech products will switch to paying news organizations in some way if the news starts to disappear but I highly doubt it seeing how people treat news today. And if it that does happen they’ll switch back again to the ai products as the centralization it can provide is valuable.
Updated how? By what? Who is going out and investigating the world to write about? An AI does not have LEGS it can not go outside and go talk to someone and interview them, it can't attend a press conference without human assistance.
You have not at all explained how an AI is going to somehow write a news post about something that has just happened.
Without money there will be no one investigating, there will be no news. If someone creates news it will be immediately ripped off so the only stable state here is no news at all
We're not just beginning to see it, it's already happened. It was enabled by digitized information, then amplified by the networking of the internet. The value of fresh information today is worth the price of a Google refresh, which for most people is effectively nothing. AI doesn't change that equation, and I'd argue it's overall impact on journalism will be less harmful than an ad-optimized economy or even the mere existence of YouTube.
Quality journalism hasn't had a meaningful source of funding for a while, now. If AI does end up replacing honest-to-goodness investigative reporting, it'll be for the same reason the internet replaced the newspaper.
Sadly, opinion pieces are what drives the news economy these days. Columnists/commentators, in effect, subsidize the hard news (at those venues that even bother to produce the latter). Filling their prime time hours with opinion journalism was the trick that Fox News discovered to become wildly successful.
> Why shouldn't the creators of the training content get anything for their efforts?
Well, they didn't charge for it, right? They're retroactively asking for money, but they could have just locked their content behind a strict paywall or had a specific licensing agreement enforceable ahead of time. They could do that going forward, but how is it fair for them to go back and say that?
And the issue isn't "You didn't pay us" it's "This infringes our copyright", which historically the answer has been "no it doesn't".
> Why using authored NYT articles is “stupid IP battles”
When an AI uses information from an article it's no difference from me doing it in a blog post. If I'm just summarizing or referencing it, that's fair use, since that's my 'take' on the content.
> having to pay for the trained model with them is not stupid?
Because you can charge for anything you want. I can also charge for my summaries of NYT articles.
If you include entire paragraphs without citing, that's copyright violation, not fair use. If your blog was big enough to matter NYT would definitely sue.
A human makes their own choices about what to disseminate, whereas these are singular for-profit services that anybody can query. The prompt injection attacks that reveal the original text show that the originals are retrievable, so if OpenAI et al cannot exchaustively prove that it will _never_ output copyrighted text without citation, then it's game over.
> "[...] the fair use of a copyrighted work [...] for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright. In determining whether the use made of a work in any particular case is a fair use the factors to be considered shall include—
(1) the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;
(2) the nature of the copyrighted work;
(3) the amount and substantiality of the portion used in relation to the copyrighted work as a whole; and
(4) the effect of the use upon the potential market for or value of the copyrighted work."
----
So here we have OpenAI, ostensibly a nonprofit, using portions of a copyrighted work for commenting on and educating (the prompting user), in a way that doesn't directly compete with NYT (nobody goes "Hey ChatGPT, what's today's news?"), not intentionally copying and publishing their materials (they have to specifically probe it to get it to spit out the copyrighted content). There's not a commercial intent to compete with the NYT's market. There is a subscription fee, but there is also tuition in private classrooms and that doesn't automatically make it a copyright violation. And citing the source or not doesn't really factor into copyright, that's just a politeness thing.
I'm not a lawyer. It's just not that straightforward. But of course the court will decide, not us randos on the internet...
I want to point out that this is an idiotic take even outside of the fact that type 1 diabetes has nothing to do with diet.
1. A lot of children get type 2 diabetes. Children are marketed to in very manipulative ways, aren't yet fully educated, have weak impulse control, and are handed insanely sugary drinks and foods.
2. Sugar is being put into everything. Unless you're cooking for yourself (hard to do when you're working a ton) for every meal it's hard to ensure you aren't getting tons of sugar.
3. The food pyramid and other systems have been encouraging a high carb / high sugar diet for decades, there's tons of misinformation about it.
At this point ChatGPT can do math by first predicting the algorithm and then handing it off to an execution engine - Python. So if that's the gap, I'd say they're closing it.
Yes, that's a fair distinction - although I think the practical implications aren't important. There's no reason why an LLM has to be AGI if an LLM + Python is AGI.
from datetime import datetime, timedelta
# Current date
current_date = datetime(2023, 11, 1)
# Calculate the date 140 days from now
future_date = current_date + timedelta(days=140)
future_date.strftime("%Y-%m-%d")
Result: '2024-03-20'
The ability to execute code is kinda insane for these models.
It’s kind of funny that they can more reliably spit out code that will give an answer than actually output the answer as text. I guess it’s a workaround that works well for many cases
Humans can also more reliably use a calculator (which is basically what python is) for big numbers than doing it in their heads. I think it makes sense.
This reminds me, I've had an alias for calc='python -ic "from math import *' for a long time now. It comes handy more often than you'd think.
No, it's actually executing that Python code. This is what allows an LLM (or an 'llm based system', I guess) to do something like "reverse <some uuid that has never been observed before>" - it can't just memorize the output and map it to the input because the output has literally never been observed. Instead, if it knows the algorithm for reversing a string, it can just use that and offload the execution to Python.
It is. It doesn’t even need an existing language. You can define your own psuedo language in the prompt and have ChatGPT “execute” it (works best with 4 nonturbo).
>We provide our models with a working Python interpreter in a sandboxed, firewalled execution environment, along with some ephemeral disk space. Code run by our interpreter plugin is evaluated in a persistent session that is alive for the duration of a chat conversation (with an upper-bound timeout) and subsequent calls can build on top of each other. We support uploading files to the current conversation workspace and downloading the results of your work.
It really feels like I'm just googling for you, you had the feature name.
I would say creating a model which is able to interpolate from training data in a way which produces an accurate output of a new input is a little impressive (if only as a neat party trick), however anybody can run a python interpreter on a server somewhere.
I’m sure there are use cases for this. But in the end it is only a simple feature added onto a—sometimes—marginally related service.
Hm, I don't think of it that way I guess. What the LLM is doing is generalizing a problem based on previous problems it has seen and then offloading the execution of that problem to a machine with some defined, specific semantics.
This is a lot more than a party trick. The model is able to describe the program it wants to execute and now it can accurately execute that - that it 'offloads' the work to a specialized program seems fine to me.
It's way more than a simple feature, this is enabling it to overcome one of the biggest limitations and criticisms of LLMs - it can answer questions it has never seen before.
Consider that right now a docker container can't be relied upon to contain arbitrary malware, exactly because the Linux kernel has so many security issues and they're exposed to containers. The reason why a VM like Firecracker is so much safer is that it removes the kernel as the primary security boundary.
Imagine if containers were actually vm-level safe? The performance and operational simplicity of a container with the security of a VM.
I'm not saying this is practical, at this point the C version of Linux is here to stay for quite a while and I think, if anything, Fuschia is the most likely successor (and is unlikely to give us the memory safety that a Rust kernel would). But damn, if Linux had been built with safety in mind security would be a lot simpler. Being able to trust the kernel would be so nice.
edit: OK OK. Yeesh. I meant this to be a hypothetical, I got annoyed at so many of the replies, and this has spiraled. I'm signing off.
I apologize if I was rude! Not a fun start to the morning.