> I don't think OS becomes any less vulnerable than usual Linux/Windows installation.
is not a good enough argument.
For the story, SIP is Apple's "rootless". Effectively the OS runs with less privileges than root. Disabling SIP significantly increases the attack surface.
That being said, I'm grateful that someone decided to do something more native for containers in macOS.
I think it's an OK argument given that most people run (and have been running with no alternative until very recently) docker in such a way that there's a trivial privesc to root. In general it seems like docker users are, overall, willing to take that tradeoff.
In general if you can `docker run` without sudo then that means you have a trivial privesc path since you can do `docker run` with the various flags that run it without any sandboxing, get a shell, and just ask to be let out of the namespace.
The way that podman and newer versions of docker get around this is using unprivileged user namespaces. Unprivileged user namespaces are not a free lunch - in fact, they're a bit of a security disaster in their own right.
In a typical installation, being in a docker group gives you access to a socket that controls docker daemon and that daemon runs as root. `sudo` is not important in this context.
Podman, too, can run in rootful and rootless mode. Rootless in podman still feels to me to be more like first class citizen, as opposed to docker case.
In both cases it's important to keep in mind in which mode you operate. Both from the perspective of security and day to day operations, as some aspects of behavior will differ between those modes.
On Linux, more or less the entire permissions system makes no assumption about SIP existing (as it doesn't there), so other protections are relied upon to secure the system (such as SELinux, granular directory permissions, etc.).
On both Linux and Windows, TPM and secure boot provide similar protections to SIP on macOS, but are optional (it's encouraged more forcefully on Windows 11).
Removing SIP from a system that relies on it as a basis for platform security is different than using a system that wasn't relying on it in the first place.
It still has them, of course, but the concern is that after ~8 years of SIP basically ~everywhere, platform security decisions have been made assuming it is present.
The "varsectomy" bug in Chrome isn't the example you think it is, because disabling SIP was not sufficient conditions for it. There were 3 other conditions that had to be met, the most notable of which is that "/" had to be writeable by tho logged-in user, which is not the default.
This is an example of defense-in-depth being present, and defense-in-depth still failing for some users who gave escalated permissions to some installers, allowing them to run roughshod over their filesystem permissions, leaving them vulnerable to a subsequent varsectomy. If one did the same thing to their Linux system, the same thing could happen.
The two other conditions listed in the page you link basically amount to "the buggy code has to run". Related, I think SSV [1], introduced last year, would also have blocked this bug even with SIP disabled. But none of that invalidates concern that areas of the OS we don't know about might not have the level of defence in depth that we would like - it's not like Apple has never cut corners or shipped bugs to hit a date.
One can have vague, general concerns about any operating system if one lets their imagination run wild, though, and one doesn't care about the presence or absence of specific examples supporting the suggestion that the "permissions system...makes assumption[s]" about SIP existing.
I think we have a rather specific example of a Chrome bug hosing OS installs. Had SIP not existed, there's virtually zero chance that bug wouldn't have been caught before shipping.
I'd like to digress to your "Encouraged more forcefully" phrasing which is quite interesting if you think about it. In my view, it would mean something like pervasive dialog box ala EULA, some UI hoops you need to resolve, alike going with local account on installation.
In reality they done basically everything to force users to use secure boot.
If they disabled normal boot altogether, OS adoption would suffer heavily.
They could've obscured that option, but it would be found out, and enterprise users would be pissed at them because they didn't gave them a provisionable way while the way exists. So it came down to normal variables in installer registry.
However modifying, e.g making users "hack" the ISO is really as forceful as it gets without market loss.
Note: There may be more normal way today than modifying the registry of ISO, I installed 11 once when it came out.
Sounds like if Apple wants developers who wants to use containers natively, they need to address the problem of not being able to offer this feature without disabling SIP.
No matter what you or I think about what's needed for adoption, technical problems get in the way of the tool working with SIP, so seems it's in Apple's ball court really.
Understatement of the year. I am sure there are some places where being caught doing something like that (without authorization) could result in one of those “my hands are tied, I have to fire you” situations.
Think places where security is a big deal, like finance, military, aerospace, critical infrastructure etc.
And what is the benefit of that? Who would use that and for what? Containers solved deployment, and software distribution problems related to diversity of runtime environments on linux. It also has some security benefits, but their adoption was successful because it solved real world everyday problems. It eventually allowed better utilization of servers.
They did not have such success on windows, despite Windows also having a container subsystem, as windows servers already did this with IIS web app containerization.
On MacOS desktop software distribution is largely a solved problem since ages. On MacOS/Darwin servers... are there such in industrial use apart from some research installations?
Docker For Max marshals filesystem events over a VM host/guest boundary which can grind the most powerful computers to a halt if you’re sharing directories between the host and guest. For example, at my last company we developed Python apps and ran them in Docker for Mac containers by mounting the source code directory into the containers (so we wouldn’t need a build step) but as our project grew the filesystem event marshaling became exponentially slower until we eschewed Docker from our dev iteration loop entirely (the fidelity benefits weren’t worth the performance hit). Note: there are lots of projects and hacks that claim to solve this problem but none made an ounce of difference.
The alternative is to use a performant file sharing system through the VM isolation. Some people use NFS, I personally use VMware Fusion + vmhgfs.
Sure it does not attempt to map fsevents<->inotify 1:1 but honestly I can live with that limitation given that it's a 10x performance increase compared to the DfM kitchensink.
I do go native darwin when I can / it makes sense.
Is that something you can configure in DfM or are you saying you sort of roll your own DfM alternative? Also, what does "VM isolation" mean here? How is NFS or VMware Fusion more "isolated" than DfM?
We used native MacOS processes and it worked out fine. In practice I don't recall any bugs due to different behavior between our dev environment (MacOS) and our production environment (Linux) and we saved a ton of time during development. Docker just wasn't worth it. This worked for us because our app didn't depend on any platform specific behavior.
Of course, if you are using Go or some other language with a fast, static cross-compilation step, you don't need to mount a source code volume into your container, you can just rebuild the whole container image or rebuild on the host and `docker cp` the new binary onto the target container.
Technical limitations aren’t excuses for a bad design. If it’s not a good design due to a technical limit, the answer isn’t to sacrifice security for functionality.
If it really isn’t technically possible (which I think you might be able to do in a Darwin VM), then maybe this approach isn’t a good idea.
> Technical limitations aren’t excuses for a bad design.
Tomorrow Apple might decide it is safe to chroot with SIP enabled (I actually do not understand why they restrict it, chroot is a tool to increase security). Does that suddenly convert bad design into a good design? But this is exactly the same design.
Design operates within the contexts of a system. A perfectly viable design can be made bad by a change in the system for which it was designed.
Does this current design require disabling SIP? Then I don’t think it is worth my effort to use (for my use case). If Apple changes the system in the future, my opinion might change.
But a design cannot be judged as good or bad outside of the context for which it was designed.
If you feel this is a better way to tackle the problem, then talk to Apple about it.
> I don't think OS becomes any less vulnerable than usual Linux/Windows installation.
A modern Linux with SELinux enabled (the default in e.g. Fedora) running apps inside rootless containers (Podman doesn’t even need a daemon) is likely much more secure than your default MacOS or Windows.
Well Linux powers just about all the most important systems in the world and SELinux was originally developed by the NSA, but I'd love to get your insights.
Apple relied on it being turned on and started introducing permission checks where you'd have things like "do thing as root" and check for some flag being set that is protected by SIP. If you have it turned on, there are no issues, because the check blocks both root and non-root users! But if you turn it off, now the "do thing as root" is available to any account because the flag is editable.
I'm more interested in this as part of a CI/CD process for releasing macOS software. For an isolated build worker - yeah, sure, I'll go ahead and disable SIP.
Not every workload is running on an endpoint connected to a human via keyboard and screen.
>For an isolated build worker - yeah, sure, I'll go ahead and disable SIP.
Isn't this especially dangerous on a build worker? All your source code goes in and you (presumably) use the binaries that come out across the rest of your infrastructure. Compromising a build worker in a persistent fashion due to lack of SIP seems like it could do some serious[1] harm...
Depends on your threat modeling. Are you running untrusted code on the worker (maybe you have an org with thousands of engineers, maybe you're running builds from a public fork)? Sure, that's an issue. Are you a small startup? Take the convenience and focus on the bigger fish to fry.
Does OrbStack allow you to create MacOS Containers? It doesn't seem like it from their website. It seems like it just runs on MacOS to let you create Linux containers.
This kind of makes me wonder why you'd have a Mac at all (I'm sure there are use cases).
Wouldn't a Linux device, or Linux running on a Mac suit you better?
For me, the security picture is one of the main features of the eco-system even if it's very restrictive - disabling SIP undermines it more or less completely.
>Wouldn't a Linux device, or Linux running on a Mac suit you better?
Maybe they want a unixy desktop with working sound ?
Half joking, but that's my use case - homebrew is pretty great, most developers use a Mac in my domains of interest so it's always supported.
Linux is just too much work (and I'm using Fedora on my desktop). SIP is just false positives and annoyance.
I'm on the fence about M/ARM switch since I still see a lot of friction with containers so I might be looking at framework for my next device. Or just go all in on client/server development model.
I can use it to get stuff I need to do work and I don't remember the last time I couldn't. Upgrades sometimes leave me broken but it's usually ironed out fast because everyone is using it. If my dependencies are that locked down I'm using docker and special environments anyway. Linux can get tedious with upgrades breaking or dependency mismatches. Homebrew is probably the largest homogenous community - if you're doing something relatively popular there will be a lot of noise when stuff breaks. Linux is spread out across various distros/repos/package managers, overlap of users with your problem is a lot smaller.
Beats all the package management experiences I've had on Windows, admittedly I have not tried to use Windows for work for >1 year.
OK but what I like about homebrew on mac is that when I'm having an issue with "popular stack X broke after updating" it's probably me and >10k other people out there, so by the time I hit the problem it's already under investigation on GH. I'm not sure the same would apply to homebrew on Linux - even if you ignore the differences between distros - how popular is homebrew on linux and linux desktop in comparison ?
I wish more of those 10k people would help get others off of a package manager that is so fragile and convoluted that updating so often leads to popular things breaking.
Things like macports and pkgsrc do things in an arguably much simpler, more unixy way, without the contortions that so often seem to leave homebrew in a bind after routine operations like updating.
The comment was in response to parent's stated complaint, namely having to wait for someone else to resolve issues with popular packages being broken after an update, which has been the experience of more than one user.
If you need to build something from source (my use-case: Vim, so I can change which language bindings exist in the resulting build) it can sometimes be a lot easier than cloning and using the "raw" C/Make build system.
Also, assuming a downstream distro like Debian or Ubuntu, what's in Homebrew is likely a more up to date package. You could fiddle with adding/using Debian testing or some PPA, or... you could just use Homebrew.
(FWIW: I use Arch and the AUR on my desktop Linux installs these days, and it's essentially the same process. But still using Homebrew on the Mac, and occasionally in Linux when I'm not on a desktop)
I mean, it installs stuff more or less reliably for me so I’d say that’s pretty great. I’m sure it has issues (I’ve had problems trying to install old versions of packages for example) but I don’t think I’ve ever had “brew install <whatever>” fail to install what I wanted
The experience is really seamless, mostly it just works… which IMO is nothing special as they mostly support installing the latest and maybe a couple of other versions, and software that requires weird stuff will likely be packaged in a self contained way.
I switched to it after using both Fink and macports and it was the first time I felt like I had a package manager as reliable to apt on Linux. Both Macports and Fink would routinely break my whole system by putting extra versions of system libraries where other applications could find them. Since, I’ve switched to nix.
For example people do not use homebrew to install python3 on a fresh install of mac os and get a python that spews a bunch of messages to stderr about the version of libc6.so and cannot speak TLS.
Not sure I agree, when I use brew it still regularly updates random unconnected crap through multiple major releases. And of course it takes ages doing so
I use AMD64 containers on my
M1 Pro under Docker Desktop with zero problems. It’s about 10-20% slower than my AMD64 Linux machine on average, which is usually fast enough.
depending of what you mean by friction with containers, there may be different solutins. for example, as a temporary solution on Docker when you don't want/have time to support both arm and x86, you can do something like this:
FROM --platform=linux/x86_64 ubuntu:23.04
i haven't found any issues with it that i could not get over in the past 2+ years of m1. most of the containers are available on both architectures anyway. the performance improvement was totally worth it, i won't even talk about the heating issue with intel.
> i haven't found any issues with it that i could not get over in the past 2+ years of m1.
I'm currently running a Journal of Open Source Software x86 container on aarch64 and it's terribly slow. Takes 12GB of RAM and 3 minutes to build a LaTeX document, see https://github.com/openjournals/inara/issues/30. Any tips?
Building a single-page LaTeX document as a PDF can be done in less than 0.3s when the environment is set up correctly. I don't know anything about this particular container.
> Maybe they want a unixy desktop with working sound ?
In my experience, this has not been an issue for the past 10-15 years atleast. Before that there were some problems with few (external) soundcards or random cpu spikes with the mixers.
However, the UX can still improve. Switching audio outputs with multiple outputs like external displays etc is not very smooth or intuitive.
Some bluetooth headsets have issues but I've had those with a mac as well.
I'm running Linux everyday and I really wouldn't recommend it for any serious audio stuff.
There is great audio software coming to Linux (Bitwig, Reaper, etc) which is great but the underlying infrastructure is a mess.
There are like 3-4 audio subsystems running, I never know which one is it, setting latency is wizardry and sometimes it doesn't run at all. It's usually fine when I run stuff like Spotify, VLC, or Youtube in Firefox, so for user-level audio, Linux is fine IMO. But when I run something where I care about latency and multichannel output, it's hit or miss. It runs fine one day and then I get no sound on another or distorted sound or sound playing at wrong speed and wrong pitch (yay, 44,1 vs 48).
Maybe it's the distros I'm using, maybe there are some that work better, but the UX isn't as great as with macOS. On Manjaro, update sometimes get audio notification removed from tray and I can't change volume using mouse or dedicated keys. Then I have to look for few hours for a solution only to have the same thing happen again three months later (same with brightness keys on laptop). On Ubuntu Studio with an external soundcard, I get randomly distorted sound or no sound at all. So it's easier to use some shitty onboard sound, great.
I like Linux, I use Linux daily, but sound on Linux is terrible. It's much better than it was, yes, but still terrible. For anything more than "play a song here", macOS is much better.
Fedora Silverblue with Pipewire will just use a single subsystem and that's it. Inmutable OS and the rest of software it's Flatpak. The issues are gone. Oh, you need a proper devel environment with dnf/rpm? Just use "toolbox enter" and install all the complex envs under that container.
It sounds like your specifically fighting your distros problem, or something that its not your understanding.
I would not consider Linux sound Terrible, but to be fair, I only use it every day for regular development tasks for the last 10 years. Maybe I've become accustomed to whatever problem you see that I don't.
I moved from Linux to M1 MacBook recently. I know my greps and vims, but I was tired of audio glitches during high CPU usage, system not waking up from sleep, total OS freezes, super loud fans, and so on.
Now I get none of that. I don't think I've ever heard the fans. Audio just works, everything is super snappy. It always wakes up. I'm no longer afraid of bluetooth.
And on top of that, setting my $DAYJOB VPN took three minutes and it just works, where on Linux I had constant problems with DNS breaking, and setting it up was always an hour of work, praying I got the config files right this time.
It really seems to be "unixy desktop with working sound", the best of both worlds.
Exactly my experience. After 15 years, I became an apple fanboy in 15 days. I still do hate losing my muscle memory on some bash shortcuts, but I'd say it was very much worth it.
There's only a 1% chance these days on Linux that your sound won't work or your computer won't sleep when you close the lid or your wifi won't work, or your ethernet, or your cooling, or a peripheral, or CPU/memory spikes.
And a 90% chance it'll be at least one such thing.
All of those things work fine on every computer in my household that runs Linux. This spans thinkpad, dell and ASUS laptops, Dell desktops, home-built gamer type desktops, a few raspberry pi's, and a SFF PC we use to run Kodi on the main TV.
I do find it amusing in a thread about how you have to turn off a core security feature to be able to use containers properly on a Mac that the discussion immediately turns to how bad Linux sound drivers supposedly are. Honestly, I went in the other direction (Mac to Linux) and I've found the waters to be just fine. I don't know if I just have the magic touch or something, but ¯\_(ツ)_/¯.
Haha now that you mention it touch (trackpad) absolutely sucks on Linux when I last tried it - I couldn't even get palm rejection to work.
But this isn't bashing on linux desktop (I would use it if Mac wasn't an option) as much as giving a reason why people would use MacOS despite being annoyed by SIP.
The sound is an internal meme where the linux devops would regularly have to drop off calls to talk (restart to unmute :)).
On my desktop I couldn't even boot installer without running with safe mode, otherwise I'd just get stuck on a blank screen (ancient 1050 TI GPU and standard desktop components otherwise, so not exotic/new stuff).
I've used linux desktop for >decade and Gnome shell feels like home but these days I feel like I don't have the time for linux adventures. Maybe I'll mix it up with my next device, but I'm not reading great things about AMD power modes and Linux.
> The sound is an internal meme where the linux devops would regularly have to drop off calls to talk (restart to unmute :)).
Hmmm once every full moon MS Teams running on ungoogled chrome do not seem to realize my Bose BT Headset is paired and available (and in that case I just use the internal soundcard) but I have seen people having sound issues on MS teams and needing to reboot regardless of the OS they were using. Windows, Linux, even some MacOS users so I wouldn't use that as a generalization.
> In my experience, this has not been an issue for the past 10-15 years atleast. Before that there were some problems with few (external) soundcards or random cpu spikes with the mixers.
May be some confusion. To run linux on a newer Mac with "Apple Silicon" (ARM based), you need to go through a lot of hoops and much work needs to be done still for a stable environment. Check out https://asahilinux.org/about/
Or maybe you thought they meant running linux in general on a PC (Intel x86 32/64 bit)? In that case I agree - driver issues like that have been mostly ironed out by now.
I'm getting early PulseAudio vibes with PipeWire though. BT audio devices stuttering and sometimes losing audio completely regardless of output devices unless I restart the daemon. I guess it stabilizes again during next few years.
> Wouldn't a Linux device, or Linux running on a Mac suit you better?
No? Maybe you're preferring Mac OS for getting stuff done, exchange work with the outside world and/or use non-historic software (like any commercial desktop app such as idk Photoshop, Sketch, Audio, 3D, CAD s/w, etc., etc.) and still are a developer?
Or even doing something esoteric such as using office software without wanting to throw your notebook out of the window?
I'm not your parent post but surely the easy answer to why use a mac at all is simply how great the hardware is. The 2020 M1 Air is nearly 3 years old and still a brilliant machine.
No, the hardware drivers are very polished and never give any problems. And the hardware itself is beautiful but I suspect wouldn't integrate as well with the OS if it were running Linux.
In my opinion, MacOS is just a nicer experience on the whole than (Ubuntu or Red Hat) Linux (I have limited experience with other distros, but they are all pretty comparable to the big distros at best.
I only ever really had one goofy driver/deep OS bug in Mac - something with the location daemon would cause the wireless internet connection to cut out repeatedly. That bug was left behind with that machine when I left that company, and didn't appear in my next macbook pro.
Linux is just always a struggle with drivers, subtle bugs, and other misc friction. It's not a dealbreaker - ubuntu 22.04 is still my daily driver, but it's very much enough that I would prefer a mac for most development.
For example, if I run a software update, it quietly breaks the fn keys to change screen brightness, and when the machine wakes from sleep, the screen stays black. I figured out after much trial that running ubuntu-drivers fixes it, but it's a pain. I'd rather just turn off auto-updates.
Also the Command key for keyboard shortcuts is brilliant and just works across the whole system. On linux I have to use ctrl-shift to copy/paste and I haven't found a good workaround yet.
I ran desktop Linux (Manjaro) for years in an enterprise job with certified hardware, as a Linux sysadmin. I regularly had issues and spent 5-10% of my time troubleshooting the OS or a janky application.
Conversely, macOS is broadly 'production grade'. It mostly 'just works' (with a number of tweaks - including SIP -, hacks etc on initial config, most power users automate with dotfiles). It has a drastically better UI, first class terminals and unixy support, and most code built for it has a higher level of shine.
I am confident enough to deploy alternate security implementations for the convenience of full FS control, as I know many power users are. Disabling SIP is a bad idea for those who don't understand it, the same as disabling Windows Defender or forwarding NAT on your router.
SIP on consumer laptops/desktops is security theater. It only really makes sense on web-servers. For people for whom this is a deal breaker, can you describe a concrete scenario where SIP would actually protect you?
I don't know whether it is popular everywhere, but it is certainly popular everywhere I go (anywhere in the US, and western Europe). I absolutely love it.
SIP is a feature that protects you from malicious actors with root (admin) access on your device. After they've encrypted your photos and drives and changed your passwords, it prevents them from making your machine unbootable by deleting or altering system binaries. As a side effect of this protection, you give up certain freedoms to customize your system.
Don't disable security features because of random comments from randos.
Also, don't install shit anywhere but your home directory as you unless you want to break your system in an irreparably, unmaintainable, or unsupported manner.
If you're using sudo, you're already doing it wrong.
> Also, don't install shit anywhere but your home directory as you unless you want to break your system in an irreparably, unmaintainable, or unsupported manner.
I disable SIP because I don't want to use an OS where I cannot debug programs which did not consent to be debugged. macOS makes it impossible to inspect failure states I encounter in normal usage (like I can on Linux with debuginfod enabled, or Windows with .pdb files) without first rebuilding the program in debug mode.
My experience is that I was unable to debug Maestral (a Python program) crashing in native Cocoa bindings in LLDB, because the interpreter did not allow itself to be debugged. Mozilla says Firefox acts the same way at https://firefox-source-docs.mozilla.org/contributing/debuggi.... It's possible there's some workaround I'm not aware of, but I ended up disabling SIP just like every non-macOS computer I own.
I understand that of course, I'm much more curious why you think this is a concern for people, and/or why you consider this an effective protection against said unknown actors.
I want that feature on cause I dont want stuff I dont know about looking and changing stuff I dont know about, whether it's running with escalated privileges or not
I doubt you (or any human) is capable of enumerating what you don't want looked at. Frankly, I doubt most of this unknown area is covered by SIP at all, and it would be extremely odd if it did. Perhaps you might consider arguing for actual permissions rather than arbitrarily walling off the OS in a way that tangentially benefits the monopoly Apple holds over their own computers.
Wouldn't it be far easier to enumerate what you want an app to access?
I feel like you're assuming that applications have to be honest about what they are when they request a user-prompted permission. SIP makes that irrelevant.
No, I'm assuming that you know what you install and that apps run with the same rights your user has. Your user can't touch /System, so shouldn't the app
Why would it be sarcasm? If I am root and want to write to /bin or /System, I should be able to, even if it might be dangerous. I get why Apple does it for normal consumers but for people who know what they're doing, it should be available, which it is by disabling it.
Hopefully if Darwin containers take off and produce an ecosystem of containers this could pressure Apple to implement process namespacing in Darwin in a SIP compatible way.
You're free to disable SIP, so your post makes no sense.
It's more akin to locking your front door. Keep it locked if you want the extra protection but feel free to crank the door open if you want the breeze to come in, but knowing that more than the breeze might do so.
Have you ever tried modifying the filesystem a couple levels under /?
Last I used a Mac I remember trying to create a new directory in / and writing to /bin (or something like that). I was appalled that Mac doesn't let you do anything other than read-only operations for select paths a few levels under root, and as I remember, there way no way to disable this asinine behavior.
I say this as someone who has used Unix, MacOS and Linux for most of his long life: not being able to write to /bin and other system directories is a feature and I really don’t understand how anybody in 2023 could see it otherwise.
Freedom does not just imply the ability to do the things you want, but also the ability to avoid the things you don’t.
That specific example doesn't seem to be an issue anymore. The Nix Installer creates /nix/store on macOS 13.5 just fine without disabling SIP. You do need admin rights of course.
I opt-in to a prison with trade-offs I deem reasonable for what I get out of it. I have a couple of Linux devices too.
Being interested in this project and simultaneously disappointed it requires SIP to be disabled are not conflicting views, even if a little unfortunate for me personally.
Seems like all counterarguments to this fact sound like people who aren't competent enough and/or don't understand the basics. There doesn't need to be a higher cognitive load using Linux and security arguments make even less sense.
Fundamentally, containers are about namespace/isolation of a bunch of OS interfaces, so file system functions, network functions, memory management, process functions, etc, can all pretend like they're the only game in town, but crucially without having to virtualize out the kernel.
Does XNU have such namespacing functionality across all its interfaces?
Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.
The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out: they actually run a Linux virtual machine (using Hypervisor.framework/hvf), and have that just provide the container environment.
Is that what this project is doing? But then, how could it run a macOS container?
[1] based on the foundation that Linux, unlike BSDs, has a stable syscall interface!
This introduces a long-standing problem that is really the main issue I have with Docker/Podman/containers on anything other than Linux - you have a dedicated VM that needs memory to be set aside for it exclusively.
This is fine if you have a 32/64GB machine, but less so on an 8GB non-upgradeable laptop.
I get it - memory is relatively cheap these days - and manufacturers that are building memory-limited devices are really only doing it to fleece you on obscene upgrade fees at the time of purchase - but it would be nice if there was a more elegant solution to this on Windows and macOS.
WSL 1 had a solution to this that clearly took a lot of work to put together, wherein they'd have a Linux kernel running side-by-side as a Windows sub-process so that the memory pool was shared. Unfortunately it might have been too much work as they scrapped it entirely for WSL 2 and just used essentially the same VM route.
If anyone knows of any projects trying to work around that problem I'd love to hear about it. If Apple really wanted to bring the development community back on board, focusing on these kind of use cases would be great, sadly it seems someone over there has taken the view that scrapping butterfly keys and the touchbar is "enough".
Say what you will about Microsoft, but they've focused really hard on developer use cases for decades, and it shows.
> If anyone knows of any projects trying to work around that problem I'd love to hear about it.
Containers are namespaced processes. These processes exec against the corresponding kernel they require. There is no workaround: if you have an ELF binary calling Linux syscalls it can only run on a Linux kernel†, so to run that you need a VM††. It's not as bad as it appears thanks to memory ballooning†††.
Conversely if you want to exec a Windows binary in a container, the Windows kernel needs to provide process namespacing features (which it does). And if you want to exec a Darwin binary in a container, then the Darwin kernel needs to provide process namespacing features (which it doesn't).
† WSL1 was implementing the Linux syscall API on the Windows kernel, which proved to be much more complex than it appears to be.
The challenge of running OCI containers in every OS is a bit similar to the problem of running the same binary in every OS.
I think the only true solutions are (a) OS vendors develop their own native container platforms with UX similar or better than Docker (b) OS vendors agree on some common ABI standard
> running a x86 windows binary - looking for 32bit stuff under SysWOW64 - under x86_64 wine in a x86_64 Linux container AOT translated to aarch64 by Rosetta 2 for Linux that switched the ARM M1 memory model to Total Store Ordering on a aarch64 Linux kernel under Virtualisation.framework on arm64 darwin.
> WSL1 was implementing the Linux syscall API on the Windows kernel, which proved to be much more complex than it appears to be.
I've long wondered, and again now that Chat GPT is proving so adept at coding, if this translation layer could be automated. Do humans actually have to hand-code each syscall? Or are there just enough edge conditions that can't be automated?
The hard part isn't so much writing the code as it is testing the code. The WINE server is decades old, porting API calls to another kernel isn't an entirely new or novel process. Getting all the software to work right is an ongoing process though, and it requires careful deliberation over what works and what doesn't. WINE code is still being refined to this day in response to new titles and old bugfixes.
So, I won't preclude the idea of AI helping, but I think human effort is still the bottleneck for projects like this. Even if AI could write perfect code 100% of the time, testing and troubleshooting would probably still be the larger timesink.
This is supposed to be possible on macOS, but it's basically completely broken (just doesn't work) due to a bug. Apple is aware of it but unfortunately hasn't said much more.
Obviously that limits the options, but I'll still be taking one last shot at using creative workarounds to tackle the memory problem in OrbStack (another containers-on-macOS product).
Recent versions of Docker Desktop have a "Resource saver"[0] mode. It will de-allocate memory and CPU when containers aren't being used. If there's a leaf icon next to your Docker icon then it's in resource saver mode.
I imagine over time it will get smarter too. Right now it waits for no containers to be running for 30 seconds and enables resource saving mode but who knows what could happen in the future. Maybe it can internally profile and estimate load based on evaluating runtime stats of your contains and dynamically change the VM's resources on the fly and then expose a +% over provision threshold option or a way to turn off dynamic resource saver mode.
One of my issues too. The tools/ecosystem are Linux driven and doing anything on Windows requires a Linux 'stub' of some sort (VM, WSL, etc). I am comfortable with both OS's but all of my coworkers are Microsoft OS tied. 0% Linux experience. Cannot really introduce a managed kub/docker/containers into the picture without ending up being 'support' - not happening! The cloud is an option but that still leaves the developer experience (local laptop) short changed.
Edit: It has been a while since I last looked at this. Looks like containerd is, perhaps, a native option
WSL2 doesn't really use VMs in the traditional sense. msft have invested heavily in lightweight virtualisation (eg better memory management) for certain security features, which allowed WSL to perform well enough without the maintenance/support overhead of WSL1
rund is an experimental containerd shim for running macOS containers on macOS.
rund doesn’t offer the usual level of container isolation that is achievable on other OSes due to limited macOS kernel API.
What rund provides:
- Filesystem isolation via chroot(2)
- Cleanup of container processes using process group
- OCI Runtime Specification compatibility (to the extent it is possible on macOS)
- Host-network mode only
- bind mounts
So essentially a chroot with a bit of make-up and a lot of marketing?
Except for bind mounts (not even overlayfs...) there isn't much interesting.
> - Host-network mode only
Yeah expect a lot of things to break in subtle ways... most containers are developed kinda expecting you have your own network namespace (and that no one else is using ports)
1. It is not trivial to properly set up a chroot on macOS. If you try to find a working guide/tool that works with modern macOS, I doubt you'll find anything (at least, I failed, even though tried very hard)
2. I believe that ability to package stuff into a Docker image distributable via already existing infrastructure and compatible with already existing tools maybe "a bit of make-up", but it is an important makeup.
3. Kubernetes recently got HostProcesses for Windows: https://kubernetes.io/blog/2022/12/13/windows-host-process-c.... They are even less isolated from host than chroot and still, people find them useful for certain scenarios.
Great effort. I get why you call it container - but sounds more like jail or cheroot would give more appropriate expectations; like "tooling to build and run Darwin containers in a macOS chroot"?
I didn't want to use "jail" term because it is mostly unheard of outside of FreeBSD.
Container definition is very stretched nowadays. Look at Windows HostProcesses in Kubernetes [1]. They don't have neither process, network nor device isolation from the host.
I also plan to try macOS sandbox-exec tool, which should offer additional isolation from the host.
If the parent process of the container here changes its bootstrap port to itself or disinherits it then it could also create an isolated mach namespace, restricting access to mach/XPC services.
- bind mounting solves the exposition of filesystem within the root pivot.
- overlayfs solves the persistence efficiency issue using a layered union fs.
> most containers are developed
Most Linux (and Windows) containers. Since these are macOS containers there are no containers developed yet so by definition there is nothing to break.
DfM is more like running the CLI locally to a remote Linux machine, and all it does is conveniently expose /Users in the same place through the VM folder share so that you have the convenient illusion that it happens locally.
If Darwin had process namespacing features it would not make it magically able to run Linux processes.
> > Does XNU have such namespacing functionality across all its interfaces?
> I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK
Theoretically, probably, for coarse-grained yes/no things? I don't think it's able to go much further than "you can use the local network and/or internet" and "you can read/write to the filesystem location corresponding to you bundle identifier `com.foo.bar`" but not "hey let me present you with a namespaced view of loopback or process list".
Also not sure if it can be dynamically set by a parent process for a child? Seems like it's very bundle oriented (except maybe for Apple processes) so not very practical.
I played with it some time ago, can't recall the context but it was about build systems / packaging (maybe nix?), doing the configure/make/make install with reduced privileges.
> Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.
There is more to the container ecosystem than Linux containers; Windows native containers function much the same way (well, in two ways, with VM-backing or the traditional kernel syscall interface, but with Windows syscalls).
I am also interested in the API boundary. It seems that there are too options:
1. Rely on system call stability. This is like Linux containers but unlike Linux macOS doesn't provide a stable system call API. So this would break when the system updates with a change that modifies the system call API.
2. Install the host libraries into the container at runtime. This should provide as much stability as macOS apps usually have. It may also be beneficial as you wouldn't be embedding these into every container.
It seems like 2 would be preferable. However it may be a bit weird when building as the libraries you build against would be updated without the container being aware, but this is unlikely to break anything unless they are copying them to new paths which seems unlikely.
Generally speaking macOS does not guarantee syscall stability, and does not generally guarantee compatibility for any binaries not linked to `libSystem.dylib` (that is the supported ABI boundary)[1]. This has a number of implications, including (but not limited to):
* The most obvious is the commonly mentioned fact that syscalls may change. Here is an example where golang program broke because they were directly using the `gettimeofday()` syscalls[2].
* The interface between the kernel and the dynamic linker (which is required since ABI stability for statically linked executables is not guaranteed) is private and may change between versions. That means if your chroot contains a `dyld` from an OS version that is not the same as the host kernel it may not work.
* The format of the dyld shared cache changes most releases, which means you can't just use the old dyld that matches the host kernel in your chroot because it may not work with the dyld shared cache for the OS you are trying to run in the chroot.
* The system maintains a number of security policies around platform binaries, and those binaries are enumerated as part of the static trust cache[3]. Depending on what you are doing and what permissions it needs you may not be able to even run the system binaries from another release of macOS.
In practice you can often get away with a slight skew (~1 year), but you can rarely get away with skews of more than 2-3 years.
I do not. I'm not really a macOS user but have heard this mentioned many times. I remember one recent example is Golang switched from doing raw syscalls to using the macOS libc due to this issue. That was probably ~5 years ago now so you may be able to dig up examples there.
I can't help but feel like this is an X/Y problem. Apps on MacOS shouldn't need containerization to function.
I get the point of isolation for build/test situations. But Apple provides a neat virtualization framework, and you get security + isolation + reproducibility + decent performance.
It seems like if you feel the need to containerize the userspace on MacOS you're using MacOS wrong. It's not the same thing as the Linux userspace, and doesn't have the same kernel features that would let you do so cleanly or performantly.
Orbstack is moving mountains to provide Linux-native perf and support for containers and it still makes me beg the question: why are devs allergic to just using Linux natively? At least I understand why Orbstack is useful, I don't know why containerizing MacOS itself is.
> But Apple provides a neat virtualization framework, and you get security + isolation + reproducibility + decent performance.
You also get limits on how many VMs your machine can run, each VM needs gobs of storage and locked-out RAM blocks, and sharing directories between the host and guest, compared to bind mounts, is something that makes me remember for my root canal dental jobs wistfully.
I've only used it with Linux kernels, but that has not been my experience with the new Virtualization.framework. The perf of virtio shares is reasonably fast.
I can see how you'd need a crap ton of disk for MacOS virtualization, but again, why do you need it?
If it's isolation for builds, fix your build. If it's isolation for tests, live with it. If it's for running your app, write your app to properly run in the app sandbox.
MacOS apps are designed to be self contained and not require isolation, unlike most Linux distros, which are designed to use FHS and share their state/dependencies with everything else on the system.
Further you can sandbox processes just fine on MacOS without needing to mock the whole userspace, like you do on Linux. This will give you the same degree of isolation that a container does.
Neither is using macfuse as a stand in for overlayfs and bind mounts.
My point is "as fast as it gets" is using MacOS how MacOS is designed: ie, through sandboxing and not containerization that pretends the MacOS userspace is Linux. It's not Linux.
There is a fundamental trade off between isolation and performance. You cannot securely share resources without overhead.
What's the licensing situation on this? Would I be distributing parts of macOS in my containers? I don't think Apple is OK with that.
Or is this just the fully open source Darwin core? That wouldn't likely be super compatible with a ton of production software? I need more explanation of what is actually going on here because it sounds like a good way to get sued.
> Would I be distributing parts of macOS in my containers?
Unless you're producing fully static binaries (or static enough that they don't bind to non-redistributable things) it'd be a yes (it would not be much of a container if it needed non-packaged things)
There are Cirrus Runners which is a service of managed GitHub Actions Runners powered by M2 chips. But there is no free option for OSS yet. https://tart.run/integrations/github-actions
This one is ridiculous. This should already exist. Until GitHub builds it, you can use GitHub Actions to kick your builds off but run them remotely on Earthly Cloud (https://earthly.dev/). Even the free tier includes arm64 remote runners.
Note: I work at Earthly, but I'm not wrong about this being a good, free, arm64-native workflow for GitHub Actions.
It's intended to prevent malware from changing system files due to rogue permissions or escalation. With SIP enabled, even the root/sudo user doesn't have rights to change these files.
It also refuses to boot a system with drivers that are not signed by Apple, so as to deter malware from using drivers as an attack vector.
IMO that’s not good enough, especially when disabling SIP is involved.
We don’t even have certainty that the human running the account is who they say they are (anyone can make a GitHub account and make it look like a real person).
Not everyone who wants to use a container system understands the underlying code of that container system. If I’m a web developer using Docker Desktop or podman to build my PHP app, I’m not necessarily going to understand the code written in Go when my specialty is PHP.
yes it does. There’s only one contributor for most of it and you can click to see his profile.
With the source code available and the primary contributor clear, what more could anyone want? Certainly it’s a bit much for one to ask for a security audit they themselves won’t do
It clearly links to the GitHub where you can click to see all contributors
I suppose the answer to your question is “people who want macOS containers”, whoever they are. As far as malware, I’d employ whatever your standard practices are for installing GitHub projects
I bought an Apple Silicon machine after their presentation claiming that they would have first class docker support, but the reality has been that while the first docker worked well as it was translated, now it wants to default to arm containers and it has become very difficult to use because it doesn't want to use Rosetta 2 containers.
The whole point of using docker is to use the same containers in production as you use in development, so having docker default to these random arm containers means that my containers aren't exactly production, because they are arm based and the servers are not.
I understand that docker is the developer of docker software, but I really wish I could just click a button and force intel based containers in docker as the default and have to opt-in to arm.
If anyone has an easy solution to this let me know. I don't want to spend hours and hours figuring out docker on my mac.
macOS apps have to be signed and notarised to run without a warning, which is a pretty big part of the defence picture for this software - the certificates can be revoked at any time to block the software if malicious behaviour is identified.
However, if I install Homebrew, then install python, then install a pip package, there's really no kind of scanning/notarization/checking happening at all. I wonder if this is something Apple has ever looked into - it seems like the exact scenario where you'd want to sandbox it away from the rest of the system.
> However, if I install Homebrew, then install python, then install a pip package, there's really no kind of scanning/notarization/checking happening at all.
There is: you are running that pip package in a chain of processes: Terminal (or iTerm, or whatever) - your shell - python - pip package. In this chain, Terminal has "Developer Tools" privilege, which allows you to run software, that does not meet the system's security policy.
You can disable this privilege in System settings, Privacy and Security panel.
EDIT: Having looked at this again, I'm not sure the Developer Tools privilege is even required to run essentially arbitrary software in Terminal. Homebrew still works without Developer Tools installed, as do interpreted runtimes. If I can even run an arbitrary shell script then the privilege is pointless, which makes the case for a dedicated sandbox for Terminal even more important.
This is my point, Terminal with Developer Tools privilege is essentially granting carte blanche privilege to every sub-process running there, which is not really necessary.
If I could pop a Terminal with a restrictive scope, allowing read and write access only within that directory tree it would be a much better situation than we have now. macOS' permission system isn't really that far away anyway - I can already disable Terminal's permissions to access anything else same as other apps.
Essentially a basic and very limited not-even-a-chroot would be a vast improvement and would still offer an adequate sandbox.
Yeah, but we need to analyze picture as a whole. And by default, chroot is banned while running as root arbitrary stuff downloaded from the internets is not.
At the same time, I don't truly understand why anyone would need to use it. If your preference is to totally work with macOS, then I'm sure this would be perfect for that. Otherwise, what's the advantage?
VMs have really come a long way. Every major OS today has a virtualization framework that makes running another OS extremely performant. Docker on macOS uses a virtual machine, but so what? Performance of individual containers, in my experience, isn't really a problem unless you're doing something with the GPU, and even then there are ways to deal with that. Even a fully-emulated VM using QEMU (without hypervisor or KVM) won't have any noticeable performance penalties in many cases.
IMO, there's a much greater advantage to sticking with Linux. Even if the host isn't Linux, developing and deploying with Linux guests provides a tremendous level of consistency and portability.
But maybe I'll be proven wrong by this project someday soon!
What my dream is that the User Mode Linux is made into a cross-platform userspace binary that translates syscalls transparently between itself and the host. So you might get "drivers" that talk to Windows, Linux, *BSDs, Darwin, it manages memory in an efficient (for the host) way, and enables you to run any kinds of wild experiments with, say, virtualized and passed-through serial devices, USB devices, networking, bind-mounting from the host and image mounts. And yes, containers. All of that without needing host root in most cases.
Of course the drawback would be that the host would see just a fat Linux process and its child processes, much like you can see qemu, but it could be an interesting thing nonetheless, if even for shits and giggles of it.
When macOS runs on Unix kernel and Linux systems are the best supported for containerisation and I assume are much more lightweight than macOS, I personally don't see any reason to run macOS in a container.
I think I get that part :) but realistically if anyone wants to run macOS software they are most likely macOS users already. Whereas containerisation is useful to aid with development and deployment. Have you come across anyone who runs production software on macOS? :)
One might develop a game that by some kind of a miracle releases for macOS too. So they way to run CI tests on macOS. Or they might target iOS. And use macOS build machines to produce builds. The world is not only about web, you know?
Can anyone speak to how the macOS runners on GitHub actions work? It would seem from this post that containers of any kind for macOS are a brand new thing..
It is the same for any OS. Virtual machine boots a separate instance of the whole OS. This is slow, this is often too much isolated (you can't easily/effectively share files between host and guest), you need to set artificial limits on VM disk/memory/cpu. On the other side, containers work in the context of host OS, what means less overhead and easier interaction with host.
the amount of engineering hours wasted making macos usable for backend dev work and then wasted again from inefficiency due to that failure is staggering.
linux is great. macos is great. windows is great too. for their intended purposes.
caveat: this is based on rund. Extract from the readme:
rund is an experimental containerd shim for running macOS containers on macOS.
rund doesn’t offer the usual level of container isolation that is achievable on other OSes due to limited macOS kernel API.
What rund provides:
Filesystem isolation via chroot(2)
Cleanup of container processes using process group
OCI Runtime Specification compatibility (to the extent it is possible on macOS)
Host-network mode only
bind mounts
I use MacOS and am very positive about it. I have lots of reasons to run Linux containers. What are some reasons I might want to run a MacOS container?
I would imagine it would enable you to run MacOS specific pipelines like building a project and have it work semi-portably across different machines/users.
Yes, I can see it could be useful for a company making MacOS software. But for a company whose developers use MacOS but whose product has nothing specifically to do with MacOS, my instinct is that if local environment reproducibility concerns get to that point then linux containers are the answer since they'll be more closely related to CI/prod envs. Or are you thinking maybe that's wrong for large companies with lots of developers using MacOS?
Unrelated to containers themselves: how do you make a patch when no version was released? I mean, people call this "semantic" versioning, but then spit in the face of those semantics...
Original author here. I wanted to clearly indicate early-prealpha-unstable-not-for-production-yet state of this software. Using "1.0.0" and even "1.0.0-alpha" would give false expectations about maturity of this project.
This was more of a stab at "semantic" part of the semantic versioning (which similar to <div> in semantic Web... found its own semantics that don't follow from its definitions). You are definitely not the only one using it like this.
I believe this is the problem with the format of semantic version which seem to assume that releases only happen to software ready to be... released :)
My preferred course of action in such situations is not specify a version at all.
0.1.0 might make more sense, but still not completely (backwards compatible with what?). Then again, it doesn't really matter. It's not like someone's going to accidentally install 0.0.1 because of semantic versioning not being 100% correct.
A normal version number MUST take the form X.Y.Z where X, Y, and Z are
non-negative integers, and MUST NOT contain leading zeroes. X is the
major version, Y is the minor version, and Z is the patch
version. Each element MUST increase numerically. For instance: 1.9.0
-> 1.10.0 -> 1.11.0.
So, no leading zeros, ta-da!
Oh, wait. The spec was written by some... big brain:
Major version zero (0.y.z) is for initial development. Anything MAY
change at any time. The public API SHOULD NOT be considered stable.
So... my reading of this "definition" is that there's really no need for three digits, if major is zero... Then why on earth would you have two digits? Also, if no pubic API at this point, then why have versions at all? I mean, you clearly shouldn't be specifying anything with zero major version as a dependency because it should be illegal to depend on a library w/o public API... Then, again, why have versions in this situation? And if the argument is that its for internal use, then why standardize it for external use?
CI/CD workflows most likely. And devshops that have standadised on docker containers for their stacks (mac-based devs in such places suffer a fair amount of papercuts today). Then I guess there are people that are very security minded that might want to run all userland executables in containers (although this project here is not for them I'd say).
It's sad to see so many negative comments for this. I get it's not an ideal place to start for macOS containers, but it's a start. Apple isn't doing it, so the community has to. Once you have a start, you can iterate on it. It might not be great now, but hopefully this makes it possible in a year or so. Who knows, maybe this is the kick Apple needs, and maybe they'll hire the devs of this project to fully work on this.
Sorry, not disabling SIP for something that I can already do without needing to nobble security policies (and have them reset/impossible due to MDM). If there was user/networking space in Darwin then maybe I'd be interested but...
I mean in terms of functionality, this doesn't give me anything extra to what using docker would do on macos (granted it's via a linux based VM). From an end user perspective there's no real difference, but I don't have to entirely disable SIP just to use it.
Something like namespaces or proper jails on darwin would be super cool, but not at the expense of other security measures and chroot-ish outcome imho. Maybe this works for some, but not me :)
> I mean in terms of functionality, this doesn't give me anything extra to what using docker would do on macos (granted it's via a linux based VM)
Ah, I understand your angel, in that your use case is to run namespaced processes that achieve some functional purpose irrespective of the underlying kernel/platform, which is totally fair.
> not at the expense of other security measures
Not for me either... that is, not in a host OS, maybe a dedicated VM; I consider this to be as it says on the tin, 0.0.1, a thing that would help bootstrap an ecosystem of containers, which would push towards Apple adding namespaces or jails (oh, hell yeah, JailKit!) to darwin.
It’s remarkable that Apple doesn’t have a first party solution to this yet. They used be, or aspire to be, at the forefront of OS research.“The most advanced Unix”.
UNIX was already winning the server room and workstation market before Apple, that is why they came up with A/UX in first place.
The Hollywood studios that now use Apple, would be using SGI previously.
On iDevices, UNIX APIs aren't even that relevant for app development, even basic stuff like networking has been superceeded by Objective-C specific APIs.
So no, I don't see anything UNIX related where Apple has helped to caught on.
Moving beyond UNIX, now that is a thing NeXT and Apple have done a lot.
“But first, let's clear a few things up: is this kind of logic new? No, it certainly is not. The most prominent system that works like this is Apple's launchd system: on MacOS the listening of the sockets is pulled out of all daemons and done by launchd. The services themselves hence can all start up in parallel and dependencies need not to be configured for them. And that is actually a really ingenious design, and the primary reason why MacOS manages to provide the fantastic boot-up times it provides. I can highly recommend this video where the launchd folks explain what they are doing. Unfortunately this idea never really took on outside of the Apple camp.”
Other than that, your answer has nothing to do with what I wrote.
I’m sure they did, it’s even stated in the first paragraph I quoted: “But first, let's clear a few things up: is this kind of logic new? No, it certainly is not.”
It’s just that influence is not all about being first.
I mean, I won't be allowed to install it side by side on my work laptop in a million years. GPU acceleration would be nice in a VM if macOS can pass it through, which I've no idea if it can.
It's not really intended to be run as a standalone distro (there's a fedora version though) and (afaiu) the point is to understand the bootloader process and stuff like GPU support from linux.
I know company policy moves glacially with these things (been there, got the faded t-shirt!) so yea, you're probably right there. Technically you could probably still use linux MDM instead of MacOS for mgmt, but getting that past IT is nigh on impossible imho also.
The point of Asahi is to provide linux drivers for Apple hardware. What would be the point? You can already run Linux in a VM on arm macOS today with good performances. You don't even need to disable SIP.
"macOS native containers"
Cool, this sounds interesting.
"Disable System Identity Protection."
Eesh.