Hacker News new | past | comments | ask | show | jobs | submit login
Run FreeBSD 13.1 for ARM64 in QEMU on Apple Silicon Mac with HVF Acceleration (gist.github.com)
135 points by codetrotter on Aug 3, 2022 | hide | past | favorite | 50 comments



You can also use UTM which is a GUI for qemu with HVF turned on already. Pretty useful, I've run everything from Windows 11 to Snow Leopard in it.

https://mac.getutm.app


Is HVF Apple-specific? Is it related to ARMv8.3/8.4 Nested Virtualization? Does Graviton 3 or some other commercial offering have this support too?


Yes. HVF is the QEMU driver for Apple/macOS Hypervisor.Framework


So basically KVM for Mac then? So like on Linux you have QEMU+KVM, on Mac you instead have QEMU+HVF?


Pretty much!


When I saw qemu/utm on m1 mac, I was wondering how hard it would be to run the same utm arm images on windows with qemu, surprisingly, not bad. Hardest part was the missing documentation to get everything working, as UTM is pretty automated on osx.


What does this offer that BSD-mutant MacOS native on Apple Silicon lacks?


>What does this offer that BSD-mutant MacOS native on Apple Silicon lacks?

The ability to test applications in a native FreeBSD environment? This is just running another OS within macOS same as one might run Windows or Linux images. It's useful for development all on one system, certain kinds of isolated deployments and so on. Also I use it for doing another layer of runs of certain things before pushing out to deployment, you can virtualize a router/firewall like OPNsense or VyOS for example, load your live config, and experiment before sending it out to metal. This is just someone playing around with yet another way to do it. I use VMware myself right now on an x86 Mac but there are lots of choices and it's nice to see more stuff heading over to AS as well.

From your comment though it's not clear to me if you're confusing this with running an alternative OS on the metal, as the Asahi Linux folks are working on? If that's what you were thinking, then the answer would be that longer term having Linux and FreeBSD on AS hardware could help extend its life (Apple won't support them forever anymore then they do any Macs), Mac Minis at least might be useful little appliance systems (eyeballing numbers so far an M1 Mini could in principle compare favorably as a firewall say to a lot of x86 systems in the same power/heat/physical size), and of course some people simply would prefer to stick to another OS they're at home in but also like the Apple hardware. Nothing wrong with that.


For all the criticism MacOS gets, it can't be denied that it actually is a certified Unix.



But what is the importance of this - compared, say, to a popular Linux distribution? (MacOS may be more popular on the desktop, but Linux seems to be way more important in the grand scheme of things.)


Being – or rather not being – Unix seems to be important enough to the GNU project that it's in their name:

GNU's Not Unix


The ability to simulate and setup infrastructures. For example, here's one[1] in VirtualBox that can't run on M1.

[1] https://blog.uidrafter.com/scripted-virtualbox-vm-installati...


FreeBSD is a good server OS in various ways macOS isn’t. macOS’s networking is good at being a cell phone more than a server.


Could you provide some details on this?


In terms of the market share, at least, according to Wikipedia [1],

Less than 1% is confirmed to be UNIX or Unix-like and non-Linux. The top operating systems in order are: 0.3% BSD (97.8% of which is FreeBSD),[251] <0.1% Darwin,[252] <0.1% HP-UX,[253] <0.1% Solaris,[254] and <0.1% Minix.

[1] https://en.wikipedia.org/wiki/Usage_share_of_operating_syste...


Thanks for quoting wikipedia on market share, but I was hoping for technical details regarding networking on macOS, preferably based on firsthand knowledge.


One example I experienced recently: got my 10G fiber and was torrenting was few things. I was pulling down about 25MB/sec and my mac mini started acting really weird. ARD locked up and the connection dropped and various other things. Did the same thing on Linux, transmission on both but in docker on Linux, and it didn’t flinch. Everything else in the system worked fine, and I have quite a lot on that Linux system, including a plex server.


Interesting - what do you think the root cause was?


I think the networking stack in macOS is just not nearly as good as on Linux. I think it's fine for a small number of maxed out sockets, but the test that broke it was on another level. Actually, I think it was more than 25MB/sec, it was more like 55MB/sec, total, from 100's of peers.

Honestly, until I got the 10G fiber, I never really tested macOS like this. I have no local tests which could generate that sort of throughput.


Doesn't have syncookies or other DoS protection.


Can you do something equivalent using pf?


Darwin lacks a great userland. It's really just the kernel.

And macOS itself I don't use anymore, it's become too locked-down and too minimalistic for me. I want full control over my computer. So I'm on FreeBSD now for my main driver.

However running it on an ARM Mac would be an option for me if it were well supported.



Yes I know macOS is POSIX certified.

But the entire OS is certainly not FOSS and it limits me very much in what I can do with my computer, unless I switch off its security completely. It's an all or nothing model.

POSIX compliance is not something I actually care about. Having full control of my computer is.


Certain features in FreeBSD are not available in Darwin, such as ZFS and jails. Also, if you’re doing software testing, it’s nice to be able to spin up a VM of the OS that you’re testing your software on.


>such as ZFS

FWIW, ZFS has actually been available on macOS for a long time in various iterations (MacZFS, Z-410/ZEVO, and now OpenZFS [0]). I ran it for around 11 years on a number of systems with almost all of my data on it (home folder, applications, /opt for MacPorts etc) and it was a tank. Fun project and well worth checking out. I'll always regret the various factors that meant Apple didn't adopt it as their native FS.

----

0: https://openzfsonosx.org/


I wonder, comparing with APFS do you miss it?


I miss not being able to move disks from Apple to Linux and freebsd. That is, that's what I miss using APFS. No other aspect of APFS appeals to me over ZFS, except that Apple prefers it for time machine. That's why I use it.

If Apple supported ZFS as a first class in-kernel fs type, I'd be delighted.

I have already moved large zfs systems between FreeBSD and debian repeatedly. Snapshots are just ace! (by move I mean zfs export, move hard disks or re install OS and zfs import. zfs snap; zfs send | zfs recv is a different kind of "move" which I also do a lot)


I haven't given it up, I'm stubborn :). What I do now though at the desktop is have my user folder and data that I previously had on an internal or DAS ZFS fs instead live on an APFS formatted iSCSI LUN which in turn is a zvol on a NAS. So it's all still ZFS underneath, I still get all the data protection and ease of backups and so on, the performance is decent [0], but macOS thinks it is running perfectly native, and I don't need to worry at all about upgrades since the ZFS layer underneath is transparent. Well, at least not so long as I can still use iSCSI under macOS of course, right now that is still fine but if Apple kills the ability to run initiators down the road with no replacement option I'll be in trouble once again.

I do miss some aspects anyway in terms of the ease of spinning up/cloning independent FS for projects, VMs etc and having them right there instantly, with their own snapshot policies and rollback chains. Obviously can still do that on a NAS and expose it but it's not at all as frictionless a process. I also miss not being able to do restores as easily, but Apple has steadily killed a lot of the convenience Macs had there and really wants you on the rails of their scheme, so it's better to just try to move as much custom stuff as possible outside the root fs entirely then overlay mount or point to different paths instead. I do still use it on my older MBP but I don't know how that'll survive transition to the AS era. Previously you could through some effort boot off it even, but my assumption is that will be utterly dead going forward so at most it'll be partitioning the internal drive.

From my admittedly biased perspective APFS kind of sucks, it still sees data corruption, it's much harder to repair than HFS if something breaks since Apple doesn't bother to document or stabilize stuff anymore. I still think moving away from normalization was stupid. Overall though it is what it is, and it's Apple's native choice and it mostly doesn't get in the way. I know there's some clever stuff to do with CoreStorage as well and some people have played with using that along with ZFS but I have not.

So yes, guess that turned into a very long way to say that I would indeed miss it if it ever comes to that.

----

0: No issues saturating a 10G fiber link or 8g FC, and I've seen demoes hitting over 11GB/s (88 Gbps) on a Mac Pro 2019 with a higher end FC that I can't afford.


I'd really like to see jails/BSD containers on macOS.

Running docker containers in a Linux VM works OK, but it makes me sad because I don't actually need or want another kernel beside Xnu. For that matter, I don't even need or want docker itself, just a container feature.

Though speaking of Linux, I actually wouldn't mind having BSD's Linux compatibility layer on macOS either.


> I'd really like to see jails/BSD containers on macOS.

That's not how jails work.


They’re just asking for an equivalent feature.


Indeed. Also note post I was responding to.

Note that macOS' Xnu kernel supports BSD system calls (e.g. chroot(2)), and macOS includes a BSD userland which I think is largely derived from FreeBSD.

In principle it should be possible (though non-trivial) to add support for the relevant BSD system calls such as jail(2) and jail_attach(2), as well as the appropriate sysctls to enable BSD jails to actually run natively on macOS.

(For comparison, Linux implements containers primarily via the unshare(2) and setns(2) system calls, as well as /proc and cgroups. It's more of an a la carte approach since there are a number of independent namespaces supported by unshare(2). And bind mounts, overlayfs and the older unionfs, and the venerable chroot(2). Etc.)

To me it seems that supporting the BSD system calls and infrastructure would be a reasonable evolution of macOS' BSD subsystem.

Moreover, as you note, macOS could add an Apple-specific native container feature or framework. Consider that macOS already has a lower-level Hypervisor framework as well as a higher-level Virtualization framework which provided an easy to use API for creating VMs. They could implement their own Container framework or perhaps adapt or extend the existing Virtualization framework.


This could be "an exciting new hardware platform for FreeBSD people" more than "an exciting new OS for Apple hardware owners."


The actual FreeBSD and *BSD parts of macOS are ancient.


The stuff with licenses Apple likes is relatively up to date; it's definitely not bleeding edge, but no worse than you'd get from Debian or RHEL. The criticism macOS gets for shipping ancient software is because one of the licenses they don't like is the GPL3. So there's a GPL2 version of bash from 2007, rsync is from 2006, etc. The end user can, of course, install newer versions of these separately.


The FreeBSD and *BSD parts of XNU are ancient, which is what I'm referring to, as is some of the userland.


It can't be that old considering they even have pf.


I'm looking at it from the perspective of XNU. Most of FreeBSD's interesting and modern features are not in its userland, but in its kernel and/or exposed to the userland from there. The BSD parts of XNU are literally decades old, and were lifted before jails were a thing.


jail(2) and jail_attach(2) pls.

Also looks like macOS still supports union mounts and chroot.


fewer strings attached?


Funnily enough, this tutorial uses the linux-style dd numeric syntax (lowercase M) which fails on macOS (I think it inherited dd from freebsd a looong time ago) and needs bs=1M (not bs=1m).


All of the commands given in the guide are provided exactly as they should be run, and you can copy paste all of them directly into your terminal on macOS without any changes.

That includes all of the dd commands. These run as they should on macOS with the syntax given in the guide.

Assuming you are using the same version of macOS as me; macOS Monterey Version 12.5


That's true for most versions but I just tested on Monterey and bs=nM now works. Since this is a M1 specific we just need to check if Big Sur behaves the same.


This is super cool, but I see folks having problems with networking. Maybe I'll try in a month.


Most of the comments are from past versions of the document. I debated removing them but the issue with ping acting weird remains so I decided to leave the comments for now.

But networking in general appears to work, aside from the ping strangeness.

For example, using pkg to install packages works fine. Likewise, portsnap fetch extract, and when you build packages from ports the system is able to fetch over the network without apparent issues.

Edit: I cleaned up the comments on the gist a little, leaving the ones that are still relevant (the ones about ping), while removing some comments that were talking about problems that are out of date.


Cool. I'll give it a try then.

I didn't even know you could remove comments on a gist. TIL.


It looks like the vmnet patches made it into QEMU - maybe try setting the network device to use that:

-netdev vmnet-macos,id=net,mode=bridged -device virtio-net-pci,netdev=net

Note: You'll need to run QEMU as root to use vmnet, non-root usage requires an entitlement.


QEMU needed to be patched for network to work on M1 Mac hosts back then. I don’t know if the patches landed in newer versions of QEMU.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: