One very cool thing that you can do with it is to use binfmt_misc to tell the kernel to use `qemu-arm` to run ARM binaries, then you can chroot in to an ARM device's filesystem from your x86 workstation, and all of the ARM binaries just work.
I was just dealing with this today. QEMU was too slow on my MacBook Air though.
Do you have a link to a comprehensive guide on doing this by chance? I was thinking tomorrow I’d just launch an arm instance in AWS and figure it out but I have a dual Xeon workstation at work (windows) that I might try as well.
(Some additional steps needed if you want to use regular chroot instead of nspawn.)
Sometimes qemu shows an error saying some operation isn't supported, but this hasn't broken anything yet for me, even after I did a whole Raspbian Stretch -> Buster upgrade this way.
BTW, with Debian buster and later, you won't have to copy the qemu-arm-static binary around, since the Linux kernel will now use the file from outside the chroot/container.
On a distro like debian you can even use it to build-and-run userspace binaries for an unrelated architecture (some chroot magic was required last time I checked).
You (1) use binfmt_misc to tell the kernel to use `qemu-ARCHITECTURE` to run binaries for that architecture, then (2) make sure you also have all of the libraries that the binary is linked against, then that binary executable should just run seamlessly.
Now, if your ARM binary was compiled to look for libc at /lib/libc.so, but /lib/libc.so is the host's x86 libc, then that obviously won't work; and the easiest way to get the libraries all sorted out is to use a chroot with OS install of the target architecture. If you do go the chroot route, you need to make sure that `qemu-ARCHITECTURE` is statically linked, because it won't have access to the x86 libraries it needs to run after the chroot(2) call happens (which is why most normally-dynamically-linked distros have a "qemu-user-static" package in addition to their normal "qemu-user" package).
But with a multilib scheme like Debian's, where all libraries get installed to /lib/ARCHITECTURE-TRIPLET/ instead of /lib/, then it should be possible to install all of the appropriate target libraries on the host system without a chroot! You "should" just need to configure APT to let you install packages built for that architecture. (I haven't actually tried this; I'm not a Debian user, but I am envious of their multilib).
I've used this to run some 32-bit Linux binaries under Windows Subsystem for Linux (WSL), which only natively supports 64-bit binaries. (Recompilation for 64-bit was not an option.) It wasn't ideal but it did work smoothly for the most part. I just used `dpkg --add-architecture i386 && apt update && apt install libc6:i386` rather than creating a separate chroot. I did have to edit the binfmt registration to remove the 'OC' flags set up by the qemu-user-binfmt package, since these aren't supported by WSL, and manually enable the i386 binfmt which is blocked by default on amd64 platforms. There is also a persistent SIGSEGV in one particular binary which may not be related specifically to running under QEMU.
