
ARM Mac: Why I'm Worried About Virtualization - bmalehorn
https://bmalehorn.com/arm-mac/
======
adrianpike
I'm actually not worried, for a few reasons;

\- I already do cross-arch development day-in and day out between x86 and ARM,
and have only run into hard blockers on a library or tool a handful of times.
The solve was generally pretty straightforward to either use an ARM-compatible
alternative, or to cross-compile it myself.

\- We've done this many many times before and it's not that bad. I know I'm
not the only one old enough here to remember the days of having heterogeneous
fleets across PPC, SPARC, and x86. Or even more recent - different extensions
for x86 with different chipset manufacturers.

~~~
DaiPlusPlus
But back in those very heterogeneous days (don’t forget go throw in Alpha and
MIPS!) - computers running more exotic processors like SPARC were workstation-
class and their manufacturers were responsive to requests from the SE
community - so while I share your lack of concern about heterogeneity - I am
concerned that Apple won’t be making their ARM platform the best for developer
workloads (at least, non-iOS, non-macOS workloads) - remember that besides
freeing them from Intel’s slower release schedule, Apple’s other main
incentive for adopting ARM is to ensure their computers have a great
performance-per-watt ratio but with a low power budget. I know their latest
A-series chips are very, very competitive (sorry, I mean: mopping the floor)
with Intel’s current mainstream chips, Apple’s expertise is still with low-
power mobile devices - I’m not convinced Apple will be switching away from
Xeon Chips in the MacPro or i7 (i9?) chips in the high-end MBPs - but most SEs
I know using MBPs have their 13-inch models which are moving to ARM right
away.

In short: I feel Apple’s consumer-oriented direction is starting to be at-odds
with what they need to do in order to remain a compelling general development
platform.

Remember that macOS became a favourite for web-application development only
around 12-13 years ago (prior to that it was seen as an OS for creative-types)
- because they were selling nice hardware with an equally nice Unix-family OS
with a compelling desktop experience - take a look at typical Linux desktop
distros from around the same time: visual eyesores and incompatible with most
laptops thanks to OEM driver issues. Apple wasn’t specifically targeting
software developers at all - they were even showing ominous signs of
disinterest by discontinuing their X Windos server and going-back on their
promise of establishing Java as a pillar of the OS.

With the move to ARM on laptops I think Apple will just lock-down the
bootloader and won’t look back.

What’s funny now is Windows 10’s WSL and Windows Terminal, Docker support, etc
are suddenly making Microsoft look good as an OS for writing code for non-
Microsoft platforms. And at least with a Windows laptop - even ARM Windows
laptops - you can tinker with the bootloader and fire-up Slackware if you
really wanted to.

~~~
adrianpike
Yep, I think we're of one mind here. I'm not worried about the ARM
architecture shift, but the whole direction of the platform makes me expect I
won't be using a Macbook as my primary dev machine much longer.

Interestingly enough - for personal hacks (mostly cross-compiling Golang to
ARM, natch) I'm actually using WSL lately, and it's definitely good enough.
Not perfect, but nothing much is.

~~~
DaiPlusPlus
> Not perfect, but nothing much is.

Has anyone ported Plan9 to ARM yet?

~~~
nix23
Shure!!

[http://9front.org/releases/](http://9front.org/releases/)

 _.386.iso 386 pc

_.amd64.iso amd64 pc

 _.pi.img arm paspberry pi 1, 2 and 3

_.pi3.img arm64 raspberry pi 3 and 4

------
timsally
It is very likely that ARM-based Macs will lack a performant hypervisor upon
release. We will have to see how VMWare responds. I'd bet it will inspire new
products and innovation and the desktop space will move towards a less x86-x64
centric world. In the end it is a short term problem. Someone will respond and
provide a performant hypervisor that can run on an ARM host and virtualize
x86-x64 and ARM guests.

It's true it will cause some pain in the first year or two, but even as a
heavy VMWare Fusion user I am really looking forward to the benefits of a
vertically integrated laptop.

~~~
mister_hn
With just less than 10% of market share, do you really think it will change
the whole thing? Unless Microsoft pushes for ARM too, I don't see any changes
soon

~~~
cwhiz
Microsoft has been dabbling with ARM for a long time now.

It will all come down to whether this move gives Apple a significant
performance and/or battery life advantage. If Apple pulls it off it will force
Microsoft and other vendors to respond.

~~~
mister_hn
They announced there will be no Bootcamp, I hope there will be a possibility
to install other OSes as well on their new machines.

This is critical especially when they make their older devices "EOL" and there
are no more OS updates.

~~~
cwhiz
VMWare and Paralells have both said they will have support. I don’t know what
that means, yet.

I would not personally buy a Mac product for the next 2-3 years.

------
013a
I'd say that I'm excited for ARM. That doesn't mean the transition will be
seamless or easy.

I know that a big complaint about the move is "great, now I'm doing ARM
locally and deploying to x86". I think this is a legitimate concern, for now,
but I also strongly believe it is inevitable that, within the next decade,
deploying to x86 in the Cloud will be as "weird" as ARM would be today. The
benefits are way too numerous.

Well, more accurately, I think it'll be a "I'm on Fargate, oh wow, Fargate
runs on ARM, I had no idea" kind of thing. Ok, the article outlines why you
may need _some_ idea, but come on; we're talking about one line where I'm
downloading the x86 version of a dependency instead of an ARM version. That's
an easy fix.

I don't know what this means for open accessibility of hardware. Right now, I
could go buy and run locally the Intel Xeon chip powering my app in the cloud;
when things move to ARM, it absolutely will be "AWS Graviton" (not sold
outside AWS) or "Azure ARM Whatever" (not sold outside Azure). This sucks for
accessibility, but, actually, does it? ARM enables the cloud providers to do
this; they could never design their own x86 chips. As long as we're all
standardized on the same ISA, and the chips generally have the same
characteristics, I'm looking forward to a very bright future where vendors are
now also competing against one-another _in the silicon_. And I may not be able
to buy an AWS Graviton, but I'm sure (well, hopeful) that one day I'll be able
to build an ARM desktop that isn't a Raspberry Pi. AWS will have their chips,
Quallcomm has theirs, Apple has theirs, Microsoft and Google have some, and
they're all competing against one another.

Ok, maybe this is a pipe dream. But, I'm definitely in the short Intel camp,
at least on the long-term.

~~~
klelatti
This touches on an interesting question which I think underlying some of the
concerns here today: Who will build ARM chips comparable to an i7 say that I
can go out and buy and plug into my machine at home?

No-one does now and it's not obvious who would as we speak today. But if the
demand is there then even with lots of obstacles to overcome, of course, then
they can and will.

~~~
smspf
Not sure about the socket used (it might be soldered down), but aarch64-based
workstations are already available for the general public, e.g. [1].

[1] [https://www.anandtech.com/show/15737/arm-development-for-
the...](https://www.anandtech.com/show/15737/arm-development-for-the-office-
unboxing-an-ampere-emag-workstation)

------
bazizbaziz
This seems like a weird benchmark, reading from /dev/urandom and gzipping
random data does not seem like something most folks will want to do. It even
appears like /dev/urandom speeds differ greatly on various architectures [0]
and there are issues with /dev/random being fundamentally slow due to the
entropy pool [1] (but I guess this is why the author uses /dev/urandom).

It would be better to measure something more related to what docker users will
actually do, like container build time of a common container, and/or latency
of HTTP requests to native/emulated containers running on the some container.

One reason to feel positive about the virtualization issues is that Rosetta 2
provides x86->ARM translation for JITs which an ARM-based QEMU could perhaps
integrate into it's own binary translation [2].

[0] [https://ianix.com/pub/comparing-dev-random-speed-linux-
bsd.h...](https://ianix.com/pub/comparing-dev-random-speed-linux-bsd.html) [1]
[https://superuser.com/questions/359599/why-is-my-dev-
random-...](https://superuser.com/questions/359599/why-is-my-dev-random-so-
slow-when-using-dd) [2]
[https://developer.apple.com/videos/play/wwdc2020/10686/](https://developer.apple.com/videos/play/wwdc2020/10686/)

~~~
bmalehorn
Author here.

I'm glad somebody said something! Yes the gzip perf test is pretty silly, but
illustrates a significant difference. /dev/urandom throughput on this setup
was about 100 MB / s so it wasn't a bottleneck for this test - the bottlneck
was gzip.

Feel free to come up with a performance test yourself! I personally want to
know what an HTTP test would look like. You can run an ARM image by running:

    
    
        docker run -it arm64v8/ubuntu
    

Unfortunately, Rosetta 2 is not going to help here. Rosetta 2 translates x86
-> ARM, but only on Mac binaries. It does not translate Linux binaries, and
cannot reach inside a Docker image.

~~~
sirn
Was your emulation done with qemu user space emulator[1] (the syscall
translation layer) or qemu system emulator[2] (the VM)? If it was qemu-system
you might have better numbers with qemu-user-static, which does binary
translation similar to Rosetta 2 rather than a being a full system emulator
with all its overhead.

You can probably use qemu-user-static to translate x86-64-only binaries in a
Linux container on an ARM machine, too, but I have never tried.

[1]:
[https://www.qemu.org/docs/master/user/main.html](https://www.qemu.org/docs/master/user/main.html)

[2]:
[https://www.qemu.org/docs/master/system/index.html](https://www.qemu.org/docs/master/system/index.html)

~~~
bmalehorn
I ran this on a Linux laptop - it looks like it's running qemu-user-static:

    
    
        root        9934  103  0.0 125444  6664 pts/0    Rl+  12:25   0:12 /usr/bin/qemu-aarch64-static /usr/bin/gzip
    

So it might be that Docker already runs a native x86_64 Linux, then uses qemu-
static binary translation.

~~~
sirn
That's strange, in my experience it shouldn't have 6x slowdown. Probably might
be due to several factors, but here's your test, running on my system without
Docker:

Ryzen 3900X (host machine)

    
    
        $ dd if=/dev/urandom bs=4k count=10k | gzip >/dev/null
        10240+0 records in
        10240+0 records out
        41943040 bytes (42 MB, 40 MiB) copied, 1.02284 s, 41.0 MB/s
    

qemu-aarch64-static

    
    
        $ dd if=/dev/urandom bs=4k count=10k | proot -R /tmp/aarch64-alpine -q qemu-aarch64-static sh -c 'gzip >/dev/null'
        10240+0 records in
        10240+0 records out
        41943040 bytes (42 MB, 40 MiB) copied, 3.33964 s, 12.6 MB/s

------
grahamlee
It didn’t take months, the time I did it (running Docker on a Pinebook, which
was not a great experience). It took a couple of hours to flip some base
images away from Alpine, as Debian already has a load of ARM packages built.

~~~
thomaslord
This assumes that your Docker workload can run on an ARM system without lots
of hacking, and also that you trust the ARM-compiled version you're running
locally to function identically to the x86-compiled version running on your
server.

~~~
grahamlee
No, it doesn't. If I'd made the statement "all you need to do is…" then it
would have involved some assumptions. What I said was "I did this and all it
took was…" no assumptions, just experience.

------
smspf
So many wrong assumptions ...

1\. If emulating aarch64 (arm64) on x86_64 is 6x slower (on your system, btw,
it's not an universal constant), it doesn't mean emulating x86_64 on aarch64
will be 6x slower. It'd probably be worse, or at least that's my gut feeling.

2\. Generic container images like the Ubuntu mentioned usually have aarch64
(arm64) support, so running the x86_64 image makes no sense for the presented
use-case.

3\. You won't be able to use most software because they don't release ARM
binaries ... and the example uses `wget` && `tar xf`, with no binary signature
check. As someone who has been porting stuff from x86_64 to aarch64 for a
couple of years, I admit I've seen this pattern frequently. The most obvious
solution is to build from sources, which would have been better off on x86_64
too, instead of fetching a prebuilt (and unverified) binary from the internet.
Maybe there are some CPU flags the compiler could notice and apply
optimizations which are not included in the prebuilt binary.

I'm not an Apple fan and I'm certainly not a fan of cross-architecture
development either. I do agree with the general idea behind the article,
however I find it a bit hand wavy.

~~~
thayne
> Generic container images like the Ubuntu mentioned usually have aarch64
> (arm64) support, so running the x86_64 image makes no sense for the
> presented use-case.

I think the argument here is you can't build your own docker images that you
use in production and run them on your mac without emulation (unless your
production workload also runs on ARM).

~~~
smspf
That's a fair point. Emulation implies other limitations too - code compiled
on your machine might leverage only the CPU features emulated, which would
lead to sub-optimal binaries, not to mention much slower builds.

------
eberkund
Are there any excited embedded developers in the crowd? I have done a little
embedded work and cross compiling has always been a huge pain in the ass to
setup. I know some people have even gone as far as purchasing expensive niche
workstations with ARM CPUs specifically to avoid this problem. I feel like
having a mainstream ARM platform like the MBP will make compiling software for
ARM-based single board computers a breeze.

~~~
detaro
I'd much rather have a more powerful x86 workstation for the same money than
an ARM laptop. Never really had problems with cross-compile. And without
support for running Linux natively, it doesn't get me much for even for the
parts of testing that don't need the specific target (well, VMs maybe).

~~~
TheNorthman
To be clear, we don't know if the ARM MacBook will be able to run Linux
natively. We only know that Apple won't continue support for Boot Camp and
therefor Windows anymore. Linux was never supported.

~~~
sigjuice
There won't be native Linux.
[https://news.ycombinator.com/item?id=23640746](https://news.ycombinator.com/item?id=23640746)
(Craig Federighi confirms Apple Silicon Macs will not support booting other
OS)

EDIT: fixed link

~~~
TheNorthman
I don't understand your point. Native Linux isn't restricted to x86_64.

EDIT: Your new link doesn't tell a different story. From the comments:

> It is still possible to disable secure boot using csrutil. Apple has never
> officially supported booting Linux on a Mac.

>
> [https://twitter.com/never_released/status/127585087215369011...](https://twitter.com/never_released/status/1275850872153690114)

~~~
detaro
I think they didn't mean to link to that specific comment, but the overall
submission.

------
jeremyjh
My 2017 Macbook pro still has quite a lot of life in it, but it seems unlikely
I will replace it with another Mac in a few years. Before the Mac I had a
Lenovo X1 Carbon running Linux and it was great; and even then was a better
development environment in some ways (docker has better filesystem
performance, pacman is much better than homebrew). I do use some audio
processing applications and my kids play a few games that do not run at all on
Linux. I may try WSL again instead of going straight back to Linux, but its
hard to imagine Mac will be the best OS for me.

~~~
drewrv
I recently had to get a new laptop and went with windows for the first time in
a decade. Macs are still great to develop on for now, but looking at the
trajectory apple has taken, the growing pains ARM will likely bring, and also
the trajectory of Windows, it seemed like Windows would be the safer choice
over the next few years.

------
pwinnski
If the worst of every possible thing happens _and_ you avoid the most obvious
solutions _and_ one is very, very slow, then yes, you're right to worry.

Or, you could use already-extant Debian ARM releases and spend minutes rather
than months switching over.

------
edw
I'd like to advocate for remote development environments. Most of my day is
spent typing into a tmux session on a cloud-hosted box. (I picked up a Magic
Keyboard for my 11" iPad Pro, and thanks to Blink it's a great glass terminal.
It's not going to work if you're debugging let's say a React app, but I've
been very happy on it the last several days churning out Golang.)

Running stuff on your laptop makes it run slow, get hot, and burn battery.
I've considered getting a small x86 or ARM media appliance as a (physically
local) remote server for when I can't count on an Internet connection. A media
PC costs how much? The big holdup has been the tyranny of choice I'm
confronted with. (Suggestions are welcome!)

I think very few people would be surprised if the coming of ARM Macs will,
along with AWS's ARM moves (and Microsoft's), drive acceptance and adoption of
ARM-based server computing. The mechanism won't be anything formal, just the
vague pressure that comes from people wanting their programs and libraries to
compile locally.

~~~
chooseaname
I have a small server at home running proxmox. I have a couple containers
(lxd) running for personal dev projects. I agree that if you can do it like
this, it's nice. I can be pretty much anywhere and open up a terminal, vpn in,
and pick up where I left off thanks to tmux.

------
sjs382

        I would expect about a 5x slowdown running Docker images.
        
        Docker on a Mac utilizes a hypervisor. Hypervisors rely on running the same architecture on the host as the guest, and are about about 1x - 2x as slow as running natively.
        
        Since you're running ARM Mac, these hypervisors can only run ARM Linux. They can't run x86_64 Linux.
        
        What will happen instead? These tools will fall back on emulators.
    

Most of the software I run in Docker already supports ARM. I'd imagine that _a
lot_ of (most of?) us that use Docker do, too.

~~~
jayd16
It'll be annoying maintaining multiple docker images. Kind of defeats the
purpose.

------
binarynate
The loss of Boot Camp is huge. One of the reasons I develop on a Mac is so
that I can use a single machine for all development (including macOS, iOS, and
Windows development). Most of the time, developing for Windows on Parallels
works fine, but there are some cases where it's necessary to boot directly
into Windows to test or debug adequately. I hope Apple is able to reach an
agreement with Microsoft or at least continues shipping Intel-based Macs until
then.

~~~
beagle3
It will ship Intel based laptops for at least a couple more years (at the very
least, the models that will already be out at the switch), and will support
them for a lot more; so just buy the best intel based once following the
switch, and it will last you 4-5 more years.

But also: Getting a cloud windows station or an el-cheapo-$500-under-the-desk-
when-you-really-need-it Windows machine is probably worth it if you're doing
professional work. It would quickly cost much less than the time you lose when
rebooting to the the other OS, from my experience.

------
m000
Apple will singlehandedly make 2021 "the year of Linux on the desktop".

~~~
octorian
Every time Microsoft or Apple majorly screws something up, people say this. It
still hasn't happened yet.

However, I think Apple has been a far greater threat to Linux adoption than
Microsoft. Why? Because it gives techies the *nix environment they want, with
the software and hardware support no one will give them on Linux.

There is real value in proprietary commercial end-user application software.
Most companies who make such software couldn't care less about supporting
Linux. So if you want to use Linux, you have to use F/OSS alternatives and
continue to try convincing everyone that somehow they're better than the
commercial options... even when the rest of the world has agreed that they're
really not.

The whole incentive structure around F/OSS development really doesn't work for
software where the profit motive is in the product itself... Not some nebulous
"support contract" that you don't actually need. (Which is a far bigger issue
for end-user applications.)

~~~
ryandvm
> Because it gives techies the *nix environment they want, with the software
> and hardware support no one will give them on Linux.

The UNIX experience on the Mac is pretty shitty. Ancient versions of all the
tooling. Command-line utils have that weird BSD well-water flavor. No package
management. Funny Docker quirks.

The hardware used to be pretty nice, but honestly I'm still having trouble
forgiving them for getting rid of the physical ESC key and turning volume
control into a two-step routine on the TouchBar.

Honestly if I'm doing server-side development, I much prefer using my ThinkPad
(Ubuntu) over my MacBook. About the only thing I miss is the far superior
touchpad on the Macbook. That's it.

~~~
mbreese
>> Because it gives techies the _nix environment they want, with the software
and hardware support no one will give them on Linux.

> The UNIX experience on the Mac is pretty shitty. Ancient versions of all the
> tooling. Command-line utils have that weird BSD well-water flavor. No
> package management. Funny Docker quirks.

It doesn't have to be the best _nix environment. Hell, it doesn't even have to
be a good one. It just has to be "good-enough". For this, they still have an
advantage over Windows. And compared with Linux, they still have the advantage
that by-and-large, things "just work". I have never personally been able to
say that about a Linux desktop I've had. There is always one more thing to
tweak, one more knob to turn, etc...

I'm with you on the ESC key and touch bar though... which they thankfully
fixed the missing ESC key.

~~~
meddlepal
Exactly what advantage does macOS shitty Unix environment have over Windows 10
w/ WSL2?

~~~
saagarjha
It's not running in a virtual machine.

~~~
meddlepal
Hardly an advantage with all the other negatives.

------
lachlan-sneff
This is ridiculous. Your code will just have to support multiple
architectures, which is very easy with modern languages and tooling.

~~~
bobalob_wtf
I think you missed the point. What about all the dependencies of your code
that are only compiled for x86_64? The article isn't talking about native apps
on the laptop, it's about apps that run on a server but that you are
developing locally.

You can't run your x86 docker image on your ARM mac without emulation. You
can't run your x86 Windows VM without emulation etc.

Of course there are solutions like using a remote server or a VM in the cloud,
but if you're buying a decent machine then you would normally expect to be
able to run these things locally.

~~~
acdha
Do you have any examples of this? The last time I tried an AWS ARM server, it
was literally no modification other than changing the server type — Linux has
run on ARM for many years and Apple is far from the first company to use the
platform.

For example, back in 2017 Cloudflare was basically looking at this as a
question of which hardware ran most cost-effectively rather than having
engineering heroics first: [https://blog.cloudflare.com/arm-takes-
wing/](https://blog.cloudflare.com/arm-takes-wing/)

~~~
user5994461
I want to say Oracle database clients as an example.

My company definitely had problems to get database drivers to work generally
speaking and on both 32 bits and 64 bits. Have a look at postgres, oracle,
cassandra, redis, sybase to name a few, I am not sure which one was worse, it
wasn't me doing the work. But I've seen some of the C and C++ dependencies
that needed to compile with the errors that happened and that was horrendous.

~~~
jbverschoor
That’s a good moment to make your c/c++ code more robust, and cross-platform,
like the languages they are

~~~
user5994461
The problem wasn't in our code, it was in the database code that was either
from open source or from a vendor.

If you want a sample. Try to install the cassandra client library in python.
It will pull in and compile all sort of shit. That's supposed to be python and
easily cross platform.

~~~
jbverschoor
Yes, so time for them to clean up their mess ;)

~~~
etaioinshrdlu
Reading the comments here flow something like this: 1. complain that switching
to ARM is a mess 2. no it's not a mess, it's easy 3. no, see it's a mess 4.
fine, clean up the mess.

Yes, we should clean up the mess! It doesn't mean it's not a mess. And I think
the mess is actually still understated.

------
gtrubetskoy
I remember the days when having to switch between x86, ppc, sparc, etc was a
thing (not to mention the many flavors of UN*X) and we survived. In fact I
think it was more fun back before the x86/Linux server domination.
Architecture diversity is good.

------
rgovostes
I'm more convinced dropping dual boot and supporting virtualization is the
right move.

Only the host OS is going to have the right drivers for the trackpad, wi-fi,
GPU, power management, etc. etc. Through virtualization, the guest OS doesn't
have to worry about constantly evolving hardware models.

Virtualized OS performance is already very good, and USB passthrough has
existed for a while. Snapshots are a godsend.

What won't work are things like CUDA for eGPUs over Thunderbolt 3, and you'll
have to share disk and RAM with the host OS.

But for most use cases it's probably the right choice. (This doesn't address
the author's concern about moving away from x86.)

------
core-questions
> Why can't you update the Docker image to also support ARM? You theoretically
> could switch your backend to run ARM Linux. However, this would take months
> - renting out ARM instances, re-building all repositories, and a tense
> switch over.

I don't see why this would be so hard. If anything, I expect to see a massive
upswing in things like AWS Graviton2 uptake, and a lot of common Docker images
being built with ARM versions out of the box. It might be about a year or so,
but eventually we'll be able to just go ARM-native the whole way.

What Apple needs to do is make a first-class, WSL-tier implementation of
Docker for Mac for ARM.

~~~
user5994461
>>> expect ... a lot of common Docker images being built with ARM versions out
of the box.

This has no chance of happening. The common cloud CI systems do not support
ARM at all (travis, circle CI and co). There is only a minority of developers
with macbook and the rest is not going to spend $2000 to buy one just to build
some docker images.

~~~
rbanffy
They will. Trust me. And you can buy a cluster of RPis to build your images
for $2000

~~~
user5994461
It's ludicrous to assume that anybody has $2000 to spend on fantasy hardware.
It's month of disposable income outside of the SV bubble.

Better question might be, how many of the most common open source projects are
managed by volunteers in their spare time? These will not build for ARM,
unless there is a free tool doing it automatically for them. Currently GitHub
+ travis/circle can do for x64 on every push to master.

~~~
core-questions
You don't need a cluster of them, one $50 machine is enough for home use.
Travis et. al. will be buying those 96-core Marvell ARM machines and plowing
through builds.

~~~
rbanffy
Do they run their own metal? I thought they were on AWS.

~~~
duskwuff
Well, if they are, that makes things even easier -- Amazon has offered ARM
instances since 2018 [1].

[1]: [https://aws.amazon.com/blogs/aws/new-
ec2-instances-a1-powere...](https://aws.amazon.com/blogs/aws/new-
ec2-instances-a1-powered-by-arm-based-aws-graviton-processors/)

~~~
rbanffy
I think they have both. They offer POWER and LinuxONE as options and those
aren't available in your average cloud provider. They probably have a sweet
deal with IBM Cloud.

I'd love to have a POWER and an IBM LinuxONE in my shed, but that's not going
to happen.

Maybe the POWER, but the Z most likely not.

------
Uehreka
For my most recent project[1], I wanted to see if Amazon’s Graviton instances
would be a good choice for my docker deployments (I was deploying MongoDB, an
Express server, and several instances of the Janus WebRTC server). I was
developing in Pop OS on an x86_64 desktop (since we’re gonna have to start
specifying now) and found the toolchain around building ARM64 images to be
pretty simple once I got it set up.

I benchmarked some `t2a.nano`s against some `a1.medium`s and found that the
`nano`s were sufficient for my needs, so I went with them (they are cheaper
than `a1.medium`s, even if the `a1`s have a better price-to-performance
ratio).

I didn’t find it too difficult to rebuild any of these projects for cross-
architecture usage. Even Janus, which has a TON of C/C++ dependencies (some of
which have to be compiled from a particular version of the source) easily
built for ARM with no change in the Dockerfile.

So I kind of feel like OP is exaggerating the effort required to migrate
servers to ARM. Sure it might be a hassle when you have tons of microservices,
but you can move them incrementally, and most things recompile with no
changes. And regardless of what architecture your dev machine is, you’ll want
to be able to compile for and work with both architectures if you want to get
the most out of the infrastructure on offer in 2020.

[1] Shameless plug: [https://chrisuehlinger.com/blog/2020/06/16/unshattering-
the-...](https://chrisuehlinger.com/blog/2020/06/16/unshattering-the-audience-
building-theatre-on-the-web-in-2020/)

~~~
bmalehorn
Cool, thanks for sharing. It's these kind of experiences that I was hoping to
gather from making this post.

Did you notice at the end that you did NOT end up choosing ARM? You ended up
going with x86_64 because that's what made more sense for your backend. That's
part of my point - developers should choose their backend architecture based
on the performance and pricing of their backend, not their development laptop.
And if that decision is "we should keep using x86", then there will be a big
performance hit in development.

~~~
pjmlp
Back in the UNIX glory days, I was responsible for keeping a software stack
running across Windows NT (later 2000), Aix, HP-UX, Solaris, each with its own
CPU architecture.

This is just another CPU story, no big deal.

------
paride5745
As a Linux tech, I welcome this Apple move honestly.

Having a proper competitor for x86/x64 is a good thing.

The fact docker is slower on ARM (at the moment!) is mostly due to the lack of
interests for optimizations.

With Apple starting to produce MacARM machines, and maybe more ARM servers in
the wild, docker (and other platforms/frameworks) will start to get more
performant on ARM as well.

------
dlivingston
Do we know that Boot Camp isn’t supported by Big Sur? Or is it just that one
can’t run an x86 OS on an ARM architecture? In other words - will I still be
able to dual-boot into something like ARM-flavored Linux?

~~~
leecb

      one can’t run an x86 OS on an ARM architecture
    

This is the limitation. There is an ARM version of Windows, but the comments
from Microsoft don't sound terribly promising:

    
    
      “Microsoft only licenses Windows 10 on ARM to OEMs. We have nothing further to share at this time.” [1]
    

And Apple has more firmly stated that this won't be an option:

    
    
      “We’re not direct booting an alternate operating system,” says Craig Federighi, Apple’s senior vice president of software engineering. “Purely virtualization is the route. These hypervisors can be very efficient, so the need to direct boot shouldn’t really be the concern.” [1]
    

1: [https://www.theverge.com/2020/6/24/21302213/apple-silicon-
ma...](https://www.theverge.com/2020/6/24/21302213/apple-silicon-mac-arm-
windows-support-boot-
camp?utm_campaign=theverge&utm_content=chorus&utm_medium=social&utm_source=twitter)

~~~
gumby
And the iPhone didn’t permit native third part apps when launched. Not because
they weren’t ready to announce it yet but because at launch time they figured
web apps would be enough.

I have no idea what plans they might have but I would be surprised if you
couldn’t install some ARM Linux distros on their laptops sometime next year.

------
jjoonathan
If you stay back on x86 virtualization will be slow, but if you jump to ARM
this is great!

> However, this would take months

(infomercial arms)

I'm sure it won't make sense for everyone, but I'm just as sure it will make
sense for many.

------
outworlder
On the flip side, I guess ARM Macs will now allow the use of hypervisors for
Android simulators, instead of a full hardware virtualization.

... thus making Android development better on Macs?

~~~
lfy_google
Android Emulator developer here. In addition to what the other comments said
about android emulation with hypervisors existing already for x86, we're also
looking into the Hypervisor.framework API for Apple silicon.

It won't be a trivial task (hoping for pre-existing code to port over maybe?)
but we have the other pieces like using Hypervisor.framework for x86 already,
and being able to cross compile the other code for arm64, so that would be the
only major task left.

On the subj. of better GPU support, it depends on what it's actually like
using the drivers, but from previous experience with the GPUs and drivers
shipped with macOS, there shouldn't be any special kind of trouble at least.
We may have to use Metal if Apple also gets rid of opengl support on those new
machines, but there are also existing translators for gles and vk to metal.
The graphics hw itself, is actually the least of our worries due to how
consistent the hardware is likely to be---we'd have to deal with a much fewer
set of hw/driver quirks versus other host OS platforms.

~~~
my123
Hello,

OpenGL is deprecated but still supported on Apple Silicon, even for arm64
apps.

~~~
lfy_google
Nice, that's good news.

------
phamilton
> ec2 only offers 6 general-purpose ARM instance sizes

m6g, c6g, r6g each support 6 sizes for a total of 24

~~~
_msw_
Disclosure: I work at AWS building cloud infrastructure

C6g, M6g, and R6g (powered by AWS Graviton2) each support 8 sizes, along with
bare metal. A1 instances (powered by AWS Graviton) have 5 sizes, along with
bare metal.

That's a total of 33 distinct instance sizes.

~~~
bmalehorn
Thanks, I didn't know about c6g, m6g, r6g. I've updated the post to remove
this mention - I only counted a1 instances.

That still leaves storage optimized and GPU optimized instances missing. I'm
guessing storage should be easy enough to add, but what about GPU?

From my novice experience with GPUs, they need finicky drivers that must be
ported by the GPU manufacturers, so I figure it might take a while to get
competitive ARM GPU instances.

~~~
_msw_
Configurations of these instances with NVMe local storage are coming.

NVIDIA is supporting Arm for CUDA development, see
[https://nvidianews.nvidia.com/news/nvidia-brings-cuda-to-
arm...](https://nvidianews.nvidia.com/news/nvidia-brings-cuda-to-arm-enabling-
new-path-to-exascale-supercomputing) and
[https://blogs.nvidia.com/blog/2019/11/18/ngc-containers-
arm/](https://blogs.nvidia.com/blog/2019/11/18/ngc-containers-arm/)

------
ris
If you hadn't sold yourself out of the free market, you would be able to
_choose_ what architecture machine you bought.

~~~
Spivak
A free market wouldn't have saved you because the market has every incentive
to gravitate to a single architecture because vendors and customers want the
best software compatibility. The more popular an architecture (or really any
platform) gets the more software that's written for it until it starves out
competitors because they can't run the software their customers want.

------
Skunkleton
> Docker on a Mac utilizes a hypervisor. Hypervisors rely on running the same
> architecture on the host as the guest, and are about about 1x - 2x as slow
> as running natively.

That doesnt sound right to me. Perhaps on IO bound tasks, if you are using
emulated devices. On CPU bound tasks you should see near native performance.

------
harpratap
There's another unintended consequence of this virtualization - docker is
already has very high CPU usage on my macbook, anywhere from 50-100%. Because
of which it is always hot and toasty. This is caused already caused my screen
to start deteriorating
([https://www.ifixit.com/Answers/View/567125/Horizontal+line+o...](https://www.ifixit.com/Answers/View/567125/Horizontal+line+on+bottom+of+MacBook+Pro+2017+\(Due+to+overheating\)))
and the battery has degraded considerably too, even when I'm not coding on it
and docker is shut down. This means a significant hit to the longevity of such
devices as they not meant to be pushed so hard 40 hours a week. With ARM macs
I see it getting even worse.

------
pazimzadeh
On the bright side, it looks like low-latency streaming is good enough that as
long as you have internet connection, Boot Camp is not really necessary. This
works well for me: [https://shadow.tech/usen/](https://shadow.tech/usen/)

------
snapetom
A lot of developer-centric focus discussion on how Docker would work (hint: it
does), but VirtualBox is still pretty common in the sysadmin world and other
industry circles. Moreover, there seems to be no way it will ever work. It
will be interesting to see how that turns out.

~~~
bmalehorn
Author here. That's a major point of the article - "are we screwed?" I'm not
an expert on virtualization but I wanted to see some discussion on this topic,
because it feels like we might be screwed and nobody is talking about. Anyway
I was happy to see Docker worked, at least on a basic level.

~~~
snapetom
Cool. Thanks for writing it. It summarizes and collects a lot of issues we
were all grumbling about here and there. The main hurdles for Docker are
organization, not technical. However, the other issues you bring up are going
to be more technical (same as you, though, not a hypervisor expert and/or
we're going to be at the mercy of big vendors like Apple, Oracle, and
Microsoft. Those are much harder problems to overcome.

------
seanparsons
Any company with a load of binaries built without any effort towards
supporting cross platform builds that uses Docker is gonna have a bad day with
this. They buy a bunch of new MacBooks and then find they can't use them until
they spend a few weeks porting everything.

~~~
neilalexander
I suppose that depends what they’re written in. Some languages (e.g. Go) far
simpler than others.

------
lsllc
My guess is that Apple will end up copying Microsoft and providing a WSL style
Linux kernel "shim" into Darwin (pretty easy as it's already UNIX) and use
Rosetta2 to translate any x86_64 containers to aarch64). No need for a
hypervisor.

------
justaguy88
I wonder if Apple will still allow a dual boot with a native (arm64 in this
case) Linux

~~~
gardaani
There's no official support for ARM-based Macs:
[https://news.ycombinator.com/item?id=23640746](https://news.ycombinator.com/item?id=23640746)

------
bitwize
Once again -- Apple will do what the entire industry without Apple couldn't
do. In this case, force a migration to ARM-based servers, so that prod is
running on the same architecture as the developer's machine.

Apple is finally killing x86.

~~~
tjoff
Hooray, we have successfully fixed the mistake with an open platform and will
now be relegated to incompatible hardware without any competition.

At last, the future will surely be bright!

~~~
duskwuff
How was x86 any more "open" of a platform? If anything, x86 is a far more
"closed" platform, as there are only two remaining manufacturers of x86 parts,
and there is no licensing process to join them. Meanwhile, there are hundreds
of ARM licensees, and the process for becoming a licensee is all documented
online [1].

[1]: [https://www.arm.com/why-arm/how-licensing-
works](https://www.arm.com/why-arm/how-licensing-works)

~~~
tjoff
How? It is absolutely inconceivable how much more open it is.

Also, the CPU is but a minor part of the puzzle. But still that is still twice
as many as apple (good luck exchanging that apple-arm with any other brand).

Please let me know how open you think the next apple ARM platform is when you
try to boot any OS not written by apple.

Please compare that with a computer built from AMD/Intel with a motherboard
out of dozens of manufacturers etc. Any ATX power supply etc. Pretty much any
PCI-E graphics card etc.

~~~
duskwuff
You are confusing the x86 _CPU architecture_ (which is closed) with the PC
platform (which is relatively open).

> Also, the CPU is but a minor part of the puzzle. But still that is still
> twice as many as apple (good luck exchanging that apple-arm with any other
> brand).

Even on x86, interchangeable CPUs are the exception, not the rule. Intel and
AMD CPUs haven't even used the same socket since the 1990s, and even within
those manufacturers, socket incompatibilities are common.

Software interchangeability is more of an operating systems issue than an
architectural one. With appropriate software shims, though, there is no reason
to suspect that (for example) Linux ARM software could be run on an Apple ARM
CPU. In fact, it's quite likely that tools like the Android emulator will do
exactly that.

> Please compare that with a computer built from AMD/Intel with a motherboard
> out of dozens of manufacturers etc. Any ATX power supply etc. Pretty much
> any PCI-E graphics card etc.

Server-class ARM hardware generally does use similar parts as x86 servers,
including power supplies and PCIe peripherals.

~~~
tjoff
I am not. But also, how many x86 processors are sold outside of the PC
platform? They have a symbiotic relationship. The foundation of everything we
have today.

> Software interchangeability is more of an operating systems issue than an
> architectural one.

Not if the architecture is designed around keeping others out. But just not
telling anyone how to do it is enough in 99% of cases. Some hacker might post
a buggy proof-of-concept for an obsolete device that no one will run.

> Server-class ARM hardware generally does use similar parts as x86 servers,
> including power supplies and PCIe peripherals.

Wanna place a bet on what apple is going to do?

------
fluxem
But linux can be run on arm natively. Moreover, most packages are also
compiled for arm. So apt-get install will work just the same. I'm sure they
will be able to target Apple's specific arm chips when they come out.

~~~
donarb
Apple released a list of open source projects that they have ported to ARM,
they intend to upload patches to each of these projects:

    
    
      - Bgfx
      - Blender
      - Boost
      - Skia
      - Zlib-Ng
      - Chromium
      - cmake
      - Electron
      - FFmpeg
      - Halide
      - Swift Shader
      - Homebrew
      - MacPorts
      - Mono
      - nginx
      - map
      - Node
      - OpenCV
      - OpenEXR
      - OpenJDK
      - SSE2Neon
      - Pixar USD
      - Qt
      - Python 3
      - Redis
      - Cineform CFHD
      - NumPy
      - Go
      - V8

~~~
tonyedgecombe
It’s interesting that Electron is on that list.

~~~
asadlionpk
Crucial piece of tech for many products like VSCode, Slack, Discord, etc.

~~~
pjmlp
Given how much Microsoft's React Native team bashes Electron with their
performance bar charts (300x more bloat than RN), I look forward that, as soon
as it is mature across Linux, macOS and Windows, they replace Electron with RN
on VSCode.

------
lowbloodsugar
If I were to describe my job, or programming in general, it might be "problems
that have no obvious solution". This is a sad article that just seems to be
the opposite in spirit to "Hacker" ethos.

------
jmull
> Should I get an ARM Mac? ...if you use virtualization often, I wouldn't
> recommend it.

I use virtualization continuously, but not for anything that needs to be as
fast as possible.

I won’t hesitate to get an ARM Mac once I can run x64 Windows VMs on it.
(Presumably VMWare Fusion or Parallels, and for once I won’t feel ripped off
by the upgrade pricing.)

Docker on Mac doesn’t work that well today, so I don’t have any workflows that
depend on it.

~~~
jki275
There is no reason to expect that virtualbox, parallels or Vmware will emulate
x86-64 to run a guest OS. None of them does any emulation today, they are
virtualization platforms.

------
leoh
This is silly. Most stacks will have counterparts on both architectures. Just
run CI with the same configurations as prod. Problems due to differences will
be rare for most stacks -- modern languages are well defined and run against
extensive spec tests on all major platforms -- and will smooth out over time.

------
rcarmo
All my Docker stuff is multi-arch these days. Here’s one of my sample
pipelines, written precisely to show how easy it is:
[https://github.com/rcarmo/azure-pipelines-multiarch-
docker](https://github.com/rcarmo/azure-pipelines-multiarch-docker)

------
quux
I'm not a heavy docker user but why can't I develop docker containers on Arm
(as native containers, no emulation) and deploy to x86_64? Or vice versa? I
understand that some packages are binary only and wouldn't be necessarily
available for Arm, especially initially, but the majority should be.

~~~
Teknoman117
It's not that you can't, you just have to be mindful. Because docker
containers contain compiled applications, you have to be aware of what CPU
architecture they're compiled for. ARM can't natively run x86 binaries, x86
can't natively run ARM binaries.

If you want to develop containers for x86 systems on an ARM system, you'll
have to cross compile your containers, which I'm not sure if docker actually
supports outside of emulation.

If you are only a consumer of containers, most of the popular ones have been
compiled for multiple architectures.

------
marricks
It will be extremely interesting to see what the dev's say with the test
boxes. Curious if there's any magical fixes like switching over to ARM Linux,
since the software of an image would likely be compiled for x86 I really doubt
it...

Perhaps this will spur some people over to running ARM servers in the cloud...

~~~
lukevp
Has anyone received an invite into the program yet? I applied on day one, but
don’t have a Mac Store app currently published, so I’m not sure if I will get
accepted.

~~~
Vomzor
I don’t have a published Mac app either and I got accepted.

------
gigatexal
It’s still early days. bhyve which is likely what they’re using or whatever
the hypervisor is will just run arm docker images — unless you have a hard x86
dependency many of our http micro services should run just fine on arm images
at least my workflow will be just fine.

~~~
gigatexal
Furthermore this is speculation. We don’t have ARM Macs yet to test. This is
like all the nerds in the forums speculating on hardware leaks how the next
gen GPUs will perform: wait for hardware and reviews. Making any sort of claim
as to what to buy or not at this point seems disingenuous.

------
AkihiroSuda
> Why can't you update the Docker image to also support ARM? You theoretically
> could switch your backend to run ARM Linux. However, this would take months

No need to take months. `docker buildx` can build multi-arch images without
using real ARM instances.

~~~
Matthias247
Does that help if the software you build inside the container doesn't build on
ARM? Imagine a 3 digit count of legacy C libraries which do so far not compile
on ARM for a variety of reasons. You would need to spend a significant amount
of time to make them compile and run.

------
ed25519FUUU
I use docker every day and I guess I’m just not worried about this. The
container pushes come from a CI host, so I’m not worried about compatibility.

And it’s 20% slower? Well, most of the build time is slow for all sorts of
reasons. I honestly don’t think I’ll even notice.

------
Aqueous
this is why intel should be scared - very scared - that the PC world seems to
be converting to ARM en masse. PCs might be a relatively small fraction of
intel’s total sales, but it’s the second order effects they should be worried
about. if it becomes less convenient for developers on ARM machines to develop
and deploy software to x86 cloud architecture, they will begin to demand that
the cloud architecture be shifted to ARM as well.

------
user5994461
Didn't think of that. If running Docker is 10 times slower on ARM and virtual
box doesn't support the architecture at all, this might indeed end developers
using Mac.

~~~
armagon
I've been developing on a Mac for years, and I've never needed to use Docker
or virtualisation to do it. (I've been doing game development, and now web
front end development. I'm sure the major game engines and browsers will be
ported to run on ARM chips (although it may take a while)).

Honest question: What sort of development regularly requires using docker or
virtualized OSes?

~~~
neurostimulant
Almost every server applications (databases, webservers, webapps) are
available as docker images. It's very easy to deploy those apps using docker,
which is why it's taking over sysadmin world by storm. Previously, handling a
big web application deployment is a complex task that requires a dedicated
team. Now, you can just package your app as a docker image, and other people
that know docker will know how to deploy your app without having to know its
internal dependency graph first.

~~~
anthk
If you use Docker as a production tool instead of prototyping several shit
will happen soon.

~~~
neurostimulant
Certainly not for production, but when you can't run x86-64 images, just
simple prototyping or building images that requires x86-64 will require using
external servers instead of using locally installed docker, which will be a
huge barrier for casual prototyping.

------
jbverschoor
We have qemu on arm. It’s fast enough to software render half-life. Servers
will follow desktop, so in a few years a lot of things are arm

------
cbsmith
Hey guys... we have done emulation before, and it's not nearly so bad.

Also, there ARM images for Docker. You don't HAVE to run x86-64 binaries.

------
saxonww
Develop on the platform you want to deploy to.

~~~
yjftsjthsd-h
I am not running Darwin in prod.

EDIT: I suppose I should clarify; I don't totally disagree. I personally run
Ubuntu on my laptop and servers. But plenty of people are quite happy
developing on Darwin and deploying to some sort of GNU/Linux.

------
zekrioca
I guess someone will port Docker to JVM, and use the JVM optimized to whatever
ARM processor there will be.

~~~
neilalexander
Docker is written in Go which already has first-class cross-compilation
support, so getting the Docker tools running on ARM is practically a non-
issue.

~~~
zekrioca
Yeah, I thought about the conceptual idea, similar to what MS did:
[https://threatvector.cylance.com/en_us/home/teardown-
windows...](https://threatvector.cylance.com/en_us/home/teardown-
windows-10-on-arm-x86-emulation.html)

------
nojito
Apple has their own virtualization offering to share and they are being quite
coy about

------
__warlord__
Has Apple mentioned something about Thunderbolt 3 or USB 4 on this new Macs?

------
lowbloodsugar
I had an ARM based desktop in 1988. Be thrilled to have one again.

------
huslage
Hypervisors are by no means 1-2x slower. Testing I/O is not testing the
performance of a hypervisor. It's testing the I/O stack.

------
monadic2
Tl;dr they want to run x86 linux for their own reasons rather than arm.

------
julienfr112
Mac book was never really a dev platform. Maybe for front or nodejs, or
definitly for native apple apps, but seriously, brew and so are so subpar.

~~~
chrisseaton
I develop low-level code like compilers just fine on a MacBook.

~~~
fortran77
What's "low level" about a compiler?

~~~
chrisseaton
> Mac book was never really a dev platform. Maybe for front or nodejs, or
> definitly for native apple apps, but seriously, brew and so are so subpar.

I'm not sure what you're asking? It's lower level than the examples which were
given.

There's nothing stopping you using a MacBook for almost any development task.
It's not just for front-end tasks. You can do work that runs directly on the
architecture.

------
old-gregg
When I ask Mac-loving developers, why they chose to run MacOS when developing
non-MacOS software, they used to give me good reasons.

I think their reasoning is no longer valid. The hardware has gotten worse
(keyboard, touchbar), the OS has gotten more hostile, meanwhile the state of
Linux on Laptop has gotten a lot better. So... I used to understand them, but
I no longer do.

[https://i.kym-
cdn.com/photos/images/newsfeed/001/016/674/802...](https://i.kym-
cdn.com/photos/images/newsfeed/001/016/674/802.jpg)

