
Bottlerocket, an open source Linux distribution built to run containers - sshroot
https://aws.amazon.com/blogs/opensource/announcing-the-general-availability-of-bottlerocket-an-open-source-linux-distribution-purpose-built-to-run-containers/
======
dpryden
I'm confused about how the documentation recommends using a Kubernetes
operator to manage OS updates. That seems weird and backwards to me. I would
rather see an immutable OS AMI in an auto-scaled group, and just replace the
node instance whenever there is an update.

I can see a place for managing OS updates on an instance, but that seems more
like "pets" than "cattle"... and I've always treated Kubernetes nodes like
cattle, not pets. Isn't that the most common approach anyway?

~~~
freedomben
This is how OpenShift 4 does things. I too thought it was strange at first but
now with some experience it's quite pleasant.

Can be a beast to debug though if you haven't done it before.

~~~
dan_quixote
I can assure you that OpenShift doesn't take this path because it is "better".
It does so because bare-metal is a significant part of their market and there
isn't a better option to automate the process currently.

I once worked on a competing product (before the OS update operator was
available) and the update-in-place model was always a disaster. Various
problems like dns, service discovery, timeouts, breaking changes to dependency
pkgs, etc make for a problematic process. Combine that with the frantic pace
of k8s develeopment, short node compatibility window (2-3 minor k8s releases)
and various CVEs - you end up debugging a lot of machines in unknown states
that fail to rejoin clusters after reboots.

~~~
spicyusername
This has definitely not been my experience running many hundreds of Red Hat
CoreOS nodes in production.

So far, aside from a few small flakey issues, having the cluster nodes _and_
the OpenShift cluster update in lock step has been dramatically simpler to
manage.

------
solatic
As strong as the engineering behind Bottlerocket seems to be, I'm not entirely
sure who they built it for, except as a foundational component for AWS's
managed offerings.

If you, as an AWS customer, decide to fully embrace AWS lock-in, then why
would you run this yourself on an EC2 instance instead of running ECS or EKS?
If you're trying to avoid AWS lock-in, why would you choose an OS that's
locking you into AWS Systems Manager and Amazon Linux 2 for debugging needs?

~~~
NathanKP
Hi I'm a developer advocate in the container engineering org at AWS. I think
there are a few misunderstandings here that I may be able to explain better.

First Bottlerocket is not Amazon Linux 2, it is its own minimal operating
system, with most components built from the ground up in Rust. This is totally
different than the Amazon Linux 2 you may be familiar with (and most other
operating systems for that matter). Bottlerocket is optimized for running
containers with high security isolation. The host OS is extremely minimal, it
does not come with bash, an interpreter, ssh, or anything beyond the system
basics needed to run containers. In fact it uses an immutable root filesystem.
You aren't intended to run or install things directly on the host at all.

Everything that is installed and runs on Bottlerocket runs as containers,
which are kept isolated from each other and the host with best practice
security features such as SELinux. For example you can connect to a container
on the host via AWS Systems Manager, or you can optionally enable a container
that lets you connect to it via SSH. Once again the thing you are connecting
to is the container on the host though, not directly to the host.

For this initial release of Bottlerocket we are focusing on providing image
variants that are prepackaged with the requirements to serve as underlying
container hosts for an ECS or EKS cluster. However we also intend Bottlerocket
to eventually be something that anyone can use, anywhere, even outside of AWS,
if they wanted to benefit from the secure by default, container first design
of Bottlerocket.

You can read more about the security features of Bottlerocket here:
[https://github.com/bottlerocket-
os/bottlerocket/blob/develop...](https://github.com/bottlerocket-
os/bottlerocket/blob/develop/SECURITY_FEATURES.md)

And you can find a bit more of the charter / goals for the project here:
[https://github.com/bottlerocket-
os/bottlerocket/blob/develop...](https://github.com/bottlerocket-
os/bottlerocket/blob/develop/CHARTER.md)

~~~
azinman2
I just installed Proxmox on a home server, and I’m using its CT containers
(LXC) to run various services. Could I use this as a replacement for Proxmox?

~~~
rubyn00bie
Nope. Proxmox is mostly just an administrative UI, containers are a Linux
feature. You don't need Proxmox for anything, you can just run containers
"natively" in Linux or VMs with KVM + QEMU. The linked above is mostly just a
Linux distro geared towards a pretty specific set of use-cases.

------
trishankdatadog
Little-known fact: like Google Fuchsia, Bottlerocket uses The Update Framework
(TUF)[1][2] to securely update itself!

[1] [https://theupdateframework.io/](https://theupdateframework.io/)

[2] [https://github.com/awslabs/tough](https://github.com/awslabs/tough)

~~~
kapilvt
tough is one of the actual oss gems behind bottle rocket thats reusable in non
aws contexts.. bottle rocket probably can be but likely that will always be a
second class citizen, and afaics completely undocumented for usage outside of
aws atm.

~~~
jhaynes
Really glad you like our tough (TUF) library!

As far as running in non-AWS contexts, we haven't had time to head down that
path yet, but as I mentioned in an earlier post, we've tried to build
Bottlerocket in a way that it can be extended to work outside of AWS either as
VMs or even bare metal. In fact, a few of the engineers on the team have been
playing with getting it running on their RaspberryPi's at home :)

------
adolph
Firecracker, Bottlerocket, starting to see a trend here

[https://aws.amazon.com/blogs/aws/firecracker-lightweight-
vir...](https://aws.amazon.com/blogs/aws/firecracker-lightweight-
virtualization-for-serverless-computing/)

~~~
statictype
Both are written in Rust. Looks like Amazon is investing in the language quite
a bit.

------
daxfohl
So difference between this and Firecracker would be that the latter is boot-
speed and overhead optimized, and this one is a bit heavier but more capable?

If choosing between this and say Kata Containers plus Firecracker, the latter
would be more secure because of VM isolation but this would be more efficient
because multiple pods could go in a single VM?

Is Bottlerocket secure enough to host multi-tenant workloads within the same
VM?

~~~
chromedev
Firecracker and Kata containers are ways to run containers inside lightweight
VMs that boot directly into the Kernel, they are not Linux distros in
themselves. You are trying to compare apples to oranges. You could run
Firecracker or Kata containers on top of something like Bottlerocket, however
Bottlerocket is geared more towards the container-sandbox and isolation crowd
while Kata/Firecracker is for those who think you can only get isolation using
VMs.

------
andrewrynhard
For a project similar to bottlerocket, checkout [https://github.com/talos-
systems/talos](https://github.com/talos-systems/talos). It is geared for
cloud, VMware, and bare metal users. We have integrations with Cluster API for
each, with the bare metal provider being our implementation:
[https://github.com/talos-systems/sidero](https://github.com/talos-
systems/sidero). Full disclosure, I am the CTO of Talos Systems.

------
peterwwillis
I haven't dug into the engineering behind this yet, but my main concern with
any custom Linux distribution is it often ends up as a waste of engineering.

It's pretty easy to write your own Distro and pair it down to the essentials,
and pairing it down that way allows you to strip out complexity, making it
easier and more reliable to patch it. But that then means you are now
maintaining this custom thing forever. If you're Amazon, that might be fine,
but I suspect that this will be dropped when it is no longer profitable or a
competing project supplants it - meaning in 5 years this thing might be gone.
(A common theme of custom Linux distributions)

And then there's troubleshooting. With a stripped-down distro, you will
eventually need more tools to debug the thing, meaning you have to build and
maintain packages to do that. Bottlerocket's answer to this is "run them in
containers and use our API!", but I'm not sold on this. Have you ever tried to
do debugging between host and container, or container to container? There's a
lot of b.s. you have to hop through, and most Unix tools were not written with
it in mind. I highly doubt that it will magically work for everything. If
that's the case, then this "don't worry, because _magic_ " idea is not really
saving you work over maintaining a traditional OS.

Moreover, you don't need a custom distro to do live patching. There are simple
tricks you can use to juggle files and processes during a live patch, to say
nothing of "checkpoint process && 'mv old_file new_file' && thaw process",
etc. Kernels support live patching too. So if the argument is "well it's
easier to patch", i'm not sure you're not trading away "easy" in one place for
"pain in the ass" in another (see above). All of this also argues that it's
just as effective to treat live-patched systems as you treat immutable
infrastructure, and I'm not convinced of that argument either. The former is
just more complex, and complexity attracts failures.

Ultimately I think what you'll find is Bottlerocket will get a niche
following, but also some people will get annoyed by it and go back to regular
distros which already have well defined methods.

------
moondev
Cluster API does both! The operator rolls out immutable machine images that
make up clusters. It's badass.

------
jimmcslim
So, is this available as an ISO? I currently run an Ubuntu VM in bhyve on my
FreeNAS home-server to host various containers for experiments, etc... could I
run this instead or is it tied to AWS?

~~~
jhaynes
It isn't today, but we've built Bottlerocket in a way that it can be extended
to work on bare metal or other places outside of AWS. There are a few issues
on our github[1][2] that are similar that you're welcome to watch or
contribute use cases to.

[1] [https://github.com/bottlerocket-
os/bottlerocket/issues/841](https://github.com/bottlerocket-
os/bottlerocket/issues/841) [2] [https://github.com/bottlerocket-
os/bottlerocket/issues/1097](https://github.com/bottlerocket-
os/bottlerocket/issues/1097)

------
robomaker2
At my previous company we discarded the AWS-Linux distro and used rancherOS
for container hosting because the version of yum they used was too flaky. They
were unwilling to move to DNF to try and fix it. We've long badgered them for
something like this (rancher style AWS-Linux dsitro) and they seem to have
finally listened. Too bad, I moved to a different company and a different role
to benefit from this. At least my old colleagues will be happy

------
haunter
Wasn't Amazon Linux 2 something similar? Or I'm mixing it up
[https://aws.amazon.com/amazon-linux-2/](https://aws.amazon.com/amazon-
linux-2/)

~~~
acdha
Amazon Linux 2 is an AWS-optimized Linux distribution but it's a full
distribution you're used to — RPM-based, generally compatible with
RHEL/Centos, etc. — and that has the usual benefits and costs. You can
customize it as much as any other Linux distribution but you're taking on the
corresponding level of effort to secure and operate it. If all you run are
containers, that's overhead for things you're not really using and it also
means that you're going to be slower to update things like the kernel and
system libraries since there's more to test and backwards compatibility is a
big concern.

------
booleanbetrayal
Can anyone at AWS comment on how this fits into the Fargate roadmap and
Compute pricing? Presumably, a slimmer OS for things like EKS nodes could
translate into some sort of Compute discount.

------
0_gravitas
So, I'm a little confused; is this not what NixOS is all about, or is there a
difference? (as my question probably suggests, im not all that knowledgeable
about nix)

------
waheoo
I wish the push to containerise everything would just die.

I need the pieces of my system to work together, not against one another, not
contend for files and permissions.

------
aex
Free project idea: A Qubes OS alternative built on Bottlerocket.

~~~
chromedev
Why Bottlerocket? Why not just use Alpine, Void, NixOS, Arch, etc instead?

~~~
geek_at
I've fallen in love with Alpine. I'm using it on my bare bone servers running
from a ram disk (USB drive) hosting Docker containers and even qemu VMs. All
on encrypted zfs volumes. Totally love it

~~~
pcw888
Can you recommend any tutorials or blog posts on some of this? Really
interested in it. Otherwise I'll search for it, thanks for the information.

~~~
chromedev
Here are some links for you that I think would be useful for a related setup,
however I can't speak for the user that posted that. The headless installation
should follow the same principles of using a RAMDisk though.

[https://wiki.alpinelinux.org/wiki/Raspberry_Pi_-
_Headless_In...](https://wiki.alpinelinux.org/wiki/Raspberry_Pi_-
_Headless_Installation)

[https://wiki.alpinelinux.org/wiki/Alpine_Linux_with_root_on_...](https://wiki.alpinelinux.org/wiki/Alpine_Linux_with_root_on_ZFS_with_native_encryption)

[https://wiki.alpinelinux.org/wiki/Docker](https://wiki.alpinelinux.org/wiki/Docker)

[https://wiki.alpinelinux.org/wiki/Install_Alpine_in_Qemu](https://wiki.alpinelinux.org/wiki/Install_Alpine_in_Qemu)

~~~
pcw888
that's great, thank you!

------
yarrel
How exactly would they make a proprietary Linux distro?

------
spicyusername
So basically Amazon CoreOS.

------
viztor
This reminds of CoreOS

------
LeSaucy
Look out Rancher!

~~~
chromedev
Rancher has been around a long time and isn't cloud-specific. Until this
distro can prove itself useful outside of the AWS ecosystem, then Rancher will
still dominate.

~~~
throwaway3neu94
Bottlerocket seems more like a competitor to RancherOS, which is abandonware.
([https://github.com/rancher/os/issues/3000](https://github.com/rancher/os/issues/3000))

I liked RancherOS's very few moving parts approach (even your shell is a
Docker container). Hope Bottlerocket will be somewhat similar.

~~~
chromedev
I don't see any confirmation based on that link that it is abandoned. There
was an update in June.

~~~
skolsuper
> RancherOS 1.x is currently in a maintain-only-as-essential mode. That is to
> say, it is no longer being actively maintained at a code level other than
> addressing critical or security fixes.

quoting support.rancher.com, taken from the 4th comment down in the linked
Github issue

~~~
chromedev
It looks like that link is broken and I can't find anywhere on their website
that states this anymore

------
senthilnayagam
I remember reading last week linux plumber conference agreed to allow rust in
linux kernel .

but these guys have built an OS in rust.

~~~
chromedev
Did you even read the blog post? This is running Linux, and only the AWS
software-components running on it are largely written in Rust. It is still
Linux and the OS is not written in Rust, nor the container daemon and probably
none of the non-AWS software packages.

~~~
pjmlp
Most available userspace seems to be a mix of Rust and Go, except for existing
stuff written in C like LinuxSE.

~~~
chromedev
What userspace do you see written in Rust?

~~~
pjmlp
From this thread
[https://news.ycombinator.com/item?id=24346300](https://news.ycombinator.com/item?id=24346300)

~~~
chromedev
That honestly sounds terrible, and apparently only works on ECS. What a shame.
I was thinking that AWS was actually going to provide something useful to the
open source community.

