
Bottlerocket: An operating system designed for hosting containers - ecliptik
https://github.com/bottlerocket-os/bottlerocket
======
whalesalad
A link to the actual source code (90% Rust) and README:
[https://github.com/bottlerocket-
os/bottlerocket](https://github.com/bottlerocket-os/bottlerocket)

And here is a post from AWS with more technical details:
[https://aws.amazon.com/blogs/aws/bottlerocket-open-source-
os...](https://aws.amazon.com/blogs/aws/bottlerocket-open-source-os-for-
container-hosting/)

~~~
rfrfee
Given that Red Hat recently killed CoreOS, it's great to see new alternatives
coming up. I cannot wait to give it a spin!

~~~
ailideex
Not even once ... [https://docs.fedoraproject.org/en-US/fedora-
coreos/](https://docs.fedoraproject.org/en-US/fedora-coreos/)

~~~
jbirronw
That first needs to prove itself in practice, so far it seems more like a
desperate try to tie it to Fedora and have it catch up with the popularity in
cloud of the likes of Ubuntu and to eventually have people paying by locking
into RHEL CoreOS.

~~~
joana035
No, let's pay a premium and lock ourselves into a trillion dollars company's
last day project instead :-)

------
ranman
I’m here for the puns like updog: [https://github.com/bottlerocket-
os/bottlerocket/blob/develop...](https://github.com/bottlerocket-
os/bottlerocket/blob/develop/sources/updater/updog/README.md)

~~~
jhaynes
Glad you're enjoying them! We have a glossary just in case:
[https://github.com/bottlerocket-
os/bottlerocket/blob/develop...](https://github.com/bottlerocket-
os/bottlerocket/blob/develop/GLOSSARY.md) (my personal favorite is Laika, the
first dog in space and our pre-init binary)

~~~
paulannesley
Honestly, reading the glossary gives me Urbit [1] vibes :(

> bork: A setting generator called by sundog to generate the random seed for
> updog, determining where the host falls in the update order.

[1]: [https://urbit.org/docs/glossary/](https://urbit.org/docs/glossary/)

~~~
alkonaut
Agree. It's cool so long as the number of names is small, and the names
actually are a pun on the function and not just e.g. names of planets. If
"updog" is what brings something "up" that's a good name.

WiX (windows installer creation) has a multi-phase command line interface
where the compiler/linker/.. has different names indicating the order they are
applied: candle, light, smoke... Also a working system I guess.

~~~
steveklabnik
"bork" is dog-talk for "bark", and so something that randomly gets an updog
going being named such makes sense too, it's just a slightly more obscure
joke.

------
nstart
Opening line from their announcement blog post:

>It is safe to say that our industry has decided that containers are now the
chosen way to package and scale applications.

Curious how the HN community feels about that statement. Not so much about the
truth of the statement but about the fact that containers are becoming the de
facto method of packaging applications.

~~~
hardwaresofton
As far as I'm concerned, this is an obvious truth. Linux containers are
processes with better sandboxing -- who would _not_ want this?

As kinks in the kernel support and tech get worked out, and OSs deepen support
I can't imagine that it will ever make sense to say something like "I could
have run the process with cgroup and namespace isolation but I chose not to,
choosing to make a new user-level isolation or run everything as root
instead".

Arguments against containers as the future based on the complexity _may_ have
weight but not for long.

~~~
sourcesmith
"I could have run the process with cgroup and namespace isolation"... using
systemd.

~~~
kchr
Or skip the million non-container related dependencies introduced by systemd
and focus on a container centric init system...

------
jrhill85
The subject says "for hosting containers" but the README says "for AWS EKS
Kubernetes" which sounds a little less general...

How tied in to the AWS model is this? Are the places that would need to be
expanded known?

Also at least at a glance, this is a neat use of real-world Rust

~~~
jhaynes
While our first variant is focused on Kubernetes and EKS, we have designed
Bottlerocket in a way that new variants can be built that work with other
orchestrators, or even without one (we have ECS support on our roadmap
already). Also, we really enjoyed working in Rust for big chunks of this!

~~~
bogomipz
>"While our first variant is focused on Kubernetes and EKS..."

So is the idea that people create a Bottlerocket AMI and use that as their EKS
worker node images? Is that correct?

~~~
jhaynes
Correct! Or you can spin up one of the AMIs we've already built. You can find
current AMIs via public SSM parameters: [https://github.com/bottlerocket-
os/bottlerocket/blob/develop...](https://github.com/bottlerocket-
os/bottlerocket/blob/develop/QUICKSTART.md#finding-an-ami)

------
jontro
Their take on using cargo for packaging is quite interesting:
[https://github.com/bottlerocket-
os/bottlerocket/tree/develop...](https://github.com/bottlerocket-
os/bottlerocket/tree/develop/packages)

~~~
joana035
They are using tradicional RPM for packaging.

The cargo.toml workspaces relates more to make IMHO.

~~~
orf
Wouldn’t something like Bazel make more sense? Using cargo to track inter-
dependencies seems a bit weird to be honest.

~~~
merb
well it's interesting since they only need to learn the rust toolchain, not
something else.

------
sciurus
Is this intended to become AWS's version of GCP's Container-Optimized OS?

[https://cloud.google.com/container-optimized-
os](https://cloud.google.com/container-optimized-os)

------
jahkeup
Project is up on GitHub along with its other components!
[https://github.com/bottlerocket-os](https://github.com/bottlerocket-os)

------
eikenberry
Looks like it is free software. Dual MIT/Apache license.

[https://github.com/bottlerocket-
os/bottlerocket](https://github.com/bottlerocket-os/bottlerocket)

------
ajxs
This is somewhat false advertising. This is not an operating system in the
sense of being a new kernel. It looks like it's a set of build tools for
building a Linux distribution.

~~~
tasogare
An operating system is not defined only by its kernel. It's kernel + APIs +
user land. If one of those components is changed radically, it's indeed a new
operating system. That's why Ubuntu and Debian are distinct, even if both are
based on Linux.

~~~
ajxs
You're correct. Debian and Ubuntu are technically different operating-systems.
Due to the similarity of 'operating-systems' built around the Linux kernel
they're typically referred to as 'Linux distributions'.

In the context of software development, if you tell someone you're developing
a new operating-system you're probably going to conjure up images of writing a
new kernel. If you tell people you're developing a new Linux distro, this is
closer to what they'll imagine.

~~~
akvadrako
It’s not that simple. I wouldn’t use the word distribution for GUIX, NixOS,
Android and several OSes that use the Linux kernel.

~~~
ajxs
This is true. Android is a good example of where the line between a Linux
distribution and an outright OS blurs. I'm not convinced that this really
affects my overall point all that much though.

------
islanderfun
Fascinating project! Anyone know of possible overlap with Firecracker? Really
digging these Rust projects.

~~~
jhaynes
No overlap per se, but we are looking at how to integrate Firecracker as a
potential target for "container" launches:
[https://github.com/orgs/bottlerocket-
os/projects/1#card-3386...](https://github.com/orgs/bottlerocket-
os/projects/1#card-33865383)

------
zapita
How does this compare to Linuxkit? At first glance it seems almost identical
but I may be missing something.

[https://www.github.com/linuxkit/linuxkit](https://www.github.com/linuxkit/linuxkit)

~~~
lifty
Linuxkit allows you to build your own appliance like OS, while Bottlerocket is
more of an end user project. A project that is more similar to Bottlerocket is
[https://www.talos.dev](https://www.talos.dev) or
[https://www.projectatomic.io](https://www.projectatomic.io)

~~~
zapita
I see. Thank you.

------
eliaspro
Any insights on the design decision to go with Wicked instead of systemd-
networkd which is already provided by systemd and better integrated with the
remaining systemd tooling/components/conventions?

------
jpeeler
It looks like this supports automated updates within a specific time window,
but it's not clear to me how the "waves" are defined. (Note that this is
something that is currently lacking in Fedora CoreOS:
[https://github.com/coreos/zincati/issues/34.](https://github.com/coreos/zincati/issues/34.))

I do wonder if the dual partition approach was deemed more stable than using
OSTree or why the latter wasn't used.

------
bogomipz
Does anyone if or where Bottlerocket is being used at AWS? Is it used to run
their Fargate/ECS/EKS service?

------
dpeterson
Question for any Amazon folks here that may know. Is this akin to something
like Atomic or CoreOS that is used within Openshift as the Master or Worker
node OS or is this more like the UBI (Universal Base Image) that can be used
as the base image of a container via "FROM" within a dockerfile?

~~~
antonvs
You don't need Amazon folks, just read the material.

This is not a container base image, it's a container host OS. It is somewhat
similar to Atomic or CoreOS, but in some ways it seems to be a bit more of a
radical redesign than those.

------
lysium
So, the main feature is that updates happen for all packages at once and not
for each package individually. Sounds interesting, even for non cloud setups.

How does that work? The explaining image does not explain that. How is that
different from rolling back on file system level?

~~~
tjkirch
The update system is image-based; when an update is downloaded, it's written
out to an alternate set of partitions, and then it can flip over to those
partitions with a reboot. That makes it easy to roll back with the same kind
of single flip, too.

It's different than filesystem-level rollbacks because it's all-or-nothing, so
you don't have to worry about update failures after a few packages, and
because all of the components in a given image are guaranteed to be tested
together, whereas with package-based systems, your combination of packages may
have never been used together by anyone else. In addition, for builders, it's
easier to sign, distribute, and verify a single image.

~~~
yjftsjthsd-h
How does this compare to something like nix or Fedora Silverblue?

~~~
arianvanp
Nix has upside that you just need to flip a symlink to do the same . Downside
is that you don't have thinks like dm-verity that can prove that your update
wasn't tampered with.

In nix the nix store is remounted over itself read-only, but nothing stops
someone from ripping out the disk and flipping bits. This is not possible with
these kind of 2-partition schemes if you have dm-verity set up

~~~
the_duke
This is possible in nix with "nix-store --verify --check-contents" .

~~~
arianvanp
That's different though. that's verification _after_ the fact, whilst dm-
verity does it any time during block-level access.

Also an attacker could modify the nix-store sqlite database and spoof the
hashes, rendering this check moot

------
mwcampbell
I'm curious about why Bottlerocket is building base packages like glibc, bash,
util-linux, etc. from source, rather than just pulling binary RPMs from CentOS
or Amazon Linux.

~~~
tjkirch
Amazon Linux and CentOS are general-purpose distributions and need more
features built into the base packages than we do for Bottlerocket. We’re able
to simplify the spec files and produce smaller RPMs with only the content and
dependencies necessary for our more narrow goals.

------
peterwwillis
So it's AWS's version of CoreOS?

I wish somebody'd take some VC or R&D money and build distributed computing
features into the kernel itself, so we could quit wasting our collective
engineering talent, time, money and energy on distributed applications that
run on non-distributed-operating-systems. It's like nobody wants to work on
creating a round wheel, so instead we're spending all our time building custom
roads for square wheels.

~~~
johnmarcus
Erlang is probably the closest thing to this, but unfortunately everything it
runs also needs to be written in erlang. If erlang could run containers as
processes it would take over the world, all of the features from kubernetes
has been baked in since the late 90's.

------
bbgm
I am from AWS. Could we please change the title to say “Bottlerocket from
AWS“. Like Firecracker [1] it’s explicitly not AWS branded.

1\. [https://github.com/firecracker-
microvm/firecracker](https://github.com/firecracker-microvm/firecracker)

~~~
ignoramous
Speaking of branding, I've some questions:

1\. When are services branded as AWS (AWS Fargate) vs Amazon (Amazon
DynamoDB)?

2\. Is BottleRocket a nod to SkyRocket [0] or a movie of the same name?

3\. Why is it called _Fargate_ [1]?

[0]
[https://en.wikipedia.org/wiki/Skyrocket](https://en.wikipedia.org/wiki/Skyrocket)

[1]
[https://youtube.com/watch?v=ye3-gUwu9tI&t=44m28s](https://youtube.com/watch?v=ye3-gUwu9tI&t=44m28s)

~~~
ayberk
To answer your first question, AFAIK, standalone services that can be used on
their own are prefixed with Amazon (eg S3, EC2, DDB), whereas services that
are deeply integrated into AWS ecosystem are prefixed with AWS.

Disclaimer: I work for AWS, but this is not an official answer. What I said is
correct to my best knowledge, but I cannot guarantee its correctness/accuracy.

~~~
valzam
This is the correct answer, it was mentioned in an AWS Cert training.

------
rhatr
This looks pretty sweet. It seems to be a continuation of the same trend of
"just enough Linux to run containerd" that CoreOS started, linuxkit continued,
then Project EVE expanded to cover virtualization.

It is also interesting to note that every step on that journey seems to have
picked the coolest runtime to implement it in (C/early Go, established Go, and
now Rust)

------
rhatr
Started to dig into the implementation. First impressions so far: loving all
the Rust harness - really nicely done and way better than buildroot/yocto/etc
for creating tight, single-purpose linux images. Speaking of tight, here comes
the bad news: really NOT loving all the over-engineered upstream components
like D-Bus and systemd that seem to be there by default. In that sense
#linuxkit with its Alpine base and strong attention to how bit the image is
still comes way on top.

One more thing on the good side: the TUF implementation in Rust seems really
interesting. I'll be digging some more and may actually steal it for linuxkit
(and by extension Project EVE)

Fun fact: a lot of the patches you will find in more system level packages
like grub seems to trace their lineage to CoreOS (and potentially Project EVE)
but I haven't seen acknowledgments anywhere. This is of course all fine from
licensing perspective -- but I still would be curious to know whether it is
indeed where it was taken from.

~~~
Conan_Kudo
> really NOT loving all the over-engineered upstream components like D-Bus and
> systemd that seem to be there by default.

I'm happy with the usage of systemd if they take advantage of the hardening
features in systemd units for core system services. I'm a bit less happy about
the continued usage of Docker, but I get why that's happening for this (EKS
and ECS both use it, so it helps support that infrastructure).

~~~
mmzdnp
Bottlerocket does not package docker. It packages containerd instead for its
container runtime.

~~~
justincormack
What is the docker-engine package for then? [https://github.com/bottlerocket-
os/bottlerocket/tree/develop...](https://github.com/bottlerocket-
os/bottlerocket/tree/develop/packages/docker-engine)

~~~
mmzdnp
Sorry, I should have been more clear. The docker packages are there for the
development build of Bottlerocket. The Kubernetes variant does not use the
docker packages for its build. See more about variants here:
[https://github.com/bottlerocket-
os/bottlerocket/tree/develop...](https://github.com/bottlerocket-
os/bottlerocket/tree/develop/variants)

~~~
justincormack
Ah ok, thanks, still finding my way around.

------
dirtydroog
Remove SSH? Over my dead body!

What is container specific about all this? It just seems to be minimal images?

~~~
orf
If you need to SSH into your cattle you’re either not in position to benefit
from something like Bottlerocket or you’re doing things wrong.

