On: Large-scale cluster management at Google with Borg

[dcsommer]

Does Kubernetes use chroot jails like Borg, or has it advanced to LXC containers? Nowadays LXC isolation seems strictly preferable.

[wffurr → dcsommer]

Borg also uses cgroups, just like LXC.

From the paper: "We use a Linux chroot jail as the primary security isolation mechanism between multiple tasks on the same machine… all Borg tasks run inside a Linux cgroup-based resource container."

[samkone → wffurr]

Remember that Borg is an old thing. At the time they probably started out with chroot jails, because cgroups weren't ready yet. But Kubernetes, Mesos use docker and cgroups(Mesos).

[KaiserPro → samkone]

Cgroups have been around for a long time. Wikipedia says 8 years.

I think was in RHEL 6.2 possibly even 6.0

[powera → wffurr]

cgroups were largely written by members of the Borg team.

[jbeda → dcsommer]

Kubernetes uses Docker (or now rkt) to do the actual containing. The focus is on scheduling and managing lots of containers.

You may run O(10) containers on a single machine. When you run O(10k)+ containers in a cluster you need new tools to manage things.

[jhspaybar → dcsommer]

LXC uses chroot + other things. chroot is essentially the foundation of any containerization tool.

[zobzu → jhspaybar]

Not really- chroot provides no isolation, namespaces do. choot is the visible portion (changes root.. ie changes /mnt/blah/bleh to /) but not "the foundation"

Things can in fact work isolated without chroot, too. Its just convenient.

[exacube → jhspaybar]

Note that chroot will provide containerization, but without isolation (processes can always fork out of their chroot jails). You still need things like pid namespaces etc.