
Iocage – A FreeBSD jail manager - HugoDaniel
https://iocage.readthedocs.io/en/latest/
======
jzelinskie
I'm deeply involved in the Linux container ecosystem (docker, rkt) and I'd
like to understand the difference in workflow between that and this. Is anyone
familiar enough to speak to both?

~~~
crest
Jails are a general mechanism to run multiple userlands on one kernel. They
started with a change root directory, an IPv4 address and no access to
anything dangerous (e.g. /dev/mem). Jails grew a lot features with the biggest
bump in FreeBSD 8. Starting with FreeBSD 8 jails can be nested, have multiple
IP addresses etc. but they still lacked System V IPC. This changed with
FreeBSD 11.

Now the only thing jails can't have is their own IP stack. Jails share the
hosts IP stack which improves efficiency and simplifies most deployments, but
it prevents them having administrative access to the IP stack. There is an
experimental kernel feature (VIMAGE) to jails to run their own instance of the
IP stack but there are still some nasty bugs hiding in this code, because
nobody thought about how to tear down the IP stack. After all it was
initialised once during the boot process and kept running until the power went
out.

The largest difference is in the mindset behind jails. Jails are designed as
secure operating system level virtualisation. Docker on the other hand is
fairly fragile and offers neither secure isolation between containers nor
between containers and the host. Jails can contain a complete userland and
this a very common setup.

A full FreeBSD userland + some ports/packages to make it useful is about one
1GB. This used to be a lot 15+ years ago when jails where created and the
older jail management tools like ezjail reduce the per jail storage
requirements with nullfs and unionfs hacks. These days 1GB isn't that much for
a simple container and most FreeBSD servers run on ZFS. ZFS offers a much
simpler and cleaner way to reduce storage requirements: just clone a snapshot
(the template) create a new jail and copy a few config files into the clone.
The only problem is that you can't rebase your clone.

Docker is designed around the idea of single purpose containers without stable
storage. You can use jails to implement this idea, but FreeBSD jails support
more than that. Also all the FreeBSD jail managers I used try to stay out of
your network configuration as far as possible and at most configure alias IP
addresses on existing interfaces.

Docker is very opinionated software fighting against limitations imposed on it
by the Linux kernel. Jails are FreeBSD kernel feature touching multiple parts
of the kernel with a minimal userland interface in the FreeBSD base system .
Multiple higher level jail managers are available in the FreeBSD ports tree.

There is no reason why you couldn't implement a docker like jail manager and
the jetpack projects started doing exactly this. Keep in mind that docket
images are the worlds new statically linked binaries for people who can't
figure out how to define and reproduce the relevant parts of their development
environment in their production environment. Executing existing docker images
with their Linux binaries would probably require a massive update to the Linux
compatibility layer (a reimplementation the Linux syscall ABI).

~~~
fnj
Perhaps you have the knowledge to inform me on a question to which I have
never been able to find a plain answer. Do FreeBSD jails have anything like
UID/GID virtualization and namespace isolation and resource quotas that
LXC/LXD gives you on linux via cgroups?

Why is this NEVER brought up when FreeBSD and linux are compared?

~~~
crest
UIDs and GIDs are just numbers to the kernel. Root inside a jail become any
UID/GID and chmod/chown files just like root outside the jail, but even root
inside a jail is locked down into the jail root directory.

You can prevent root inside a jail from modifying files with several
mechanisms. The simplest is to not allow (re-)mounting from inside the jail
which is the default and mount the relevant file systems read-only. BSD
extended flags offer a finer granularity and by default root inside a jail
isn't considered privileged by chflags(2). You can disable the security
feature in which case the normal rules apply and access is controlled by the
secure level (jails have their own secure level).

Processes running inside a jail have access to the full UID and GID namespace
without any mappings.

PIDs exist in a global namespace and are allocated by the kernel. Processes
inside a jail can't address PIDs unless they are inside the same jail or a
child jail. Root inside the host can always address all PIDs. Non-root
processes in the host (and jails) are subject to the
security.bsd.see_other_uids and security.bsd.see_other_gids sysctls which can
be disabled to protect the obvious ways to spy on other users. There is also
support for various forms of mandatory access control. Have a look at the
`mac_*` manual pages for more information on the available MAC disciplines.

Restricting the jails to just their processes in a shared namespace is a much
cleaner alternative to loosing the PID as unique global process identifier.
And mapping between different scopes is even worse than extending the
identifier from an integer into a tuple.

FreeBSD also supports hierarchical resource limits. Jail ids are one possible
subject type to limit. That way you can limit resource consumption per jail.
See
[https://www.freebsd.org/cgi/man.cgi?rctl](https://www.freebsd.org/cgi/man.cgi?rctl)
and
[https://www.freebsd.org/cgi/man.cgi?rctl.conf](https://www.freebsd.org/cgi/man.cgi?rctl.conf)
for more details on that.

~~~
fnj
Every LXC/LXD container can have its own PID 0 init process, and none of them
even have visibility of the others. That's what I am not seeing in jails. Root
on the host cannot even see any of the LXC/LXD PIDs.

Am I not seeing it because I am not looking hard enough, because FreeBSD guys
don't really understand this whole issue, or because it's not there?

Re: resource limits: can the CPU load per jail be limited?

~~~
langseth
Across jails, pids are not visible. I can't comment on the ability of a user
on the root of a linux system being able to see the pids in a LXC, but I bet
there is a way. After all they are running on the same kernel. Is the non-
visibility of the proceses just a userland trick on linux?

Resource limits for Jails can be done with rctl

~~~
zzzcpan
It's important to understand, when people talk about cgroups, they do mean not
only CPU/memory limits, but also network and disk I/O limits, that freebsd
doesn't offer out of the box.

~~~
crest
The firewall can match on the jail id for local sockets and apply traffic
shaping based on these matches. Hierarchical resource limits on
CPU/disk/memory/... can be configured per jail.

------
_paulc
There are lots of jail wrappers but in most cases it is just as easily to just
take the time to understand how jail.conf works and use this (see FreeBSD
jails the hard way [1]). It's fairly easy to just use zfs clones to create
jails and customise the exec.prestart/exec.start functions to do automatic
provisioning and configuration (eg. I store the port forwards in a variable
for each jail and process this and automatically setup pf rules when the jail
starts).

[1] [https://clinta.github.io/freebsd-jails-the-hard-
way/](https://clinta.github.io/freebsd-jails-the-hard-way/)

------
cyphar
Is there anyone from the FreeBSD community interested in standardising FreeBSD
containers inside the OCI specification? Currently only Linux, Solaris and
Windows have been included in the standard -- which is a bit disappointing
(I've always been fond of FreeBSD).

If anyone is interested, please contribute to the Open Container Initiative.
[https://github.com/opencontainers](https://github.com/opencontainers)

~~~
tachion
Not exactly OCI, but as an example of what can be done with FreeBSD, check out
[https://github.com/3ofcoins/jetpack](https://github.com/3ofcoins/jetpack)
that's a APC implementation using Jails/ZFS and other FreeBSD goodies.

------
smkelly
iocage is great, but according to the README in GitHub it is not being
developed anymore. The author is working on a rewrite in Go. The rewrite isn't
done yet last I checked.

~~~
synchronise
How does it compare to ezjail?

~~~
crest
Ezjail is fairly old. It works with the older less capable API to jails and is
restricted to features supported by the old API. It was also designed for use
with UFS on tiny disks.

Lets say you are a webhoster around the year 2000 and your servers have a
handfull 36GB or 72GB SCSI disks. You want to protect each customer from all
other customers and protect yourself from all customer scripts. This was
before IA32 CPUs offered the features to support efficient transparent
virtualisation and even if they did the resource demand per VM would have been
too high. As long as your customers are happy with static file hosting
everything is fine, but as soon as way want to execute some useful server side
scripts you have a problem. FreeBSD offers a way to run one HTTP server per
customer inside jail, but keeping a full FreeBSD userland (base + http server
+ databases + scripting language + customer code) per customer would quickly
fill your puny little disks. Ezjails offers a neat solution to the problem:
store a template just once and instantiate it with a nullfs read-only mount.
Now your storage requirements are manageable at a reasonable price with
hardware of the day and your buffer cache hitrates are better too. All of
these indirection and aliasing hacks make ezjail more complicated than modern
jail managers, because ezjail had to work around the operating system
limitations instead of taking advantage yet to be invented operation system
features.

~~~
citrin_ru
Ezjail now supports ZFS.

Sharing one basejail via nullfs is useful feature, can Iocage do this?

Nullfs not only allows to save space on disk but also allows faster updates
(extract new basejail then switch all jails to it, without full upgrade of
each jail).

Also in software old doesn't mean bad, and newer is not automatically better.

~~~
crest
I didn't mean to imply that. I just explained why ezjail looks so convoluted
to a new user. Ezjail still solves the same problems today as it did 10 years
ago. On the other hand since not even bloatware did keep up with storage cost
decreases users can afford more comfortable trade-offs today.

------
baobrien
I'd love to see a combined FreeBSD jail and bhyve manager. Can libvirt do
that?

~~~
ketralnis
Have you looked at
[https://github.com/pr1ntf/iohyve](https://github.com/pr1ntf/iohyve)?

------
voltagex_
iocage is easier to use than ezjail or warden, but I still managed to end up
with a broken system where the system thought it was FreeBSD 11 but still
expected to use packages from FreeBSD 10 (I get an ABI error when doing pkg
upgrade).

The distinction between jails and basejails is tricky to follow.

I don't know where the rewrite went.

~~~
jlgaddis
What do your _pkg.conf_ entries look like? Do they point at 10 specifically?

~~~
voltagex_

        FreeBSD: {
          url: "pkg+http://pkg0.nyi.FreeBSD.org/${ABI}/quarterly",
          mirror_type: "srv",
          signature_type: "fingerprints",
          fingerprints: "/usr/share/keys/pkg",
          enabled: yes
        }

------
t_tsonev
Works well for me. The only thing I dislike is the GUIDs for the containers
that can't be changed to something shorter.

At the time I had to decide ezjail didn't work with FreeBSD 10, not sure if it
has been updated.

~~~
kchoudhu
...Am using ezjail on a mix of 10.3 and 11 now. Works flawlessly.

