Use LXC and LXD. It's the best of both worlds. Docker is very limiting and already falling short in features. For instance daemon must run as root which sucks. LXC supports unprivileged namespaces so all users of a system can have their own set of containers. Docker insists on being the parent process, That means when docker dies, so do all your containers.. This is really bad...
How do you get boot2LXC working? Any docs, or does everyone have to DIY it all again?
Is there an ecosystem around LXC that provides things like Flynn?
I can do it all myself, but I can't do it, my development job, and be home for dinner at night. Like most tools today, the value is in the ecosystem, not the tool itself.
LXC is generic container technology like KVM OR Xen are generic virtualization technologies. LXC virtualizes the OS environment and gives you lightweight containers that you can seamlessly transition your VM workloads to, or use as a lightweight portable alternative to VMs, so its use case is general and not a narrow focus on paas or deployment centric technology.
Users can then decide how they want to deploy. Docker takes that base container and adds layers of aufs, constrains the container OS template to single app by modifying the container OS's init, gives you the dockerfile and focuses on deploy centric functionality with immutability idempotency etc, and this makes it much more complex to use than LXC. Its a use case built on Linux containers, not containers itself.
LXC is not 'low level kernel capabilities' [1] as Docker misleadingly refers to it on it's website. This has resulted in a lot of confusion about LXC in the Docker ecosystem with folks thinking its 'difficult to use' or 'just low level stuff'. A tad unfair to LXC given Docker was based on it till 0.9 and knew exactly what it was, and is as accurate as referring to docker or nspawn as low level capabilities.
That would be kernel namespaces and cgroups that LXC uses to give end user containers, like Docker uses post 0.9 directly with libcontainer and systemd-nspawn uses for its containers.
Docker builds on containers to deliver additional functionality. There is an additional cost in complexity but if that is your use case the trade off may be worth it, but for other use cases the complexity may be overkill.
You can simply make a VM image of LXC installed and you have boot2lxc, the vast ecosystem of orchestration technology that works in VMs and systems works in LXC, you don't need specific tools to be designed just for LXC. its not opinionated or exclusive like the tools built around the Docker ecosystem that are finely focussed on a specific use case and typically support Docker only.
So, you're saying it's about a month or six to replicate the conveniences that docker has provided for our organizations?
Docker isn't anything special. There's nothing that is magical or revolutionary there, but what you describe is quite a bit of work to replicate the ecosystem. That's time that could be used creating novel value.
Not sure saying LXC is "generic" is helpful either. It is opinionated that you should be virtualizing the whole OS environment, like Docker is (fairly) opinionated that you should not be.
I think some of the confusion arose as people used to use LXC as a generic term for the container technologies as well as the product, before there were any other products. namespaces+cgroups is not such a nice name. And using "linuxcontainers.org" as the website does not make it clearer.
