Example : this company had part of their site built by some abstruse static site generator with a million dependencies written in a language that no one at the company had installed by default, nor knew (I think it was Ruby). We put it all in Docker, and the README changed from 10 lines of install x, y, z version foo to /random, to just “run docker build, then docker run.” Most of the people working on those docs didn’t need to know about ruby or gems or lock files or any of that.
A year later I overheard someone say thank god that thing was dockerized because I was absolutely not looking forward to installing all those dependencies just for a typo fix.
That’s one of the areas where docker shines: app delivery.
I'm also still not on the docker-wagon and would like to know what I'm missing out on. Most of my stuff is python. Why is docker better than virtualenv with a requirements.txt file?
From this description it seems to solve the same issue.
I could imagine something like environment variables, but on the other hand that's something I've learned not to keep in version control, and putting it in docker would be exactly that, no?
> Why is docker better than virtualenv with a requirements.txt file?
It's not, it solves a different issue. If you only have Python as dependency, requirements.txt are fine (well, user needs to install correct version of Python, pip and virtualenv / pipenv, but that's doable). But as soon as your app is actually composed of nginx / apache, python, some background process in Rust, bash scripts for cron jobs,... then you have a problem with app delivery, which Docker solves nicely. Just package eveything in a Dockerfile and distribute the image. Bonus point: you can now test it locally, with the same installation.
I jumped on Docker wagon very early for this exact reason. I don't care about hype, but it does solve these kinds of problems.
I'm also python dev and I'm embracing docker for delivering my app to users.
> Why is docker better than virtualenv with a requirements.txt file
One of issues that `docker-compose build` solves better is that venv is superconfusing to your non-python users. I've had people:
- skip it and do `python setup.py install` instead
- skip it and install requirements globally `pip install -r requirements.txt` because thats how it worked for them last time they used python
- version control their `venv` directory
- backup and upload their `venv` directory on new server when moving
Now they just do `./appctl setup` that asks them for few things, writes `setup.env` file for docker-compose, and then runs `docker-compose build`
But this is only beginning. Thanks to `docker-compose` that `build` step also installs:
- PostgreSQL database
- Redis for caching and message queue
- UWSGI
- Celery
- Nginx proxy
The TL;DR is a typical web app is often a lot more than just running Python, and there's a huge amount of value in being able to run the same code across different operating systems without any installation changes.
The OP says app delivery but they actually talk about development builds. Dev builds are another area where Docker/containerisation helps. Dev builds are just as annoying with Golang as Ruby or Python. In fact build awkwardness is orthogonal to static vs dynamic typing.
> Dev builds are just as annoying with Golang as Ruby or Python.
Hm, I'm pretty sure I can take any Go project out there and build it in a matter of minutes on my current OS (depending on the build time of course). Just a matter of setting the environment on the right compiler binary and retrieving dependencies (which just came better 2 weeks ago with 1.11's modules).
Interpreted languages have their uses, of course, but there's a clear difference in both ease of deployment and development between the realms of interpreted languages (Ruby/Python/PHP/etc.), VM/IL/JIT based languages (JVM/.NET/etc.) and 'plain old' compiler based languages (C/C++/Rust/Go/etc.).
> In fact build awkwardness is orthogonal to static vs dynamic typing.
I said "interpreted languages' ecosystems", not "dynamically typed languages".
I think you're mixing concepts here (or just maybe you're adding info not directly related to my comment).
Just a matter of setting the environment on the right compiler binary and retrieving dependencies
Which is exactly how it works in Python.
I'm pretty sure too I'll very quickly run into dependencies hell by picking randoms ruby/python programs. Just take a look at that SO question:
That SO question is about a tool for "setting the environment on the right compiler binary and retrieving dependencies". But you can also do it manually just fine; that's how I usually did it, in fact. Just retrieve the dependencies to a directory and call Python with PYTHONPATH set to it.
Pyenv is essentially the same as Go's new modules tool, it's just not called through the same binary as the interpreter/compiler.
Interpreted languages have their uses, of course, but there's a clear difference in both ease of deployment and development between the realms of interpreted languages (Ruby/Python/PHP/etc.), VM/IL/JIT based languages (JVM/.NET/etc.) and 'plain old' compiler based languages (C/C++/Rust/Go/etc.).
Yes; with Python, you can both run directly from source without wasting time compiling after each change, or you can produce a standalone binary (dependent only on libc) for distribution, using a tool like PyInstaller. Plain old compiled based languages are much more limited.
Yes, you can have everything in every language. After all it's turtles all the way down.
What makes us successful as software engineers, in the end (and IMHO), is the efficiency level at which we reach our goals. To my eyes, when deliverability is the goal, then a language where native (cross)compilation is first class citizen is the most efficient path. But don't get me wrong, Python is really great and still a good choice for that scenario (I'm thinking of dropbox, for ex).
Mainly because I currently work on long term projects, building entreprise grading systems. Here you want simple, dumb technologies for which you'll still be able to put up build OS in the years to go.
I can build any python project in 10 mimutes (and not pollute my global environment). I can't do that with Go or C or C++ or Ruby or php. I can if they are dockerised.
You said at first "Dev builds are just as annoying with Golang as Ruby or Python" and now you're saying "I can build any python [...] I can't do that with [...] Ruby"?
I mean that I am knowledgable about Python builds. You are knowledgable about Golang. Neither skill is special. With Docker I don't need to learn as much about FooLang's build process to contribute to a project using FooLang.
Thanks, it gets clearer. As a matter of fact I'm both knowledgeable in building things in Python and Go, Node and PHP too. And others. But, admittedly, for 'big' things I tend to lean on Go nowadays (edit: mainly because I need to deliver on-premise, OS-aware applications).
It seems like you think there's a "build process" with Go because you're accustomed to python/pyenv (or whatever is your env manager of choice), in which cas you need to known that there's no such thing with Go.
The two following course of actions are the same:
- "apt-get docker; docker pull {project-url}; docker start {project-url}"
- "apt-get golang; git pull {project-url}; go run .go" (the last Go 1.11 will take care of dependencies at build time)
You can't as easily do "apt-get python; git pull {url}; python .py", unless the program is really simple, because there's such questions as "python2 or 3?" or "pyenv, virtualenv or anaconda?", obligatory XKCD here: https://xkcd.com/1987.
But it's totally ok not to bother learning the compilation command of a given project's language and rely on Docker, really I'm ok with that, doing it myself from time to time. My initial point was on app delivery to final users.
You're right that the multiple solutions for Python make things confusing and prevent you from using the same method for every project, hopefully we'll start to standardize on pipenv, which makes the process:
I get that docker makes this simpler but it’s not really a feat only docker could accomplish, a makefile or bash script could have accomplished the same thing.
A bash script would have to automatically install all the dependencies. Over time, there is a growing chance that some of the versions required will conflict with whatever is already installed on the machine, and someone will have to go in and fix them... then fix them in a way that it works on everybody's machines.
With docker, you can just ignore all that. As long as there is a single person capable of updating the dependencies on a single machine with docker, it'll work the same everywhere, always.
A bash script would be nowhere near as practical. First of all it would be much more complicated to deal with various environments, and in practice, docker run/docker stop is much easier to upgrade.
If everybody in your team uses same distribution of Linux (say latest Fedora, for example), then all dependencies can be packaged into RPM or installed from system repository. RPM and docker are very similar in general idea: they are both image of result of installation of a program(RPM)/system(Docker).
Every member of team will need to run same version of Fedora directly or in Vagrant or in (ta-da) docker container.
From my experience, it turned out that for small Dockerfile's, RPM packages are unnecessary burden, so usually I just start with just docker. But later, when Dockerfile growths, it much harder to track dependencies and installation instructions, when they are interleaved, so it much easier to return to individually packaged programs and replace almost whole content of Dockerfile with single "dnf install meta-package" command.
I was like that and ignored Docker for a number of years. The app I'm working on has one database, one queue, one API server and one web front end. Some of these will scale when in production, but for eveloperment it's quite simple. I used to just run them at the command line and get on with development.
But using docker has been an eye-opener. I can now get my front end developer to have a fully working database, queue, API server running with a simple `npm run docker`. They are free to develop the UI without having to worry about the underlying stack. I am free to change that stack without having to worry about telling them what commands to run on their laptop to get it up to date.
As we add more services, such as email sending, image processing, video compression, it wont require anything special to configure from the front end perspective.
We're now moving it into the cloud and are now seeing the benefits of Kubernetes with our docker containers to deploy, have a blue/green deployment, not worry so much about what the servers are doing but focusing on deployments. This is very clever stuff and will reduce our hosting cost without having to duplicate infrastructure.
So yeah, I didn't get docker for a long time. It does depend on the types of apps your building though. My use case fits it perfectly.
Docker is surely not for everyone. If you don't get why you want to use Docker, chances are you don't need it.
Differently people use Docker differently, I see Docker could be useful for several purposes:
1) scalability
This is the main advertisement point for Docker I guess. Well it enables your stateless application to scale automatically.
2) being able to deploy without caring about dependencies
There are times we want to deploy something into customers' servers. We have no control over their machines' environments. But with Docker, the only thing I need to make sure is that they have it installed.
3) Infrastructure as code
All our source code is checked in git and version controlled nicely. When we need to run it again, just clone from git. How nice! But our infrastructure, meaning server formation and service relationship? Not so much. Before using Docker, we basically need to ssh into servers and install dependencies according to the documents. It was not only slow, but unreliable because we all know how documents can lie. With Docker and docker-compose, the infrastructure can be written in code and checked in git. Most importantly, code doesn't lie.
You can surely use other tools to achieve those. I'm not saying Docker is the only way. But I believe it's a good tool to consider when you have those needs.
Before using Docker, we basically need to ssh into servers and install dependencies according to the documents.
There's an entire cottage industry of not-docker tools for that. Not to take anything from docker but the options aren't 'docker' and 'the pre-docker dark ages where you ssh'ed into servers one by one and carefully typed commands from a faded printout'.
> There's an entire cottage industry of not-docker tools for that.
Not the least of which are the distros own packaging systems (e.g. rpm/yum or dpkg/apt). Full investment in the complexity of the ecosystem isn't even necessary for basic dependency handling.
On 2, perhaps I misunderstood how docker works but don’t you need to deploy a docker image that is binary compatible with the type and version of the host OS? Then don’t you just move the problem? (Two customers with different types of OS require you to recreate the docker images). And what if these customers need to upgrade their version of linux, do they need to contact all their software vendors to reissue new docker images?
I never understood how creating the tight dependancy between the host and the docker image is not a problem.
Well, you need to get a binary compatible docker image for the hardware. But frankly almost everyone uses x86-64 now, so this is not usually a problem. If these customer upgrade their version of Linux, the new Linux version usually comes with Docker, and you don't need to update the image.
I believe the running containers share the host's kernel, so maybe if you were doing certain low level kernel level stuff, maybe the version of the kernel matching would matter?
I'm a front end dev, and at my latest contract I wanted to help out our back end developers with some .NET stuff. Can you imagine trying to setup a full .NET stack on a Macbook a few years ago? This time what I had to do was:
* install Docker
* install Visual Studio Community and .NET Core
* run docker-compose ... up
Then I had their entire .NET software stack up and running, including MSSQL, ElasticSearch, Kibana, redis, all with one command.
I don't like doing DevOps personally, but I definitely appreciate how Docker helps me in situations like this!
You don't need docker to run a single program on a single computer... but the moment you either want to run multiple copies, or distribute it to multiple computers, particularly if they're not your computers, then being able to encapsulate stuff and reduce the number of dependencies is a huge benefit.
For example, it lets you run software on your computer without worrying about installing stuff that might affect other software, and without even having any interactions with other software aside from what you allow it.
Old timer to old timer: it's how the rest of the computing ecosystem re-invented Java after it got uncool to use Java. Just slam all your dependencies into an image and it can run anywhere [-]
[-] Until it has an external dependency and then it breaks, which is where docker turns out to be rather fragile by comparison.
Old timer to 2 other old timers: Docker does solve a problem, and this is quite close to it... We had a solution that we needed to install on servers for multiple clients. Before Docker, we had to make sure the libraries were correct (on CentOS, Debian, Ubuntu,... you name it), which was a nightmare. We were literally debugging on customers' installations to see why it broke there.
Docker solves this nicely. Apart from the kernel you take all dependencies with you.
In this case, it provides an easy installation of all the dependencies in a single command, while still allowing for customizations, and all in a standard way. Not to mention that certificates and similar are all solved.
Forget about hype and just try Docker. It's a tool like any other. It can be abused, sure, but it's useful too and it's here to stay.
EDIT: difference between Java and Docker is that containers depend on kernel capabilities, not on some 3rd party JAR maintainers. In my experience Docker simply works unless you are doing something really weird.
I've been using Docker for lots of things. To be honest, the dependency part has been helpful but still weak: we inevitably end up finding that our containers are built from things that go stale (apt-get sources, pip, broken URLs, stale SSL certs, etc). I guess there is a skill to doing it well so that this doesn't happen, but there you are - it didn't really solve the problem it's now an extra skill we need to have to solve a problem we had already mostly solved.
What has really sold me is more the orchestration side, with docker-compose allowing us to easily launch a whole fleet of independent services. I assume swarm and kubernetes etc., make this even better. It's really this that lets us decompose our software into finer grained independent components which has a number of other benefits. I don't think we could manage it otherwise.
You still need to do a frequent build to detect broken dependencies, but at least setting up a daily docker build in CI is so much simpler and faster (and cross-platform!) than doing this with vagrant or live systems. At least you'll catch problems a lot sooner.
If you want to protect yourself from breaking dependencies, mirror them locally, and use a network-isolated docker build to verify you've actually got everything you need.
The build process might break because of stale links, like you say, but already built images should continue to function everywhere unless you're doing something really weird.
Remember that docker = lxd/lxc =~ bsd jails/solaris zones =~~ enhanced sandboxing/isolation for processes (containerization).
When thinking about it that way, it's the pre-docker world that seems unnecessarily dangerous -- Why would you just let some random process run on your system with all it's dependencies and possibly cause havoc without containerizing it?
lxd/lxc containers are a tad more flexible than docker in my experience, notably because you tend to run a full distro in there instead of just a single app.
I agree -- I'm personally super excited to try my hand at using LXD[0] on Ubuntu, in particular I want to see just how awesome/well-isolated system images are. If everything I'm reading is right, lxd/lxc system containers are going to make running VMs seem outdated.
Unfortunately it's a little bit annoying because the lxd docs don't make it easy to find what they define as a "system image" -- do you have some links to more more good layouts? I actually found a good talk from DebConf '17[1].
I mostly rely on system images from Proxmox, they're fairly okayish to run even without PM.
I still need VMs for some things, especially when the kernel of the host is too old or missing modules I need or when I need to mount things via FUSE (or similar ops).
containers keep all that junk away from my system.
how many apps need node.js or ruby and stuff just to preprocess their css? ever managed some legacy php app? instead of putting that junk directly on my machine i just put it in a container. dont need it anymore? cool - docker rmi that shit
Yeah well maybe but some docker config file imply that you know how to configure the postgress, redis, node package manager, nginx and the SSL chain that live in the container by the means of a yaml config file that is in the end another custom layer of abstraction over the original config files. Of course some docker based projects are better than others.
I like to keep my servers free of garbage. Installing bajillion dependencies for every application you run kind of violates that. Bonus points for Python module management.
Ye I don't use it for scale. It's great just to have a declarative file which is easy to read and understand. I guess I use it like people use Puppet and Ansible, I just find it better for my use case. And I don't always have to rely on what's in the package manager for my OS. So I get a nice stable core OS, and up to date apps.
Have you used a Windows program that comes with all its dependencies (e.g dlls)? Have you used OS level isolation to deny some resources to some processes?
With 'built-in' imaging in LXD I think a lot of the use-cases LXC isn't quite as fluid for will become more realistic. (i.e. just download this image and run it)
For local dev environments I still think Vagrant (and pick whatever underlying provider you want) is a better solution.
docker is basically a reproducible lxc/lxd, but you can get very close to actually running lxc-like machines with docker
for me, docker actually solves the problem that lxc/openvz were trying to solve in a better way. we already had the data on separate volumes, and docker is a much nicer interface than bootstrapping and then updating a chroot for lxc
