I went through sweat and tears with this on different projects. People wanting to be cool because they use hype-train-tech ending up doing things of unbelievably bad quality because "hey, we are not that many in the team" but "hey, we need infinite scalability". Teams immature to the point of not understanding what LTS means have decided that they needed Kubernetes because yes. I could go on.
I currently have distilled, compact Puppet code to create a hardened VM of any size on any provider that can run one more more Docker services or run directly a python backend, or serve static files. With this I create a service on a Hetzner VM in 5 minutes whether the VM has 2 cores or 48 cores and control the configuration in source controlled manifests while monitoring configuration compliance with a custom Naemon plugin. A perfectly reproducible process. The startups kids are meanwhile doing snowflakes in the cloud spending many KEUR per month to have something that is worse than what devops pioneers were able to do in 2017. And the stakeholders are paying for this ship.
I wrote a more structured opinion piece about this, called The Emperor's New clouds:
I started my career in a world where we did everything using shell scripts running directly on bare metal servers, usually running Solaris, and later SuSe or RedHat. I never understood the "how would you reproduce your setup without Docker (or X, where X is some other technology)". The scripts were deterministic. The dependency versions were locked. The configurations were identical. The input arguments were identical. The order of execution was identical. It all ran on a deterministic computational device. How could it not be reproducible?
Well that's exactly the point! Creating complex cloud resources with, for instance, Terraform, is less reproducible than a shell script on an LTS system like Ubuntu or RHEL - that's because the cloud provider interfaces drifts and from time to time stops accepting the terraform manifests that previously worked. And to fix it, you have to interrupt your normal work for yet another unplanned intervention in the terraform code - this happened to my teams several times.
This does not happen with Puppet + Linux, because LTS distributions have a long release cycle where compatibility is not broken.
I tried to explain this topic in the article linked above. Not sure how far I succeeded.
Like 12 years of life cycle is not enough for you to plan a transition?
You can use the entire life cycle but not one is forcing you to. You can update from one LTS to another every 2 years, or 4 years, or 5 years... you decide.
I don't really think we're in disagreement here. The longer you wait, the harder the transition will be. LTS is a good foundation, and usually the right choice for "enterprise" or "business" settings, but you should not rely overmuch on any one LTS release's way of doing things, when the wider Linux ecosystem moves much faster.
The longer you wait the harder the pain. The less you wait the more frequent the pain. So it depends on the function that converts intensity and frequency to suffering :p But, most importantly, the fact that LTS gives you a choice is what I was highlighting.
For the scope I operate, which is pretty standard Linux packages (PostgreSQL, MariaDB, Nginx, Docker, OpenVPN, OpenSSH) the changes between 16.04 and 22.04 have been quite OK to deal with.
It's a tradeoff. Doing a big effort once every 4 or 5 years, vs a hopefully smaller effort every year. Sometimes the intermediate smaller steps help you move forward, sometimes it just means more migrations. Sometimes the software/hardware you need means you can't use a LTS OS at all.
If possible, it's nicer to pick established, mature software for as much of your stack as you can, so that there's less of a difference in APIs over longer time frames. But it's not always possible.
I went through the migration from CentOS 6 to 7 and never want to do anything like that again. The good news, I guess, is that it never will happen again: CentOS is basically dead anyway, and it's not likely that so many core pieces of system software will change that drastically anymore.
I can't imagine you leaned into any one of those releases, then. That sequence involves major changes to the kernel, the init system, the configuration management tools, the core libraries, Apache, Python, Perl, etc. Any one of those alone could (and did, in my experience) trigger a major rewrite of configuration and/or code.
I'm glad it was painless for you. In my experience, it was not, and most of the reasons were beyond my control.
Every major release of every major distribution makes choices. These are choices about what software to include in the first place, what versions of that software to pin (especially for LTS releases), what default configuration to provide, recommendations about how to solve certain problems, etc. These choices are made based upon the experience and opinions of the distribution maintainers. However, those maintainers are (usually) not major contributors to the software they're distributing. This means distros can make "bad" choices, choosing for example to focus on software that eventually dies out, or recommending configurations that eventually get deprecated or removed, etc. Sometimes, these choices are even made in a way such that they exclude what will become the winning alternative, leaving no migration path except complete and total overhaul.
If all Linux is to you is a place to run some application software, these choices are mostly irrelevant. As long as the software you care about continues to run, the other things are just picayune details. If this comes off as derisive, I apologize, because I'm actually broadly endorsing that view of things, as much as it is possible to achieve. But if you start really taking advantage of the things which the distribution provides out of the box and recommends, especially around large-scale multi-system operation, you end up buying into the distibution's choices. When a large organization you're a part of does it too, now the sunk costs really start to mount. As the Linux ecosystem continues to evolve, especially in different directions than the distribution chose at the time, the cost of migrating to later releases grows. This is all a good reason to me to not marry oneself so tightly to those particular choices, but that isn't always feasible with deadlines and compliance requirements and so on bearing down on the sysadmin.
There's also an even bigger problem that can arise, the distribution can just end, such as the termination of CentOS, leaving lots of people hanging. In that case, I know some who started to pay Red Hat for RHEL, but most seem to have moved on to other distros, like Ubuntu. That kind of migration has a lot of the same issues, too, once again leaving me to recommend not to lean into the particulars too much.
Using debian and bash and perl for setup and config. There is almost no work involved in the past decades; everything still works fine. I do not like busy work; trying to do things the hard way is not making me money or giving me happiness; running saas products (on non cloud cheap hardware that never dies) is and that's what I have done for the past 25 years.
There is no need to adapt things that work as they are already.
> But if you start really taking advantage of the things which the distribution provides out of the box and recommends, especially around large-scale multi-system operation, you end up buying into the distibution's choices.
You mean management interfaces and repo mirroring stuff provided by the OS vendor, like cockpitd and Satellite and whatever?
Sure, that's part of it, if those tools are used. Daemons like the particular flavor of syslog and cron are also part of it. Patched kernels used to be more common, too. I listed a bunch of things that actually broke for me before in a sibling thread; sometimes it was down to e.g. the Python packages that were in EPEL vs. the Python packages that were actually being maintained by their original authors in PyPI, or various security tools configured around paths that changed, etc. There were usually workarounds or alternatives, but they were more difficult to set up than doing things the "native" way.
I see! Thanks for referring to your sibling post, that definitely made clearer what you're talking about.
And yeah if you package stuff against, e.g., the Python libs included in the distro (or EPEL), you essentially need to maintain a repo as a downstream repo of the distro, then rebuild the whole repo with whatever subsequent release as a new upstream when it's time to upgrade. That kind of thing is doable but it's substantial integration work, and if it's aomething you do once a decade nobody is ever going to be fluent in it when it's time to be done.
I think I'd rather just maintain two repos— one against the latest stable release and one against the upstream rolling release (Fedora Rawhide, Debian Unstable, openSUSE Factory or Tumbleweed, etc.)— and upgrade every 6 months or whatever than leap the wider chasms between LTS releases.
And yeah the Python and Python libs shipped in a distro are generally there for the distro's integration purposes, which may involve different goals and constraints than app developers usually have. Building against whatever a distro ships with is not always the best way, as your painful migrations demonstrated.
> There's also an even bigger problem that can arise, the distribution can just end, such as the termination of CentOS
If you are doing something serious you probably want to chose suppliers in such a way that you can demonstrate you have security and business continuity under control. That means you probably want to use RHEL, Suse or Ubuntu, distributions for which commercial support exists.
(Ubuntu is particularly interesting because you can start with an LTS release for free and activate commercial support if business goes well, without changing your processes.)
You can think about this beforehand or wait until customers require some kind of certification and the auditors ask you for your suppliers list + the business continuity plan, among other things. You will face this if you deliver to a regulated market or if your customers are large enough to self regulate this kind of thing.
LTS not good enough? Well, cloud native does not have LTS comittement and Pipy does not provide security fixes separated from logical changes.
Try to keep your Terraform code stable for two years in AWS, or try to understand the lifecycle of AWS Glue versions from the docs. Or trust that Google will not discontinue their offers :-)
I mean, maintaining software is never easy or effortless but I respect the effort done by LTS Linux providers - they sell stability and security for a fraction of what you pay for cloud native.
What is it that people do that breaks so often due to lack of backwards compatibility from the OS?
IMO, the lure of an LTS is that you don't need to keep testing if your computer is still working every week when a set of updates come. Not that things that your software depends on the details remain frozen. If your software depends on the details of something, you should add it as a dependency.
The bigger problem IMO is not that things break, it's that if you depend on one LTS release too heavily, and you wait too long to migrate from one LTS to another, everything breaks all at once.
What should be a gradual migration as new things develop turns into a singular nightmare.
What are you depending on the OS that isn't extremely backwards compatible?
Once in a decade you get something like a breaking upgrade of nginx, or the glibc debacle of 2003. That may take a person-week to fix[1], what can hardly be called "herculean".
1 - If you go with 1 person * 1 week, if you try to go with 7 people * 1 day, it will suddenly cost 7 person-weeks. But the only way upgrading is such a hurry is if you borked a lot of things prior to it.
Off the top of my head, some of the things that have broken at an LTS transition that I've been involved with are out-of-tree kernel module builds, C code using OpenSSL, Puppet config, Salt config, RPM specfiles, Python code, Perl code, Apache configs, shell scripts, Java code, bootloader configs, bootstrap scripts, and init scripts/configs (esp. sysvinit to systemd). Any one of these things is not a problem in isolation, the problem is due to having to fix all of them all at once. Too much complexity put into any one of them (often arising from external requirements or rushed implementations) also makes migrating harder. Waiting until the 11th hour on the EOL clock just adds to the stress of the process.
Many of my bad experiences were because of corporate policies and lack of proper prioritization at levels above system administration. However, the sysadmin does have some choice in the matter, especially when greenfielding. You can turn stability into a vice if you're not careful.
You said it: Your versions were locked. Therefore it is not constantly up-to-date.
I was pinched myself: Security.
- With the cloud threats, everything needs to be constantly up-to-date. Docker images make it easier than permanent servers that need to be upgraded. We used to upgrade every week, now we’re upgraded by default. So yes, sometimes our images don’t start with the latest version of xyz. But this is rare, downgrade is easy with Docker, and reproduction on a dev engine easier.
- With the cloud threats, everything needs to be isolated. Docker makes it easy to have an Alpine with no other executable than strictly necessary, and only open ports to the required services.
I hate the cloud because 4GB/2CPU should be way enough to run extremely large workloads, but I had to admit that convenience made me switch.
We did upgrades periodically, each time a conscious choice after reviewing the release notes of the dependency. Occasionally a script would need to be updated, but that was it.
To be fair there's real issues with this approach, too. For example, shell scripts aren't actually very portable. GNU awk vs nawk vs... multiply that by all your tools, and yeah those scripts don't run deterministically (they rely too much on the environment). This alone was a big reason why systemd exists today.
But there's a middle ground here too. To me there's a HUGE gap between Kubernetes distributed systems and shell script free for all.
reproducibility isn't just on your deployments, it's for development too. got old REAL fast when your fancy build doesn't work the same on every devs device or some one off issue with how your dev has setup their environment steals hours from everyone.
it was a big reason why we moved to containers at the bare minimum, because its quick and easy to spin up and destroy and you are guaranteed what runs locally runs on prod. no more "well it worked on my system!".
>reproducibility isn't just on your deployments, it's for development too
Absolutely. Adhoc configurations should be forbidden! It is easy to ensure dev env reproducibility when you run Linux. If you have config management your devs can have VMs that subscribe to the same exact configuration that the staging prod and dev environments have. They can literally have a deplpyment server in their machine, as a VM. Since the configuration is stored on a server and applied continuously, it is hard to screw it.
You can achieve this with Docker as well, if the arrangement is not too complex.
The problem, at least in my experience, comes when you start depending on several cloud native components where local emulations are always different from the real cloud env in tiny details that are going to screw the deploys over and over.
Wouldnt there be slight differences in different Unix flavors so that the script couldnt run in all of them? If it only worked on Solaris, what would happen if Solaris retired? (Like what happened to Centos)
POSIX never specified things like disk partitioning or package management, so this still requires something else to give you a working system in the first place.
You will likely have to adapt your scripts for OS-specific or installation-specific tasks like package management and modifying filesystems. In the past I've used Nix (either via `nix run` and `nix shell` or templating in Nixpkgs' `writeScript` or similar) for this stuff to guarantee that I'm always running the same tools regardless of what's installed on the base system. This can free you up to use a different shell, rely on recent features of Bash, use GNUisms in coreutils, sed, grep, find, etc., fix a specific version of jq, use external templating tools, etc. For systemd-based distros, you can even use Nix to manually install system-wide services: just install a package to the default or system profile, and then symlink the included unit files from the profile (not the direct store path) into /etc/systemd. `systemctl daemon-reload` and you can manipulate them in all the usual ways one would with systemd.
Other Unix distros don't have first-class support with Nix so you may need to take some additional care when working out your script (especially the part of it that installs Nix), but if you don't need to set up services this way you can write portable scripts with few limitations that will work across all Linux distros, macOS, probably FreeBSD and maybe NetBSD.
I've never been so lucky as to work at a place that used any Unix flavors other than Linux and macOS, though.
Depends on where you are in the ecosystem. If you're running your own service, the only flavors that matter and the ones you're using.
If all my machines are FreeBSD 4.11, I don't care if my scripts don't run on Linux or Solaris or SCO or even FreeBSD 4.8 or 14. I might care someday, but not today.
Maintenance scripts need to run on all the versions in the fleet (usually), but setup scripts can often be limited to the latest version, because why not use the latest OS if you're setting up a new machine.
If you're distributing software, yeah you've got to support a lot of variation. If you're at a shop that runs lots of different flavors, you have to support lots of variation. But a lot of people just pick a flavor and update the scripts as needed when the flavor of the day changes.
Trying to keep dependencies and running services as tight and small as possible helps a lot with keeping up to date on security. Don't need to update things that aren't installed, and may not need to update things that are installed but not running (but sometimes you do).
I feel like Kubernetes is always randomly mentioned in rants like this. Instead of saying your hardened VM has Docker you could have just said it has kubelet on it. Then instead of a bunch of ad hoc "docker services" you could pay pennies for a k8s control plane that gives you control over everything on those VMs. I fail to see how your way is anything but worse.
The bad cloud infrastructure is when people try to use every single thing AWS sells and their whole infrastructure is at super high levels of abstraction that they could never migrate to another platform. K8s isn't that at all.
Unfortunately in air-gapped systems you cannot simply pay pennies for a managed k8s platform. In these cases you have to bootstrap and manage k8s on your own in your data centers. While I do not think bootstrapping and managing a cluster is difficult at all (especially if you only handle stateless workloads) it may still not fit or integrate well with a companies overall management infrastructure.
While I am a happy cloud infrastructure user in private, I have to go through some extra hoops to deploy applications at work, regardless of if k8s is used or not.
In think in either case, if you already have code that's done, using that is going to be less effort than switching.
However, I ran kubeadm on a hetzner server and it's just sat chugging along forever basically. I use the cluster to run ephemeral apps where I build and deploy 1 golang service, a couple of node services in about 60 seconds ( with cache, obviously ).
As someone old enough and skilled enough to do the same with puppet, why bother when it's simpler easier that even the kids who don't understand TLS can do it with k8s?
With k8s you get a way of saying 'WHAT YOU WANT' without 'HOW TO DO IT', and this is applies not only to the actual infra aspect, but the people maintaining it too. Any cloud platform and devops worth their salt can maintain a k8s system. Good luck finding someone to understand what that 'custom Naemon' plugin is doing.
> Good luck finding someone to understand what that 'custom Naemon' plugin is doing.
You Kubernetes people get triggered very easily. I was already lucky to have found several juniors that worked in this kind of thing with minimal training. The 'custom Naemon plugin' is 30 lines of bash and you can adapt it to any monitoring system.
Of course this is scary and complicated. I might consider switching to 'Kubernetes operators', which sounds simpler :-)
I've done all of this and then some. I used to deploy websites by FTPing into the server and copying files. Then it was bash scripts, then Ansible. IMO Kubernetes hits a very good level of abstraction. You can totally deploy 30 lines of bash to every server, you just have to wrap it in a docker container. That's all k8s asks for for a workload. You don't have to use operators. That would be something to explore much later. Honestly I just think you should be more generous and not assume people have created this stuff just for fun. K8s really does address real problems around deployment and it's very well thought out.
To be fair in other comments OP made an effort not to get involved into those endless Kubernetes vs VM discussions. However either side eventually posts a snarky comment and there goes.
I think everyone just has to acknowledge that there are use cases for both. Also Kubernetes and "classic" configuration management via Ansible (or others) are orthogonal to each other. So these discussions are somewhat misguided in the first place.
For example: you might want to deploy a VM or auto-install and configure a physical machine with custom tooling and something like Ansible or Puppet and _then_ configure said machine as a Kubernetes node that handles the actual workloads. In other cases some Dev might want to install and run an application without the k8s layer using Nginx as a webserver. In this case, too, Puppet/Ansible might or might not be involved in configure the application but only handle the "OS layer" if there is such a thing. And in yet other cases you get away with a simple cloud-init script that makes your machine a k8s node and leave out other configuration management tools altogether.
Guess what: All of this is fine. Evaluate solutions based on what you need, not what other people working in giant corporations urge you to use. And then go and build it, ideally having fun doing it.
Representing either tool as a one-size-fits-all is misguiding at best and seems to be overly simplistic to the complex problem of deploying your applications.
> Honestly I just think you should be more generous
I am generous in the context for generosity. Turns out that engineering is not about being generous but rather about choosing the most efficient solution for problems that in the end need to be business driven. This requires evaluating requirements, context and tradeoffs. That takes a cold, rational mind more than generosity.
> K8s really does address real problems around deployment and it's very well thought out
It's great where it makes sense. It's less than great elsewhere.
Not everything is SaaS, not everything needs scaling, not everything needs 99.99% of uptime, not everything needs a CDN, not every company is VC backed operating at high risk / high reward, etc, etc. Context is better than ideology. If you read the article I posted you will see that stated clearly.
I completely agree that most people don't need that. This is always what people say when k8s comes up. This is also what people said about git 15 years ago (you're not the kernel etc). But the thing is you don't have to use any of the bits you don't need. At first I listened to the naysayers and was wary of k8s thinking it would create more problems than it solves. That simply hasn't been the case. It's not a no-brainer, there are tradeoffs, but I really think it makes sense especially if you're doing docker anyway. Like I said in another comment, people tend to talk about two different things. There's k8s which can be as little as just a single node k3s server which is basically docker compose with a few extras like automatic rollout etc. Then there's the over the top "cloud native" stuff. One does not imply the other.
The questions are short but the answers would be long. Puppet manages all fine grained OS resources (files, dirs, repos, cronjobs, sudo declarations, firewall rules, etc) and you aggregate those resources into classes which are then pushed to different machines. The classes are parametrizable for the differences between systems.
If I was to write an idempotent script for each native resource I would finish in some years :-)
You chose whatever monitoring system you like the most.
For offline nodes you use whatever the level of criticity of your node justifies. This is something people struggle to understand: not every business needs 99.99% uptime. That said, I never had a downtime in Hetzner. On Digital ocean I had one short forced reboot in 4 years. YMMV so protect yourself as much as necessary.
Deploying on a different provider than Hetzner is the same as deploying on Hetzner except the part of launching the machine which is trivial to script - the added value is making the machine work and Ubuntu/Debian/RHEL are the same everywhere. You don't have vendor lock in with this.
If K8s works for you, enjoy it. Nobody is telling you to stop :-)
Serious question for you, why use Docker at all? You can just get rid of the clunky overhead.
You mentioned Python backend, so literally just replicate build script, directly in VPS: "pip install requirements.txt" > python main.py" > nano /etc/systemd/system/myservice.service > systemd start myservice > Tada.
You can scale instances by just throwing those commands in a bash script (build_my_app.sh) = You're new dockerfile...install on any server in xx-xxx seconds.
I mentioned Docker because it interests many developers but on VMs that I control I do not need Docker at all. Deploying with Docker provides host OS independence which is nice if you are distributing but unnecessary if the host is yours, running a fixed OS.
For Python backends I often deploy the code directly with a Puppet resource called VcsRepo which basically places a certain tag of a certain repo on a certain filesystem location. And I also package the systemd scripts for easy start/stop/restart. You can do this with other config management tools, via bash or by hand, depending on how many systems you manage.
What bothers me with your question is Pip :-) But perhaps that is off topic...?
s/host OS independence/a certain level of host OS independence
And getting containers to run depends on the OS - if you don't control the host, leads to major ping-pongs.
Even within Linux (Ubuntu, Debian, RHEL, etc) when you are distributing multiple related containers there are details to care about, not about the container itself but about the base OS configuration. It's not magic.
OP is talking about substituting a Kubernetes setup. FreeBSD was never in the cards. 99% of companies in the cloud don’t run or care about anything other than Linux.
If you really want to open that can of worms, here it goes:
Pipy is an informal source of software that has low security levels and was infested with malware many times over the years. It does not provide security updates: it provides updates that might include security-related changes as well as functional changes. Whenever you update a package from there, there is a chain reaction of dependency updates that insert untested code in your product.
Due to this, I prefer to target an LTS platform (Ubuntu LTS, Debian, RHEL...) and adapt to whatever python environment exists there, enjoying the fact that I can blindly update a package due to security (ex: Django) without worrying that it will be a new version which could break my app. *
Furthermore, with Ubuntu I can get a formal contract with Canonical without changing anything on my setup, and with RHEL it comes built-in with the subscription. Last time I checked Canonical's security team was around 30pax (whereas Pipy recently hired their first security engineer). These things provide supply-chain peace of mind to whoever consumes the software, not only to who maintains it.
I've just doubled down on "making my own Debian packages".
There's tons of examples, you are learning a durable skill, and 90% of the time (for personal stuff), I had to ask myself: would I really ever deploy this on something that wasn't Debian?
Boom: debian-lts + my_package-0.3.20240913
...the package itself doesn't have to be "good" or "portable", just install it, do your junk, and you don't have to worry about any complexity coming from ansible or puppet or docker.
However: docker is also super nice! FROM debian:latest ; RUN dpkg -i my_package-*.deb
I don't mean this as a rebuttal, but rather to add to the discussion. While I like the idea of getting rid of the Docker layer, every time I try to I run into things that remind me why I use Docker:
1. Not needing to run my own PPA server (not super hard, it's just a little more friction than using Docker hub or github or whatever)
2. Figuring out how to make a deb package is almost always harder in practice for real world code than building/pushing a Docker container image
3. I really hate reading/writing/maintaining systemd units. I know most of the time you can just copy/paste boilerplate from the Internet or look up the docs in the man pages. Not the end of the world, just another pain point that doesn't exist in Docker.
4. The Docker tooling is sooooo much better than the systemd/debian ecosystem. `docker logs <container>` is so much better than `sudo journalctl --no-pager --reverse --unit <systemd-unit>.service`. It often feels like Linux tools pick silly defaults or otherwise go out of their way to have a counterintuitive UI (I have _plenty_ of criticism for Docker's UI as well, but it's still better than systemd IMHO). This is the biggest issue for me--Docker doesn't make me spend so much time reading man pages or managing bash aliases, and for me that's worth its weight in gold.
Yuuup! I'm super-small time, so for me it's just `scp *.deb $TARGET:.` (no PPA, although I'm considering it...)
Really, my package is currently mostly: `Depends: git, jq, curl, vim, moreutils, etc...` (ie: my per-user "typically installed software"), and I'm considering splitting out: `personal-cli`, `personal-gui` (eg: Inkscape, vlc, handbrake, etc...), and am about to have to dive in to systemd stuff for `personal-server`, which will do all the caddy, https, and probably cgi-bin support (mostly little home automation scripts / services).
I'm 100% with you w.r.t. the sudo journalctl garbage, but if you poke at cockpit https://www.redhat.com/sysadmin/intro-cockpit - it provides a nice little GUI which does a bunch of the systemd "stuff". That's kindof the nice tag-along ecosystem effects of "just be a package".
I'm definitely relatively happy with docker overall, but there's useful bits in being more closely integrated with the overall package system management (apt install ; apt upgrade ; systemctl restart ; versions, etc...), and the complexity that you learn is durable and consistent across the system.
In situations at work where we use something as an alternative to Docker as a deployment target, it's Nix. That has its own problems and we can talk about them, but in the context of that alternative I think some of your points are kinda backwards.
> 1. Not needing to run my own PPA server (not super hard, it's just a little more friction than using Docker hub or github or whatever)
Docker actually has more infrastructure requirements than alternatives. For instance, we have some CI jobs at work whose environments are provided via Nix and some whose environments are provided by Docker. The Docker-based jobs all require management of some kind of repository infrastructure (usually an ECR). The Nix-based jobs just... don't. We don't run our own cache for Nix artifacts, and Nix doesn't care: what it can find in the public caches we use, it does, and it just silently and transparently builds whatwver else it needs (our custom packages) from source. They get built just once on each runner and then are reused across all jobs.
> 2. Figuring out how to make a deb package is almost always harder in practice for real world code than building/pushing a Docker container image
Definitely depends on the codebase, but sure, packaging usually involves adhering to some kind of discipline and conventions whereas Docker lets you splat files onto a disk image via any manual hack that strikes your fancy. But if you don't care about your OCI images being shit, you might likewise not care about your DEB packages being shit. If that's the case, you can often shit out a DEB file via something like fpm with very little effort.
> 3. I really hate reading/writing/maintaining systemd units. I know most of the time you can just copy/paste boilerplate from the Internet or look up the docs in the man pages. Not the end of the world, just another pain point that doesn't exist in Docker.
> 4. The Docker tooling is sooooo much better than the systemd/debian ecosystem. `docker logs <container>` is so much better than `sudo journalctl --no-pager --reverse --unit <systemd-unit>.service`. It often feels like Linux tools pick silly defaults or otherwise go out of their way to have a counterintuitive UI (I have _plenty_ of criticism for Docker's UI as well, but it's still better than systemd IMHO). This is the biggest issue for me--Docker doesn't make me spend so much time reading man pages or managing bash aliases, and for me that's worth its weight in gold.
I don't really understand this preference; I guess we just disagree here. Systemd has been around for like a decade and a half now, and ubiquitous for most of that time. The kind of usage you're talking about is extremely well documented and pretty simple. Why would I want a separate, additional interface for managing services and logs when the systemd stuff is something I already have to know to administer the system anyway? I also frequently use systemd features that Docker just doesn't have, like automatic filesystem mounts (it can do some things fstab can't), socket activation, user services, timers, dependency relations between units, descri ing how services that should only come up after the network is up, etc. Docker's tooling really doesn't seem better to me.
> Docker actually has more infrastructure requirements than alternatives.
I was mostly comparing Docker to system packages, and I was specifically thinking about how trivial it is to use Docker Hub or GitHub for image hosting. Yeah, it's "infrastructure", but it's perfectly fine to click that into existence until you get to some scale. I would rather do that than operate a debian package server. Agreed that Nix works pretty well for that case, and that it has other (significant) downsides. I'm spiritually aligned with Nix, but Docker has repeatedly proven itself more practical for me.
> Definitely depends on the codebase, but sure, packaging usually involves adhering to some kind of discipline and conventions whereas Docker lets you splat files onto a disk image via any manual hack that strikes your fancy. But if you don't care about your OCI images being shit, you might likewise not care about your DEB packages being shit. If that's the case, you can often shit out a DEB file via something like fpm with very little effort.
I'm not really talking about "splatting files via manual hack", I'm talking about building clean, minimal images with a somewhat sane build tool. And to be clear, I really don't like Docker as a build tool, it's just far less bad than building system packages.
> don't really understand this preference; I guess we just disagree here. Systemd has been around for like a decade and a half now, and ubiquitous for most of that time.
Yeah, I don't dispute that systemd has been around and been ubiquitous. I mostly think it's user interface is hot garbage. Yes, it's well documented that you can get rid of the pager with `--no-pager` and you can put the logs in a sane order with `--reverse` and that you specify the unit you want to look up with `--unit`, but it's fucking stupid that you have to look that stuff up in the man pages at all never mind type it every time (or at least maintain aliases on every system you operate) when it could just do the right thing by default. And that's just one small example, everything about systemd is a fractal of bad design, including the unit file format, the daemon-reload step, the magical naming conventions for automatic host mounts, the confusing and largely unnecessary way dependencies are expressed, etc ad infinitum.
> Why would I want a separate, additional interface for managing services and logs when the systemd stuff is something I already have to know to administer the system anyway?
I mean, first of all I'm talking about my preferences, I'm not trying to convince you that you should change, so if you know and like systemd and you don't know Docker, that's fine. And moreover, I hate that I have to choose between "an additional layer" and "a sane user interface", but having tried both I've begrudgingly found the additional layer to be the much less hostile choice.
> I also frequently use systemd features that Docker just doesn't have, like automatic filesystem mounts (it can do some things fstab can't), socket activation, user services, timers
Yeah, I agree that Docker can't do those things. I'm not even sure I want it to do those things. I'm talking pretty specifically about managing my application processes. But yeah, since you mention it, fstab is another technology that has been around for a long time, is ubiquitous, and is still wildly, unnecessarily hostile to users (it can't even do obvious things like automounting a USB device when it's plugged in).
> ... dependency relations between units, descri ing how services that should only come up after the network is up, etc. Docker's tooling really doesn't seem better to me.
Docker supports dependency relations between services pretty well, via its Compose functionality. You specify what services you want to run, how to test their health, and how they depend on each other. You can have Docker restart them if they die so it doesn't really matter if they come up before the network (but I've also never had a problem with Docker starting anything before the network comes up)--it will just retry until the network is ready.
Docker's tooling is better in its design, not necessarily a more expansive featureset. It has sane defaults, so if you do `docker logs <container>` you get the logs for the container without a pager and sorted properly--you don't need to remember to invoke `sudo` or anything like that assuming you've followed the installation instructions. Similarly, the Compose file format is much nicer to work with than editing systemd units--I'm not huge fan of YAML, but it's much better than the INI format for the kind of complex data structures required by the domain. It also doesn't scatter configs across a bunch of different files, it doesn't require a daemon-reload step, the files aren't owned by root by default, they're not buried in an /etc/systemd/system/foo/bar/baz tree by default, etc.
Like I said, I don't think Docker is perfect, and I have plenty of criticism for it, but it's far more productive than dealing with systemd in my experience.
Hey I missed this reply at the time but I wanted to say thanks for your thoughtful reply, although I don't have the energy or time for much of an answer at the moment. You raise some good points, and I better understand where you're coming from now even though my preferences are just different in some cases.
This is the way. And truthfully if you can learn to package for Debian, you already know how to package for Ubuntu and you can easily figure out how to package for openSUSE or Fedora or Arch.
How is this different than a Dockerfile that is creating the venv? Just add it to beginning, just like you would on localhost. But that is why I love to code Python in PyCharm, they manage the venv in each project on init.
We have a simple cloud infrastructure. Last year, we moved all our legacy apps to a Docker-based deployment (we were already using Docker for newer stuff). Nothing fancy—just basic Dockerfile and docker-compose.yml.
Advantages:
- Easy to manage: we keep a repo of docker-compose.yml files for each environment.
- Simple commands: most of the time, it’s just "docker-compose pull" and "docker-compose up."
- Our CI pipeline builds images after each commit, runs automated tests, and deploys to staging for QA to run manual tests.
- Very stable: we deploy the same images that were tested in staging. Our deployment success rate and production uptime improved significantly after the switch—even though stability wasn’t a big issue before!
- Common knowledge: everyone on our team is familiar with Docker, and it speeds up onboarding for new hires.
I think a lot of (justifiable) Docker use comes out of being forced to use other tools & ecosystems that are fundamentally messy and not really intended for galactic-scale enterprise development.
I have found that going all-in with certain language/framework features, such as self-contained deployments, can allow for really powerful sidestepping of this kind of operational complexity.
If I was still in a situation where I had to ensure the right combination of runtimes & frameworks are installed every time, I might be reaching for Docker too.
Python, Ruby, and to a much larger extent PHP are the Docker showcase!
For example, if you have a program that uses wsgi and runs on python 2.7, and another wsgi program that runs on python 3.16, you will absolutely need 2 different web servers to run them.
You can give different ports to both, and install an nginx on port 80 with a reverse proxy. But software tends to come with a lot of assumptions that make ops hard, and they will often not like your custom setup... but they will almost certainly like a normal docker setup.
Yeah, not caring about systemd is a big win for me. And I don't just mean the cryptic systemd unit syntax, but also the absolutely terrible ux of every CLI tool in the suite. I'm tired of having to pass half a dozen flags every time I want to view the logs of a systemd unit (or forgetting to type `sudo` before `systemctl`). I'm tired of having to remember the path to the systemd unit files on each system whenever I need to edit the files (is it `etc/systemd/system/...` or `etc/system/systemd/...`?). Docker is far from perfect, but at least it's intuitive enough that I don't have to constantly reference man pages or manage aliases.
I would love to do away with the Docker layer, but first the standard Linux tooling needs to improve a lot.
I am under the impression that those using Docker are those using shitty interpreted languages that fail hard on version incompatibilities, with Docker being used for version isolation as a workaround. How would a bash script help?
I deploy to pared down bare metal, but I use containerization for development, both local and otherwise, for me and contributors.
So much easier than trying to get a local machine to be set up identically to a myriad of servers running multiple projects with their idiosyncratic needs.
I like developing on my Qubes daily driver so I can easily spin up a server imitating vm, but if I’m getting your help, especially without paying you, then I want development for you to be as seamless as possible whatever your personal preferred setup.
Once you do it for long enough it might be worth it to consider configuration management where you declare native structured resources (users, firewall rules, nginx reverse proxies, etc) rather than writing them in shell.
I use Puppet for distribution of users, firewall rules, SSH hardening + whitelisting, nginx config (rev proxy, static server, etc), Let's Encrypt certs management + renewal + distribution, PostgreSQL config, etc.
The profit from this is huge once you have say 20-30 machines instead of 2-3, user lifecycle in the team that needs to be managed, etc. But the time investment is not trivial - for a couple of machines it is not worth it.
Honestly not having to use Puppet or Ansible are among my reasons for using Docker. I do some basic stuff in cloud-init (which is already frustrating enough) to configure users, ssh, and docker and everything else is just standard Docker tooling.
The point of this discussion is clear: complexity adds extra ops work, so the gains obtained from additional complexity need to compensate for that extra work.
Detailed config management has a learning curve and pays off only from a certain fleet size on.
Dedicated hardware pay off at a larger scale.
Complex cloud native arrangements pay off when... [left as an exercise for the reader].
Yeah, I mean you're kind of eating shit either way. You either have to deal with cloud friction or Linux friction, and at least cloud friction is mostly stuff like IAM where the friction is mostly about nudging you toward a better security posture. In Linux the friction is boring stuff like "every component has a different configuration file format and different expectations about where those configuration files are on disk and where it keeps its application data and how it names its command line arguments" and so on.
This isn't bad for a small team, but it becomes increasingly painful as you scale, but it's really hard to make it work smoothly for bigger teams (the sysadmin team becomes a bottleneck for everyone's deployment, deployments slow to a crawl so everyone builds these enormous, buggy releases, testing becomes a once-a-month thing instead of a continuous thing, etc). And the teams that do it well basically end up reinventing a big chunk of the cloud without any of the benefits of a standard, well-documented, widely-understood cloud platform anyway.
Because it seems unobvious but docker always saves you. It's actually quicker than running pip install requirements.txt once you get a year in. (Trust me, I used to take your approach).
Forget about "clunky overhead" - the running costs are < 10%. The dockerfile? You don't even need one. You can just pull from the python version you want e.g. Python1.11 and git pull you files from the container to get up and running. You don't need to use container image saving systems, you don't need to save images, or tag anything, you don't need to write set up scripts in the docker file, you can pass the database credentials through the environment option when launching the container.
The problem is after a year or two you get clashes or weird stuff breaking. And modules stopping support of your python version preventing you installing new ones. Case in point, Googles AI module(needed for gemini and lots of their AI API services) only works on 3.10+. What if you started in 2021? Your python - then cutting edge - would not work anymore, it's only 3.5 years later from that release. Yeah you can use loads of curl. Good luck maintaining that for years though.
Numpy 1.19 is calling np.warnings but some other dependence is using Numpy 1.20 which removed .warnings and made it .notices or something
Your cached model routes for transformers changed default directory
You update the dependencies and it seems fine, then on a new machine you try and update them, and bam, wrong python version, you are on 3.9 and remote is 3.10, so it's all breaking.
It's also not simple in the following respect: your requirements.txt file will potentially have dependency clashes (despite running code), might take ages to install on a 4GB VM (especially if you need pytorch because some AI module that makes life 10x easier rather needlessly requires it).
life with docker is worth it. i was scared of it too, but there are three key benefits for the everyman / solodev:
- Literally docker export the running container as a .tar to install it on a new VM. That's one line and guaranteed the exact same VM, no changes. That's what you want, no risks.
- Back up is equally simple; shell script to download regular back ups. Update is simple; shell script to update git repo within the container. You can docker export it to investigate bugs without affecting the production running container, giving you an instant local dev environment as needed.
- When you inevitably need to update python you can just spin up a new VM with the same port mapping on Python 3.14 or whatever and just create an API internally to communicate, the two containers can share resources but run different python versions. How do you handle this with your solution in 4 years time?
- If you need to rapidly scale, your shell script could work fine, I'll give you that. But probably it takes 2 minutes to start on each VM. Do you want a 2 minute wait for your autoscaling? No you want a docker image / AMI that takes 5 seconds for AWS to scale up if you "hit it big".
Sorry, but you've got no idea what you're talking about.
You can also run OSI images, often called docker images directly via systemds nspawn. Because docker doesn't create an overhead by itself, its at its heart a wrapper around kernel features and iptables.
You didn't need docker for deployments, but let's not use completely made up bullshit as arguments, okay?
I have no idea what I am talking about? Docker is literally adding middleware between your Linux system and app.
That doesn't necessarily mean there aren't Pro's to Docker, but one Con to Docker is - it's absolutely overhead and complexity that is not necessary.
I think one of the most powerful features of Docker by the way is Docker Compose. This is the real superpower of Docker in my opinion. I can literally run multiple services and apps in one VPS / dedicated server and have it manage my network interface and ports for me? Uhmmm...yes please!!!! :)
Docker's runtime overheads on Linux are tiny. It's pretty much all implemented using namespaces, cgroups and mounts which are native kernel constructs.
Well designed, written and efficient...middleware. It's a wrapper around linux and a middle between my OS and my app! A spade is a spade.
There are cons beyond performance. For example Docker complexity - you need to learn a new filetype, a new set of commands, a new architecture, new configurations, spend hours reading another set of documentation. Buy and read another 300 page O'Reily book to master and grasp something that again has Pro's and Con's.
For me? It's not necessary and I even know some Docker Kung-Fu but choose not to use it. I do use Docker Desktop occasionally to run apps and services on my localhost - it's basically a Docker Compose UI, and I really enjoy it.
> It's a wrapper around linux and a middle between my OS and my app
No. Docker doesn't "wrap" anything, and it certainly does not wrap Linux. Please reconsider looking at the documentation. It uses native kernel features. SystemD does a similar thing.
> For example Docker complexity - you need to learn a new filetype, a new set of commands, a new architecture, new configurations, spend hours reading another set of documentation
Yes, but containers do not incur overhead because of the daemon. It is there for management purposes. In other words, system calls / network access / etc are not going "through" the daemon.
> Docker is literally adding middleware between your Linux system and app.
Not really, no. Docker just uses functionality provided by the Linux kernel for its exact use case. It's not like a VM.
> it's absolutely overhead and complexity that is not necessary.
This is demonstratively wrong. Docker introduces less complexity compared to system native tools like Systemd or Bash. Dockerfiles will handle those for you.
> I have no idea what I am talking about
I wouldn't say that. You seem to have strong puritarian opinions tough.
>have it manage my network interface and ports for me
...and bypass the host firewall by default unless you explicitly bind stuff to localhost :-/
I don't particularly love or hate docker, but when I realized this, I decided to interact with it as little as possible for production environments. Such "convenient" defaults usually indicate that developers don't care about security or integrating with the rest of the system.
A lot of system applications on a standard Linux machine run as root or run with rootful permissions. This problem is solved by sandboxing, confining permissions and further hardening.
> requires sudo to use
Yes. However, this is a security plus and not a disadvantage.
> turns off all system firewalls
This statement makes no sense.
> has no way of doing security updates for containers.
There isn't a "Docker runtime", and the daemon is not a runtime any more than systemd is a runtime. They're both just managing processes. If you want to argue that Docker containers have an overhead, you could maybe argue that the Linux kernel security features they employ have an additional overhead, but that overhead is likely to be marginal compared to a less secure approach and moreover since you're Very Concerned About Security™ I'm sure you would prefer to pay the security cost.
Duplicating a base Linux distribution a thousand times for every installed piece of software absolutely is overhead.
(Theoretically you could build bare images without pulling in Alpine or Ubuntu, but literally almost nobody ever does that. If you have the skills to build a bare Docker image then you don't need Docker.)
> Duplicating a base Linux distribution a thousand times for every installed piece of software absolutely is overhead.
You're not duplicating an entire distribution, just the user land that you want. Typically we use minimal user lands that just have certs and /etc/passwd and maybe `sh`. And to be clear, this is mostly just a disk overhead, not a CPU or memory performance overhead.
> Theoretically you could build bare images without pulling in Alpine or Ubuntu, but literally almost nobody ever does that
Yeah, we do that all the time. Google's "distroless" images are only about 2MiB. It's very commonly used by anyone who is remotely concerned about performance.
> If you have the skills to build a bare Docker image then you don't need Docker.
Building a bare Docker image isn't hard, and the main reason to use Docker in a single-host configuration is because Docker utilities are just far, far saner than systemd utilities (and also because it's just easier to distribute programs as a Docker images rather than having to deal with system package repos and managers and so on).
I'm with you, but for me Cloud does have one major benefit:
If you use it as IaaS, it's a lot quicker to get prototypes working than if you use anything else, including VPS's from other providers.
Google Cloud in particular has very few vectors for lock-in, and follows more principle of least surprise.
But once you have prototyped, you should ask the question about rebuilding it somewhere that is cheaper.
Near infinite scalability of disk drives is nice, and snapshotting, and cloud in general can allow you to extend your prototype into taking production load and allowing you to measure what you will need; but leaning in to "cloud magick" (cloud run, lambdas, etc) will consume almost as much time to learn and debug as just doing it the old school way anyway. In my lived experience.
I am not against the cloud. VMs are also cloud, unless you run them on your own servers. For instance, the Hetzner Cloud (mostly VMs, plus load balancers and disks) is so cheap and has such a nice CLI API that it competes aggressively with dedicated servers - I would definitely start any with VMs, not with iron.
The biggest problem is the so called cloud native stuff which is both more expensive and more complex. There are contexts where it makes sense but for startups they are doing more harm than good.
Thing is, by the time the cloud native stuff makes sense most companies are at a scale where it'd be cheaper to just hire a good devops team, and start building your own cloud infra on own hardware.
Probably so. And that would be likely my approach at such scale.
Still, my most benevolent interpretation of current reality is, rather than saying "that cloud native stuff is crap", accepting that there are cases where it may make sense.
For instance, large companies might have trouble hiring a good ops team because they have in general trouble hiring and retaining talent (another conversation topic).
Ops people are a scarce good because univs do not train people for that and most people prefer coding. I am leaving the work devops out because the market completely perverted its meaning.
Most of my money comes from companies unable to handle even simple setups - and having trouble to find the right people, so I somewhat agree too. But it's mainly an education problem - it's pretty much impossible to find good people with that skillset, but it is possible to find people straight out of University willing to learn.
I fully agree with you: it is mostly an education problem and you can find people willing to learn right out of univ. Indeed, that is exactly my experience: I successfully onboarded several (carefully selected) junior people into the ops skillset over the years and I have seen them do wonders with customer systems, while enjoying their "ops life", without having fires every day.
The connection of this to the replies above it: I am not sure if this kind of junior people would be easy to retain in a large corporate environment. We certainly can do that in niche consulting.
We're a tiny company doing ops as services for large corporations - with one customer now coming close to a decade. That solves the retaining problem as we have limited exposure to all that big corporation nonsense, and have the option for individuals to go on a vacation in other projects without losing their knowledge in the organisation.
And somehow I feel these cloud native services keep breaking. Again Azure Container Instances found a interesting new way to fail. I have to check on Monday is it still in booting itself more often than usual(dev environment so have not tried any fixes)...
While the VMs that run some parts of the system have been rock solid giving zero issues... Should have just thrown the stuff on one of them or added third one. Cost would have been same.
Apart from the operation side, there is a development side parallel too.
Two examples that I came across
- "Test" mean if it passes on CI, it is good. Failing to run test on local? Who do development on local anyway?
- Teams so reliant on "AI" because this is the future of coding. "how to sort a list in python" became a prompt, rather than a lookup on the official documentation.
I’ve just recently gotten into ansible and find myself building the same thing. I wrote a script to interact with virsh and build vms locally so I can spin up my infra at home to test and deploy to the cloud if and when I want to spend actual money.
I’m still very much an ansible noob, but if you have a repo with playbooks I’d love to poke around and learn some things! If not, no worries, I appreciate your time reading this comment!
> while monitoring configuration compliance with a custom Naemon plugin.
While I absolutely agree with you and your approach, would you mind elaborating what kind of configuration compliance you are referring to in this statement? I suppose you do not mean any kind of configuration that your Puppet code produces as that configuration is "monitored", or rather managed, by Puppet.
I don't mind elaborating - the fact that people are asking me questions reminds me that I need to invest a bit more effort on some articles.
This case is actually pretty simple.
Puppet applies the configuration you declare impotently when you run the Puppet agent: whatever is not configured gets configured, whatever is already configured remains the same.
If there is an error the return code of the Puppet agent is different from that of the situations above.
Knowing this you can choose triggering the Puppet agent runs remotely from a monitoring system, (instead of periodical local runs), collecting the exit code and monitoring the status of that exit code inside the monitoring system.
Therefore, instead of having an agent that runs silently leaving you logs to parse, you have a green light / red light system in regards to the compliance of a machine with its manifesto. If somebody broke the machine leaving it in an unconfigurable state or if someone broke its manifesto during configuration maintenance you will soon get a red light and the corresponding notifications.
This is active configuration management rather than what people usually call provisioning.
Of course you need an SSH connection for this execution and with that you need hardened SSH config, whitelisting, dedicated unpriviledged user for monitoring, exceptional finegrained sudo cases, etc. Not rocket science.
Thank you for your thorough explanation. Interesting to see that you basically use your monitoring system as a scheduler to run Puppet and it sounds beneficial to closely integrate it with your monitoring to have it all in one place.
At my place of work we went the "traditional" way of running Puppet locally. It has been our experience that Puppet failures due to user misconfiguration or some such do not require our immediate attention (e.g. after hours), so we just check Puppetboard a few times per day to identify failing nodes.
Another reason why we use Puppetboard to monitor Puppet nodes is that every alert that our Icinga monitoring system produces is automatically interpreted as an incident which needs immediate attention. We are currently in the process of changing that so we are able to process non-critical alerts in a saner way.
Anyway, interesting to see how a fellow Puppet user manages their setup. Keep it up!
Thank you as well, for sharing these notes about your setup. Indeed concentrating everything in the same monitoring system is very helpful as it reduces the cognitive load. You can likely do the same with Icinga.
Feel free to reach out on Linkedin if you need some more details. More than happy to share.
I can't remember the last time I've seen a position description for a software developer (or anything tech related for that matter) that didn't include a requirement for skills in some cloud related tech.
Sometimes the job descriptions are boastful in their reference to those technologies, and other times you can detect some level of despair.
I currently have distilled, compact Puppet code to create a hardened VM of any size on any provider that can run one more more Docker services or run directly a python backend, or serve static files. With this I create a service on a Hetzner VM in 5 minutes whether the VM has 2 cores or 48 cores and control the configuration in source controlled manifests while monitoring configuration compliance with a custom Naemon plugin. A perfectly reproducible process. The startups kids are meanwhile doing snowflakes in the cloud spending many KEUR per month to have something that is worse than what devops pioneers were able to do in 2017. And the stakeholders are paying for this ship.
I wrote a more structured opinion piece about this, called The Emperor's New clouds:
https://logical.li/blog/emperors-new-clouds/