The Kubernetes API is itself a leaky abstraction over many different concepts. Putting docker-compose over top of that is just an even leakier abstraction.
Even their example of using Ingresses is already leaky because there is no similar concept of Ingresses in docker-compose, so they're hacking it with labels. What if I need to set a higher request timeout for my app? I have to set a docker-compose label that converts into an Ingress annotation (the leaky part of the Ingress abstraction)? And this is somehow less cognitive load?
Like all similar attempts to simplify deploying apps, it only works if all your apps fit in a nice box. Anything slightly non-standard (like needing different classes of storage volumes) will expose the leakiness. Even Kubernetes kinda still expects all your apps to be stateless web services and will fight you otherwise.
I thought maybe I was just getting old and crusty, but this really does capture my primary complaint with K8s - everything that isn't standard, is so special and unrelated to the underlying technology that you need to relearn how K8's interacts with every layer of abstraction just to have half a chance of it working.
YAML practices… YAML is not a language, it’s a syntax tree. It pretends to be configuration and what it really becomes is some kind of a gimped template engine which people put layers and layers of code generators on top of to manage the complexity. XML was better and I’m willing to die on this hill.
Schemas and includes are great features, but you will have to defend the syntax hill on your own - imo the syntax is neither human nor machine friendly
oh, like azure k8s vs aws k8s vs gcp k8s? each has their own way to like... attach a volume for block storage or something? i thought that was all abstracted away/standardized?
> Once you get to the non-standard, cloud specific configuration
Use the service resource in Kubernetes to order a load balancer in different clouds. You need different tags, they function and are configured differently, and you still need to learn how it works in your cloud.
Contrast this with the contract* K8 tries to sell you, which is "describe the intent and we'll figure it out for you" and you'll realize the cloud providers bunged up this whole thing.
That's not to say K8 is perfect, but if you think for even a second that wiring up resources outside your kubernetes cluster will be lower-friction, kubernetes is falling short of it's goals.
I've always felt the standards are more about organization and common practices, whereas the implementation is still the wild west. Unfortunately in my experience, abstraction, even if structured, doesn't solve the main issue: it turns out the engineering operation of infra is still pretty hard. At least you know where to look to find the chaos though.
> whereas the implementation is still the wild west.
in my own experience, i would've thought "ok, let me aptinstall k8s and then apply some yaml to it so i can avoid going to the cloud for a massive k8s bill for almost no reason"
standing up kubeadm on a single master + worker node setup on a VM (like a droplet or EC2) is basically like, extremely non-standard + frowned upon
then, does anybody actually write pure k8s yaml? or do they write jsonnet/kustomize/helm charts? do they apply with kubectl or the helm CLI or the argocd CLI?
> then, does anybody actually write pure k8s yaml?
You can think of plain k8s yaml as a sort of machine code for k8s. You _can_ write yaml by hand, but it’s not generally how things are done outside of initially learning how k8s works.
The tools you’ve mentioned all make working with k8s resources a lot more pleasant. Something simple like parameterizing the name of a secret instead of hardcoding it in N different yaml files saves a lot of time if you ever need to refactor, and being able to provide those values via a “standard” CLI tool makes automated deployments a lot easier compared to hacking together some yq commands. helm in particular tracks the history of any charts you’ve installed, and has a very simple “helm rollback” command which can get you back to the last working version of your application if things end up going wrong.
The exact tools used by each team will differ; you could use any of them. But no one in my experience is writing plain yaml manifests and kubectl applying them in production contexts.
Yep. And the end result is a hodge podge of labels that make up an API that is much worse than the original interface, and requires significant amount of bounds/error checking in order to provide useful error messages until the codebase for it is no longer simple.
I just give developers access to k8s and teach them about it. That's really just my admission that the Kubernetes API isn't perfect, but it's better than many alternatives. If I were to build something to hide k8s from developers, eventually I'd have to add so many extra options that I'd probably end up re-inventing the Kubernetes API anyway.
That may just be unique to my job. Maybe another company could force all apps into nice boxes that prevents their bespoke platform API from leaking.
Honestly, I love compose files for local development. Launch and init a database, a DB admin tool, middleware, and a UI server in a single shot.
New team member: "How do I get the stack running?"
Existing member: "Clone this repo and run docker-compose up -d"
20 minutes and a cup of coffee later:
New member: "Cool! Ready to start!"
Does it look like our AWS stack? Only superficially. Which can be a good thing. The compose file shows you the broad strokes, not the extreme details. Compose files help with the basic mental model. Want to know more? Read through the CDK stack code.
But using a compose file for a cloud deployment interface? No thank you.
Yeah recently at work I created a compose.yml for our kafka pipeline. We have a collector and an enricher which produce to and consume from a kafka topic, respectively. The enricher then runs some functions on the data it consumes and produces to another topic. These services are all super microservicey, and so testing locally outside of docker was difficult (not to mention the PITA that is setting up a zookeeper broker on Windows)
The compose.yml runs everything on a docker network and uses redpanda for the kafka broker. Iteration became SUPER easy. Previously, we were just writing config files locally and uploading them to S3, then restarting the dev instance of whatever we were working on in k8s to re-ingest. This was, as I'm sure you can tell by now, tedious
In theory, it works everywhere. In practice, for any sufficiently large project, teams can't (either in terms in possibility or ability) size down the running services so that the whole stack runs locally.
And then starts the hunt for tradeoffs, with a million pitfalls.
I have seen the pattern fail too often to be only optimistic on the approach. I now tend to favor ephemeral deployments with only one locally running service connected to a bunch of remote services.
I gave up on compose since I couldnt figure out VSCode autocomplete for Python and JS, with the library files living within the container. Once I needed to run a command on a docker container (wasn't allowed to commit it) to make it work. To update pip/npm stuff, I do it outside of docker and then update the images.
Has anybody found a better way with just open source tools?
We had DevOps for a while, but now systems have gotten so complex (and our tooling hasn't kept up) that we're back to pre-DevOps days with Dev throwing stuff over the wall to Ops.
"Progress" is that now also ops gets to throw half-functional operators, underdocumented CRDs, and pluggable CRI/CNI/CSI all not quite configured to work together properly back over the wall to devs who have no hope to debug anything anymore!
I would be more interested in a stripped down JavaScript runtime for configuration. Some tooling already supports this paradigm. Embed something like QuickJS or a Lua interpreter and have the script export a config object. It feels too much like we want these plaintext config files to turn into fully-fledged scripts.
It's not stripped down, but Pulumi is a version of this. Regular code, and all the benefits and drawbacks of that approach, in your language(s) of choice to create infrastructure.
I haven't used Pulumi, but I have used Terraform (years ago). That is pretty much exactly what I am talking about, but instead of creating a new language, just use an already established language meant for scripting.
We're trying to do something very similar to what the OP proposes at our current company (Kurtosis), and we explored YAML, Jsonnet, Dhall, CUE, and general-purpose languages and eventually settled on Starlark: https://docs.kurtosis.com/explanations/starlark#why-did-kurt...
BLUF: Infrastructure seems to live in this weird "configuration code" space between pure config (YAML) and actual code. You want reusable config, but you don't want to have to understand the complexities of a real codebase. Jsonnet, Dhall, and CUE all have the simplicity you want, but we found the DSL we'd need to write to be rough. Starlark hits this very nice balance of safety and ease-of-readability (and as a bonus is basically just Python). We've been very happy with it so far.
I've started using this setup for my Nim projects. I created a project that lets me use Nim instead of YAML for a statically checked configuration system. It can be run and output json, yaml, or run as is in Nim code. Its been much nicer than pure YAML! https://github.com/elcritch/ants/tree/main
With Lua, you don't even need to export a config object, the Lua script is the config object. I do think there is some promise to this - see Neovim for example.
There are a lot of downsides though, because now you've got a leaky config language instead of a leaky config interpreter.
Like a CDK (cloud development kit)? Those tools already exist - like AWS CDK, Pulumi, CDK-TF - but they are hardly stripped down. AWS CDK for instance transpiles everything down to Typescript and those layers of abstractions make debugging painful.
Yes kind of, but instead of creating a new language (like HCL), just use a language people already know, but strip the runtime to a minimal set of features.
I share your, and all Norwegians', distaste for YAML. One saving grace I have found is that YAML is a superset of JSON. JSON isn't great, but it's certainly better than YAML.
I toyed around with dhall to generate k8s deployments. It was fantastic, but then I realized there was no chance that others would be willing to learn FP. Cue is pretty great, but it still has many sharp edges in the tooling.
Ultimately, though, all configuration files tend toward turing completeness. Just use an actual programming language. If you need to dynamically parameterize it then you can import a JSON file or whatever.
Upon further reflection, it's not actually YAML I have a problem with, so much as the (lack of) composability of YAML. As soon as you're dealing with multiple files and want to "include" something in one way or another, it seems like every DevOps software has its own semi-standard way of allowing that.
As annoying as the parsing quirks of YAML are, they don't bite me that often. What bites me is the proliferation of YAML files in an effort to DRY my config files with whatever primitive imitation of a templating language I can use as the common denominator of the various tools consuming them.
I just checked our monorepo, and we have ~93 YAML files related in some way to DevOps.
My main issue is that YAML is used as a [bad] programming language. To make matters worse, in the domains where it's typically used (DevOps) the "developer inner loop" is usually minutes long (e.g. GitHub actions).
Having to actually do the dangerous things that you typically do with YAML in order to test the YAML is insane!
As a human, I'd rather write YAML than JSON. But if I'm writing code, I'd rather write code that parses JSON.
The problem is that the two often conflict with each other, because I'm writing code to parse config written by a human.
My rule of thumb is that any autogenerated communication between services can be done over JSON (assuming some more efficient serialization format isn't available), but if a human needs to maintain it (and check it into a repo with diffs) then it should be YAML.
I agree with this point. I would also say that while HCL gets a lot of hate from the developers it really is an easy language to learn and use. Yes it's another language but the language itself I found to be very natural for declaring desired state. Last year I spent two half-days training my team on it (I have a lot of experience with it) and they picked it up rather well. Previous teams I worked with had similar results. It may not be their favorite language, but it's not as bad as cloudformation or a heavy, cumbersome CDK.
TL;DR: It has way too many opinionated things builtin, too many ways to do the same thing.
As a additional note, unlike JSON that has the awesome tooling available (`jq` is reasonably widespread, you can count on python/perl/node being almost everywhere), it has nothing that (because of the multiple opinionated ways) will reliably read your files or not mangle your desired output in some way or another -- except if you limit your YAML to just a embedded JSON.
There actually is a yq [0] which I've used effectively for basic parsing and manipulation tasks on the command line. But I've no idea how well it handles YAML corner cases or how many advanced features it's missing compared to jq.
CUE [1] is kinda-sorta "transpiled" into by Pulumi's language support facility [2]. In general the greater the domain space being expressed the more I'm leery of this kind of approach; leaky abstractions creep in faster the larger the volume of concepts embraced. Infrastructure is a pretty big concepts volume. Whatever the source system (Pulumi in this case) does not provide a hook for that the target system (CUE in this case) supports for example, hacks tend to start to emerge. If there was both this layer of abstraction and a "break glass" layer available down to something analogous to an Abstract Syntax Tree level of primitives, then the target systems at least have hooks to compose new abstractions via some kind of DSL into the source system perhaps. There be dragons there too, though; no panaceas, but at least it is better than all sorts of hacks.
I want to find out with CUE's BCL heritage to fix what they got wrong with BCL at Google being used by Borg, what the CUE team is doing to try to avoid the Second System Effect biting them as they build CUE.
Both CUE and Dhall [3] seem promising when first deployed, but some field experience is reporting challenges I typically associate with immature tooling ecosystems surrounding them [4] that come up short when trying to reason beyond what the out of the box experience can quickly support when working at scale.
One part I haven't quite figured out with these configuration expressions is how to seamlessly flow their models between organizational data models (to answer who owns what type questions, example) and operational data models (to answer who is on call type questions, for example), and gracefully manage the mistakes inevitably made in those models, without mountains of manual toil and bespoke in-house glue code.
I just want to point out that already exist a tool that convert a docker-compose.yml file to several k8s manifests: https://kompose.io . You have to adjust some details because is not perfect, obviously.
The only advantage I see is it dosen't require a kube-apiserver. Other than that it feels like just running a quick throwaway cluster with kind and writing actual manifests is more accurate and portable.
Makes a great deal of sense. It was basically the original idea (when compose was fig). Oftentimes though, people cling desperately and aggressively to existing complex systems...
Having a complex system under the hood is not inherently bad, so long as you have non-leaky abstractions that allow that complexity to be reasoned about (pods, services), sane guardrails that let you know when you're throwing away key guarantees (privileged containers), and reasonable defaults for portions of the system that you don't yet need or care to engage with (resource limits, among others).
Kubernetes may be a system that invites overcomplication, but it in no way requires it— in a world where anyone can run k3s or microk8s on their own laptop and a Kubernetes hello-world is like 10 lines of yaml, I don't think it's a stretch to posit that that's the interface that makes sense to use up and down the stack.
Intriguing idea, but I don't see how a docker compose manifest can be translated to "just work" in a kubernetes cluster without making a long list of non-intuitive implementation decisions. And if devs are supposed to now learn all the intricacies of compose when using it in an kuberentes context then it isn't very different from any other custom kubernetes config abstraction. Worse in fact due to all the confusion it will bring with it.
"Kubernetes manifests are an unnecessarily detailed abstraction for most software developers. Developers don't need as much control. Therefore, they need higher-level abstractions, which hide more detail."
Agreed... Though I think there is a different solution. Instead of coming up with all these hardcoded formats and standards (docker-compose, k8s, etc, helm, ansible) which essentially are just 'functions' that map X to Y.
Instead of defining 'docker-compose' files, just define the containers itself. And if required, map that to a docker-compose, or k8s manifest or whatever. Remove all these unnecessary layers that aren't really that helpful - because each standard or implementation has it's own parser, error handling etc - why bother with all this?
And just use something like Nix for this. Or just define it as a common python value. Otherwise we end up building all sorts of templating and ugly programming-in-yaml. You could take a look at https://docs.hercules-ci.com/arion/#_arion_compose_nix to see what this might look like - uses Nix to do something like docker-compose.
So now you can no longer use your neat docker-compose file to deploy locally on your docker as it is referring to an external secret.
So to keep the interface clean you would need to split it up in a Service model and a Deployment model that exposes the needed configuration for the target platform plus the service configuration for the specific environment.
Or something...
Thinking of something generic that works in all cases is hard and complex...and opinionated hence the many platforms, I think.
Cloud technology is so bad. Just run your programs as normal processes. No bloated images and runtimes, no slow virtual bridging layers, no barriers preventing you to access the hardware or communicating with the user.
The hardware has finite resources, and the best way to share them is to run things natively on the operating system.
It's also all simple and straightforward, everyone understands ssh and simple Linux sysadmin.
Dockerized delivery starts easy unless you realize you need a new container for every small thing you need to run, and you end up with a pile of containers a short while later.
Docker containers, in my experience, have not figured out auto updates yet. You may be running containers with critical system vulnerabilities and is not bothered to check.
So the basic premise of Dockerization 1.0, "I have exactly one thing which I will put in a single container with all of its dependencies, reproducibly built" is wrong.
I've done hundreds of nodes, all managed to a very low level (I've been to the datacenter and optimized placement of servers within the cabinet to enable shorter cables to the patch panel) which is way beyond what most companies need.
In general it's better to have fewer and more powerful and reliable nodes that you own than to have many disposable small ones, as shared instances just don't work
well and aren't cost-effective.
Cloud in general just makes everything more expensive while removing control. The costs on AWS if paying for three years upfront is the same as buying the hardware outright, except you have little say in what the hardware is and how the infrastructure is set up.
choosing serverplacement to optimize for cable length does not sound very efficient. usually you get the cables in the length you need and place servers to solve other requirements (heat, direct connections between devices, etc)
> as shared instances just don't work well and aren't cost-effective
of course ymmv from mine, but this statement being true or false depends on too many things to be of any value
Docker compose universal? Hello kafka-1, kafka-2, kafka-3 and various compose files depending on the cluster size. Thanks, compose is nice for local dev and as a „this is a minimum of a reference to make the stack talk together”, but no. Compose is missing so much. Services, rollouts, instance count, secrets.
At my current company, we use Docker Swarm with multiple compose files to run all of our services. It's incredibly easy to scale up or down services, and if you need to get more intricate with service details/configs, it allows for that. Love it.
If you're going to run in prod in Kubernetes then you need to live with the idea of applying Kubernetes manifests. So, live with it. Docker Desktop made that easy, and some of the alternatives like Rancher and Podman also make it easy. Plus, how are you going to test your ConfigMaps and Roles and RoleBindings if you aren't actually running in Kubernetes locally? How do you know if a change to your Helm chart is going to work?
If you are not running in prod in Kubernetes, but you still run in prod with Docker, what the heck are you doing? ECS? You still need a control plane to deploy the images and Compose isn't really up to that task for production workloads.
Uhh hellooo Terraform? I was just discussing docker-compose today, Terraform has the advantage of State Management..
Terraform is just as easy, but probably a combination of the two docker-compose and terraform are much more effective.. Universal Infrastructure interface is the name of Terraform's game.. They already done what you hoped docker-compose did.
Terraform deserves serious kudos for interoperation between all the clouds, docker, vmware, and literally tons of API modules.. Their Infrastructure Modules are nothing to laugh about in a JSON compatible example..
Serious, they even give Ansible a run for their money when Red Hat was all the blaze.
After pursuing a very simmilar idea several years ago, even with simple apps I quickly ran into significant complexity, where important (and very consequential) decisions had to made for the developer (aka a "strongly opinionated" framework), particularly around questions of horizontal scalability. That also locked you into a specific cloud/platform/pattern/etc that often just led to teams needing to "eject" back to enough real k8s features that the abstraction became a real source of extra complexity. We ended up abandoning the project.
I like the idea of giving developers a template format they could follow, and then those templates would be transpiled into Kubernetes manifests by the platform.
I would not use docker-compose.yaml for that, though.
Shameless plug for my own attempt at doing something like this, but creating a higher level abstraction on top of helm and terraform for composing different services and projects to quickly get projects up and running on Kubernetes with sensible defaults and the ability to quickly break out into configuring the helm charts if you need to handle something special.
Interesting idea. I wonder how some examples would look for teams which have a docker compose file which don't include some services - because they are mocked for tests or expected to be run separately etc. Would it still be possible to write a k8s operator to transform the docker compose file perfectly to a k8s manifest or would it be lacking key information?
Helm did something like in the earlier versions. Helm 3 got rid of it to the great cheering of many. It didn't ingest Docker Compose files, but it is the same architectural pattern.
One criticism I have regarding this article is the intended audience surely does not need a detailed explanation of what an abstraction is (complete with dog, golden retriever and chihuahua examples).
Kind of a weird set of examples to describe the concept of an abstraction at all... I hate contrived examples. It was my least favorite part of all of my coursework, GenEds included.
If you have application developers that can't grok what a deployment is and what a service is, fire them. Just fire them. The "run a container" part of k8s is a solved problem that we never need to revisit. There are much more interesting areas to tease apart, like networking and service mesh, and there's some room for idiot proofing those for developers (OpenShift has some nice tooling in this area, among others).
This is exactly the attitude that I've been complaining in other threads in HN.
What else should the application developer know? How to fill taxes for the company?
I hope the people who manage your k8s infrastructure know how to develop the application they are deploying. If not, just fire your k8s teams. You cannot became k8s person without knowing how to develop the application, there are more interesting areas...
If you have application developers that can't grok what a deployment is and what a service is, fire them. Just fire them.
No - fire their managers, and their managers as needed.
By definition, that's where the responsibility lies. Not with the leaf-node developers. Who are doing exactly as they've been trained and/or vetted by their managers.
At some point we also didn't know what a terminal was, how to type, or how to read. That doesn't mean they aren't baseline expectations for a software developer.
A Deployment, a CronJob, a Secret, etc.: those are the interfaces with Kubernetes. By extension, it is the interface with which you interact with the service (a cluster of machines) that your infra team maintains. (And they are the ones that need to understand how to maintain & operate Kubernetes.)
App devs need to learn the structure of those YAML interfaces. They are incredibly well documented, both in reference material and in example material.
And yet I find the same as the people above you in this thread: app eng have an absolute aversion to wanting to specify, in programmatic form, how to run their app. Yet, by definition, they're the only ones that know how to do that.
(That said, I have been known to reverse engineer applications enough to understand how they intend to be run, and then write the corresponding k8s YAML. This is an organizational anti-pattern, though.)
The YAML isn't complicated. What's complicated is that the set of data that comprises a process is inherently complicated: what user does it run as? what files does it need? what command to execute it? what env vars? etc. k8s's YAML does rather little abstracting over these.
(I sort of disagree with the "fire them" — rather, you need to ensure your onboarding trains them. But a lot of employers these days seem to do zip for training.)
What the actual problem is, in my considerable experience building teams that encountered this problem, is that they lack basic understanding of OS concepts.
K8s includes a lot of Unix. If you don't know what a user or perm is, or a mount point, you just blame k8s.
How are the developers writing Dockerfiles if they don't understand them? A deployment is a declarative way to run a (group of) container(s). There is no extra abstraction required of the developers in the trivial usecase which is what the article is suggesting is too difficult.
> A deployment is a declarative way to run a (group of) container(s). There is no extra abstraction required of the developers
You have skipped over the Pod and ReplicaSet abstractions, and the Pod part is definitely not ignorable even in the most trivial case. The container part is probably more ignorable than the pod part, since pods are what actually gets exposed for most operations.
I have to chuckle that they're still referring to docker-compose.yml as opposed to compose.yaml, and docker-compose as opposed to docker compose. The legacy files and commands will be EOL in June.
Even their example of using Ingresses is already leaky because there is no similar concept of Ingresses in docker-compose, so they're hacking it with labels. What if I need to set a higher request timeout for my app? I have to set a docker-compose label that converts into an Ingress annotation (the leaky part of the Ingress abstraction)? And this is somehow less cognitive load?
Like all similar attempts to simplify deploying apps, it only works if all your apps fit in a nice box. Anything slightly non-standard (like needing different classes of storage volumes) will expose the leakiness. Even Kubernetes kinda still expects all your apps to be stateless web services and will fight you otherwise.