The amount of vitriol in this comments section is astonishing. Can we not share fun projects without the professional Kubernetes experts thumbing their noses? At the very least, it does seem now like the complexity of K8s’s code base is a point of very thin skin for the community.
Seems like my click-baity title brought out some strong feelings :)
I do actually like Kubernetes. I recommend it to clients, I assess candidates who are to work with it, but I do really think it is a Beast. Because it is.
Kubernetes is like C++, extremely useful but it just keeps growing and nobody really knows all of it.
While Simplenetes is like... Lua, batteries not included, your boss will not approve, it doesnt't hit the buzzspot, but some people will secretely use it and be happy about it.
This is ~17k lines of shell scripting and claims to be "simple". Kuberentes is probably ~1m lines of Go. I'm not entirely convinced that this is less complex (simpler) than Kubernetes.
There are many factors in complexity, but a big one for me is what I have to hold in my head – global state. From any given point, what do I need to understand to know what's actually going on.
For Go, this is actually comparatively little. Between a fairly straightforward language, static binaries, and the focus on ease of distribution, there's not a whole lot outside of a codebase itself to think about.
However for a shell like Bash there's a relatively large amount to consider. You're very exposed to the underlying OS, to libraries and system packages, to various tools, to configurations users may have specified, to things being redefined underneath you. There's a lot of global state to consider for a Bash script.
I suspect it is simpler than Kubernetes, but I don't think it's a clear cut case.
To be fair Kubernetes itself probably has 17k lines of bash, and it is a good thing. It is the first go-to project that I use as a reference for bash scripting best practices.
edit: wc reports 47023 lines in *.sh[1], out of which 23836 are unique[2].
I can't believe you are being serious - why are you comparing a shell script to a million-line piece of production software? Apples to oranges. This was written as a joke to make fun of a common gripe against Kubernetes - its complexity. Seems like the Kubernetes community has a bit of a thin skin.
If it is a joke, the README does a poor job of conveying that. It is a bit funny at first but then gets relatively serious and recommends the project for use with up to 20 containers, with the beta caveat.
Either way, setting aside the laudable amount of work put into this, talking about the issues of complexity and different forms it takes seems quite on topic. (I don't have a cow in this fight, not using K8s or anything of the sort.)
If we take out the argument of OS dependent setup, supporting only stateless services etc.
One of the key benefits of kubernetes are API server and custom resource definitions. If you don't like how kubernetes does stuff. You can change the default behavior and orchestrate the way you want.
It's easier to get contributors and responses when most people already have github accounts. When not many habe have gitlab, it adds some friction to contributing and a lot won't bother
i dunno. autotools is hairy, but it also solved some incredibly hairy problems over an enormous breadth for a very long time in a very reliable manner.
portability across all the different unices with their weird c compilers and differing libcs that actually worked out of the box every time was a gargantuan task.
Building a platform to deploy stateless services are the easy part. This might even work nicely at a smaller scale. It is gets super complicated when you want to mount volumes, re-use those volumes. This is like a nice weekend project, contrarian in nature, but it is just not a replacement for kubernetes. This is some bash script orchestration on top of podman.
Bulk of complexity in opensource distro of kubernetes comes from the fact that everything is interface driven and has to work in an generic way. For example, you don't really have to implement cri-o like thing if you are opinionated platform and only supports docker. Similarly, you don't have to build CSI, if you only support ceph or aws' storage. I love the contrarian thought, but matter of the fact is modern orchestration platforms that want to be a general purpose platform is going to be big.
It is a really nice exercise however build something like this when you want to learn about networking, containers, orchestration, storage etc. and how to tied everything together.
However, if you are just dealing with a three node cluster (Yes I know many of you are running tri-cycles with k8s stickers on it) then maybe it can be fine for many applications to not have a scheduler.
Simplenetes does however support multiple replicas of pods so if one node fails it can be OK.
If you want something simpler and use podman you can try `podc` [1] which compiles a pod yaml spec into a single executable shell script with a nice managagement CLI api to it.
The hate against spring on hacker news is so ridiculous. Spring boot is one of the easiest lowest boilerplate ways to build anything, period. Especially if you use a type dynamic language on top, like groovy or kotlin or whatever.
Some people are just masochistic or they just want to distract themselves by solving problems that have already been solved by Spring circa 10 years ago. Most of them don't even use the technology that they preach around here, it's borderline trolling.
Asking to learn, but if you were to start a Java application in the following two scenarios:
1) Monolith with some CRUD, API, persistence to a DB + search.
2) Scalable backend with an API, persistence to a DB + some caching
Not in any particular order (choose what fits your team): Kotlin or Java and a lighter framework (e.g. Javalin is to Spring as Sinatra is to Rails, with JDBI), a newer big framework (e.g. Micronaut, maybe Play, others), Phoenix/Elixir, Rails/Ruby+Sorbet, Go with libraries.
Edit: I just did a google search for "Java Framework" and get a lot of useless top-N lists with old content retitled "in 2020" with hits like: Hibernate (not a framework), JSF (JavaServer Faces), GWT (Google Web Toolkit), Struts (The Later Version), DropWizard (seems to have stagnated and docs are all over the place. JDBI was extracted and has a life of its own).
Cmon, you can't be serious, do you really recommend Play, Micronaut, etc. over Spring ecosystem? Also if you want the benefits of GraalVM you can stay in the ecosystem today with Spring Native, moreover if you are ideologically against Spring then you should go with Quarkus because that's the next technology by maturity/popularity. How can you not go with Spring when REST services are today a solved problem? Why do you want to turn it into a problem by adopting something else that is not battle tested? In the company that I work for, Spring is ruling the microservices layer and we have a combination of Scala and Python for data pipelines that are not related to the behavior of the business logic, they only know one thing and that is data. Hibernate has not been a problem for us, I can say that I totally recommend it when you have multiple people working on a project, rather than polluting the codebase with raw SQL.
If you really want something exotic in Scala that is impossible to express in Java, Go, Ruby, Elixir etc., then take a look at the Typelevel (cats, cats-effect, http4s, doobie) ecosystem or ZIO. If purely functional programming is your cup of tea.
I'm not ideologically against Spring since I mentioned Micronaut being a modern take on it, without the outdated assumptions. I'm not familiar with Quarkus but someone else recommended it so may have a look at it. I'm currently at a Rails shop, but I see patterns evolving as teams get larger toward program construction patterns like Spring, persistence, repository, application model, etc. ActiveRecord as big as it is, is still much simpler than Spring/Hibernate with an EntityManager.
BTW, I did work at a startup for years building microservices with Spring which is where I ran into it's limitations. I basically worked around the EntityManager, JPQL, and query template caching where ever it was necessary, which was frequent. I'm not against query builders or data mappers, having worked on making one myself[0]. I'm just more performance/resource conscious than most.
Anything Java-backend-ish these days, my first 2 stops would be Quarkus [1] or Micronaut [2]. It would need special requirements that those couldn't match to move me to something else.
A new Java application? Java is getting pretty long in the tooth. You might be better off building on top of Go or Rust, especially for those specific scenarios (Go in particular was designed to build network services.)
When I did enterprise sales, we had another phrase for that: "No one ever got fired for buying IBM." (Coincidentally, I also worked for IBM, back in the day.)
While it's true that tech choices are signals that enterprises do use to determine quality, if you are relying on a particular choice of programming language to sell into major enterprises that's simply a mistake. (And enterprises have invested vast sums in technology written in c# or PHP, so clearly it's not all "lol" out there.)
Rust's and Go's capabilities of compiling into a single binary, without requiring a JVM that needs to be fed and cared for, make them a particularly compelling choice for enterprise deployments. Kubernetes itself is written in Go.
Linux itself is written in C, though everything runs on it. Who cares about the language that Kubernetes is written in (by the way, it used to be Java but it was changed politically to Go)? If you are writing Java/Scala you can use GraalVM to achieve the same thing that Go does. If your service is mainly doing data processing, then Go won't help you there because its abstractions are too low-level or even non-existent. A Java service/app won't require a JVM if you you create a self-contained application package. Applications can be deployed on fresh systems with no requirement for the JRE to be installed. The advantage is that you control the version of the JRE used by the application and the disadvantage is that application updates are the responsibility of developer and self-contained applications do not have built-in support for automatic updates. Exactly why for services I wouldn't recommend it, and neither Go or Rust. I would never trade the state-of-the-art garbage collectors that the JVM provides me with the ones of Go or the borrow checker of Rust (though I would take Rust anytime over Go). I can't help myself but only think that you comment is straight from the 90s when it comes to the JVM ecosystem.
I don't disagree with your conclusion. But your comment communicates no argument against Spring and how the trade-offs it makes make it a poor option for new projects.
Anyone with sufficient experience with JPQL and Hibernate will realize their limits and how poorly it works in the latency sensitive transactions we want now. Spring and Hibernate were designed to cache domain 'entities' and manage their dirty state client-side as an optimization. Any time you go want to go near the EntityManager you're hitting this impedance mismatch.
i love this, and especially love how there are only a few things in the world that one could reasonably claim to simplify with "17k lines of shell script", and one of those is kubernetes. hilarious! (and i even see the value beyond the mere humor of the project's existence, which for me (and unlike all these salty k8s stan commenters*) would be more than enough justification on its own for creating the project in the first place). it's like both a useful project and performance art!
* for the umpteenth time i daresay that hn needs to grow a sense of humor sometimes...
17k lines is a lot of shell... but... it bypasses all the complexity of heavyweight scripting languages, bash is a lightweight interpreter that is always there and just works and i bet it's fast.
this is a cool idea.
next up, somebody will modernize daemontools for the container/cluster era...
actually no. there are a bunch of properties to properly daemonize in unix. things like special handling for stdin/stdout/stderr, reparenting, signal handling... look it up.
classic rookie mistake to just call a program that doesn't exit a daemon...
Running "in the background", handling of stdin/out/err, logging, detaching from tty, creating a new process group/session, closing fds, and much more is handled these days mostly by systemd. No need to put all this into your program at all.
I'll give you that signal handling still needs to be done in the target process, if the defaults don't satisfy you.
But you really don't need anything special anymore to make any old script into a daemon.
Without systemd you can use daemonize from shell, for example. I either run these kinds of service scripts via systemd in production, or via my special program that takes care of starting and monitoring, and restarting when the code of the script changes.
Why should I stuff daemonization machinery into every single script I might want to run as a daemon, especially when it's cumbersome in many languages where I want to do it, like bash or PHP, and in general less flexible than using some external tool?
maybe you're right. shrug. does your script change the working directory? do all systems have fancy core file management now? does it still cause issues with unmounting?
it is true that it's rare to see a daemonize flag these days...
This all depends on what the program does. If you have some app specific service that waits for postgresql NOTIFY messages, and does something in response, it doesn't matter that working dir is not /. You just stop the service and perform maintenance.
If it's something like sshd, it better not prevent any filesystems from umounting unless necessary.
Why? This is not a completely novel concept, and a daemon is not a particularly complicated thing. You can use shell scripts to easily "daemonize" small programs. It's both fun and effective to do so.
If you're looking for a simple Kubernetes alternative, take a look at the HasiCorp stack. I use that in production and it's easy to set up and maintain.
I use k3s in production, and it is great. But as far as daily use, it is not any simpler than kuberenetes. Initial setup and clustering is a breeze with k3s. But after that, you are mostly just working with the same k3s api as a full kuberenetes setup.
k3s is kubernetes. Simpler to install and get a cluster running and much smaller, but still kubernetes, with an identical feature set and API. That is part of the point of kubernetes, that it is a standard and anyone can implement it, with Google's original implementation only being a reference, not canonical.
They don't mention that this was accomplished via k3s, but I happen to know it was because it was colleagues of mine that did this. The point of being able to do something like this is you can deploy a completely identical stack of admin/controller level software to systems running in data centers and systems running in airplanes. As long as they can run kubernetes, you're good to go, and with k3s, just about any piece of hardware can run kubernetes. This has huge implications for defense because it means you can develop and test software that isn't directly responsible for specialized hardware control without having to build emulators or lab equipment identical to what is installed in the actual weapons platforms. You just need to ensure they can both run kubernetes.
It doesn't have to be kubernetes, obviously, but you get the advantage of a fairly rich ecosystem that includes things like k3s, which doesn't even require bash or a shell at all to work.
The YouTube channel [0] has plenty of introduction videos, but I find the documentation [1] to be the best place to start. You should set up Consul (service mesh) first, followed by Nomad (orchestration), and Vault (secret management) if you need it. The only thing I initially struggled with was the systemd-resolved config on Ubuntu, to make the service discovery work. I plan to write an article about it, should I ever find the time for it :)
Btw. the cluster consists out of three servers on Hetzner [2] and runs our web analytics platform Pirsch [3].
1. Anything that offers effective competition to Kubernetes is a Good Thing. I'm not saying this is because I haven't look at it closely enough but competition brings about improvement.
2. Why do people write "Simplenetes has a 100x less code than Kubernetes" instead of saying it has 1/100th the code?
- Non-exported variables are by convention lower_case, while exported variables are UPPER_CASE.
- Function names are by convention lower_case.
- There's a "quite" signature string and variable; should it be "quiet"?
- The lines like `local hosts=` can be simplified as `local hosts`. You can also join such lines, as with `export`, into `local foo bar baz …`.
- `[[` is generally considered more reliable than `[` (there are a bunch of posts about this on Stack Overflow and Unix Stack Exchange).
- Naming scripts .bash rather than .sh lets `shellcheck` inspect them without telling it which shell to verify against. You do check these files using `shellcheck`, right?
- Variable names like `list` and `tuple` are unhelpful; what do they actually contain? Also, they should probably be actual arrays rather than a space-separated string to avoid relying on word splitting, and to allow entries with whitespace in them.
The scary stuff:
- All the string packing and unpacking also makes me nervous. That's not simpler than using JSON/YAML, it's just a different kind of complexity.
- Reusing the same variable a bunch of times in the same function makes it really easy to end up with the wrong value at some point. Better to split stuff into more functions or rename the variables to make them single use.
- `set -o errexit -o pipefail` would be good to have for safety.
- `kill -9` anything[1]
- Why are some of the variables inside functions not local? They effectively end up being globals, which makes for really difficult debugging.
The good stuff:
- Using local variables everywhere.
- Descriptive error messages.
- Returning early in case of errors.
All in all, Bash is not a good language for any system of this size, no matter how diligently you program. The error handling isn't comparable to most mainstream languages, the data types are incredibly limited, and there's no native support of recursive languages like JSON. You've done a great job, but I'm afraid a system like this is doomed to failure from these facts alone.
Coding in shell sure is a challenge, I try to avoid bashims where I can, that's why most scripts are .sh not .bash, since they run also with dash/ash (that's why preferring `[` over `[[`).
The only Bash requirement we have is the `podc` (pod compiler) since it parses YAML docs, I couldn't pull that one off without the more powerful Bash.
The code base as a whole absolutely needs testing and shellchecks all over the place to be labelled as mature in any sense.
Our other large shell project is Space.sh [1] where go overboard on testing, each module is tested under a bunch of distributions. [2] [3]
Next step would be to do the same for Simplenetes.
Given that in shell that the only way to protect a variable from functions down the call stack accessing it is to redeclare/shadow it as "local" brings some murky waters.
BUT, isn't it amazing what is actually possible to do with Shell??
For me Simplenetes in the long run is not about it being written in Shell, I'm perfectly happy rewriting it in Rust if we get traction on it.
Simplenetes is primarily about having a simpler architecture of what a container cluster is and is not. I think many projects just get too complex because they want to fill every single use case out there, while the interesting part is saying no to things.
The real use case is super simple clusters, which could be just 1 node, no etcd, no API (SSH managed), no iptables.
Simplenetes is not a living breathing thing which auto heals it self, it's a deterministic approach where you can see your cluster with all hosts in the git repo which represents the cluster.
Also, it's focused on making local development easy, so you can develop in a micro services architecture straight on your laptop with very quick iterations.
Despite whatever the k8s crowd misuses the term to be, 1 node is not a cluster. It might be a single node managed using cluster-capable technologies, but it's still not a cluster.
Thanks!!
We do use it in production, it is however fresh from the keyb so giving it some time before putting sensitive stuff in there.
About the state, it's perfectly doable, but requires some more Ops than k8s and isn't as flexible, naturally.
This has been bugging me for a while. Programmers love to reinvent the wheel. There a millions of make(1) clones, key-value stores, etc but I don't seem to see many projects (even toys) in the "declarative container runners" space. I think it's a shame.
While Kubernetes is hopefully Very Good, it is complicated, and I have often wondered what other approaches would yield (e.g. more imperative ones, because writing admission controllers is not exactly easy). So I'm glad to see people playing around, though I don't know if this amount of bash is healthy or if 17,000 lines is simple.
Ah; since the title was "17k" lines, and the directory was named "includes", I presumed those were includes.
So that would seem to say the title is also counting that compiled version, when in fact it is much simpler. But then also, why is the compiled version longer than the sum of the files in includes?
The line count is a rough (overestimate) summoning the source of the three projects involved, simplenetes, simplenetesd and podc, and it also includes comments and blank lines.
The compiled output pulls in some dependency modules which is reusable code (STRING, etc). So it's not clear where to draw the line :)
Reason the compiled output gets bigger is because the compilation process `(make.sh)` includes compilations of actions, which I use when connecting over SSH agentless to manage the nodes. This tough part is handled by Space.sh [1]
17k lines of shell? Hah give me the infinity lines of Go that is kubernetes — maintaining shell scripts is a nightmare. At least Go is a proper language with an ecosystem and tests and established best practices, etc., etc.
I don’t care about Kubernetes, but someone might read this post and decide that it is a good idea to “simplify” an existing project by rewriting it in 17000 lines bash script.
And some hapless person will be stuck maintaining and debugging it. It could be me, or you, your friend. You don’t want this to happen.
> I don’t care about Kubernetes, but someone might read this post and decide that it is a good idea to “simplify” an existing project by rewriting it in 17000 lines bash script.
I think the 17kloc would discourage that kind of thinking.
Haha :) Well, the learning has been great I can confirm that. My script-fu is on a whole new level now, it was already quite up there after creating Space.sh [1]
Waste of time... Well... Let's not think too much about that. I'm at least a very happy user of this :)
- "Let's put the Dev and Ops back into DevOps."
- "Simplenetes doesn't go there because that is when DevOps becomes MagicOps."
- " You enjoy the simple things in life."
Anyone taking up 17KLOC of shell script “as is”, regardless of how many cool domains it is on, to use in production deserves everything they get subsequently.
It will be a useful learning experience.
The same goes about taking up k8s with no thinking, by the way.
I don't understand your type of comment. If people involved with a project decide to advertise it on HN, specially following questionable clickbait methods, is it reasonable to demand that all posts must be exclusively comprised of high praises and compliments?
