For our small startup, I bet on Docker Swarm (mode) after previously being burned by the complexity of maintaining a K8s cluster properly. Swarm was initially a pleasant and logical progression from compose files, but has since bitten us repeatedly: Things that are downright broken, fail inexplicably, or behave differently than documented - if documented at all.
In the various repositories Docker Inc apparently likes to spread its code across, thousands of issues are open - some older than 2015. Maintainers that once commented don't even respond anymore, but seem to work on arbitrary other things. Reading about some kind of WASM runtime in the Docker daemon, or a redesigned Docker Desktop app, add insult to injury.
I have no clue what is going on inside that organisation, and more pressingly, if anyone actually still works on swarm mode. I suspect not, and this is scaring me.
Is there anyone out there in the same boat, or someone that jumped earlier? What did you do, what can we do?
While it's the closest for your use-case I can think of, I still feel it requires more resources to operate properly than what you're looking for: While there's some early functionality for Nomad to do both secrets and services independently, it still seems that you're looking at min 3xnomad servers + 3xconsul servers + 2xvault servers + whatever clients you have to actually run workloads - and make sure you have proper monitoring and alerts for consensus and cert/token rotations.
Another angle to consider would be - do you actually need the dynamic scheduling part? If not, you can simplify things a lot by going to systemd+podman and instrument it all with ansible (or what-have-you). If you want the deployment UX to be more developer-friendly you can host Dokku or something on top of that.