What about: this is now somewhat builtin into SSH, so you don't need an additional tool (or more accurately, this additional tool could be far simpler and more robust).
Ok. That is helpful, but ssh server or client? No way I would bother upgrading the server on all those boxes just to get to a point that is already working with sshpass.
Client. OpenSSH 8.4 has SSH_ASKPASS_REQUIRE that allows one to always invoke SSH_ASKPASS, notably when run from a terminal (previously, it was only invoked when there was no terminal to prompt the password).
I kinda like the way ansible does it. There is a concept of a vault. You put all the passwords in that file and they are all encrypted. You use one password when running the command or playbook and all of the keys are decrypted as needed.
I don't know if that is efficient for 30K machines though.
It isnt. I ended up building a small golang binary that ran as root and I could hit it with http calls to execute whatever I wanted. Built a message queue that would work through all the machines for eventual consistency. Worked great.
That's the whole idea of using a system based on declarative state. As soon as the system is back up, the agent can resolve state again. You also keep a central copy of the state of every agent.
You can absolutely do this by writing your own agent (or by writing a family of bash script, but they tend to grow pretty complex over time), ansible is just a framework to write that in a standardized way. It will also out of the box handle a number of common system state such as running services and sysctl triggers.
There are a number of similar systems such as puppet or salt, which are all variations of the same basic idea. 30k hosts are a lot, and will need sizing the system appropriately, but it's not an unusual configuration by any means.
That was the benefit of my system over what you are talking about, there was no dependency on a global state or centralized control surface. Each worker was autonomous and self contained and had enough intelligence to bring itself to the desired state on its own. All you had to do was one line curl|bash install my service and it would take care of the rest without any other external dependencies. No worries about having to have ansible try to connect over and over again until things were working. It would just magically fix itself.
Again, many ways to skin the cat, but at the end of the day, this solution really worked extremely well. I would do it again in a heartbeat.
