Wait, isn't a point of NixOS is that your updates are guaranteed to be smooth? I haven't used it, but your story now worries me. I don't want to be back at "just reinstall the OS" days, even if configuration is portable.
> And it wouldn't upgrade to the latest version of NixOS because of some certificate issue
This sounds like Lets Encrypt changing it's protocol. Nix itself doesn't give a shit about such things, but maybe some sort of script was supposed to double-check the cert on boot (just speculating). While NixOS can be reproduciable, whatever software Let's Encrypt runs is obviously outside the host's control, so perhaps "leap-frogging" to the latest NixOS somehow helped.
This is incidentally a good example at why I am pretty against microservices (not saying Let's Encrypt is a bad or a microservice, but it is the analogue of one in this parable).
I couldn't figure out what it was, but it was definitely a problem with NixOS itself, possibly with expired root certificates or something like that. The upgrade command couldn't connect to the nixos servers. I know there was at least one root certificate that expired in the past 3 years, so maybe I just caught a bad interval for not upgrading.
On NixOS installing any package is the same as reinstalling the operating system. Reinstalling the operating system just took a minute longer because it had to do more downloading. (edit: in theory, I actually bet that the certificates sneakily leak mutable state into the system with the expiry process, i.e. I bet that if I reinstalled the same version of NixOS with the same configuration, I'd get an install that would actually be able to talk to the nixos package servers)
But you're right to be worried a little bit, and this is exactly why many people criticize NixOS. In theory it's the most perfect operating system, but in practice it's so complicated that when something's bugged it's really hard to figure out how to recover, and the documentation is not up to the task of educating you on what to do when things are wrong.
Also this situation was a bit special because it couldn't talk to the NixOS servers for upgrading (I presume due to an expired root certificate, but I couldn't figure it out), if it was any other problem, I could simply roll back the upgrade process (which would be instantaneous because it doesn't actually remove the old packages when you upgrade).
With a cloud server it's obviously less of an issue, but even with a desktop for Nix its less of an issue than any other distro.
Each time you update your configuration it's essentially reinstalling. When you boot you have the option to boot the previous version (which includes previous app versions and etc).
Reinstalling NixOS and updating it are extremely similar actions except for full reinstalls having data loss (formatting) and taking longer (cache is empty).
