> As someone who has developed a Linux based appliance with over 100k live units across the globe, it seems insane to NOT have access to the thing you're selling and that you have to maintain.
I’ve developed Linux devices selling that many units (and more) and I’m baffled that anyone would think this is a viable way to handle things at this scale.
Units like this should have a firmly read-only Linux firmware that can only be changed by signed updates. The only data you would actually get or modify is the diagnostic data or the contents of the settings. Both of those can be sent through mechanisms that shouldn’t require SSH access.
The correct way to handle this is with a debug info feature. Put something in the app that will zip up logs and configuration files and send them in for support, with the user’s explicit permission obviously. If you can’t figure it out from logs, you can use their config files to clone the situation on a device in the office.
The bigger issue is: Who are you going to task with SSHing into customer devices? With 100K or more people filing support requests, it would be insane to have engineers handling those requests with anything having to do with SSH. It would be equally insane to hand off access to customer support people and give them the keys to SSH into customer devices.
I agree that that is the gold standard. Having an immutable Linux that is well tested on your own hardware and upgraded like that.
At the time I inherited a system that had 30-50k units deployed and was updated via Debian/APT. Older units were running Ubuntu 10.04 (it was 2016) and were hopelessly outdated. We managed to pull every single device to Ubuntu 16.04 and designed a fully automated image based update mechanism for them (I've linked it in other posts). We tried for read only base systems, but it was too tricky, so images stayed read-write, with migration of configs across upgrades.
At the time, customers even had access via SSH (similar to NAS devices these days).
I think what you are describing works for well defined hardware with a medium complexity software stack, or at least something that is limited in terms of epipheral device usage.
The appliance I was managing was heavily using raided disk, ZFS, loops, dmsetup, and many other Linux tools that we have all seen fail in horrible ways.
Not having SSH access, and not being able to diagnose lockups or hanging progress (D state issues) in a live system would have severely crippled us in being able to fix these issues. Many of them I'm sure we would not have been able to. We had failing disks, slow disks, failing RAM, hanging loop devices, corrupt loop devices, hanging ZFS, hanging ZFS, hanging ZFS, many of its bugs we fixed upstream, and and and...
On top of that, we had a "bring your own device" product that literally allowed people to use whatever hardware they want. That makes the read only firmware thing ever trickier.
As said in the beginning, I agree with you in principle, but there are many cases in which it's not as black and white. And I can fully understand the rationale of providing remote access.
Side note: I would have never expected to be down voted on HN for expressing an opinion in a respectful manner about a subject that I have knowledge about, just because it is the "unpopular" opinion. On Reddit, I'd expect to be downvoted for something folks don't like, but on HN in thought the button is just for use against trolling and such.
Re your side note, yes this is the new HN. People use the downvote as a lazy "I disagree". On the plus side, that's mainly the people who tend to read and react within the first 30 to 60 minutes of a comment being posted. After that the votes usually right themselves.
I’ve developed Linux devices selling that many units (and more) and I’m baffled that anyone would think this is a viable way to handle things at this scale.
Units like this should have a firmly read-only Linux firmware that can only be changed by signed updates. The only data you would actually get or modify is the diagnostic data or the contents of the settings. Both of those can be sent through mechanisms that shouldn’t require SSH access.
The correct way to handle this is with a debug info feature. Put something in the app that will zip up logs and configuration files and send them in for support, with the user’s explicit permission obviously. If you can’t figure it out from logs, you can use their config files to clone the situation on a device in the office.
The bigger issue is: Who are you going to task with SSHing into customer devices? With 100K or more people filing support requests, it would be insane to have engineers handling those requests with anything having to do with SSH. It would be equally insane to hand off access to customer support people and give them the keys to SSH into customer devices.