I work for a hardware vendor and their customers are basically the same as Cisco's. So here is what happened with us - we had some hardcoded root password in the equipment during early development stages. And it was also known to some of the customers support staff for debugging. Then some of our tech leads raised this issue and discussed that we had to change the root password because it was a correct thing to do and don't wait for some breach in the field. After some usual months of bureaucracy someone did the push and actually changed the password in all versions.
Can you already guess what has happened next? :)
The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome.
PS: just saying, no need to involve NSA in something that is better and more often explained by stupidity. :)
> The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome.
Did a PM or sales engineer from your org ever sit down with the customer to figure out what they were doing with root access that your solution lacked?
My naive read here is that the customer saw it as easier to keep root access because your product didn't expose what they wanted or did it in a cumbersome way.
Not OP, but over here, PM and sales usually understand that what's important to me and our other engineers (building a really great machine, with a well-architected, secure, extensible backend) is completely immaterial to my end customer.
They're trying to build a car or a chair or an appliance. To them, anything that gets in the way of that goal is bad, anything that supports it is good. Sure, the customer's engineering department had a grand vision of skilled shift leads empowered to look up the fault diagnostics, call over a QC engineer, and get two-party authorization to ignore the fault and continue...but once it hit the floor, and getting QC over to the machine took a whopping 4 minutes, the operators all memorized their lead's and QC's passwords and were rewarded for cutting TAKT times by 3 minutes and 50 seconds. Yes, bad product containment is a huge deal in the boardroom, but there's almost no will to enforce it on the production floor.
They were providing root either way, the only change I assume was switching from the same root on all machines to a per-machine root password.
I can understand how that would be a pain in the ass if your legacy architecture (e.g. administration and reporting tooling, etc…) is predicated upon a unified root, you probably don’t even have a way to report the information so that machines and passwords could be paired.
Assuming a good way to provision keys is provided, I’d make the insecure unified root password into a paid option (with lots of warnings) the price of which would increase every year. This way the existing workflow of the customer keeps working short-term, but you CY the hell out of your A in case of breach at the customer’s, and at one point the line item will get large enough that the customer allocates time & money to fix their shit.
Charging for insecure systems incentivizes selling those same systems to customers.
Soon you’ll have sales guys regaling how this unified root password system is extremely convenient and you’re one of the few companies who provides this single-signature-security-system (S4 compliance). Next thing you know someone will gain an software patent on this to ensure no one else can provide it.
The solution is trivial: no commission on this. Possibly even negative commission, can't convince your client not to take this? You get less money for the sale.
This way sales are incentivised to only talk about this if not doing so would lose them the sale entirely.
For b2b or b2c your customers don’t want to sit with you to make your product better.
Yes they want your product better.
Myself as a customer I don’t spend time filling in customer reviews. I just want to go on with my life.
I see the same in company I work for, our customers don’t have time - they have stuff to be done - either our software helps them achieve their goals or they leave.
Yeah, they did want to maintain root access because they had started to rely on it, I'm not very familiar with exact details. We have an ongoing process of exposing different debug info in the restricted CLI, but it is a never ending task so they always want something which is only visible to the root.
Oh, I'm pretty sure we can just say whatever, even "we have decided and it will be so". There is only a tiny problem when a customer is bigger (and richer) than the vendor by an order of magnitude or more :) . They tend to control a lot, with such power imbalance.
Honestly doesn't get better in the reverse either. At Microsoft I'd sit in on calls with banks and other "big names", watching them pound the table and argue their way, usually involving something dangerous and borderline malicious use of the product, all the while thinking "we could buy your company with our cash on hand".
I think the key was that they're used to being the big dog in the room, and Microsoft made a point of not correcting them when that behavior was seen.
Do you mean can't be changed on a live system? It's very easy to change a hard-coded password, just recompile the source and flash. That could definitely be what OP is referring to if they pushed OTA firmware upgrades.
There is nothing as permanent as a temporary fix...and worse people will become dependent on the specific inner workings of the temporary fix. Once given...
Or instead of changing out of the blue what exists today and is probably soaked into whole following ecosystem around the product and would be massive PITA to change it as well, you could have create parallel way for login (i.e. via certificate) and tell customer that password access will be deprecated in 1-2 years, so they have time to update their tooling.
Can confirm the hardware vendor that I work for does the same. root password is given and customers are asked to change. No one wants to. Heck I would prefer if they disabled root login completely and used keys instead. But thats now it is. Some customers even enable login from web with those passwords. The stupidity is crazy.
Would be better to have such default creds parameterized by your build system. So every dev can choose theirs and you can choose unique ones per customer.
Do I understand this correctly? Are you saying you removed that root password, such that it wasn't possible to log in with another (but different) root password?
Or are you saying that you merely changed it, to one the customer no longer knew, but was the same password across all of your install base, for every customer?
It was at that moment a simple change of the password. The benefit was that people outside of our company wouldn't have access (thus not sharing it further), and then as a future improvement we could have implemented some better key scheme. But while customers want to have root access it is not an option.
Theoretically we could implement a scheme with individual passwords or keys per device (as someone suggested in the comments above), but such thing is not being prioritized by our company, so that's our fault.
What’s their complaint? Do they have a workflow that requires the root password all the time and it creates too much of a management to not have QWERTY1234 as root password?
Our lead developers tried, and they are way smarter than me). No luck, for some reason customers don't consider this a vulnerability. Even ones who take security more seriously and do some external pentesting on our equipment.
Smart people explain things with big mental steps they deem obvious (true or not), and less smart people disagree with these steps but are unable to point the finger on them.
In this case, only one version out of three (the middle one) had this vuln, so it's very likely it was introduced by mistake. But your scenario happens all the time too.
I hope you guys got an indemnification agreement or something else that make you not liable when the inevitable happens but, knowing how these things work, I doubt it.
Can you already guess what has happened next? :)
The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome.
PS: just saying, no need to involve NSA in something that is better and more often explained by stupidity. :)