Why don't orgs test their updates? Every decent IT management/governance under the sun demands that you test your updates. How the hell did so many orgs that are ISO 2700x, COBIT, PCI-DSS, NIST CSF, etc. certified failed so hard??
(ToS/contracts will probably get you out of any damages.)
Testing for most organizations is usually either really, incredibly expensive or an ineffective formality which leaves them at more risk than it saves. If you aren’t going to do a full run through all of your applications, it’s probably not doing much and very few places are going to invest the engineer time it takes to automate that.
What I take from this is that vendors need a LOT more investment in that work. They have both the money and are best positioned to do that testing since the incentives are aligned better for them than anyone else.
I’m also reminded of all of the nerd-rage over the years about Apple locking down kernel interfaces, or restricting FDE to their implementation, but it seems like anyone who wants to play at the system level needs a well-audited commitment to that level of rigorous testing. If the rumors of Crowdstrike blowing through their staging process are true, for example, that needs to be treated as seriously as browsers would treat a CA for failing to validate signing requests or storing the root keys on some developer’s workstation.
Because historically orgs have been really bad with applying updates: either no updates or delayed updates resulting in botnets taking over unpatched PC's. Microsoft's solution was to force the updates unconditionally upon everybody with very few opportunities to opt out (for large enterprise customers only).
Another complication comes from the fact that operating system updates are not essential for running a business and especially for small businesses – as long as the main business app runs, the business runs. And most businesses are too far removed from IT to even know what a update is and why it is important. Hence the dilemma of fully automated vs manually applied and tested updates.
> Microsoft's solution was to force the updates unconditionally upon everybody with very few opportunities to opt out (for large enterprise customers only).
Not a Microsoft's fan, but this is not true. Everyone who has Windows Server somewhere, with some spare disk space for the updates, has this ability. Just install and run WSUS (included in Windows Server) and you can accept/reject/hold indefinitely any update you want.
1) the prevailing majority of laptop and desktop PC installations (home, business and enterprise) are not Windows Server;
2) kiosk style installs (POS terminals, airport check-in stands etc) are fully managed, unsupervised installations (the ones that ground to a complete halt today) and do not offer any sort of user interaction by design;
3) most Windows Server installations are also unsupervised.
> 1) the prevailing majority of laptop and desktop PC installations (home, business and enterprise) are not Windows Server;
They are not, but the point is elsewhere: that Windows Server is going to provide the WSUS service to your network, so your laptop and desktop installations (in business and enterprise) are going to be handled by this.
Homes, on the other hand, do not have any Windows Server on their network, that's true.
As a hack to disable Windows updates, it is possible to point it to a non-existing WSUS server (so that can be done at home too). The client will then never receive any approval to update. It won't receive any info wrt available updates either.
> 2) kiosk style installs (POS terminals, airport check-in stands etc) are fully managed, unsupervised installations (the ones that ground to a complete halt today) and do not offer any sort of user interaction by design;
That's fine; this is fully-configurable via GPO.
> 3) most Windows Server installations are also unsupervised.
IMHO law should require such a firm, or any firm that may impact millions of other people, i.e. including all OS developers and many others, to maintain a certified Q/A process, maintain a 24/7 coverage and spend X% on Q/A. Such companies should never be allowed to deploy without going through a stringent CD procedure with tests and such, and they need to renew the certificate annually.
These are infra companies. Their incompetence can literally kill people.
My point/problem is that EVERY company (sorry for the caps) that is ISO, PCI, COBIT, NIST CSF, etc. compliant MUST be doing this!! (again sorry for the caps)
So they drop half the 'safety' procedures once the auditor goes away? WTF! (I am semi-angry because there are so many easy solutions and workarounds to not fall for this!! (inside screaming).
How irresponsible must someone be to roll out something to 1k-5k-10k machines without testing it first??
I hope eventually law regards these companies as "infrastructure" companies, just like companies that build roads, bridges and such, that may and will kill people if not run professionally.
I'm not trying to enforce certifications because as a dev certifications always raise a bitter taste in my mouth. But those companies need certified processes that get re-certified every year. Sometimes even a cursory review from outsiders can find a lot of issues.
Updates do get tested. Windows updates can be held and selectively rolled out when a company is ready. As far as I can tell though, CrowdStrike doesn't give companies the agency to decide if updates should be applied or not.
Why don't orgs test their updates? Every decent IT management/governance under the sun demands that you test your updates. How the hell did so many orgs that are ISO 2700x, COBIT, PCI-DSS, NIST CSF, etc. certified failed so hard??
(ToS/contracts will probably get you out of any damages.)