> The .sys files causing the issue are channel update files, they cause the top level CS driver to crash as they're invalidly formatted. It's unclear how/why Crowdstrike delivered the files and I'd pause all Crowdstrikes updates temporarily until they can explain.
They weren't last time I looked. They seem to contain a bunch of different things, but you can absolutely download and parse them without needing to decrypt anything.
If you have a Crowdstrike customer ID (CID) — which you can pull from any device that has the implant — you can request any channel file you want from their file server.
Ask for metahash+/cfs/channelfiles/0000000291/<YOUR CUSTOMER ID>/C-00000291-00000000-00000001.sys and you should get something that starts with:
I woke up after they'd already pulled the bad update, and I don't have an affected system.
If I look at the C-00000291-00000000-00000032.sys version that Crowdstrike LFO serves me, I get something that looks superficially reasonable (not random garbage or full of zeroes).
I would share it, but my understanding is that channel files specifically can have different contents for different customers (as opposed to other files like their Linux kernel drivers, which is definitely the same giant .xz blob for everyone). So I'd rather not upload something that's potentially tied to a specific customer/company without asking for permission
But if you have a valid customer ID from Crowdstrike, I'm happy to point you the little tool I use. You can request old versions of channel files from LFO and look at the diff as much as you like. But I can't guarantee you'll actually be getting a file that's obviously broken or full of zeroes like some people are describing. My C291 0.32 looks superficially normal.
It's a crowdstrike update file with a bug in it, from what I gather. This makes your Windows machine go blue screen and stop working as it starts up. If you manage to remove it by various methods, it doesn't run and you're fine.
More informed people will give you more details, but this kind of AV software often has privileged access to the OS, so it can scan your files. The same privileged access also means it can really mess things up if it's not well tested.
By contrast your ordinary python or VBA script should not be able to blue screen your machine, especially not during startup.
> By contrast your ordinary python or VBA script should not be able to blue screen your machine, especially not during startup.
This is misleading. If you run a python or VBA script on startup with elevated permissions to e.g. setup a driver (with a bug) or do hardware configurations (in a way which doesn't work) then it will as much blue screen as a sys file.
The issue is not that it's a `.sys` file, but that a high privilege administrative task run at startup fails so hard that it blue screens.
(In windows the sys file extension is basically just a marker that the file is part of the "system" without being clear what it is, it can be anything from a data blob,over a data base, over python/VBA files to a device driver or a blob combining multiple of such things. They mostly contain device drivers and hardware configuration, but there are some which are basically just the equivalent of /etc/hostname)
Every Windows executable can crash a Windows system with the SE_SHUTDOWN_NAME privilege (which, contrary to the name, not only allows safe normal shutdowns) if they want to, though, which most of the time every user has.
The "if they want to" part is the difference, kernel drivers can easily cause BSoDs on accident, e.g. by writing to an invalid address, normal processes can't, they have to jump through hoops by choice.
On another note, I know nothing about cybersec, is there a reason for which antivirus on windows run at ring 0 while I read that on Linux and Mac they don't have kernel level access?
It's not an anti-virus, it's intended to monitor all and everything on the machine. You^WAn attacker might want to hide what you're doing and thus it runs at that level.
You may be right but they do market it as "Next-Generation Antivirus (NGAV)"/"Antivirus with Threat Intelligence" probably because it's a word people are familiar with
I understand that these "NGAV" must be in ring 0 (device driver) because they want to inspect more things directly. And be more protected there, avoiding being attacked. I'm not sure they can achieve this.
The cyber unit within IT is more likely, those ones are besotted with ticking compliance checkboxes, the delegation of responsibility and a game of musical chairs at any cost.
It is even more likely that IT was at loggerheads with cyber, but nowadays cyber seems to be able to trump everything and everyone.
It's a binary format specific to Crowdstrike for delivering updates. It's not a DLL or a plain patch, it's their own file format. And they have a lot of different things in there (config, regularly updated data, instructions to download the next driver version, etc etc)
These files are apparently just definitions/input for the actual CrowdStroke driver, that presumably someone named .sys so it sounds more scary and important.
Peak irony then that you can just delete them. Surely an antivirus should recognize when its being rollbacked?
I don’t have a Windows setup, let alone one with CrowdStrike installed, but for this, you have to be in recovery mode to boot, not necessarily to be able to delete that file. For that, admin access may be enough.
It likely contains a bug but is an uncorrupted file. Corrupted to me means the file was changed or modified in a way from it's intended state and likely won't run anymore.
Star is part of a command to delete any file matching a wildcard `C-00000291*.sys` meaning "starts with 'C-00000291', has any kind of text after that and ends with '.sys'". Actual files appear to differ for every machine
This exchange reminds me of a time many years ago where I was reading a computer magazine and they talked about “Windows 9x” and I thought they were referring to some version of Windows that I had never heard of.
Only a couple of years later did I understand that it was simply a shorthand for talking about Windows 95 and Windows 98.
Edit: And looking now at Wikipedia apparently it also covers Windows Me. (The Millennium Edition that came between Windows 98 and Windows XP.) https://en.wikipedia.org/wiki/Windows_9x
Also the reason that Microsoft skipped from Windows 8 to Windows 10. Too much risk of apps testing the first 8 characters of the Windows version against "Windows 9" and concluding they were running on 95/98 rather than 9.
Hm, is there any website that explains why C-00000291*.sys caused the widespread BSODs? For example, was it some kind of definition file that was accessing invalid memory locations?
New to IT, but this sure seems like a huge security risk. Even though the CEO and others have said otherwise. People who are more experienced, please let me know if I am wrong.
Wouldn't want to be the guy who pushed this particular commit. It's ironic that the company that is supposed to prevent this sort of thing causes the biggest worldwide outage ever. Crowdstrike is finished. Let's hope this will result in at least a small increase in desktop Linux market share.
When the world calls for blood against your organization, it's a test of the organization's character: will they throw a scapegoat under the bus (even if there is a directly responsible person) or will they defend their staff, accept fault, and demonstratively improve process?
More importantly, the companies that enabled auto update from a vendor to production rather than having a validation process. This sort of issue can happen with any vendor, penalising the vendor won't help with the next time this happens.
It’s both. If you’re an engineer and you push out shitty code that takes down 911 systems and ambulances, you f’ed up. Push back against processes that cause harm, or have the potential to cause harm. You are ultimately responsible for your actions. No one else. The excuse of “I was just following orders” has been dead and buried since WW2.
Yeah, ideally management should know better. But management aren’t usually engineers. Even when they are, they don’t deal with the code on a day to day basis. They usually know much less about the actual processes and risks than the engineers on the ground.
if one of the people i manage is not up to the task the fault is mine. I've hired them. I should setup a system of hard gained trust and automation to avoid or at least minimize them fucking up. When fuckups happen, they are my fuckups. Critical systems don't survive only on trust, obviously. If I don't setup the teams and the systems properly, my bosses will also take the blame for having put me in that position.
I'm not advocating for lower layers to avoid responsabilities. But if an head needs to roll you should look above. That said, peole are hardened by fuckups, so there are better solutions than rolling heads, usually.
Right. In one sense, what we're talking about is different ideas on how companies / teams work. There's a wonderful book called "Reinventing Organizations" by Laloux that I recommend to basically everyone. In the book, the authors lay out a series of different organisational structures which have been invented and used throughout the ages. The book talks about early tribes where the big man tells everyone what to do (eg mobsters), to rigid hierarchies + fixed roles (the church, schools) to modern corporations with a flexible hierarchy, and some organisation structures beyond that.
The question of "who is ultimately responsible" changes based on how we see the organisation. In organisations where the chief decides everything, its up to the chief to decide if they should place blame on someone or not. In a modern corporation, people at the bottom of the hierarchy are shielded from the consequences of their actions by the corporation. But there's also a weird form of infantilisation that goes along with that. We don't actually trust people on the ground to take responsibility for the work they do. All responsibility goes up the management hierarchy, along with control, power and pay. Its sort of assumed that people who haven't been promoted are too incompetent to make important choices.
I don't think thats the final form of how high functioning teams should work. Its noble that you're willing to put your head on the chopping block, but I think its also really important to give maximal agency to your employees. And that includes making people feel responsible and empowered to fix problems when they see them. You get more out of people by treating them like adults, not children. And they learn more, and I think that's usually, in the long run, better for everyone.
I agree that if a company has a bad process, employees shouldn't be fired over it. But I also think if you're an employee in a company with a bad process, you should fight to make the process better. Never let yourself be complicit in a mistake like this.
> It’s both. If you’re an engineer and you push out shitty code that takes down 911 systems and ambulances, you f’ed up.
This is wrong. If a company is developing that kind of software is the responsibility of the company to provide a certain level of QA before they release software. And no, it's not that "engineers are pushing out shitty code", but that the shitty company allows shitty code to be deployed in customers' machines.
Many major companies have post-mortem reviews for this kind of thing. Most of the big failures we see is a mix of people being rushed, detection processes failing, a miscommunication/misunderstanding of the effects of a small change.
One analogy is rounding - one rounding makes no difference to a transaction, but multiple systems rounding the same direction can have a large scale impact. It's not always rounding money - it can be error handling. A stops at the error, B goes on, turns out they're not in sync.
Which guy is it? The person who pressed the button? The manager who gave that person more than one task that day? The people who didn't sufficiently test the detection process? The people who wrote the specs without sufficient understanding of the full impact? The person who decided to layoff the people who knew the impact three months ago?
Unlikely, just as Solarwinds wasn't finished when they distributed malware that got government agencies hacked. You underestimate the slow turning radius of giant company purchasing departments.
Enterprise Linuxes also employ Crowdstrike or similar "security" products as mandatory part of their IT deployments. Often (always?) this is due to companies wanting certification for their secure processes, in order to sell to government or large corporations that require them.
Why the fuck didn't MSFT just do blue/green canarying? No update should be rolled out to a billion devices at once until it's baked in a million devices for a bit, and that only after baking in 10,000 devices for a bit.
Crowdstrike broke the update for Windows only this time. Although look around, they did a bad update on Linux earlier this year (although that only broke some of the Linux installs).
