Hacker News new | past | comments | ask | show | jobs | submit login

I've been deploying Microsoft based computer networks for 18 years... this would nearly top my nightmare list! I can't imagine what the alert level is at MS offices right now, but I bet they are expending every effort to get to the bottom of this ASAP :/



Is this a "lock the doors, man the battle stations" kind of problem? It seems super, super bad.


I doubt it. It looks to me a lot more like a test update that was pushed to production unintentionally rather than malware. A server compromise pushing malware as an update would presumably try to make the update look legitimate in order to maximize how long it went undetected.

Disclosure: MSFT employee, no knowledge of this beyond what I've read in this thread, though.



They've gone through the process to put the "test" update to production servers, so I'm pretty sure this is intentioned. Though they might thought this is ok wo/ harms.


> a test update that was pushed to production unintentionally rather than malware

This position seems to be based on the assumption that malware is less likely to be subject to mistakes than MSFT.

Want to check with the PR dept? ;)


It's based on the assumption that it's more likely that someone made a harmless but embarrassing mistake than that someone shipped a major security bug in a critical system, which was then exploited by someone simultaneously skilled enough to hijack the Windows Update servers to deliver malware and incompetent enough to completely screw it up.

Random text but real TLDs also just smells like test data to me. Someone smarter than me could probably do the statistical analysis to determine if this is actually random or pseudo-random garbage typed by a person. Given the fairly long runs of all-caps and all-lowercase in the description, I'd guess a person typed this out (presumably as part of creating a test or test suite).


You could rephrase that as:

> A test update getting accidentally pushed out is more likely than a compromise of the Windows Update system.


Appeared on WSUS as well...

NEVER turn on auto updates on windows. Read all the KBs, then choose to install, ALWAYS. If you have a corp network, use WSUS and stop all updates and check them. If the KB is content-free like the new ones, no install. I avoided the whole CEIP bag of shit and Windows 10 upgrade notification hell thanks to that.


I'll be sure to tell my 60-something mother to make sure she reads all KBs before deciding to install the updates that Windows is telling her is super important.

I'm sure this won't increase my load as the family technical support person at all.


Always turn on auto-updates. The likelihood of you missing or delaying an update and getting hit by an a known exploit is a lot more likely than an exploit getting through the update system or enabling a new exploit.


This is NOT good advice for reasons other than you're thinking.

Simple reason: if your computer updates it is not in a stable state until a reboot. Simply, your computer may not ask you to reboot after an update, but some software will (eventually, not every time) run very odd until you reboot.

I've seen this happen many, many times on my own machine and on many company machines I've managed.

It's best to install updates when you want to install them.


> It's best to install updates when you want to install them.

For company servers, I absolutely agree.

For corporate desktops, the administrators of WSUS (assuming an environment large enough to warrant running it) should approve them for installation after having had a chance to review them. Even so, the desktops should (IMO) be set to automatically install them and reboot once they are available.

For home PCs, just set them to automatically install and reboot and forget about it (n.b.: general rule; obviously there are/will be exceptions).

Personally, my own Windows machines (a grand total of two, running Windows 7 Professional, that are very rarely used), are configured to automatically download and install updates at 11:00 p.m. on Mondays. When an update is released that breaks things, this gives me about six days to hear about it and turn off Windows Updates until they get it fixed (assuming a typical Patch Tuesday release). A long time ago, I reviewed every update before installing them but not anymore. When one of those "drop everything and patch now!" updates comes out, I hear about them elsewhere and install them manually.


> be set to automatically install them and reboot once they are available.

I mean, it isn't the end of the world, but your helpdesk will be getting a few calls because your users are refusing reboots!


Haha, "refusing reboots". You know what happens to our work PCs when you click "postpone"? Fuck you, that was your warning. If you disregard it, in five minutes you get another 20 second warning to desparately hammer "Save" before your system force-reboots.

Not during the middle of the night, either - typically these get pushed out around 10-11 AM.


If you push them out during the day when everyone is working, sure. At 2:00 a.m.? Not so much.


No no no no no. I've watched entire networks of machines downed with auto-updates. Always read, always test.


It might make sense to pay a guy to make this his job for hundreds of computers on a corporate network, but there is no way in hell I'm keeping that close of track of updates on my home computer.

And when was this, over a decade ago? Also, what evidence did you have it was the auto-update system that caused the outage? Past performance is not a predictor of future performance.

Seriously folks, turn on auto-updates.


Seriously folks, turn on auto-updates.

I'll add my voice against this, if you have enough technical knowledge to check more carefully. I too have seen numerous occasions where something installed via Windows Update has taken out a machine and required significant action to restore it to normal functionality. My personal policy has long been security updates only, and even then I tend to do a quick web search before letting them install, which has saved me from the odd howler in the past.

On the other hand, the number of times I have seen a PC rendered inoperable or compromised because it didn't install a Windows update within 24 hours of the update being available is zero. Even if the PC is just a simple home machine, there's probably still at least some sort of firewall/router between it and the public internet, and just about any device like that is going to block unsolicited incoming traffic by default these days. To get compromised within that time frame you'd likely have to actively open something or visit somewhere that included an exploit for a new vulnerability, and while that is always a risk even on a fully patched system, it's not a big one for most people.


I guess you trust every website you visit then? And the ad networks used by the sites you visit...


Approximately nothing installed via Windows Update will protect most people from most threats they might find on web sites.

It's far more important to keep your browser and plug-ins updated to guard against those threats. Personally I also block almost all ads and other third party content, primarily on security and privacy grounds, which also significantly reduces the risk of running into malware while browsing.

If IE or Edge is your browser of choice then of course updates for those are going to be a priority for the same reasons. But even then, if someone has managed to compromise sites like Google's or Microsoft's so you can't even do a ten second web search before installing a patch without getting hit by an exploit that patch would have blocked, we're all in pretty big trouble anyway.


> Approximately nothing installed via Windows Update will protect most people from most threats they might find on web sites.

What about all the browser sandbox escapes that rely on kernel vulns?


Those are very rare. When they appear, there's inevitably enough panic and publicity to attract my attention, at which point I can evaluate and install the update myself when/if appropriate.

Fool me once, there won't be a second time, and that means you get to pull something like "UPGRADE TO WINDOWS 10 NOW!!!11!!" on me exactly once. Auto-updates are now turned off on my Windows 7 box, and they will remain that way.


The problem with this is that you are starting to treat the OS creator as hostile. This is not a good situation to be in. Microsoft has the equivalent of root on all windows machines so it is difficult to treat them as hostile. They could roll out an upgrade tomorrow that incorporated a critical kernel security update together with non-turnable off automatic updates and you would have to accept the patch or remain vulnerable. There are some that would argue that Windows 10 home editions are exactly this...

Basically in the medium to long term if you regard your OS creator as a potential threat you have very little option but to change OS...


Correct.

This is why fucking with Windows Update should have been the very last thing anyone at Microsoft would ever have wanted to do... or the very last thing they ever did do just before Security escorted them out to the parking lot.


The problem with this is that you are starting to treat the OS creator as hostile. This is not a good situation to be in.

No it isn't, but when they demonstrably are hostile to a degree, as with Microsoft's recent behaviour, that treatment is justified all the same.

It's important to separate updates that fix defects in the original product (security patches, bug fixes) from other updates that simply change the behaviour. The reason it's important is that from a legal point of view, there are often implied expectations of fitness for purpose and adequate quality when you buy something.

Software companies have for some time enjoyed a cosy position. For one thing, those kinds of rules have often not been enforced rigorously, partly because as long as the software companies were putting out bug fixes before large scale damage was done it has been pragmatic to let them carry on. Also, the law has often lagged the technology, with various loopholes meaning the same consumer protections that apply to physical products haven't always applied to digital ones and extra rights in digital products have been very rare.

However, the laws in a lot of places have been starting to catch up, just as modern trends in software have been pushing towards effectively forced updates. It would be a brave software company that rocked the boat by limiting access to security patches or other essential bug fixes in their push to get everyone upgrading all the time, though. The consequences if they push too far and the consumer protection authorities and/or business lawyers start to challenge them seriously could be extremely expensive.

Basically in the medium to long term if you regard your OS creator as a potential threat you have very little option but to change OS.

Unfortunate, but true. For now, I am still "changing" to Windows 7 for new machines on the Microsoft side. Personally, I'm betting that the inevitable backlash against ever-changing, never-owned, user-hostile, sub-standard digital products is going to pick up enough momentum over the next few years that either Microsoft or whoever actually kills their business will offer a better alternative before 2020 when Win7 support is scheduled to end.


> The reason it's important is that from a legal point of view, there are often implied expectations of fitness for purpose and adequate quality when you buy something.

I thought that software licenses and EULAs were designed to remove liability?

> Unfortunate, but true. For now, I am still "changing" to Windows 7 for new machines on the Microsoft side. Personally, I'm betting that the inevitable backlash against ever-changing, never-owned, user-hostile, sub-standard digital products is going to pick up enough momentum over the next few years that either Microsoft or whoever actually kills their business will offer a better alternative before 2020 when Win7 support is scheduled to end.

I could see the year of the Linux desktop coming eventually. But not as originally envisioned. I would not be surprised by a world where only specialists (developers, graphic designers etc) have desktops and the actual majority of computers in use are locked down iOS or Android kiosk type devices.


I thought that software licenses and EULAs were designed to remove liability?

No doubt they try, but the fact is, those kinds of documents can't override the law. In some places, the law imposes minimum standards on what is acceptable in a consumer (or even business) transaction, and software companies have tried to play the "But the EULA says..." card, and if it's actually tested in court they have sometimes lost. They often rely on people not being aware of their rights and/or not having the time or money or willpower to contest the issue.

Even that barrier may not help the software companies in the long run. Coincidentally, just today the UK introduced a sort of lightweight version of US class action lawsuits as part of a major revision of consumer protection law, as well as various other explicit consumer rights relating to digital rather than physical content.

I would not be surprised by a world where only specialists (developers, graphic designers etc) have desktops and the actual majority of computers in use are locked down iOS or Android kiosk type devices.

I'm afraid that is one all too realistic possibility. But there are reasons for hope as well.

For one thing, tablets and the like are convenient for small-scale content consumption and minor interactions, but they're awful for serious content creation or more complicated interactions. I don't think general purpose computers are going anywhere any time soon.

Perhaps more significantly, there is now a push in quite a few places to promote computer literacy and basic programming skills even at school age, and to spread the word that you can still tinker and make cool stuff, perhaps using devices like the Raspberry Pi and Arduino. We also have Linux and the FOSS community following a similar philosophy on the software side, of course, and actually one of the nicer results of so many kids having smartphones these days is that writing simple apps to run on them is now an attractive introduction to programming for kids who enjoy playing with technology. Ultimately, there is a strong human instinct to create and many people enjoy making stuff that is fun and interesting, and fortunately no amount of marketing is ever likely to change that.

Dumbed-down, locked-in devices may be the majority in the future, but I think there will always be room for powerful, flexible tools and there will always be room for innovation and creativity. It's a big world.


> No doubt they try, but the fact is, those kinds of documents can't override the law. In some places, the law imposes minimum standards on what is acceptable in a consumer (or even business) transaction, and software companies have tried to play the "But the EULA says..." card, and if it's actually tested in court they have sometimes lost. They often rely on people not being aware of their rights and/or not having the time or money or willpower to contest the issue.

On the one hand I hope you are right -- when I pay for software I have certain expectations which are often not met. On the other hand I hope that this doesn't apply to free (as in freedom and beer) projects. If the disclaimer of liability were to become invalid in e.g. the GPL a lot of good people could be put to a lot of trouble.


I have only checked up the Swedish law, but it distinguish between something given for free and when money or services are traded. The consumer protection laws are designed to identify a customer - merchant situation and then regulate it. FLOSS projects should have nothing to worry about here, and the only issue that I have heard is when projects sell CD's.


> Past performance is not a predictor of future performance.

Hogwash. If this were true, there wouldn't be the concept of reputation.


This was Oct 2014. KB2949927.


So your "read all the KBs and choose" strategy would have prevented this, really? You would have read that KB2949927 adds SHA-2 cryptographic support and said "No, we don't want that one. We'd rather stick with deprecated SHA-1"?


No we go "hmm that might fuck something up; let's try it on a test VM" or at the very least google and see if anyone else has any problems.


Do you actually deploy every update to a VM to test it? Would your testing have caught this issue (which apparently only affected people who'd explicitly disabled the bitlocker service)?

You could also just wait a week for anything noncritical to allow others to flush out any issues, which is a more time-efficient strategy than manually reviewing gobs of KB articles.

For most people, disabling auto-update is a horrible strategy. If you have a central team actively managing updates with WSUS, you can get away with this. For the vast majority of people, turning off auto-update just means they stop installing updates at all, which is the reason auto-update is the default.


auto-updates have goosed more windows systems on me than malware. I'm not even a sysadmin.

there's been a few comments in the wild saying windows 10 can install without your permission. it may even be true, a bug.

so yeah I 'seriously' disagree with you.

https://www.google.de/search?q=crash+tuesday+broken+windows+...

scroll back through the years.


> auto-updates have goosed more windows systems on me than malware. I'm not even a sysadmin.

I would hazard to say that if you were a sysadmin, then this would not be the case.


Eye-witness accounts are the least reliable source of evidence.



I have auto-updates turned off for absolutely everything. I read patch notes before upgrading anything. Especially on my personal computer.

In nearly 100% of all scenarios that I've ever, ever had issues with anything. It's because an update broke something - sometimes irreversibly. Auto-updates are a larger threat factor for me than malware or niche security threats that only attack certain features that I don't utilize (thus I'm not a potential target for that attack vector).

>Past performance is not a predictor of future performance.

In some contexts I agree with you. With programming - I disagree entirely.

Bad programming habits are a great predictor of continued bad programming habits. When the same threat vector pops up again and again in a program it's because the programmer isn't learning from past mistakes. Video game bugs are proof of this.

The first thing many glitchers do on a game I play is test variations of old, patched bugs on new updates to smuggle items out of areas that you shouldn't be able to smuggle items out of. It almost always works. Because the general, underlying problem has not been fixed. They just throw band-aid patches on it after the fact and forget to apply the band-aid patch to future updates, allowing the bug to resurface. The same variations of the same bug have been resurfacing for over a decade now.

Bugs resurface all the time in software, because programming is really tricky to get perfect and humans repeatedly make the same mistakes time and time again.


System exploits wouldn't be doing much good for the exploiter if they left your system unusable.


You're falsely equating "broken updates" and "security exploits" and I'm not sure why. I thought I was clear that I was comparing the two as separate negative occurrences with one happening more frequently than the other. Not that one would cause the other...

An upgrade provided by the company that is completely legitimate that completely renders the program unusable or destroys my workflow has happened far more often than my system being compromised has ever negatively affected me. I could count on a stub the number of times I've known my system to be compromised. I'd have to count on my hands using a binary method to count the number of times a legitimate update was botched.

I still update my programs. I just don't let them do it automatically. Leaving an extra few attack vectors up for a few days/a week to let the patch mature or for an emergency-fix patch (i.e. 30-->30.0.2 "Super major security exploit was live for 3 hours but we fixed it") to be released has always worked to my benefit. I've never had a negative outcome for waiting a few days to patch. I don't have to deal with botched releases or newly opened attack vectors. Instead I get to listen to the canaries in the mine.

Also what happens when an auto-updater gets compromised? I get to listen to the canaries. You get to be one of the canaries. So for that, I thank you.


You're right, but for reasons that people may not realize right away.

It isn't the content of the update you should be weary of (make this decision for yourself if you care this much) but it is the act of updating machines that will cause problems.

When a Windows machine updates (yes, even as of today - I had this issue just last week) it is in an indeterminable state until a reboot, even if the update doesn't require a reboot.


Are you rebooting all your windows servers for each little update too then?


No. In a perfect world yes, you would update immediately. However, it isn't practical. Define what's a good time frame (week, month, daily) for your server, its role, and your manpower and stick to that schedule.

I can definitely say that it is better to wait to update when you can reboot than to update immediately. Of course, if there is a really bad vulnerability, update immediately. Let the user know it's an exception.


This was true until Microsoft started shipping their own "exploits" (read: updates that are more for their benefit than yours.)

After the Windows 10 debacle, I'm looking to get off of Windows as soon as I can afford to. Whoever decided to turn Windows Update into an advertising platform needs to be fired -- it's that simple.


Whoever decided to turn Windows Update into an advertising platform needs to be fired

Not that I ever plan to run it, but my understanding is that Windows 10 itself, not just Update, is an advertising platform.

Nobody in Vegas is taking odds that someone will be fired. But I'm in full agreement with you. Someone should be fired.


auto-installing new security updates but having a delay of 24-48hours before installing them might be a safer alternative


lol, thanks 'moron4hire' I almost took you seriously :-)


Very good advice... I agree. Took me a long time to figure that system out, but it sure works well!




Applications are open for YC Winter 2020

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: