It's starting to feel to me like modern day Microsoft has some kind of a Teflon coating. In the last month alone they've had their flagship cloud services down for extended periods of time due to DDoS attacks, had their auth cookie signing keys compromised and abused at scale, and had a lawsuit reveal emails about how they were planning to use their cash at hand to buy all the content and destroy's Sony's gaming business.
And none of it has any impact, nobody seems to care.
The comments in this thread are crazy to me. Yes, dealing with targeted attacks by nation states is hard, and occasionally you'll fail. But that's part of the reason to pay Microsoft rather than try to host your own. In that situation the absolute minimum expectation is for MS to follow basic best practices. Nothing about this story suggests they've been doing that, it's just one wtf after another.
And their communication on this has been so bad an unclear that it has to have been intentionally obfuscatory.
FWIW, I have long asked such questions about MSFT. In the 1990s there was more open criticism of the company's practices. Not anymore. As suggested by the parent, "nobody seems to care." IMO, this sort of Teflon coating attitude has existed for many, many years. It seems that MSFT is above fault and responsibility. Feels like they have some sort of "leverage" on corporations and governments and other organisations.
I think Google and Facebook ended up surpassing the old Microsoft evil, then regulators took the eyes of Microsoft, then after slowly testing the waters, they are going hardcore into all kinds of shady shit.
Windows OS is basically just a wrapper for data mining for Microsoft’s very large, but ignored, adtech businesses (plural).
A reasonable assessment, IMHO. Each has followed the other. I have observed each of Google and Facebook seeming to believe it has the same sort of "leverage" over, e.g., national governments, as well as news organisations that produce content. A most recent instance is how Google responded to yet another government asking it to pay the country's news media for using their content as bait for ad targets^1. "OK then, we'll just prevent your citizens from accessing news through Google." This sort of "negotiation", read: lack thereof, really reeks of entitlement.
1. Purposely commercial usage, not merely "sharing a link" to some content on the web.
The news stuff is more a cascade of interrelated failures between governments, news outlets, and tech companies.
Governments do a bad job of funding news production, newspapers didn’t take the internet seriously and made a massive mess out of it, and the tech companies just sucked out the remaining value and killed news.
The fact is the big tech companies make next to nothing on news. It’s rational to just stop doing it if the value goes negative.
If government wants to levy a real tax and use that money to fund and support news, fine - but these laws are based on a mistaken premise that somebody is making buckets of money in news. Nobody is.
"The fact is the big tech companies make next to nothing on news."
Perhaps these so-called "tech" companies need the traffic generated by people reading and sharing news stories. It makes no sense for "Google News" to exist if it makes next to nothing. It would be "irrational".
Google and Facebook keep track of every user navigation to a news story. The links are not direct. Of course this is for purely non-commercial purposes. Data collected is not used for commercial purposes. Yeah right.
If it isn't valuable to Google to intermediate peoples' access to online news, then why doesn't Google stop doing it.
IMO, the issue is not that content producers such as news organsations are asking too much, it's that they are asking at all. The so-called "tech" company "business model" only works if the company does not have to pay anyone for content. If every producer whose content is being used by these companies to support and generate revenue from advertising services suddenly starts demanding payment, no matter how small, these "Big Tech" companies and their Golden Goose are cooked.
When your entire multinational runs on M365, SharePoint, OneDrive, exchange online, teams, Azure AD, you really don't want to know about its flaws because changing to another provider costs millions.
And what other provider? There's only Google and they have only a tiny share of the enterprise market. Everyone is in the same boat.
Ps speaking about productivity SaaS platforms here, not generic cloud.
If you can't get away from Microsoft, it's because you aren't trying. That is what they've counted on, and that's why they've invested as much in terms of trying to have their products taught in schools as early and widely as possible, and why I now refuse to support or teach kids on Office alone when I can teach them on LibreOffice + Office + Google docs instead.
Literacy folks, it's the key to the shackles that vendors have been doing their damnedest to slap on you over the years.
You're not serious I hope? While these apps have some basic functionality, they would not work in the modern world. Even collaborating has become so much better (remember all the emails in the mid 2000s "CAN WHOEVER HAS OPENED REPORT.XLS PLEASE CLOSE IT SO I CAN EDIT" :). And that was already a huge improvement on the Lotus 1-2-3 days. Now we can seamlessly collaborate on the same document.
Sometimes I'm genuinely surprised we got so much work done then. Things are so much more efficient now.
I literally haven’t heard of those two alternatives. Let’s be honest with ourselves. the feature set granted by Microsoft enterprise products, the single pane of glass for managing users and access controls, and the low ops overhead compared to hosting all that stuff yourself (not to mention integrating it in a compelling way that less technically inclined people can understand, and training people up on how to use it), is likely to be an absolutely ridiculous investment.
I doubt you’ll find any company with an IT department with enough resources to roll their own Office 365.
> I literally haven’t heard of those two alternatives.
That's probably because you probably weren't around in the 80s/90s. Netware was a network server tech from before Windows was a thing. All discontinued now so hanging your enterprise off this tech would be a ridiculously bad idea. Seriously, we try to get rid of tech debt, not add it :')
GroupWise still gets some token updates but really. I'm super super happy we moved off Lotus Notes which was in the end a vile piece of unstable stinking java. When it started it was pretty good, sure, but the attempts to keep up with the times made it unstable.
So no worries, they are not serious replacements for today's productivity apps. They were from a time when life and business happened at much lower speeds. When people could wait a few days for a letter to be mailed in.
Try putting a picture in Lotus 1-2-3, connecting to a remote data source, doing a pivot table etc :P Not to mention doing any kind of realtime collaborative work.
I used to consider their cloud products secure. When bings home page (plus i don't know how many tenants) was found incorrectly protected a few months ago i stopped. Google will discontinue your product or lock you out. Microsoft will let third parties in . We need better "office" platform vendors.
Hilariously, Azure PaaS offerings of Microsoft products are notably worse than what is available in AWS.
Examples:
- SQL Server Analysis Services (SSAS) has no Azure PaaS, but there is an AWS PaaS for it.
- RDS for SQL in AWS allows setting a time zone, Azure SQL Database is UTC only, breaking the majority of existing databases when migrated. (They confuse what "developers should do" with what "developers actually do".)
- Azure's PaaS for Active Directory (Azure AD Domain Services) can only be linked to an Azure AD tenant, it can't be used as a standalone authoritative domain. AWS allows this scenario.
Essentially, every Microsoft product you can think of is easier/cheaper/better on AWS instead of Azure.
If Microsoft's CEO was still Bill Gates, heads would have rolled a long time ago.
RDS for SQL Server does have some features missing from Azure SQL Database (SSAS and time zones as you said, but also SSRS). As a PaaS service it's pretty rough around the edges, though: scaling requires a long outage, copying a database between servers requires doing a backup to S3 and restoring from that, and the AWS backup/restore operates on the server level rather than the database level.
Where AWS really showed its unsuitability to host applications on the Microsoft stack was that AWS doesn't have a PaaS way to host .NET Framework web applications; IaaS is the only option.
So both .NET Framework web application nor Microsoft SQL Server database were noticeably harder on AWS than Azure for us. Standing up one IaaS server to host SSRS in Azure was a lot less work than accounting for all the gaps on the AWS side would have been.
Azure SQL database doesn’t support .BAK files. It can only import or export in BACPAC format which is SQL statements in a zip file and is an order of magnitude slower to restore.
We’ve had issues where large databases would take forever to cut over to the PaaS database and cause long outages during migrations.
It’s these random feature gaps that make these platforms difficult to deal with.
BACPACs are the only option to get a database out of Azure SQL. And yes, they do suck, for the reasons you outlined.
For the initial migration to PaaS, whether Azure or AWS RDS, it’s faster and easier to use replication instead of BACPACs.
For moves within the Azure SQL environment, they have database copy functionality that can be utilized. (Which RDS doesn’t have, and I would have sorely missed if we hadn’t pulled the plug on AWA.)
There's a better way to get data out of Azure if you can handle a bit of manual scripting (way worth if for large tables).
Setup linked-servers from your local server (via Server Objects / Linked servers), once you've done that you can just do "INSERT INTO bigtable(..) SELECT .. FROM [remotename].dbo.bigtable;" , it'll tax the transaction log but it's a magnitude faster than bacpac's,etc. Sadly Azure SQL doesn't support linked servers so one of the servers has to be outside.
There's an annoying difference between 'Azure SQL Database' and 'Azure SQL Managed Instance'. I recall the Managed Instance does support timezones and SSAS, while the ordinary offering also misses other useful stuff like Resource Governors.
Managed instance has time zones, but not SSAS. It does have SSIS, but only via Azure Data Factory. This means you need additional VMs and the performance is much worse because there’s always an extra network hop.
did their reputation ever change from being a company that produces mostly bad quality unreliable software, that is only ever purchased by someone that wion't have to use it on a daily basis?
Teams, Sharepoint, PowerBI, the ad platform that is Windows, Azure
did their reputation ever change from being a company that produces mostly bad quality unreliable software, that is only ever purchased by someone that wion't have to use it on a daily basis?
Yes. Especially on HN.
Microsoft used its video game systems and Linux flirtations as Trojan horses to convince a generation of tech-centric people that the bad old Microsoft of yore is just something that old people worry about and that today MS is all rainbows and unicorns.
HN is full of people willing to fall in their swords for MS because they weren't around for its last round of despicable behavior, and bringing it up just means you're out of touch.
Microsoft don't make tens of billions of dollars in revenue every quarter by "convincing people of rainbows and unicorns". Azure didn't pass $100Bn in annual revenue in 2022 by "Trojan horses".
> "HN is full of people willing to fall in their swords for MS because they weren't around for its last round of despicable behavior"
When was this last round? Is HN full of 20 year olds? Microsoft is really popular with the youth of today? That's nearly halfway back to when Microsoft was founded.
They went through a period (Windows 7-ish) where they weren't especially bad about anything that I can recall. Pretty stable OS and drivers, Chrome was taking over the Internet and they weren't putting up much of a fuss, they were sort of focused on phones and tablets which didn't cause harm to anyone else in the market.
All brilliant. In the sense "the user of the software is the group, not the individual". All let normal non-technical employees of companies do things in minutes that they can't do at all in the competition.
Exactly this is going on for at least the last 30 years.
Nowadays they just seem to have better PR / marketing so the usual daily scandal has exactly zero impact on them.
But form the perspective what they're doing, and how they're doing it, nothing ever changed. Microsoft is Microsoft.
---
I've read at some point some revealed internal memos or emails from Microsoft, I guess from a court case as the PDF was full of scans of printed out stuff. It was about their internal strategy regarding competition (and actually their "partners" who they count to their competition more or less, only that they see them as a kind of "useful idiots"). Frankly I can't find the PDF right now but it would be really helpful here! It was likely something famous as I wouldn't have downloaded it back than. Who has the right link?
0-day is such a poor word. We should really be calling it by the generic cross-domain term, a defect, in this case a security defect. Microsoft is selling a defective product where new defects are regularly identified and regularly cause failures and harm.
It is generally frowned upon or illegal to sell defective products. In many other industrys the discovery of multiple, repeated defects in a product allows the customer to get a full refund with protection from retaliation. Only in software do we call defects a oopsie and throw our hands up in the air as if nothing could be done and let companys off the hook.
Normalizing this process by handling trillion dollar companys with kids gloves is ridiculous. They are selling defective products and should be treated like it.
As someone with opinions on english usage (I was just eye-twitching at another front page HN story - the misused possessive on 'pilot's licence') I can see why '0 day' may not perfectly capture or convey a certain type of exploit, but I don't think this would improve clarity.
I've always distinguished a defect as a fault that was known prior to shipping (and hopefully documented) and a bug is a fault discovered after shipping. Draw your own analogies for SaaS, I guess.
But given another popular distinction between defect and bug is more around design vs unexpected failures, it may be hubris to overload either of those terms with all the extra meaning that's already captured by '0 day'.
As TFA notes, it'd be preferable if Microsoft didn't weasel away from commonly used and well understood words like vulnerability & exploit, or if they didn't eschew industry standard CVE reporting.
A defect is a fault or flaw that prevents the stated usage of a product. That is the plain meaning in literally every industry. A defective product is one fails to achieve its stated specifications, guarantees, or use.
A airbag that, unknowingly at the time of manufacture, explodes after exposure to high heat or humidity has a defect. The product, now knowing that it contains a material defect, is defective.
The use of bug, 0-day, vulnerability, exploit, etc. when making PR statements about security defects in software is a deliberate effort to downplay the severity of their failures. They deliberately avoid the use of the word “defect” because in many industrys that has literal legal consequences. At many companys, putting the word defect in a email has the general counsel descend, that is how serious it is. In contrast, calling a flaw that prevents the stated usage of your product a “bug” lets you sidestep all of that by mischaracterizing it as a legally meaningless “oopsie”.
These flaws are defects. They clearly cause the products to not achieve their stated goal and the failures cause demonstrable harm. Calling them anything less is just handling these trillion dollar companys with kid gloves which is how we got into this systemically incompetent security mess.
They need to be held to the same standards we hold other companys to and fix their damn products or get out of the businesses that they are too incapable to do acceptably.
If you are selling to a general consumer via a one-sided contract then there are implicit guarantees of fitness for the plainly advertised purpose that can almost never be waived at least in the US. Note that this does not usually apply to gifts so non-commercial OSS is likely unaffected.
If you are selling to a business customer then they are accepting those terms. If they wish to then go on and use that product in their delivery to a general consumer then they are accepting the liability of verifying or making it fit for their purpose. If they wish to provide other guarantees then they are also responsible for those.
As noted, I don't believe you have consensus on the 'literally every industry' interpretation and
use of the word defect.
But even so, you seem to be claiming that the product doesn't match "stated specifications, guarantees, or use" (I find 'use' a bit vague, but nonetheless.)
Can you point at the stated specifications, guarantees of AAD / Entra that are contraindicated by this exploit having occurred?
If written guarantees have in fact not been met - that could be really interesting.
> These flaws are defects.
You use the word flaw a few times - as something similar to a fault, here as a synonym of, or a precursor state to, a defect. If we're being persnickety about language ...
That is the plain meaning in basically every industry. That software people are allergic to it is because it would mean their products are defective and that is a no-no word because it means they are legally responsible for their failings. I am not being pedantic; coming up with highly-specialized words for the express purpose of downplaying failures and expecting others to play along is being pedantic and consumer-hostile. I am advocating consumer-friendly, industry standard language in this swamp of doublespeak.
Are you arguing that the intended functionality of Exchange is leaking emails? No.
Are you arguing that customers find their emails being leaked a desirable or neutral option? No.
Are you arguing customers will accept a product that they know will leak their emails? No.
So, a flaw that leaks emails makes the product unacceptable to the customer. The only possible defense for Microsoft here is that they accepted no liability or disclaimed all liability for leaking emails.
It is certainly possible that Microsoft did that in their contracts. I applaud the skills of their salespeople and lawyers if they tricked the customer into accepting products inadequate for their needs. That does not stop me from calling it out as duplicitous, immoral behavior that should be stamped out.
Part of this being pointing out how they have changed the words to make their statements seem less bad. The use of correct, industry-standard words makes it harder to misdirect customer intuition.
Sure, in a strictly literal, legal sense, it is quite possible that they are not selling a defective product because they lied to the customer and tricked them into purchasing a substandard product. However, in the colloquial sense, they are absolutely selling a defective product as the consumer is not getting the plain meaning of what they were advertised (even though it is not contractually binding).
I do not know about you, but I really think we should push for the consumer-friendly model over the lawyer-friendly one.
Your point seems to come down to this argument, which I don't believe is in good faith (I don't believe the vendors are acting terribly laudably, either, but that's a separate issue).
> Sure, in a strictly literal, legal sense, it is quite possible that they are not selling a defective product because they lied to the customer and tricked them into purchasing a substandard product.
So you said a guarantee was broken earlier, and I asked you to demonstrate that or point to where they've committed to guaranteeing a certain quality of service.
Now you have adopted a position of saying they're technically not doing anything wrong, but morally they are, which when talking about Microsoft or Amazon or Google or Apple or (etc) is a bit of a slippery slope and/or isn't really news.
I see you also slipped in a 'they lied to their customers' (and 'tricked them') but if you read any EULA you'll see it's all best-effort, sold as is, no warranty for any purpose, etc etc, and this has always been the case.
Should it be the case? I don't have a good answer there. If you want penalty clauses, I'm sure you can find vendors that will oblige, but a) you'll quickly have an adversarial relationship with your supplier, after b) you spend stupid numbers of monies negotiating such a contract, and c) you'll still suffer exploits and outages.
I believe they are arguing for a change in the law, so that software is held to similar standards as the other consumer facing industries. That we should not even be allowed to waive as many guarantees as we do.
The EU is currently working on making that happen. The current draft looks like it could be improved, but overall this looks pretty reasonable.
Oh, look, I get that it's a laudable goal, my point here is that we don't have that arrangement now.
The nice thing about EU doing the heavy lifting on some of these tendentious subjects is that the rest of us often get to (eventually) enjoy the benefits, just because it's easier for them to not have to maintain two 'things' (product lines, releases, etc).
I would argue that the difference is that a defect is something that an ordinary user can encounter when using the product themselves.
Triggering security bugs or zero days are typically things that require highly unusual and targeted usage which are extremely unlikely to occur otherwise.
So I would draw a distinction between failure of a product in normal usage and failure of a product when an active adversary is trying to cause it.
> Delicious irony that your post contains this abuse of single quotes:
Obviously it's not irony, but neither is it an abuse.
There are some regional variations in convention though. In my part of the world we differentiate the noun (licence) from the verb (license), and we also [can] use single quotes.
No, and your own link says otherwise, i.e. that it was an "inactive Microsoft account (MSA) consumer signing key". But in addition to letting the key get compromised, their code for validating the signatures had vulnerabilities and allowed a consumer signing key to be used to sign AD tokens.
>The actor used an acquired MSA key to forge tokens to access OWA and Outlook.com. MSA (consumer) keys and Azure AD (enterprise) keys are issued and managed from separate systems and should only be valid for their respective systems. The actor exploited a token validation issue to impersonate Azure AD users and gain access to enterprise mail.
IMHO, Azure AD has problems with AuthZ: It's too easy to accidentally only do AuthN*, it's too underdocumented, customization is non-trivial... In general, you're much better off only taking the AuthN part, and implementing AuthZ yourself. It's not as if the lower tier Azure AD allows you to look at useful logs anyway.
* The bug here being too wide a scope, or more likely, the scope not being checked at all. Not the first time this class of bug has happened with a Microsoft product on Azure:
Well, if it wasn't a zero day, then it means they aren't patching their own services in a timely fashion.
It's also possible that they have to word things carefully because international diplomacy happens in a language that looks like english, but is not. (In the same way the legal documents are not quite english.)
For instance, "China exploited zero days in our cloud services and targeted US gov't communications. We are addressing the situation." could mean "China has committed an act of war against the US, and we are responding in kind on behalf of the US government."
The last time I checked, Microsoft did all sorts of shady clandestine stuff for the US military and intelligence communities (they even supported the US CLOUD Act). Also, they manufacture stuff in China. I'm sure they have to be very careful with this sort of press release.
Oh I looove that it's some access token exchange API that is likely at the heart of this. After the constant nightmares that AD caused Azure for ages, this is just too perfect.
Shouldn't the use of zero days slightly vindicate Microsoft? It's certainly not a good look, but at least they weren't willfully neglecting to patch issues known to them.
You're right. Microsoft can not withstand _any_ cyber onslaught.
After 20 years of "improved security" they still don't get it right.
But as long as the management loves Microsoft, i guess we are stuck with it.
And none of it has any impact, nobody seems to care.
The comments in this thread are crazy to me. Yes, dealing with targeted attacks by nation states is hard, and occasionally you'll fail. But that's part of the reason to pay Microsoft rather than try to host your own. In that situation the absolute minimum expectation is for MS to follow basic best practices. Nothing about this story suggests they've been doing that, it's just one wtf after another.
And their communication on this has been so bad an unclear that it has to have been intentionally obfuscatory.