
250M Microsoft customer service and support records exposed on the web - el_duderino
https://www.comparitech.com/blog/information-security/microsoft-customer-service-data-leak/
======
deepspace
I truly believe the era of privacy is over.

There are zero consequences to anyone important when a data breach happens;
therefore there is no incentive for companies to protect their user data and
the number of breaches will continue to grow for the foreseeable future.

By 2025 at the latest, there will effectively be no such thing as privacy
anymore. All personal data belonging to everyone will have been exfiltrated
and will be available for sale.

You will be able to purchase the full medical history, all financial
transactions, all addresses, phone numbers, location history (often with
photo/video evidence), all account numbers, ID numbers (government and
otherwise), biometric data, browser/search history, every email, sms or chat
message they ever sent or received and any other information you can think of
for the vast majority of people on earth.

~~~
mathdev
I hope not. Much too late, I recently realised that I probably would not
survive my WhatsApp history getting out.

~~~
tialaramex
Because WhatsApp uses the Signal protocol design nobody ends up able to prove
anything useful about the message history. To any outsider, including the
WhatsApp server operators, the messages are opaque. To a participant,
including you, the system delivers authenticated messages but deliberately
doesn't sign them. So:

Alice knows she didn't send this message, so it's from Bob, but she can't
prove to anyone else that she didn't just fake it.

Bob knows he sent the message but he can deny it and there's no proof.

Eve has no idea what the message was, even if Alice or Bob show her the real
message she can't verify they aren't lying.

~~~
vunie
What about those messages before the signal protocol was adopted?

------
gexla
I imagine most of these are support issues handled by contractors they have
had over the years. Windows 95 through XP had Keane and Convergy's in Tucson
running their Windows support (which then forked into Canada and India.) Not
sure who they have doing it now.

The Windows parts of these records might be a good resource as it's probably
part of the documentation which builds up to become the MSKB articles. Each
support case was documented and linked to either a KB article, an internal
"not yet KB article" or you had to submit it as a unique issue. After the "not
yet KB articles" were referenced X times, then it would go to consideration as
a KB article. Collectively, all this formed their internal KB.

Worked there. Pay was terrible once Convergy's took over. Then they moved
everything to India and the support got terrible also. Too bad. They had quite
the brain drain from that process. There were a lot of Windows gurus in that
building. I learned far more than I needed to know about Windows and went way
more in depth than I ever have tinkering with Linux.

~~~
ct520
retweet. I agree worked there too, had a KB article. Can't say I learned much
only thing I remember that was new and I use to this day was (windows + pause
key) to bring up system info.

~~~
gexla
Congrats on your KB article. ;)

Looking at the dates again, records I was part of was probably before this
date. I believe I was gone by 2005. That may also be around the time the
Tucson location moved from the IBM campus to the Convergy's buildings. I
declined when they asked me if I was going to make the move.

I got there right after Convergy's took over from Keane. The training at that
time was still really good. Tony Agee and John Mott were legends (unsure of
spelling.) I credit this time with my beginnings in tech and learning how to
think for troubleshooting. They taught linear, logical troubleshooting which
was so simple and yet I still don't see much of it today. It was also a place
to develop my search skills. It's incredibly valuable to be able to sift
through a load of technical discussions and separate the signal from the
noise.

~~~
ct520
Linear and logical troubleshooting LOL ok I did learn that there too. Same
thing came in right after converge took over at ibm campus.

~~~
gexla
Who was your instructor? Do you remember any of the old guard there? What
project did you get into after training?

I got stuffed into Windows 95/98 and then sucked into XP when it blew up.

XP manager was Rohn Eloul (originally from Palestine.)

The only person I keep in touch with is Pablo Bley.

Don't remember names of anyone else. We must have worked on that same floor
though.

------
el_duderino
Microsoft released further details in its own blog post: [https://msrc-
blog.microsoft.com/2020/01/22/access-misconfigu...](https://msrc-
blog.microsoft.com/2020/01/22/access-misconfiguration-for-customer-support-
database/)

~~~
IanDrake
"Misconfigurations are unfortunately a common error across the industry. We
have solutions to help prevent this kind of mistake, but unfortunately, they
were not enabled for this database."

They need a solution to watch their solution that watches their configs.

~~~
rhizome
I bet the threat of prison would tune up their attitude. There's no reason
that these "misconfigurations" shouldn't be felonies.

~~~
logicchains
There's a massive reason these misconfigurations shouldn't be felonies. Since
such errors are so hard to avoid (no company has perfect practice), few sane
people would be willing to work in any role that involved taking such
responsibility, so a bunch of services would suddenly cease to exist, and the
economic cost from the loss of those services could well be way greater than
the cost from the occasional breach (which I've as of yet never seen anyone
here try to quantify).

~~~
tripzilch
How does it work in construction of buildings? If it got built with a
misconfiguration in a safety critical part.

Or maybe it ought to be modelled after medical malpractice? If a doctor
fumbles and messes you up.

------
mikece
An idea for someone looking for a fun "Show HN" project: build a scoreboard
that searches all of the known data breaches for this year and tells me where
I rank for how many breaches I've been in (eg: I'm 89/132 on breaches of
50,000 records or more).

Over 8.5BB customer records were exposed last year; the estimate for this year
is in excess of 10BB.

~~~
dijit
Isn't that what snusbase and dehashed are doing?

[https://snusbase.com/](https://snusbase.com/)

[https://www.dehashed.com/](https://www.dehashed.com/)

~~~
geddy
These sites really bother me sometimes. I just registered on Dehashed and it
requires me to pay for a subscription... to see my own stolen data. I reject
that on principle alone.

~~~
dijit
Mostly so it’s not abused my script kiddies. If you want to see if you are in
a breach it will tell you but just censor the password if it has been
successfully reversed.

Haveibeenpwned is the same but doesn’t reverse the passwords.

~~~
goatsi
The only difference I can see between them and businesses that have gotten
people sent to prison [0] is a slight change in the marketing materials.
Dehashed even advertises taking Cryptocurrency payments for "unlimited
searches".

[0][https://www.zdnet.com/article/company-behind-leakedsource-
pl...](https://www.zdnet.com/article/company-behind-leakedsource-pleads-
guilty-in-canada/)

~~~
ThrowAway99122
Guy mentioned in the article here using a throwaway (I'm active on this site).
Never went to trial, never went to prison, charges were withdrawn.

> An investigation by infosec journalist Brian Krebs claimed that a second
> suspect, a US man named Jeremy Wade, was also behind the service. He was
> never charged

According to American prosecutors, freedom of speech prevents prosecution in
America without intent to commit a crime.

~~~
goatsi
I wouldn't bet your freedom on it. The FBI was involved in taking down a
similar website just last week [0]. I'm sure they have a solid legal basis for
it.

[0] [https://nationalcrimeagency.gov.uk/news/weleakinfo-com-
site-...](https://nationalcrimeagency.gov.uk/news/weleakinfo-com-site-hosting-
stolen-credentials-taken-down-after-international-operation)

~~~
ThrowAway99122
I'm sure they did have a solid basis. The FBI statutes referenced are
forfeiture related and don't describe the underlying crime they are accused
of, and I'm not familiar with UK or NL law where the accused are located. USA
is ironically the only country I would hypothetically bet freedom on to
operate such a website not that I need/want to, if it's done without intent to
defraud. I'll reply why to a lower comment but the fact that no Americans were
able to be charged is a good start.

------
jsgo
I got an email from Microsoft Azure in relation to this (didn't read the
article, but people are quoting parts of the email I received here).

I appreciate that they sent something, but sometimes it'd be nice for them to
allow someone to access the data related to them that was exposed as they say
"our analysis of the support information indicates that specific personal or
organizational identifiable information related to your support case was
potentially visible." Okay, what specific personal or organizational
identifiable information of mine was visible?

I assume the representative or I may've listed said info in our communications
back and forth so let me see what was exposed so I can make a judgement of
what, if anything, I should do here.

~~~
GordonS
I got the same email, and I agree with what you said - I'd really like to know
if this is even personally relevant, and if it is, I'd really like to know
precisely what information is relevant. I'm in the EU, so I guess I could ask
under the GDPR, but I wouldn't even know _who_ to ask, and with such a large
organisation, I can only imagine there would be a lot of run-arpund, requiring
a lot of follow-ups from me :/

~~~
Darkphibre
Well, you can fill out this form: [https://www.microsoft.com/en-
us/concern/privacyrequest-msa](https://www.microsoft.com/en-
us/concern/privacyrequest-msa)

Or you could reach out to Microsoft's Data Protection Officer here:
[https://www.microsoft.com/en-
us/concern/privacy](https://www.microsoft.com/en-us/concern/privacy)

I'm not sure if that's the GDPR path, but it's what I came across.

~~~
GordonS
These seem like the kind of top-level links in a large organisation that will
lead to nowhere, but I'll at least give them a try, thanks!

~~~
jchb
Under the GDPR, that Microsoft Data Protection Officer will have 30 days to
respond to you. If they don't respond, you can complain to the supervisory
authority (in the case of UK that is [https://ico.org.uk/make-a-
complaint/](https://ico.org.uk/make-a-complaint/)).

Now, Microsoft does not necessarily have to tell you exactly what of your data
was leaked. They probably do not know! In this case, they may just respond to
your request with _all_ of the personal data they hold.

The law just says that they have to notify you of the "nature of the personal
data breach as well as recommendations for the natural person concerned to
mitigate potential adverse effects".

~~~
pbhjpbhj
Do you think the ICO will do anything? Usually their answer seems to be "well
we told the company about your case, and they said they wouldn't do it again,
so everything is fine now; you're welcome!".

------
shaabanban
Notably, elastic's Kubernetes operator which just went 1.0 defaults to
requiring a username and password (and generates one if it isn't provided). It
also doesn't seem to allow you to opt out of using TLS.

[https://www.elastic.co/guide/en/cloud-
on-k8s/current/k8s-ove...](https://www.elastic.co/guide/en/cloud-
on-k8s/current/k8s-overview.html)

------
drallison
Wow! So the person on the telephone who tells me in a soft Indian accent that
my computer has a problem can now can authenticate themselves with authority,
making it easier for them to get me (or my proxy) to enable remote
administration so they can do real damage.

Security is difficult. Microsoft is supposed to be skilled at preventing data
breeches and exploits, but apparently not. What can be done to prevent this
sort of thing?

~~~
misterhtmlcss
Nothing can be done.

It's a wildly asymmetrical relationship that means 9 billion people get a try
to knock you out and your team of what??? 25, 50, 100, 1000? Security
specialists have to see everything possible and plan for any and all
possibilities.

It's never going to happen.

This is the simple reality of the internet and I'm sure you know this, but I
saw your comment and thought I'd add this for the next person who may not
realize this.

I'm personally curious to know, because I'm no SecOps; if there is even a
theoretical solution to the internet that would have greater integrity for the
users or if this is as good as it gets.

~~~
rhizome
> _Nothing can be done._

Please don't do this. Something can be done, we all know it, but for some
reason don't think it's possible? Prison sentences should have started with
the Target CTO in 2013 (at the very latest), but the more the public is cowed
by shrug emojis, the less likely companies will protect your data for anything
other than commercial advantage.

~~~
jjeaff
Imprisoning CTOs for incompetence would be unprecedented. We don't imprison
people for that reason and I can't imagine any knowledgeable CTO ever taking
that position again once that precedent was set.

But we do need much more oversight and serious punishment for companies that
lose data like this.

~~~
616c
Security engineer here. Why?

US soldiers fight in combat knowing failure to obey orders can land them in
jail for years. The US Code of Military Justice is not fun, and you sign up
for it when you join implicitly. Is there an armed service staffing problem?

Our general US legal system is flawed, but cops can go to jail for ethical
violations and criminal behavior for actions that are integral parts of their
job function. Is there a cop staffing shortage?

I have argued this, ironically, about US Congresspeople: if we have a
volunteer army with stringent legal codes with special punishment by virtue of
their job, serving us, why are other classes of people not worthy of higher
standards and why do we suspect people will shy away from that? How can we pay
others more for higher probability of incompetence and less repercussions?

I am not trolling. When I suggested this shows the power of commitment in
volunteer armies and I wish Congress had that kind of self respect people tell
me I'm nuts. I would like a CTO and security industry jobs to mean something.

~~~
jjeaff
A CTO can go to jail for unethical and illegal behavior. But you are
suggesting they be jailed for incompetence.

And considering the fact that no amount of competence will protect you from a
sufficiently motivated and resourceful hacker, seems unfair.

Now if we come up with a framework and a security checklist that must be
followed and certified by the CTO every quarter or something, and they don't
do it, or lie, then sure, jail them.

------
whatever1
Databases that involve more than X users need to be regulated.No big database
should be deployed in public before being vetted on whether it is secured
properly. I am tired of reading every week for breaches of personal data and
passwords saved in plain text. If no company can secure our data voluntarily
then we should use the law to force them to at least meet a bare minimum of
standards.

------
Silhouette
When people ask why we're so concerned about the privacy implications and
specifically the telemetry functionality of modern software... This. This is
why.

Even if that functionality is implemented with good intentions and the data is
only intended to be used for responsible purposes, the biggest and most
technically capable organisations in the world can still make mistakes and
suffer data leaks, which are potentially a gift to criminals, commercial
competitors, and so on.

If there's anything sensitive in there -- personal data, commercial
information that was provided under NDA -- we're probably still on the hook
for it legally, too.

------
reaperducer
_250M Microsoft customer service and support records exposed on the web_

Someone should grep this to find out how many times people were told to turn
it off and turn it on again.

~~~
ehsankia
Is there any signs that this data is actually out in the wild? From the
article, it was found, reporter and fixed within 24 hours, and they claim
there's no sign of other unauthorized access.

~~~
reaperducer
_Is there any signs that this data is actually out in the wild?_

Check the dark web.

 _From the article, it was found, reporter and fixed within 24 hours_

Being fixed within 24 hours of being reported does not mean it was only
available for 24 hours. It could have been 24 days or 24 months.

 _they claim there 's no sign of other unauthorized access._

Anyone smart enough to access this would also be smart enough to cover their
tracks. When I was black hat in the 80's, this was Infiltration 101.

~~~
xixixao
Covering up is not always technically possible. It’s easy to expose data
through some unprotected end point, but that end point might still be logged,
and turning off the logging/deleting the logs might be a completely different
challenge.

~~~
thisisnico
Even more challenging if the log destination is external, and if the logging
system is an entirely independent system, even potentially provided by a third
party. Makes this hard to do.

------
jonplackett
> All of the data was left accessible to anyone with a web browser, with no
> password or other authentication needed.

Really quite incompetent. But we don’t know for sure anyone else actually
accessed it.

~~~
netsharc
I like the legally correct phrasing the MS blog ([https://msrc-
blog.microsoft.com/2020/01/22/access-misconfigu...](https://msrc-
blog.microsoft.com/2020/01/22/access-misconfiguration-for-customer-support-
database/)) said: "While the investigation found no malicious use".

If the DB server was configured so access was not logged, could you claim "We
investigated, and we didn't see any evidence of access"?

~~~
ryanlol
Surely they’d still see any exfiltration in their bandwidth graphs. And
anyways, ES spits out a lot of logs by default.

~~~
resfirestar
That assumes they have bandwidth graphs. And sure, ES generates a lot of logs,
but have you ever tried using them to investigate an exposure like this?
Unless the “xpack.security” module is on (off by default), it’s nothing
useful.

~~~
ryanlol
Linux itself gives you decent data from procfs (see /sbin/ifconfig, shows you
data transfer in/out per adapter), you can just compare data transfer from the
server to any of the boxes that are supposed to connected to.

I can’t imagine that even MS would be running ES on windows, although then
you’d probably have even more data available.

------
coliveira
My opinion is that ALL information that has ever being put online will, sooner
or later, be made public. Despite the advances in crypto, there are so many
ways to exploit security flaws and vulnerability in all kinds of software. And
now with machine learning, which can also be used to help in hacking exploits,
there not much that can be done.

------
huzaif
"I am calling from Microsoft" calls were bad enough. Now they will know some
details to a past case and sound slightly more legit.

~~~
texasbigdata
Yeah the no notice.

By the way Microsoft has absolutely terrible azure support. If you have a
legitimate issue and you dont have a dedicated support consultant good luck to
you.

~~~
Analemma_
All the cloud providers are like that though. If you're on the cheapo tiers of
AWS or GCE, you get the cheapo support. AWS might be slightly better just
because more people have used it and so there are more hacky workarounds
posted on StackOverflow, but that's small comfort at best.

~~~
keithnz
I've had good experience with Rackspace and DigitalOcean support (other than
having to repeat my problem multiple times until I get to the right person,
but at least they are keen to help).... Azure support was a disaster with too
many support staff that know almost nothing about the platform except by
reading the same websites I can read until you spam every possible support
mechanisim you can find and finally get to a "real" support person. This will
take around 2-4 weeks.

------
cobookman
Good reason why defense in depth should be used.

Simply stating that your network configuration prevents access isn't the best
answer.

~~~
wang_li
>Simply stating that your network configuration prevents access isn't the best
answer.

Right. The network should actually be configured to prevent access.

~~~
CydeWeys
And, in the event that this configuration fails to do what you expect it to
you, or your network is breached via other means, you should be utilizing
defense and depth and all of your DBs and other sensitive systems should
require authentication.

~~~
riknos314
This.

This is why google assumes that the network layer adds no security.

[https://cloud.google.com/beyondcorp/](https://cloud.google.com/beyondcorp/)

------
sorokod
"In total, the data was exposed for about two days before we alerted Microsoft
and the records were secured.

    
    
        December 28, 2019 – The databases were indexed by search engine BinaryEdge... "
    

... at least two days then.

------
vunie
The solution to leaking private information has been known for a very long
time: Don't collect beyond the bare minimum required to render a service.

Microsoft should not have collected anything beyond an email and a password.
Payment information should only be held temporarily.

Personal information is a toxic asset. It baffles me why companies willingly
hoard it.

------
bluedino
"misconfigured security roles" means the dinks that set it up never
'configured" a thing, right?

~~~
TomVDB
That's not how I read Microsoft's statement about it: the permissions were
incorrectly changed on December 5th and were corrected on December 31st.

------
privateSFacct
Does this have anything to do with Dell support info - it used to be literally
right after buying a dell product (within a week or so) you'd start getting
scam calls with your dell info.

Dell always denied it, but it was pretty funny. They had service tags and
everything - anyone else get that?

------
afinlayson
At what point do we not allow companies to own this data beyond a transaction.

Lets be honest no one can keep this data without eventually being hacked, so
maybe they shouldn't have it after that transaction.

------
ifthenelseend
How much money did you get from Microsoft for disclosing that vulnerability?

~~~
jacquesm
Nothing like working for free for giant companies that fail utterly at their
responsibilities.

~~~
owlninja
I mean if they didn't have an open bounty or posting, you should assume you
are 'working' for free.

------
Spooky23
What is it about elastisearch that dopey people stuff them with information
and leave them on the open internet, all of the time?

~~~
jacquesm
In a word: devops.

Developers are not operators and operators are not developers. The whole idea
that we can do away with this specialization and and relegate operations to
the people that create software because it is now possible to script
infrastructure and to install complex packages with a few mouseclicks does not
make it true. Operations and the complexity that goes with it is a job in its
own right, no competent operator would have left this situation as it came out
of the box.

~~~
AnIdiotOnTheNet
A combination of businesses' desire to spend less on labor and your average
developer's inherent sense of superiority mean this trend is unlikely to go
away any time soon though.

------
Pxtl
And yet all their old help forums and documentation pages somehow they can't
keep those exposed on the web.

------
huxflux
And with zero consequences I suppose?

------
meristem
When so many elasticsearch bad condiga get published, MS ought to reevaluate
their UI and default config.

------
twodave
I think Microsoft's response time to this exposure (during a holiday even) is
more noteworthy than the fact that it happened. We can sit in our ivory towers
all day and shake our heads at what an inept organization Microsoft is for
allowing human beings to make mistakes, or we can applaud the fact that once
the mistake was identified they chose to act immediately, appropriately and
transparently. What are we really expecting here? Perfection?

~~~
throwawayjava
_> What are we really expecting here? Perfection?_

No, I don't expect perfection. However, I do expect very careful
implementation of access management for very large databases containing lots
of PII and other sensitive customer information. Things like huge databases
being accessible without credentials shouldn't require perfection on the part
of some human. That sort of stuff should be continuously audited in an
automated fashion.

But the software industry is quite bad, as a whole, so even the relatively
competent actors make surprising, high-impact mistakes.

Maybe it's because the stakes are relatively low (c.f., bridge collapsing vs.
PII leak) and the competition relatively fierce? Maybe software engineering is
still very young and moving quickly?

In any case, I think it's totally reasonable to hold the opinion that MSFT is
doing things pretty well relative to the rest of the industry _and also_ that
the industry as a whole is doing a pretty poor job.

IDK, for me the story has to be one of the following:

1\. MSFT made a huge and inexcusable mistake, so maybe there's something
systemically wrong with MSFT; or,

2\. MSFT is very competent, and even very competent people are making very big
mistakes, so maybe there's something systemically wrong with the entire
industry.

~~~
Tallasatree
Architect here: from the outside looking in, you hit the nail on the head. In
addition to The industry being so young the _relatively_ low-impact when bad
things happen make things like this 'not a big deal'. When your mistakes
result in a public outcry for a day, then fades into obscurity into the night,
why change? why invest money into figuring out a better way?

When your mistake makes a building fall over...well, there's a reason why that
almost never happens.

~~~
keithnz
I don't think this is quite right. Most buildings don't get all their design
parameters tested in reality. But say when there is an earthquake, and the
building collapses and you find that various checks and balances in the design
process went wrong. I know here in NZ where we have had a number of
significant earthquakes all kinds of known and unknown things have been
discovered about buildings, either ones that have ended up killing people or
ones which now are condemned because things played out differently than the
designers thought they would

~~~
Tallasatree
Speaking about America, almost everything in a building beyond aesthetics is
designed to a CODE MINIMUM. from the hangers that hang the ACT ceiling all the
way, and especially to, the structural system. These systems have been
designed and tested ad nauseam to provide minimum life safety standards.
People in any industry can cut corners and screw up. Special situations can
arise that surpass a minimum level standard (Fires started at every exit door,
9.0 earthquake...good luck) The forest you're missing through the trees here
is the structured process that forces designers in a mature industry to design
to a minimum agreed upon standard. Ironically, I'm highlighting the benefits
of regulation...where it makes sense.

the forest I might be missing through the trees is that maybe there is an
industry agreed upon standard within the Tech industry. My understanding is
almost all of these breaches happen because comically silly mistakes (pw =
password), not super high sophisticated attacks.

~~~
keithnz
same with NZ, which has pretty strict codes as we are sitting at the junction
of 3 tectonic plates. Regulation including inspection is great, and generally
works great, but until you get an earthquake, you really don't know if all the
checks and ticking of boxes actually did its job. Microsoft and others likely
catch multiple problems through checks, but occassionally a perfect storm
happens and things break down. You then adjust your "regulations" to cover any
short comings (hopefully). The entire planet you are missing through the
forest is that all buildings aren't constantly "penetration" tested to find
where they have problems. A quick search shows that USA suffers from many live
deployed buildings that have been shown that they don't meet compliance. By
Engineers that should've known better....

------
salex89
Pretty sure at least 100M are from me...

