

Outlook.com DNS records disappeared - mrmondo
http://outlook.com/
:-(
something went wrong
Sorry, we can&#x27;t get that information right now. Please try again later. If the problem continues, contact your helpdesk&lt;p&gt;---&lt;p&gt;dig outlook.com&lt;p&gt;; &lt;&lt;&gt;&gt; DiG 9.8.3-P1 &lt;&lt;&gt;&gt; outlook.com
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: SERVFAIL, id: 62938
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0&lt;p&gt;;; QUESTION SECTION:
;outlook.com.			IN	A&lt;p&gt;;; Query time: 33 msec
;; SERVER: 10.51.20.100#53(10.51.20.100)
;; WHEN: Fri Nov 22 09:37:02 2013
;; MSG SIZE  rcvd: 29
======
mrmondo
I especially loved their status page during the outage:

    
    
      current health
    	CRM	No issues
    	Exchange	No issues
    	Identity Service	No issues
    	Lync	No issues
    	Office 365 Portal	No issues
    	Office Subscription	No issues
    	Rights Management Service	No issues
    	SharePoint	No issues

~~~
amaks
That's an executive console.

------
kristofferR
Edit: It's back up now.

Yup, it's pretty broken, though it is still up some places.

Outlook.com:

    
    
      Grapevine TX, United States (bigguy.gte.net) - Down
      Sacramento CA, United States (CalWeb) - Down	
      Providence RI, United States (Verizon) - Down
      Pasadena CA, United States (Mindspring) - Down
      Mountain View CA, United States (Google) - Up
      Vancouver BC, Canada (Radiant) - Down
      Recife, Brazil (Hotlink Internet) - Down
      London, United Kingdom (Legend Comm) - Down
      Lille, France (Nordnet) - Down
      Merzig Saarland, Germany (Probe Networks) - Up
      Milan, Italy (BT Italy) - Down
      Ankara, Turkey (TTNET) - Down
      St. Petersburg, Russia (Uni of Tech & Design) - Up
      Karachi, Pakistan (Supernet) - Down
      Delhi, India (Tikona Infinet) - Down	
      Bangkok, Thailand (TOT) - Down	
      Petaling Jaya, Malaysia (Clear-Comm) - Down
      Beijing, China (China Unicom) - Down
      Sydney NSW, Australia (Exetel) - Down
      Collingwood VIC, Australia (Pacific Internet) - Down
      Auckland, New Zealand (ICONZ)  - Down
    

[https://www.whatsmydns.net/#A/outlook.com](https://www.whatsmydns.net/#A/outlook.com)

~~~
cclogg
Did something change yet? I'm in Vancouver and have no problem going to
outlook.com... that's what is down, right?

~~~
natevancouver
It depends on what your local DNS has cached. Also, I’m finding that 1 out of
5 of their DNS servers is still responding from my location (not too far from
Vancouver).

------
bencxr
Based on what I have heard, the issue was caused by a bad active directory GPO
that caused firewall rules to refuse DNS requests. Neither outlook/xbox/etc
would have had any control over it.

I heard that the issue has since been solved internally from 4.30 PST, though
DNS changes often take time to propagate externally, taking TTL settings and
ISP caches into consideration.

~~~
wutwut
They did release a new version of their Active Directory service in Azure
yesterday. Wonder if its related.

------
mrmondo
Yeah, dropping all your NS records from your domain will do that;

    
    
      $ dig NS outlook.com
    
      ; <<>> DiG 9.9.3-rpz2+rl.13214.22-P2-Debian-1:9.9.3.dfsg.P2-4 <<>> NS outlook.com
      ;; global options: +cmd
      ;; Got answer:
      ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 8538
      ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0,   ADDITIONAL: 1
    
      ;; OPT PSEUDOSECTION:
      ; EDNS: version: 0, flags:; udp: 4096
      ;; QUESTION SECTION:
      ;outlook.com.            IN    NS
    
      ;; Query time: 28 msec
      ;; SERVER: 10.51.20.100#53(10.51.20.100)
      ;; WHEN: Fri Nov 22 09:58:48 EST 2013
      ;; MSG SIZE  rcvd: 40

~~~
natevancouver
SERVFAIL doesn’t mean they dropped the records (that would be NXDOMAIN). It
does mean their nameservers aren’t responding.

~~~
larrys
...which can be verified (as I did earlier in another thread) by querying the
nameservers directly.

[https://news.ycombinator.com/item?id=6778272](https://news.ycombinator.com/item?id=6778272)

------
druiid
Apparently it was due to Azure dying:
[http://www.theregister.co.uk/2013/11/21/azure_blips_offline_...](http://www.theregister.co.uk/2013/11/21/azure_blips_offline_again/)

~~~
jongalloway2
No, backwards. Azure outages were caused by DNS problems.
[http://gigaom.com/2013/11/21/another-bumpy-day-for-
windows-a...](http://gigaom.com/2013/11/21/another-bumpy-day-for-windows-
azure/)

[https://twitter.com/scottgu/statuses/403670409406468096](https://twitter.com/scottgu/statuses/403670409406468096)

------
tpurves
Do they never learn? I am suddenly remebering distant flashbacks to 2001 when
microsoft wiped out hotmail for 24hrs with a massive DNS screw-up:

CNET: Microsoft blames technicians for massive outage (2001)
[http://news.cnet.com/2100-1001-251427.html](http://news.cnet.com/2100-1001-251427.html)

Wired: How, Why Microsoft Went Down (2001)
[http://www.wired.com/science/discoveries/news/2001/01/41412?...](http://www.wired.com/science/discoveries/news/2001/01/41412?currentPage=all)

Can you imagine in those days, everyone was on hotmail, gmail hadn't even been
invented yet.

~~~
lucb1e
And that's why I run my own mail server nowadays. Privacy, control over data,
and at least I'm responsible for my own downtime without impacting more than a
few people. Decentralisation for the win, in my opinion.

Also credits to my ISP for allowing hosting of my own server, running native
IPv6 and of course opening port 25. Xs4all is a really cool ISP to have in the
Netherlands :)

~~~
josteink
_Xs4all is a really cool ISP to have in the Netherlands :)_

I heard they had gone downhill recent years, resembling more of the regular
commercial ISPs, but good to see they still have something going for them.

We definitely need more ISPs like this.

------
ThomW
Who's scroogled now?

[http://www.microsoftstore.com/store/msusa/en_US/cat/Scroogle...](http://www.microsoftstore.com/store/msusa/en_US/cat/Scroogled/categoryID.67575900)

This is so embarrassing.

~~~
hsod
this has nothing to do with the scroogled campaing though...

~~~
oneeyedpigeon
Maybe if Microsoft wasn't wasting its time trying to spread slurs about
competitors, it could actually spend that time creating interesting software,
building viable new products, and making sure it doesn't f*ck up its own DNS
records. Or maybe someone's making that point for them ...

~~~
curiousDog
Yes, ALL the developers (judging by your comment, who are no doubt better
coders than you) at Microsoft are working on the new marketing campaign. Jesus
christ, viable new products, perhaps they should just start making beer
delivery apps on the rails stack, that'll prove themselves right? That 4chan
thread the other day was exactly about guys like you.

~~~
oneeyedpigeon
Fair point. My original comment was poorly worded; I obviously didn't mean to
imply any of the things you're claiming here. I merely (over)reacted out of
extreme frustration with a marketing campaign that I can barely believe
someone at Microsoft sanctioned. Whatever any of us think of Microsoft (I'm
obviously not their biggest fan), you have to admit that this is a low, LOW
move, and something that probably never would have happened were Bill Gates
still in charge.

------
shanselman
Looks bigger than this. It's ns*.msft.net.

~~~
sethhochberg
A friend of mine working at Microsoft mentioned this afternoon the Redmond
campus had a full network outage.

------
pwf
I was unable to resolve any hostnames from my Azure machines from 4:20-4:40
Central time. I'm assuming this is all related.

This about sums up how I feel about Azure:
[http://i.imgur.com/ESakkWk.png](http://i.imgur.com/ESakkWk.png)

~~~
runn1ng
Wow. Your screenshot is giant.

~~~
pwf
Cmd+shift+4 on a retina MBP. Not sure why it does that...

~~~
MiguelHudnandez
It's because the metadata saying to render the image @2x is lost when imgur
serves the PNG back to our browsers.

------
mrmondo
Google's DNS cache will drop them in 250 seconds...

------
thisisdallas
It looks like a lot of their other services are down too.

Edit: Also, the Xbox One is launching tonight so if their Live servers are
down...it won't be pretty.

~~~
kristofferR
Xbox Live is down. Just posted on Reddit:
[http://www.reddit.com/r/xbox/comments/1r6axx/anyone_else_not...](http://www.reddit.com/r/xbox/comments/1r6axx/anyone_else_not_able_to_connect_to_xbox_live/)

What a bad time for this to happen for Microsoft. It's almost too badly timed
for it to be a coincidence, if I were an attacker with access to Microsoft's
DNS' this would be the perfect time to launch an attack.

------
codex
Will negative caching of DNS records prevent a quick recovery? If so, for how
long will the negative records be cached?

------
coldcode
Damn the guys at the NSA are trying to get on xbox live.

------
bluejade
This probably explains an error we were getting on a Node.js app running on
Azure websites. Simply creating a new AWS.S3 instance was throwing this:
Error: getaddrinfo ENOTFOUND at errnoException (dns.js:37:11) at
Object.onanswer [as oncomplete] (dns.js:124:16)

------
beat
It'll be interesting to find out the root cause, assuming it ever comes out.
Malice or stupidity? Did someone do an oops within the Outlook team (or some
other team), or are they under attack?

~~~
mrmondo
My guess is someone messed up when making a change to their DNS

~~~
beat
DNS problems are at least partly affecting the microsoft.com homepage as well.

------
shaydoc
unreal, i was editing a document, when it bombed on me, guess i should been
editing locally!

------
dmourati
Oh man, this is epic.

------
iLoch
Pro tip for anyone trying to check if a DNS record is working: do an NS lookup
in cmd (Windows) in interactive mode. Set the server to the IP of a root DNS
zone network (you can find these on Google) then enter that. Now do an
nslookup for the domain you want.

~~~
larrys
Actually no. The first step would be to go directly and query the nameservers
set in the whois record (not the registrars which could be wrong but the
registry).

For xbox.com that is:

Name Server: NS1.MSFT.NET Name Server: NS2.MSFT.NET Name Server: NS3.MSFT.NET
Name Server: NS4.MSFT.NET Name Server: NS5.MSFT.NET

------
beat
While looking at this, I saw some other temporary DNS fails as well, including
HN, Google News, and Twitter. But all came back quickly, except for
Outlook/Microsoft domains. (I'm in Minnesota)

------
Caspan
I have my domain at outlook.com and my MX record points to
edcb3f68479b498ee412acc8524def.pamx1.hotmail.com and I am getting no return
for that. There is a big outage somewhere

------
lotso
Not a good day for this to happen. Xbox Live is impacted as well.

------
tokenadult
It's working just fine for me. I use Outlook.com mostly as one of several
back-up email accounts. I tested it by sending a message to my Gmail account,
and all is well.

------
Kennethtruyers
Yeps, a lot of Microsoft services are down. I'm in europe and can't reach
outlook.com, microsoft.com, windowsazure.com and probably a whole bunch of
other services

------
gamblor956
I have no problem accessing Outlook.com, Office 365, Skydrive, or any other
Microsoft website. The problem many be regional...or it may be with your ISP.

~~~
jldugger
There's a lot more than one person complaining on twitter. I've seen reports
from New Zealand, Kansas, my linode in Dallas, Oregon, etc.

~~~
Einherji
Yup. I'm in Iceland and every microsoft service is unreachable, except my
regional version of microsoft.com

------
dangoldin
Yep - [http://mail.office365.com/](http://mail.office365.com/) is down for me
in New York.

------
veganarchocap
' Grapevine TX, United States (bigguy.gte.net) - Down' So we definitely didn't
hear it down the grapevine?

------
xfax
It reminds me of the following quote - don't attribute to malice that you can
attribute to incompetence.

------
abarringer
Technet DNS entries was gone for mediacom DNS servers about an hour ago in
Northwest FL.

------
swalsh
I was having some issues connecting to technet earlier, perhaps related?

------
davidcollantes
Down when accessing from Central FL (SouthEast US).

~~~
dhammack
It's up in North FL

------
mef
status page [https://status.live.com/history](https://status.live.com/history)

------
jamesjguthrie
Yep, Xbox Live is down here in the UK.

~~~
jamesjguthrie
Back up already :)

------
zachnedwich
CodePlex is down too? (in AUS)

------
jezebel6
oh no. Where will I go now to find information on gentleman enlargement pills?

------
keevie997
Can confirm Sydney is down :(

------
hughes
Still up in Toronto

------
svensenn
Down in Melbourne

~~~
peter_tonoli
Up in Melbourne Australia (on AARNet).

------
than
Down in MN.

------
vanwilder77
is up in Pune, India

------
inu3
fixed in japan

