
DDoS Attack Against Dyn Managed DNS - owenwil
https://www.dynstatus.com/incidents/nlr4yrr162t8
======
bhauer
Out of curiosity, why do caching DNS resolvers, such as the DNS resolver I run
on my home network, not provide an option to retain last-known-good
resolutions beyond the authority-provided time to live? In such a
configuration, after the TTL expiration, the resolver would _attempt_ to
refresh from the authority/upstream provider, but if that attempt fails, the
response would be a more graceful failure of returning a last-known-good
resolution (perhaps with a flag). This behavior would continue until an
administrator-specified and potentially quite generous maximum TTL expires,
after which nodes would finally see resolution failing outright.

Ideally, then, the local resolvers of the nodes and/or the UIs of applications
could detect the last-known-good flag on resolution and present a UI to users
("DNS authority for this domain is unresponsive; you are visiting a last-
known-good IP provided by a resolution from 8 hours ago."). But that would be
a nicety, and not strictly necessary.

Is there a spectacular downside to doing so? Since the last-known-good
resolution would only be used if a TTL-specified refresh failed, I don't see
much downside.

~~~
davidu
OpenDNS does this: [https://support.opendns.com/hc/en-
us/articles/227987767-Dyna...](https://support.opendns.com/hc/en-
us/articles/227987767-Dynamic-IP-Addresses-Technical-Detail-and-FAQ)

It's called SmartCache.

~~~
conatico
A shame OpenDNS used to redirect me to some spam webpage every time I tried to
resolve a domain that didn't exist--they earned a spot on my black list
forever. :(

~~~
davidu
It's been _years_ since we did that, and they were not spam pages, and easily
able to opt-out.

~~~
nly
Well, the fact that people still remember goes to show what a truly terrible
idea it really was and that it probably did permanent damage to your brand.

~~~
davidu
I'm not sure what metric you use to judge it as terrible.

I thought it was great. 10,000 companies pay for my service today. 65 million
people use my infrastructure today. Cisco bought the company for more than
$650m. It continues to innovate on the decades old DNS in secure and useful
ways.

So let me know what part is terrible.

~~~
zhengyi13
The part where you repeated Verisign's mistake in breaking a fundamental
protocol.

NXDOMAIN. Kind of a thing, and important to protocols other than HTTP.

~~~
mentat
The point is that the company did just fine even having made a mistake.
Ignoring that is just being difficult.

~~~
tripzilch
No, the point that a company doing just fine is somehow an excuse for its
actions is just the reason why we can't have nice things.

------
scrollaway
Relevant (or at least a-propos) post by Bruce Schneier, from a month ago:
"Someone Is Learning How to Take Down the Internet"

[https://www.schneier.com/blog/archives/2016/09/someone_is_le...](https://www.schneier.com/blog/archives/2016/09/someone_is_lear.html)

Edit: And to be clear: I don't mean to imply there's any connection :)

~~~
curiousgal
>We don't know who is doing this, but it feels like a large nation state.
China or Russia would be my first guesses.

Why not the USA?

~~~
amalcon
It doesn't make a whole lot of sense for the USA to take down the internet, as
they benefit the most from it. A significant fraction of that economy is based
on it, much larger than in the cases of China and Russia. It would be like the
owner of a coal mine campaigning for a carbon emissions tax: maybe there's
something we don't know, but from the information we have it seems unlikely.

Note that this wouldn't rule out the USA as such. First, it could be a
longshot preparedness thing, with no expectation that it would ever be used.
Second, they could be red-teaming the thing (looking for weaknesses so that
they can arrange for them to be shored up).

In either of these scenarios, it's no less likely that the USA would be doing
it than anyone else. If you assume that whoever is doing this is planning to
use their knowledge, however, the economic argument makes the USA less likely
to be involved.

~~~
barkingcat
There are many types of actors even within the USA nation-state / government.

For example, if a particular part of the government got wind of a data dump
about to be released by another nation-state or independent actor (for
example, a leak of some kind) - I think some parts of the USA government that
possesses the ability to do so wouldn't hesitate to take down dns to the
entire internet to avoid another similar data leak to the Snowden dump.

Be really wary of attributing intent: you do not know who will benefit the
most from taking down certain services. To claim that the US benefits from the
internet so much that it wouldn't do certain actions to protect itself from
certain types of harm is shortsighted.

Even my example could be really wrong, but the idea is that nobody really can
say - "oh the internet is too important to xyz, they'll never do anything!"

~~~
omni
> wouldn't hesitate to take down dns to the entire internet to avoid another
> similar data leak to the Snowden dump.

I don't understand how this would change anything unless you're assuming they
would take down the Internet permanently

~~~
Zancarius
I'm probably wrong, but this is how I see it (not sure about the OP).

News cycles happen fairly rapidly, so if you could take down a number of sites
that might be friendly to the dissemination of potentially damaging
information just long enough such that it's forgotten about, or the attack is
so large the media talks about the attack instead, then you might be able to
successfully avoid widespread public knowledge of such information. Though,
this would be best aided with collusion or cooperation (intentional or
otherwise) from the media. Toss in a few unrelated services as a bonus for
collateral damage, and you might be able to avoid scrutiny or, at the very
least, shift the blame to an unrelated state actor. It won't prevent the
release of information, but that's not the point--you want to prevent the
_dissemination and analysis_ of that information by the public at large.

This is all hypothetical, of course, and not likely to work. It also comes
with the associated risk that if you were discovered or implicated, public
outrage might be even worse than if you allowed the release of the information
you hoped to distract from in the first place! As such, I can't imagine anyone
would be stupid enough to try.

I'll take my tinfoil hat off now.

~~~
eridius
If you take them offline before they've managed to disseminate the info, then
it can't be forgotten because nobody knew about it in the first place. Which
means when the sites come back on, the info is still newsworthy.

------
tim_armandpour
I wanted to provide an update on the PagerDuty service. At this time we have
been able to restore the service by migrating to our secondary DNS provider.
If you are still experiencing issues reaching any pagerduty.com addresses,
please flush your DNS cache. This should restore your access to the service.
We are actively monitoring our service and are working to resolve any
outstanding issues. We sincerely apologize for the inconvenience and thank our
customers for their support and patience. Real-time updates on all incidents
can be found on our status page and on Twitter at @pagerdutyops and
@pagerduty. In case of outages with our regular communications channels, we
will update you via email directly.

In addition you can reach out to our customer support team at
support@pagerduty.com or +1 (844) 700-3889.

Tim Armandpour, SVP of Product Development, PagerDuty

~~~
pfarnsworth
I had the privilege of being on-call during this entire fiasco today and I
have to say I was really really disappointed. It's surprising how broken your
entire service was when DNS went down. I couldn't acknowledge anything, and my
secondary on-call was getting paged because it looked like I wasn't trying to
respond. I was getting phone calls for alerts that wasn't even showing up on
the web client, etc. Overall, it caused chaos and I was really disappointed.

~~~
JPHPJ
"It's surprising how broken your entire service was when DNS went down." lol

------
jssjr
I'm a GitHub employee and want to let everyone know we're aware of the
problems this incident is causing and are actively working to mitigate the
impact.

"A global event is affecting an upstream DNS provider. GitHub services may be
intermittently available at this time." is the content from our latest status
update on Twitter
([https://twitter.com/githubstatus/status/789452827269664769](https://twitter.com/githubstatus/status/789452827269664769)).
Reposted here since some people are having problems resolving Twitter domains
as well.

~~~
cddotdotslash
I'm curious why you don't host your status page on a different
domain/provider? When checking this AM why GitHub was down, I also couldn't
reach the status page.

~~~
ToastyMallows
+1

The only way that I could check to see if Github knew they were having
problems was by searching Google for "github status", and then seeing from the
embedded Twitter section in the results page that there was a tweet about
having problems. Twitter also being down for me didn't help the situation
either.

~~~
redbeard0x0a
The attack is on the DNS servers, which take names like www.github.com and
resolve them to ip addresses (i.e. 192.30.253.112 for me). Their status page
is status.github.com - it is on the same domain name (github.com) as the rest
of the site. Normally this isn't a problem because availability is usually
something going on with a server, not DNS.

In this case, the servers (DNS server under attack at Dyn) that knows how to
turn both www.github.com and status.github.com into an IP address were under
attack and couldn't respond to a query. The only way to mitigate this would be
to have a completely different domain (i.e. githubstatus.com) and host the DNS
with a different company (i.e. not Dyn).

~~~
cddotdotslash
Right, this was my point. Hosting "status.domain.com" doesn't help much when
it's "domain.com" that's having the problem. I think today's event will make a
lot of companies consider this a bit more.

~~~
Rapzid
Hiiiinnnnndsiighhhttttt!!!!! Yeaaaahhhhyeahh!

Anyway, for them to take the github.com nameservers out of the mix they would
need a completely separate domain name; would you know to look there?

You can delegate subdomains to other providers, but the NS records are still
present in the servers listed in the registrar. So, you'd already need
multiple DNS providers.. And you wouldn't have been down. Just sayin. I'm not
sure anyone rated a DNS provider of this status getting hit this hard or
completely as high enough risk to go through the trouble.

It's easy enough to look at a system and point out all the things you depend
on as being a risk. The harder part is deciding which risks are high enough
priority to address instead of all the other work to be done.

~~~
mnordhoff
I mean, some organizations do take precautions against this point of failure
and use a separate status domain. Most don't.

[https://www.dynstatus.com/](https://www.dynstatus.com/) (using Route 53, at
least today)

[https://www.cloudflarestatus.com/](https://www.cloudflarestatus.com/) (using
Dyn, ironically)

------
elwell
To get on github you can add to your /etc/hosts:

    
    
        192.30.253.113  github.com
        151.101.32.133  assets-cdn.github.com
    

And it seems faster than normal right (less users).

Edit; for profile pics include:

    
    
        151.101.32.133  avatars0.githubusercontent.com
        151.101.32.133  avatars1.githubusercontent.com
        151.101.32.133  avatars2.githubusercontent.com
        151.101.32.133  avatars3.githubusercontent.com
        151.101.32.133  avatars4.githubusercontent.com
        151.101.32.133  avatars5.githubusercontent.com

~~~
xyclos
how about npm?

~~~
puddintane
I was able to access everything by changing DNS as mentioned in the other
posts [1].

[1]
[https://news.ycombinator.com/item?id=12762841](https://news.ycombinator.com/item?id=12762841)

 _edit_ Of course this is if your local policy allows you to change this!

------
Animats
So who was prepared for this? Pornhub:

pornhub.com:

    
    
        Name Server: ns1.p44.dynect.net
        Name Server: ns2.p44.dynect.net
        Name Server: ns3.p44.dynect.net
        Name Server: ns4.p44.dynect.net
        Name Server: sdns3.ultradns.biz
        Name Server: sdns3.ultradns.com
        Name Server: sdns3.ultradns.net
        Name Server: sdns3.ultradns.org
    

ultradns.biz:

    
    
        Name Server: PDNS196.ULTRADNS.ORG
        Name Server: ARI.ALPHA.ARIDNS.NET.AU
        Name Server: ARI.BETA.ARIDNS.NET.AU
        Name Server: ARI.GAMMA.ARIDNS.NET.AU
        Name Server: ARI.DELTA.ARIDNS.NET.AU
        Name Server: PDNS196.ULTRADNS.NET
        Name Server: PDNS196.ULTRADNS.COM
        Name Server: PDNS196.ULTRADNS.BIZ
        Name Server: PDNS196.ULTRADNS.INFO
        Name Server: PDNS196.ULTRADNS.CO.UK

~~~
BinaryIdiot
When your business depends on your infrastructure being up and running you try
to prepare for anything. Then again Twitter is down so...

~~~
Animats
Twitter is still 100% on Dyn.

twitter.com:

    
    
        Name Server: NS1.P34.DYNECT.NET
        Name Server: NS4.P34.DYNECT.NET
        Name Server: NS2.P34.DYNECT.NET
        Name Server: NS3.P34.DYNECT.NET
    

They're probably using some geographically based DNS distribution scheme which
they can't quickly move to other DNS servers.

------
dEnigma
I was not aware of the attacks going on until this happened:

1\. Tried to download "Unknown Horizons" (game featured recently on Hacker
News) binary, github-link doesn't work.

2\. Think "Ok, might be an old link", google their github-repository, github
appears down.

3\. Try accessing github status website, is down.

4\. Interested, try to visit github status twitter account, twitter is down.

Really weird experience, normally at least the second source of news on a
downed website I try during an attack works.

~~~
jeppebemad
Had a similar experience. When I went to confirm on twitter, that was down
too. I was able to acces Twitter from my phone though, where I found a ton of
tweets saying "Twitter down!". Strange.

~~~
alexmorenodev
Funny how we're able to tell that twitter's down on twitter, but here in
Brazil, when whatsapp was down, nobody could use their own whatsapp to ask or
tell if whatsapp was down.

------
foobarbecue
According to Fortune, Hacker News "reported" on the incident. Are we
journalists now?

"Popular tech site Hacker News reported many other sites were affected
including Etsy, Spotify, Github, Soundcloud, and Heroku." \--
[http://fortune.com/2016/10/21/internet-
outages/](http://fortune.com/2016/10/21/internet-outages/)

~~~
mxuribe
No, not necessarily journalists; rahter, an information source...Fortune - a
site/company known for journalism/reporting - now just gave HackerNews more
legitimacy as an _official information source_...Now with this power, please
use it responsibly. ;-)

~~~
stingraycharles
Too bad that the majority of the readers will think that HackerNews is somehow
related to the "Hackers" that took down the internet.

~~~
figgis
Eh the name implies (at least in that context) this website would be used to
keep track of the hackers.

------
meshko
Very funny guys, can you stop now? We have a demo in 4 minutes.

~~~
iagooar
How did the demo go?

~~~
meshko
Well, the parts that relied out outside services hooked up via SSO were not
demoed, but majority of it worked fine because demo server was misconfigured
to not actually rely on the external services. It is pretty funny.

~~~
praneshp
It's Friday. Story/short write up appreciated

~~~
meshko
I am a bit paranoid about disclosing details, but basically our SAML IDP was
down, so the sales person couldn't log in at all. I was messing with the demo
server to convince myself that it is 100% IDPs fault and we can't do anything
about it, and discovered to my surprise that the form-based authentication was
not disabled on it (normally our servers are in one mode or the other, but not
both, even though this is an artificial separation). So I gave them the direct
link to the form based entry point and most of the demo could be done.

------
chromaton
I can't currently get resolution on www.paypal.com.

$ dig @8.8.8.8 www.paypal.com

; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.paypal.com ; (1 server found) ;; global
options: +cmd ;; Got answer: ;; ->>HEADER<<\- opcode: QUERY, status: SERVFAIL,
id: 17925 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;www.paypal.com. IN A

;; Query time: 29 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Oct 21
12:35:33 2016 ;; MSG SIZE rcvd: 32

~~~
chromaton
And it's back again. I'm on AT&T in Atlanta.

    
    
      $ dig @8.8.8.8 www.paypal.com
    
      ; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.paypal.com
      ; (1 server found)
      ;; global options: +cmd
      ;; Got answer:
      ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40999
      ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
    
      ;; QUESTION SECTION:
      ;www.paypal.com.      IN  A
    
      ;; ANSWER SECTION:
      www.paypal.com.    266  IN  CNAME  www.paypal.com.akadns.net.
      www.paypal.com.akadns.net. 29  IN  CNAME  ppdirect.paypal.com.akadns.net.
      ppdirect.paypal.com.akadns.net.  299 IN  CNAME  wlb.paypal.com.akadns.net.
      wlb.paypal.com.akadns.net. 29  IN  CNAME  www.paypal.com.edgekey.net.
      www.paypal.com.edgekey.net. 20  IN  CNAME  e3694.a.akamaiedge.net.
      e3694.a.akamaiedge.net.  19  IN  A  23.73.8.114
    
      ;; Query time: 146 msec
      ;; SERVER: 8.8.8.8#53(8.8.8.8)
      ;; WHEN: Fri Oct 21 13:05:48 2016
      ;; MSG SIZE  rcvd: 198

~~~
roland-s
Paypal and others still down for me at University of California (I think we're
our own ISP?)

------
sly010
I am confused. Are so many big websites using Dyn, or does Dyn have some
special role in the DNS chain in the US?

~~~
redm
They sell premium services, have a large sales team, and are very aggressive.
I get emails from them weekly discussing millisecond savings of their DNS
solutions and the value increase in customers and sales.

Squeaky wheels get grease and their sales team squeaks a lot.

~~~
dijit
Realistically they compete with Neustar which is shockingly expensive and has
less features and is harder to use.

I chose Dyn over Neustar (UltraDNS) when it was time to renew contracts
because it was 60% cheaper, had a better latency, their support was great and
the interfaces were clear.

Not a fanboy or anything, I really don't like how aggressively they hound me
now (even though I have nothing to do with DNS for my current employer), but
it's cheap and effective so it's not surprising people use them.

~~~
pyvpx
they are cheap compared to Neustar. And Neustar is priced like a Bugatti. Dyn
is more Porsche pricing.

~~~
Filligree
Way, _way_ back in time, they offered lifetime DNS hosting for a relatively
low price.

I bought that, and they've honored the deal. Admittedly it comes with limits
that would make it useless for any large site, but it's just great for
individuals.

~~~
jonah
I have a similar deal with UltraDNS. Nice to have "enterprise" DNS for my
little personal sites.

------
jtmarmon
I'm updating a list of confirmed outages as I see them here
[https://news.ycombinator.com/item?id=12759520](https://news.ycombinator.com/item?id=12759520)

So far twitter, etsy, soundcloud, spotify, github, pagerduty...crazy that this
can even happen

~~~
colanderman
I'm surprised; I would have thought such large sites would use more than one
DNS provider? I mean:

    
    
        $ host -t NS twitter.com
        twitter.com name server ns4.p34.dynect.net.
        twitter.com name server ns3.p34.dynect.net.
        twitter.com name server ns2.p34.dynect.net.
        twitter.com name server ns1.p34.dynect.net.
    

I would have expected at least one of those to be somewhere else. What is the
reason they would not have a backup provider?

~~~
komali2
I know a lot about some things, but almost nothing about networking, so excuse
me if this is a really dumb question but - would your physical location
determine what hosts you returned from that query? Like if you were in Asia
would you get different ones back?

~~~
snug
Yes because DNS typically uses ANYCAST networking. The DNS request routes to
the nearest location.

------
danyork
Journalist and security researcher Brian Krebs believes this is someone doing
a DDoS as payback for research into questionable "DDoS mitigation services"
that he and Dyn's Doug Madory did. Doug just presented his results yesterday
at NANOG and Krebs believes this is payback. Read more:
[https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
twit...](https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-
spotify-reddit/)

~~~
LordHumungous
If so that's a quick turnaround.

~~~
danyork
Well, Krebs sees this as an extension of the attacks that took down _his_ site
a few weeks ago after he wrote about this research. So he wrote about it,
attackers take down his site. His co-author Doug Madory speaks about it,
attackers take down Madory's employer's site.

Krebs indicates in an update at the end that a source had heard rumors in
criminal channels that an attack against Dyn was being planned.

Doug Madory's presentation was on the agenda for NANOG and so attackers would
have had plenty of time to know about it.

------
rybosome
I'm wondering, from a regulatory perspective, what might be done to mitigate
DDoS attacks in the future?

From comments made on this and other similar posts in the past, I've gathered
the following:

1) Malicious traffic often uses a spoofed IP address, which is detectable by
ISPs. What if ISPs were not allowed to forward such traffic?

2) There is no way for a service to exert back pressure. What if there was?
e.g. send a response indicating the request was malicious (or simply unwanted
due to current traffic levels), and a router along the way would refuse to
send follow up requests for some time. There is HTTP status code 429, but that
is entirely dependent on a well-behaved client. I'm talking about something at
the packet level, enforced by every hop along the way.

3) I believe it is suspected that a substantial portion of the traffic is from
compromised IoT devices. What if IoT devices were required to continually pass
some sort of a health check to make other HTTP requests? This could be
enforced at the hardware/firmware level (much harder to change with malware),
and, say, send a signature of the currently running binary (or binaries) to a
remote server which gave the thumbs up/down.

~~~
tim333
One thing that occured to me regulations wise is to require IoT devices to
have some minimum level of security such as a unique hard password rather than
it just being "admin" or some such. You could enforce it for items sold in the
US or EU and the Chinese manufacturers would probably follow so their goods
could be sold easily.

~~~
nkristoffersen
I've noticed a lot of wifi routers are doing this now. Which is pretty great.
All appear to be unique passwords for each router.

------
Animats
Analysis of the Mirai botnet: [1]

This is worth reading. It has links to copies of the code and names the known
control servers. Quite a bit is known now about how this thing works.

The bots talk to control servers and report servers. The attacker appears to
communicate with the report servers over Tor.

[1] [http://blog.level3.com/security/grinch-stole-
iot/](http://blog.level3.com/security/grinch-stole-iot/)

------
Mizza
Although I don't like to to recommend Google products, they provide a provide
a public DNS-over-HTTPS interface that should be useful for people who want to
add specific entries into their /etc/hosts files:
[https://dns.google.com/query?name=github.com&type=A&dnssec=t...](https://dns.google.com/query?name=github.com&type=A&dnssec=true)

------
Animats
"digikey.com", the big electronic part distributor, is currently inaccessible.
DNS lookups are failing with SERVFAIL. Even the Google DNS server (8.8.8.8)
can't resolve that domain. Their DNS servers are "ns1.p10.dynect.net" through
"ns4.p10.dynect.net", so it's a Dyn problem.

This will cause supply-chain disruption for manufacturers using DigiKey for
just-in-time supply.

(justdownforme.com says the site is down, but downforeveryoneorjustme.com says
it's up. They're probably caching DNS locally.)

------
newsat13
Switch to OpenDNS servers - 208.67.222.222 and 208.67.220.220. Even google NS
are down it seems. Heroku works after switching to opendns.

~~~
deathanatos
Google's DNS has been working all day here. The problem is that Dyn's DNS
server is being DDoS'd; if you request a record that the authoritative DNS
server for is hosted by Dyn, then when you query Google's DNS for that record,
then Google's server needs to make a query to Dyn, which is down, and thus,
your query fails. But queries to Google for non-Dyn domains will continue to
work just fine.

OpenDNS works because, as another poster notes, that, for better or worse,
they don't strictly obey TTLs:
[https://news.ycombinator.com/item?id=12762429](https://news.ycombinator.com/item?id=12762429)

------
bgentry
If you're having issues with people accessing your running Heroku apps, it's
likely because you're running your DNS through herokussl.com (with their SSL
endpoint product) which is hosted on Dyn.

If you can update your DNS to CNAME directly to the ELB behind it, it should
at least make your site accessible.

~~~
asdf333
thanks for the tip! how did you determine the ELB address behind the ssl
endpoint?

edit: figured it out. What i did was do:

nslookup your-SSL-endpoint.herokussl.com

then you'll see the elb address.

Switch to the openDNS servers helpfully pointed out by someone above first...

~~~
Artemis2
Presumably with something like `dig @208.67.220.220 -t CNAME <your
site>.herokussl.com`. This uses the OpenDNS nameservers, that people have been
reporting as working. Haven't tested it as I am on the go.

~~~
asdf333
thanks! figured it out but appreciate the help!!

~~~
natashabaker
I'm seeing "connection timed out; no servers could be reached". Anyone else
seeing that when trying to run the above command?

~~~
asdf333
did you switch your computer's dns servers to openDNS?

208.67.222.222 208.67.220.220

(or specify dns server in the command)

~~~
natashabaker
Yes we did dig @208.67.220.220 -t CNAME <ssl-endpoint>.herokussl.com. And we
got the following SERVFAIL error:

; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<\-
opcode: QUERY, status: SERVFAIL, id: <id> ;; flags: qr rd ra; QUERY: 1,
ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;<end-point>.herokussl.com. IN CNAME

;; Query time: 1226 msec ;; SERVER: <server>#53(<server>) ;; WHEN: Fri Oct 21
12:27:55 2016 ;; MSG SIZE rcvd: 44

~~~
asdf333
try nslookup your-SSL-endpoint.herokussl.com

the dig command does not work for me either...

=================================

nslookup iwate-2009.herokussl.com Server: 208.67.222.222 Address:
208.67.222.222#53

Non-authoritative answer: iwate-2009.herokussl.com canonical name =
elb030330-152447250.us-east-1.elb.amazonaws.com. Name: elb030330-152447250.us-
east-1.elb.amazonaws.com Address: 54.225.242.254 Name: elb030330-152447250.us-
east-1.elb.amazonaws.com Address: 54.225.217.226 Name: elb030330-152447250.us-
east-1.elb.amazonaws.com Address: 54.235.181.244

~~~
natashabaker
In that case I get:

;; Got SERVFAIL reply from 10.17.100.2, trying next server Server: 10.17.100.2
Address: 10.17.100.2#53

 __server can 't find <sslendpoint>.herokussl.com: NXDOMAIN

~~~
asdf333
maybe try this and replace w/ ur sslendpoint? see if that works? this works
for me.

[http://network-
tools.com/nslook/Default.asp?domain=iwate-200...](http://network-
tools.com/nslook/Default.asp?domain=iwate-2009.herokussl.com&type=255&server=208.67.222.222&class=1&port=53&timeout=5000&go.x=21&go.y=14)

------
cm3
Just to be clear, this is a DDoS against Dynect's NS hosts, right?

I'm confused because of the use of "dyn dns", which to me means dns for hosts
that don't have static ip addresses.

I'm actually surprised so many big-name sites rely on Dynect, which I hadn't
heard of, but more importantly don't seem to use someone else's NS hosts as
2nd or 4th entries.

~~~
detaro
The company is just called "Dyn", DynECT is a product name, but yes.

~~~
cm3
Thanks, I was really confused when I read the title "Massive Dyn DNS outage"
and how that affects Twitter or Github.

------
ohblahitsme
Twitter and Github are still down here in LA (and confirmed on isup.me)

~~~
jessemillar
Both are down here in Utah too.

~~~
flowless
New attack incoming
[https://www.dynstatus.com/incidents/nlr4yrr162t8](https://www.dynstatus.com/incidents/nlr4yrr162t8)

------
andmarios
OpenDNS servers seem the only ones that still work. Kudos.

It may not be the proper action but this kind of soft-fail scenario (use the
old DNS until you can contact the DNS servers and get new ones) is much
better.

    
    
      echo "nameserver 208.67.222.222" | sudo tee -a /etc/resolv.conf

------
ljosa
AWS says "We are investigating elevated errors resolving the DNS hostnames
used to access some AWS services in the US-EAST-1 Region." Is that
coincidental, or are they being DDoSed also?

~~~
colanderman
Apparently us-east-1 is backed by Dyn (and only Dyn) as well?

    
    
        $ host -t NS us-east-1.amazonaws.com
        us-east-1.amazonaws.com name server ns3.p31.dynect.net.
        us-east-1.amazonaws.com name server ns1.p31.dynect.net.
        us-east-1.amazonaws.com name server ns2.p31.dynect.net.
        us-east-1.amazonaws.com name server ns4.p31.dynect.net.
    

That's… utterly bizarre to me. us-east-2 has a more diverse selection:

    
    
        $ host -t NS us-east-2.amazonaws.com
        us-east-2.amazonaws.com name server u4.amazonaws.com.
        us-east-2.amazonaws.com name server u6.amazonaws.com.
        us-east-2.amazonaws.com name server u3.amazonaws.com.
        us-east-2.amazonaws.com name server u2.amazonaws.com.
        us-east-2.amazonaws.com name server u1.amazonaws.com.
        us-east-2.amazonaws.com name server u5.amazonaws.com.
        us-east-2.amazonaws.com name server ns2.p31.dynect.net.
        us-east-2.amazonaws.com name server ns1.p31.dynect.net.
        us-east-2.amazonaws.com name server pdns1.ultradns.net.
        us-east-2.amazonaws.com name server pdns5.ultradns.info.
        us-east-2.amazonaws.com name server ns3.p31.dynect.net.
        us-east-2.amazonaws.com name server ns4.p31.dynect.net.
        us-east-2.amazonaws.com name server pdns3.ultradns.org.
    

Not that anyone should be running a service whose availability they care about
_solely_ in us-east-1 anyway…

~~~
sandinmyjoints
AWS may have updated this, I now see

    
    
        $ host -t NS us-east-1.amazonaws.com
        us-east-1.amazonaws.com name server pdns5.ultradns.info.
        us-east-1.amazonaws.com name server ns3.p31.dynect.net.
        us-east-1.amazonaws.com name server pdns1.ultradns.net.
        us-east-1.amazonaws.com name server pdns3.ultradns.org.
        us-east-1.amazonaws.com name server ns4.p31.dynect.net.
        us-east-1.amazonaws.com name server ns1.p31.dynect.net.
        us-east-1.amazonaws.com name server ns2.p31.dynect.net.
        us-east-1.amazonaws.com name server u1.amazonaws.com.
        us-east-1.amazonaws.com name server u2.amazonaws.com.
        us-east-1.amazonaws.com name server u3.amazonaws.com.
        us-east-1.amazonaws.com name server u4.amazonaws.com.
        us-east-1.amazonaws.com name server u5.amazonaws.com.
        us-east-1.amazonaws.com name server u6.amazonaws.com.

~~~
colanderman
Me too. Someone realized their oopsie :)

------
tedmiston
Anyone else spend the morning thinking the problem was their setup? I've been
flushing my system DNS cache, Chrome's DNS cache, changing DNS servers,
rebooting my router, turning VPN on/off, etc.

~~~
stuntmachine
Yeah. :/

It happened to be at the same time I was getting things configured to connect
to a new VPN that I hadn't used before for the first time. Until about 7am
today my home network was a 10.0.0.0/8 network. VPN kept bombing in the last
phase of connecting and I couldn't figure out why, so I thought it was an IP
conflict with my internal network range.

So naturally, I then went into my router and changed my subnet for my entire
home network to the more common 192.168.1.0/24 range to see if it'd help. It
didn't. Until suddenly VPN "just worked" \-- which makes me wonder if I needed
to change my network at all to begin with.

Then I started experiencing all sorts of weird issues where the Internet
seemed to disappear from one minute until the next.

Then I hit IRC when things finally stabilized and see "Did you hear about
Dyn?".

My reaction: wut.

TL;DR: I rearchitected my home network at 7am for no reason.

------
nodesocket
I've been singing the praise of AWS Route53 for a long time, they up and
running. I can't believe major multi-million dollar companies (Twitter,
GitHub, Soundcloud, Pagerduty) would not run a mix of multiple DNS providers.

Also what is happening is a cascade effect, where a 3rd party being down
effects others.

~~~
crftr
> I've been singing the praise of AWS Route53 for a long time, they up and
> running.

I'm a fan of Route53, too.

But can we say that it weathered the attack? Or was it just lucky that its
systems weren't targeted?

~~~
stevekemp
One of the reasons why Route53 is good is because they give different
nameservers to each hosted zone - unless you choose to use a branded record-
set.

I've seen them be hit by dDos attacks in the past, but never had any
significant impact.

(I wrap Route53 and handle storing DNS records in a git repository over at
[https://dns-api.com/](https://dns-api.com/) Adding support for other backends
is my current priority to allow more redundancy.)

------
Supersam654
OpenDNS DNS Servers (208.67.222.222 and 208.67.220.220) are still resolving
websites while my typical fallback to 8.8.8.8 is not.

~~~
foxhop
I noticed the same pattern.

------
artursapek
Twitter, Reddit, wow. I was so confused for a moment. Thankfully HN is here to
explain.

~~~
tunap
I had several, sporadic 'secure connection could not be established' yesterday
while trying to open HN, amongst others. Painfully slow page load times across
the board, too(Craigslist, Monoprice,weather.gov, etc) Still may be my buggy
phone SIM...

~~~
dlg1416
wait, buggy phone sims is a thing?

~~~
tunap
Sorta. When I changed phones I cut my micro SIM down to nano size. Cut a wee
bit too much off and it now can slide off contacts if jarred... gotta get a
new SIM.

------
jread
Seems to be impacting POPs in US East most severly. We use Ripe Atlas to
assess the impact of DNS outages, and in the past hour have measured about
50-60% recursive query failure from a few hundred probes in that region:
[https://cloudharmony.com/status-for-dyn](https://cloudharmony.com/status-for-
dyn)

~~~
jread
Now impacting multiple regions including both US East and US West with very
query failure ratios - 50-70%

------
jrochkind1
Is it time for everyone to actually start using secondary name servers/DNS
resolvers too from a different provider from primary? DNS _is_ built for this,
for the very purpose of handling failure of the primary resolver, isn't it?
Just most people don't seem to do it -- including major players?

Or would that not actually solve this particular scenario?

~~~
sinap
The attack is on the authoritative name servers, not a DNS resolver. A public
DNS resolver will query the authoritative name server for a record if it
doesn't exist in it's cache.

~~~
mgamble
Agreed, but there is nothing stopping you from having the authoritative name
servers for a domain with different providers. As someone previously said, DNS
was designed for this.

~~~
Symbiote
It's used to be common for universities to do this, mine still does:

    
    
      ic.ac.uk.		45665	IN	NS	ns1.ic.ac.uk.
      ic.ac.uk.		45665	IN	NS	ns2.ic.ac.uk.
      ic.ac.uk.		45665	IN	NS	ns0.ic.ac.uk.
      ic.ac.uk.		45665	IN	NS	authdns1.csx.cam.ac.uk.
    

(and Cambridge use Imperial College as a secondary) but the best-known
American universities are on cloud providers now.

------
wnm
Heroku also seems to be affected. I'm getting this when I run 'heroku status':

>> We are seeing a widespread DNS issue affecting connections to our services
both internally and externally.

------
altyus
For me redirecting my DNS to Google public DNS 8.8.8.8 and 8.8.4.4 did the
trick.

~~~
dijit
that's not going to help much if the authoritative name servers (which is what
dyn is, btw) go down for more than a day.

Max record cache time is 86400s (24h), so if the attackers can keep it down
for 24h then google will have to have custom instructions in place (or cache
more aggressively than the RFC allows)

~~~
wongarsu
Since the attacked dyndns DNS servers are evidently anycast, the google server
you are reaching might connect to a different dyndns server than you do. If
google has luck to reach a less overloaded server, they might get an answer
where you get none.

~~~
TurningCanadian
Side note:

In addition, Google Public DNS engineers have proposed a technical solution
called EDNS Client Subnet. This proposal allows resolvers to pass in part of
the client's IP address (the first 24/64 bits or less for IPv4/IPv6
respectively) as the source IP in the DNS message, so that name servers can
return optimized results based on the user's location rather than that of the
resolver. To date, we have deployed an implementation of the proposal for many
large CDNs (including Akamai) and Google properties. The majority of geo-
sensitive domain names are already covered.

from [https://developers.google.com/speed/public-
dns/faq](https://developers.google.com/speed/public-dns/faq)

------
danyork
There's a bit of exquisite irony in the fact that just _yesterday_ an article
on the Dyn blog was:

Recent IoT-based Attacks: What Is the Impact On Managed DNS Operators? -
[http://hub.dyn.com/traffic-management/recent-iot-based-
attac...](http://hub.dyn.com/traffic-management/recent-iot-based-attacks-what-
is-the-impact-on-managed-dns-operators)

It's a good piece about how IoT-based DDoS attacks are carried out. And now
Dyn has the answer...

HN thread about that article at:
[https://news.ycombinator.com/item?id=12764650](https://news.ycombinator.com/item?id=12764650)

------
devy
Is Zendesk being affected? Their status page is reporting external DNS
provider is having DNS issue [1] and most of their sites are being affected.

[1] [https://status.zendesk.com/](https://status.zendesk.com/)

~~~
jorts
Yes, they were affected.

------
Legogris
Microsoft's visualstudio.com's build servers fail to resolve Github and New
Relic. So much for my Friday night deploy to staging.

------
mjpa
Is it really an internet wide outage?

Only 2 of the points in the US are affected on
[https://www.whatsmydns.net/](https://www.whatsmydns.net/) for the domains
we've got on Dyn - same for Twitter etc

~~~
owenwil
If it's under a denial-of-service it's possible that it may respond correctly
part of the time.

~~~
toast0
Since many (all?) of Dyn's authoritative server IPs are anycast, attack
traffic is probably not well distributed either. If you're routed to a server
that's getting a lot of attack traffic, you're likely to have problems, but a
server without much attack traffic will work fine.

------
danyork
Other HN threads on related articles:

Krebs on Security:
[https://news.ycombinator.com/item?id=12761859](https://news.ycombinator.com/item?id=12761859)

NY Times:
[https://news.ycombinator.com/item?id=12765652](https://news.ycombinator.com/item?id=12765652)

Bloomberg:
[https://news.ycombinator.com/item?id=12763501](https://news.ycombinator.com/item?id=12763501)

Dyn:
[https://news.ycombinator.com/item?id=12764650](https://news.ycombinator.com/item?id=12764650)

------
patmcguire
Any quick script to see if a given domain ultimately resolves to them? My SaaS
company has a lot of custom domains from whatever DNS servers pointed at us
and I'd like to be able to tell people whether it's our fault or not.

~~~
mdavidn
`dig NS $domain`

Query for the root domain, without any subdomains like www. That is, you need
to check the "zone apex," the shortest name purchased from a registrar and
potentially delegated to Dyn. Look for dynect.net in the list of authoritative
name servers.

~~~
patmcguire
Yeah, I tried that. I don't see dnyect in a lot of domains that are failing,
and it's clearly related _somehow_ , they didn't all break at the same time by
coincidence.

------
CodeSheikh
Let's assume, that foreign countries such as Russia or China would be trying
to sabotage our elections on Nov 8th night. What are the severe economic and
political backlash that we can deal with if we cut off the traffic coming in
from those region (not in a "we control the internet" kinda way)? I am sure
they already have nodes operating within the USA. A lot of major tech
companies use CDNs that can still serve traffic globally to the consumers of
those countries. Even better, how about we regulate and slow down all of
incoming traffic for say half day on election day? Is it even possible?

~~~
transfire
But then why would they be doing it right now? I'm sure they already know if
they can do it or not. I don't think they need to do a large scale test run
that would put people on high alert. They'd keep their head down until
election day.

But then, what is China or Russia going to get out of doing something like
this? It isn't going to change anything. Hillary is the next president
regardless. Hell, even if no votes could be counted I am sure the Supreme
Court wouldn't have a problem calling it for her.

So to me the idea that China and Russia is doing this for political reasons
doesn't make any sense.

------
_ar7
Almost every website I visit except HN seems to be down...

~~~
chrisabrams
Same here, that's odd isn't it ;)

~~~
pyvpx
well, first thing one does to use CloudFlare is migrate to their DNS. A CF-
hosted site isn't going to be using Dyn...

------
devnull42
Dyn reporting another attack started at 15:52 UTC.

------
edcastano
The great irony:
[http://www.isitdownrightnow.com/yesware.com.html](http://www.isitdownrightnow.com/yesware.com.html)

------
pawal
DNS was designed so that you can have multiple operators for your
authoritative name servers.

Who would have thought adding a spof to your infrastructure would ever be a
problem?

------
emmet
Is it just me or are these kind of attacks becoming way more frequent
recently? This kind of widespread outage seems so new, but again, that might
just be me.

------
dudul
Damn, I've spent the past 30 minutes trying to update my DNS and playing with
my router config! :)

No GitHub, well, it's gonna be a fun Friday...

~~~
piquadrat
putting something like

    
    
        192.30.253.113 github.com
    

into your /etc/hosts (or other appropriate location for your OS) should get
you going again.

~~~
geekamongus
Careful...some people may be trying to get out of work today.

------
mirekrusin
They should do it once a year and call it Friday without Internet Day.

~~~
Asparagirl
Better yet, have one day a year that is "Red Team Day" where people hunt for
vulnerabilities so that assessments can be done, and companies can later fix
any issues noted. Like how earlier this week there was a statewide earthquake
drill in California, local emergency sirens were sounded, schoolkids practiced
hiding under desks, etc. The Internet needs periodic tests like that too.

~~~
jessemillar
I absolutely love this idea! Would be a bit tricky to implement, but would
definitely improve security in the long run.

------
shortstuffsushi
In (well, after) attacks like this, and really any other massive DDOS,
shouldn't it be possible to identify potential botnets and try to take them
out (notify their owners that they're being used, notify their hosting
providers, etc) so that they can't be used again in the future?

------
azaydak
Quick question for you all. Just two days ago I registered two domain names at
dynu (not dyn). Early this morning I a cold call from a company in India who
knew the domain names and my phone number and was calling to ask if I wanted
them to help me manage my website cheaply. Also, this morning I got a spam
text from someone who claimed to by godaddy offering the same thing. Now I
protect my number really well so this is the first time in 5+ years that I
ever got spam texts or calls to my number. Do you think Dynu was also hacked?!
Or maybe Dynu sells client numbers (which is how the guy in India claimed to
get my number) and it was just by random chance that this happened at the same
time as the Dyn hack.

~~~
shortstuffsushi
Fwiw, this isn't a hack, this is a DDoS (denial of service). It seems almost
certain that your information was either given out by dynu, or your WHOIS
record isn't protected. Check your domains out with your favorite WHOIS tool
first. Otherwise... time for an awkward conversation with dynu.

~~~
azaydak
Right, hack wasn't the right word. Anyway, thanks for the info.

------
atsidi
I've been having the same problem accessing github in particular. Just for
fun, I opened the Opera browser and activated the built-in VPN. That got
everything going again. At least for browsing, not so useful for my git pulls
and pushes.

------
LeanderK
Can someone explain why this is so bad? I think the internet handled the
downtime of Dyn pretty great, not reaching github wasn't exactly pleasing, but
i added the ip temporary to /etc/hosts and the problem was solved. Isn't the
best strategy to accept that attacks will continue and systems may go down and
design for resilience? If so this attack can serve as a warning and as a check
that we can handle these types of attacks. I am a bit exaggerating, but i
would imagine that constant attacks keep the internet resilient and healthy.
An unchallenged internet may be the greater risk.

~~~
rak00n
You're assuming we'll build immunity fast enough. What if we don't?

Attacks at this scale can bring a significant part of Internet down. The
economic affect can be just as bad as a war.

~~~
LeanderK
but just pretending no one will try bad things won't help either. It may speed
up the progress and will prevent bringing a significant part of the internet
down in the future and also remind everybody that these things can (and
probably will) happen.

------
adamrights
We were affected @WSJ as well.

------
DenisM
The DDoS problems, at least those not related to spoofing IPs, could be
curtailed if we provide a strong incentive to the ISPs to work on it.

Let's hold the ISPs financially liable for the harmful traffic that comes from
their network. If a client reports a harmful IP to the ISP, every bit of
subsequent traffic sent from that IP to this client carries a penalty.

Yeah, I know, routing tables are small, yada yada. If we put thumbscrews to
the ISPs they will find a way to block a few thousands IPs of the typical
botnet, even it requires buying new switches from Cisco & co.

Incentives drive behavior.

~~~
Florin_Andrei
Put the thumbscrews on the IoT manufacturers instead, so they don't release
widgets with bad security, so the problem is eliminated at its root.

You wouldn't allow car manufacturers to sell cars with faulty airbags, why do
we allow device manufacturers to provide plentiful firepower for bad actors?

~~~
DenisM
With ISPs it's a lot easier - you know who your ISP is, so either they respect
your blacklist or they automatically owe you money.

How would you even start chasing a manufacturer of a cheap IP cam from China?
How many of them can you chase at once?

------
jtmarmon
Semi related: I noticed this incident right when it began, but not because I
was trying to access a website. This started happening to me:
[http://imgur.com/PPlaY5o](http://imgur.com/PPlaY5o)

Then when I went to push to github out of fear my computer was about to soil
itself, that failed too, and I noticed the outage.

Does anyone know if the above errors could be related to the outage? I'm using
vim inside tmux with zsh as my shell. Maybe zsh does some kind of
communication with gh while running?

I restarted my computer and it's still happening

~~~
scrollaway
The zsh default git plugin definitely doesn't touch github, or the network in
general.

Are you using some oh-my-zsh github plugin by any chance?

~~~
jtmarmon
plugins=(git rbenv nvm gitfast zsh-autosuggestions github)

~~~
scrollaway
So, yes.

[https://github.com/robbyrussell/oh-my-
zsh/wiki/Plugins#githu...](https://github.com/robbyrussell/oh-my-
zsh/wiki/Plugins#github)

oh-my-zsh is, imho, way overengineered and bloated. Leads to all sorts of
issues like the one you're encountering there.

I would recommend sticking to a plain zshrc file that you can read, edit and
fully understand.

The one I wrote and am using day to day is available here, with documentation:
[https://github.com/jleclanche/dotfiles](https://github.com/jleclanche/dotfiles)

~~~
jtmarmon
removed the github plugin, reloaded zsh and happened again 5 min later. I
believe it has to do with slack, because the issue resolved itself after
closing. maybe slack got pwned and all slack users are being used as part of
the botnet lol

------
mdtancsa
Anyone know any details of what the attack looks like ? I had a quick look in
my (albeit small) network to look for odd flows going to their ASN33517, but
didnt see much that looked odd on first glance...

------
elmigranto
I've managed to (seemingly) save my browsing with Yandex DNS:

    
    
        77.88.8.8
        77.88.8.1
    

[https://dns.yandex.ru](https://dns.yandex.ru)

~~~
elwell
I'm sure yandex is safe, but I'm wary of using a anything dns.*.ru to route my
traffic to potential phishing versions of sites.

~~~
elmigranto
If it makes you more at ease, use this one :)

[https://dns.yandex.com](https://dns.yandex.com)

------
peatmoss22
Need to get in to dyn.com to download your zone files add this to your hosts
file: 204.13.248.106 www.dyn.com 204.13.248.106 dyn.com 216.146.41.66
manage.dynect.net 151.101.33.7 static.dyn.com

------
metaverse
While my app isn't resolved using DYN, we are relying on APIs on our EC2
backend that use their DNS. Is there a Linux DNS caching server that will
serve from a local cache primarily, and do lookups in the background instead
to update the local cache? During the period DYN was down, it would've
continued severing from the local cache and retried the background lookups,
keeping my app up. I can also see it improving performance as my servers
currently do lookups to the EC2 DNS on each http request...

~~~
nixgeek
If you're in us-east-1 then you potentially do actually rely on Dyn even for
the amazonaws.com instance hostnames.

[https://gist.github.com/agh/4e20df0d2d3bfa189477569b77f72e24](https://gist.github.com/agh/4e20df0d2d3bfa189477569b77f72e24)

~~~
metaverse
Seems then that ELB has a local cache because http requests were reaching my
app servers throughout the outage.

------
octoploid
It is spreading to other DNS providers, too:
[https://status.fastly.com/](https://status.fastly.com/)

www.ft.com is unreachable for example.

~~~
snug
Fastly is simply putting up a status page so they aren't contacted about
issues, and letting them know it's about DYN. And they are having internal
issues with communications like zendesk.

------
mmaunder
Third attack underway:
[https://twitter.com/AlexJamesFitz](https://twitter.com/AlexJamesFitz) (as of
10 mins ago)

------
anonymousjunior
No idea if this would work, but could people theoretically just ping flood the
IOT devices involved to mitigate the attack?

They run some sort of web server since most devices provide some web
interface, so clearly there's a port open which could be hit if the IP is
know, and with the shoddy security in these devices I'd wonder if their local
(likely low performance) hardware would be susceptible to something as simple
as a ping flood attack.

------
leesalminen
Boulder here. Can't resolve Wufoo or PayPal using 8.8.8.8

------
paulddraper
I thought DNS (particularly public) was basically immune to DDoS?

If one DNS server is down, use the cached result or another server.

DNS is some of the most distributable, cachable data I can imagine.

~~~
snug
Depends on how many PoPs they have. Looks like they have 4 easter US.[0] If
they are seeing large attacks that Krebs saw a few weeks ago, that could
certainly be enough to take down one or two, and then causing redirected
traffic to take down the other two.

I used to work for a DNS/DDoS provider, and this was a very real problem.
Leave the PoPs that are being affected out, or risk overloading the other PoPs
by overloading real traffic.

Before moving the other traffic, you also have to worry about blocking the
DDoS traffic otherwise you're just redirecting them to the other PoPs.
Mitigating DDoS attacks are not fun, and hard to block.

[0][http://dyn.com/dns/network-map/](http://dyn.com/dns/network-map/)

------
r1ch
Surprised to see so many big names relying on a single provider. DNS is
designed to be distributed, it should be possible to avoid a single point of
failure.

~~~
mayli
This question came to my mind when I saw this post. The possibility might be
the management cost. Synchronize between different providers can go wrong and
might hard to debug when end users get different replies.

------
fatherzeus
For people in need of the IPs for their respective services. You can find them
here: ipaddress.com or any of the other similar services

------
wweiss1230
How can I, a proficient web developer but one with little experience working
directly with its underlying infrastructure, help in whatever effort is being
down to thwart this and related attacks? I feel a moral obligation to help as
these attacks seem a grave threat to our economy and could cause unrest given
the current political climate. Thanks.

~~~
dmourati
Read all the analysis you can to form a better understanding of how this all
works. Use that information to design and run more resilient services in the
future. Teach what you have learned to others.

------
danyork
[https://cloudharmony.com/status-for-dyn](https://cloudharmony.com/status-for-
dyn) is now (12:43pm EDT) showing Dyn's "US East" and "US West" centers as
being down. Anyone know anything about this Cloudharmony service? How often
does it update? and what is it monitoring?

------
djhworld
At work earlier we was seeing hostname resolution errors with applications
trying to contact amazon s3 from on premises infrastructure.

This was in eu-west-1, but it coincided with a bunch of other systems in the
organisation having problems at the same time.

Additionally CloudWatch logs seemed to be completely broken for about 30
minutes on the Amazon Console.

------
arp
Here's how to add static mappings temporarily to survive through the outage:

[https://www.reddit.com/r/sysadmin/comments/58o5mp/dyn_dns_dd...](https://www.reddit.com/r/sysadmin/comments/58o5mp/dyn_dns_ddos_pt_2/d923yvw/)

------
dudul
And there is no twitter to tweet about it!!!

------
x2398dh1
Currently I am able to get into every site on the web, including GitHub, by
using a VPN service based in Hong Kong.

------
RRRA
Those distributed alternatives look better everyday... if only there was a
working group and a transitional path.

------
cyberferret
Hmm... Seems to be quite widespread. Some of our Amazon AWS services (located
in the US) that rely on SQS are reporting critical errors. Intercom.io is also
down at present, which we use for support for our web apps. Not looking very
good from here (in Australia).

------
pmuk
I'm getting DNS errors on my PS4 when trying to download stuff, I guess it's
related!

~~~
pmuk
Switching to Google's public DNS seems to have fixed it!

------
foxhop
So I had hardcoded my DNS server to googles, aka:

    
    
        dig @8.8.4.4 github.com +short
    

I was not getting an answer.

However using my routers/dhcp/ISP to set my DNS server, I am able to get
answers:

    
    
        dig github.com +short
        192.30.253.112

~~~
snug
Cached locally?

dig +trace github.com

------
edgartaor
I'm curious. What kind of infrastructure you need to make this massive attack?

~~~
saidajigumi
via [1]: "Dyn says today's DDoS are in part being caused by Mirai botnet,
which recently caused record-sized attack [2]"

[1]
[https://twitter.com/AlexJamesFitz/status/789562789920636928](https://twitter.com/AlexJamesFitz/status/789562789920636928)

[2] [https://krebsonsecurity.com/2016/10/source-code-for-iot-
botn...](https://krebsonsecurity.com/2016/10/source-code-for-iot-botnet-mirai-
released/)

tl;dr: a substantial part of this attack is a botnet of IoT devices.

------
dmalvarado
This may be dumb, but someone enlighten me:

If this kind of attacking does escalate, wouldn't it be possible to simply cut
off requests from outside the United States at the points of entry? Basically,
turning the US into an intranet?

~~~
danyork
We don't know yet, but the attack very easily could have been coming from a
botnet of devices entirely inside the US. Geographic borders don't matter much
at all for the Internet.

~~~
dmalvarado
But even if it were, the creator of the botnet would first have to gain
control and then issue a command, right? How would either of those things be
possible from outside if there was no connection into the US?

~~~
danyork
Well, there's pretty much no way to impose the geographic borders of the US
onto the Internet. Our networks here in the US are all global and integrated
with other networks all around the world. The only places where this kind of
geographic control is possible are countries like Iraq, Iran, China and others
where the government controls all the ISPs. Countries with more freedom have a
free flow of information and packets - and to me that is a very GOOD thing.

------
dboreham
What this event shows is that using DNS as a load routing/balancing mechanism
is a bad idea (because that's why folks have low TTL and an inability to
specify truly redundant secondaries).

------
cultavix
Not sure if related but circleci.com is down for us do to a "DNS issue" !

~~~
jessemillar
Definitely related. Can confirm.

------
kilroy123
Interesting. Lots of sites have been down for me, here in Mexico City.
Twitter. Github. Loads of other random sites. When I turned on my US based
VPN. It all started working again.

------
wav-part
Why is there even a concept of managed DNS ? Arnt we already paying >$1M/yr so
that we can get 32 bit integer from a string ? This does not make sense.

------
dev_1024
How come you can access these sites from some countries? I imagine there are
lots of name servers and that the attackers are specifically targeting servers
for US?

------
nbrempel
It's a strange coincidence that Hover DNS was down for same reason a week ago.

[http://hoverstatus.com](http://hoverstatus.com)

------
Rapzid
Looks like github and braintree both got AWS dns servers mixed in about the
same time. Did they both switch over or is Dyn working with AWS on this?

------
lips
How many DNS services ala Dyn exist? Is it not still massively significant
that a successful attack can be launched on even one of these?

------
adobrawy
Twitter and GitHub is down on Scaleway (AS12876) and Tiktalik (Warsaw, Poland,
Europe, AS198717) network too (no response from dynect.net).

------
Kluny
Highrise seems to be having problems, as seen by email errors when we forward
email to Highrise dropboxes.

------
llamataboot
Heroku is still having problems as well

------
alexmorenodev
Here in Brazil things are pretty slow.

"Oh, maybe its our shitty ISP screwing up everything again."

No, it's in a bigger scale.

------
tbarbugli
Github does not work for 100% the time

~~~
bananicorn
Weird, works for me - from Italy (not sure if there isn't just some caching
going somewhere down the line and I can see it because of that) edit:
nevermind, it's almost certain i've got it cached

~~~
owenwil
Definitely a DNS cache on your computer (Or even in Chrome)

~~~
detaro
I can query the authoritative ns*.p16.dynect.com DNS servers from Europe
(Germany in my case), and the traceroute looks like it's near Frankfurt. So
the anycasted copies here seem fine.

~~~
detaro
and now these seem down as well. EDIT: and up again half an hour later

------
Animats
Github is currently inaccessible. Can you still compile Rust programs that
depend on Github files?

~~~
steveklabnik
First of all, the only thing that'd matter is for modifying your dependencies
at the moment. If you've previously built the project, and don't touch your
deps, GitHub won't be hit.

Second, Cargo only depends on GitHub for the index. For more:
[http://integer32.com/2016/10/08/bare-minimum-crates-io-
mirro...](http://integer32.com/2016/10/08/bare-minimum-crates-io-mirror-plus-
one.html)

That includes a link to a mirror run by integer32.

------
zappo2938
Explains why the Heroku API is down.

------
kakarot
Don't be a dick. I'm sure their staff has a giant collective migraine right
now.

------
Kaedon
What other providers would you recommend than Dyn? Route53? Cloudflare?
Something else?

------
ifelsehow
Reposting imglorp's comment on the root of the comment tree, as it's buried
currently. This should restore service for those desperately needing to access
Github etc ;)

> ....point your machine or router's DNS to use opendns resolvers instead of
> your regular ones: 208.67.222.222 and 208.67.220.220

------
tbarbugli
I am very surprised this is not getting that much attention on national news.

------
im3w1l
Fascinating weak spot!

------
d--b
Looks like at least some of it is resolved. spotify is back

------
Raed667
You can add Netflix to the list.

    
    
        GET https://art-s.nflximg.net net::ERR_NAME_RESOLUTION_FAILED
    
        GET https://assets.nflxext.com net::ERR_NAME_RESOLUTION_FAILED

------
BlackGuyCoding
Anyone having any issues with WhatsApp? Mobile text seems to work fine but all
images fail, Desktop & web browser aren't connecting at the moment (west
coast)

------
CarVac
Using Google Public DNS fixed things for me.

~~~
ssebastianj
I'm using Google Public DNS too. I don't really know if there is a relation
with DynDNS but I'm still experiencing issues in GitHub and Twitter, like
partial loading of images.

~~~
dichardson
Perhaps Google had old (but valid) records still in their cache for a while.
Google DNS was working for me for a while, and then stopped. Apparently Dyn
has the problem fixed, but maybe there is some TTL based propagation delays
still. I updated my internal network to use Dyn's internet guide/public DNS
and the problem is fixed.

Maybe this is their strategy: we break it, you buy it ;)

[https://help.dyn.com/internet-guide-setup/](https://help.dyn.com/internet-
guide-setup/)

If you can't load that page, the public DNS servers are: 216.146.35.35,
216.146.36.36

------
mirekrusin
Github doesn't work again for me :(

------
invisiblep
Why not use:

OpenDNS - recursive DNS

Cloudflare (DNS only) - authoritative DNS

Both services are free and distributed across the world.

~~~
q3k
Dyn was supposed to be distributed too.

The takeaway here should be to use multiple authoritative DNS providers, not a
single (even if better) one.

------
Artemis2
PayPal, Braintree, Spreedly down. Some companies are going to lose money
today...

------
darkmouth
and its down again

------
eredi93
and the attacker are back. DDoS v2 is here

------
halayli
github.com seems to be down because of this.

------
middleman90
Shopify is down

------
transfire
Oo oo, I know! Iran did it!

------
piker
CNN.com is knocked out by this attack as well. I could see that as a useful
target.

------
transfire
Must be trying to stop the latest Julian Assange leak.

~~~
alasdair_
The Wikileaks twitter feed is putting out some really weird stuff lately. They
are claiming their "supporters" are behind the attacks, which makes little
sense.

------
ilostmykeys
The Internet is so resilient. LOLz.

~~~
wongarsu
The internet is resilient against being completely taken down (as demonstrated
by everything working just fine for me from Germany). It's explicitly not
resilient against taking parts of it down. You can take some continents off
the internet entirely by cutting half a dozend cables, but it's extremely hard
to make the internet unusable for everyone.

------
chatmasta
I'd like to see proof of this attack from an outside network observer.

Is it possible the government could force a DNS provider to pretend to fall
victim to a DDoS attack, as a form of a false flag cyber attack?

------
brooklyndude
Why does it always have to be a "Nation State", have been hanging out with 17
year old's that knew far more about DNS configs than a room of "Cyber-
Security-Professisonals", they were clueless, these kids could run circles
around them.

Kids.

------
raemike123
USA cyber defenses are NOT up to the task of defending our critical electronic
infrastructure. Letting every company that runs critical services decide their
own security posture is not scalable and has left us vulnerable. While no one
is getting hurt, we are taking cyber missile hits from our enemies and
eventually the damage will be worse. Other countries with more central
controls will be less vulnerable than we are to crippling infrastructure take
downs.

~~~
bluejekyll
No. What we need are new techniques for creating back-pressure to all the
routers which are forwarding on this type of attack. The issue is that our
Routing technology does not give downstream nodes any way to push back on the
flood of packets.

Cisco could step up to the plate here. And no, I'm not talking about
firewalls. We need newer ICMP type packets to create this back-pressure, so
that we can stop floods like this.

~~~
andrewmchen
It looks like we used to have something similar but it got deprecated.
[https://tools.ietf.org/html/rfc6633](https://tools.ietf.org/html/rfc6633)

~~~
Animats
I was a big fan of ICMP Source Quench in the early 1980s, but it wouldn't help
now. It doesn't have authentication.

