
Ask HN: How did Dyn fail to fend off DDOS? - ruler88
I&#x27;d imagine that DDoS attacks is something that DYN and other DNS providers would spend a lot of resources to prevent. Was there something specific about this DDoS attack that DYN was unprepared for? Or is there some reason that distributed natural of DNS makes it hard to prevent DDoS? Anyone know of any steps that DNS guys are taking to prevent another DDoS?
======
45h34jh53k4j
I would like to remind those that think all is lost with this:

A serious conversation with vendors about default passwords and backdoors post
this incident will help prevent recurrence. This has forced this talk and we
are better for it.

There was a time when your windows box would get popped from being online for
more than 4 minutes. We recovered from this. Conficker in 2008. Blaster in
2003. It was a 'BIG BOTNETS OH NO', but we cleaned up, recovered, hardened.
Microsoft went from being botnet enabler to an active force in dismantling
bots and crime rings. It sucks, and some of us have a bad day, but we recover
ever stronger.

XiongMai Technologies may well find themselves in some international hot water
over this incident, and I think they deserve it. They sold a faulty product
that caused billions of dollars in lost revenue to some very large internet
properties for a day in October 2016. I would encourage vendors look at these
incidents from last decade and how these were turning points for upping their
security game. I would encourage its victims to investigate legal recourse.

Specifically the current vulnerable nodes of Mirai, i am sure these will be
removed from the internet pretty soon. One only gets to fire something like
this a few times before the feds are on the door.

Your regularly scheduled program will commence shortly.

~~~
dpweb
Just eliminate default passwords completely. The first person that opens the
box, or applies a license key, sets the password and it must be strong.

~~~
webmaven
A better solution is for each physical instance of a device to have a default
password that is _strong and unique_ (and encoded in the firmware, such that a
factory reset of the device doesn't make it default to a non-unique PW).

There are a few other ways to handle the problem of securing endpoint devices.
For example, for devices that are intended to use a local aggregator, gateway,
or proxy of some sort you can get around the issue (and improve the UX) by
avoiding passwords entirely, and requiring that the device instead be paired
with a base station through a physical action the user performs (pressing a
button on both, knocking them together, etc.) instead.

------
Animats
It's time to apply some serious pain to the junk IoT manufacturers, retailers,
distributors, and importers. A nice big billion-dollar lawsuit against Amazon
for gross negligence would be a good way to start. US consumer law allows
suing everybody in the supply chain. (They can then sue each other and try to
sort out who pays, but that's not the victim's problem.)

We also need some big recalls. If Homeland Security tells the Consumer Product
Safety Commission this is a national safety issue, the CPSC can order a
recall. Something like this worked with those exploding "hoverboards". CPSC
ordered recalls, Amazon took the junk back, and Amazon refused to pay
manufactures in Shentzen. The manufacturers were furious, but hoverboards with
crap batteries disappeared from the market very fast.

~~~
verroq
I think the more realistic solution is that a vigilante group of hackers
continuously scan and take over vulnerable IOT boxes with the intention of
bricking and/or disabling their network access would be the most feasible.

~~~
snowwrestler
The problem with this idea is that it is illegal, and federal agents are much
better at tracking people down on the Internet than they were even 5 years
ago. So while I think a lot of us would cheer the vigilantes on, they would be
taking a serious personal risk.

~~~
jazoom
If they were so good at that then this wouldn't be a problem in the first
place. The hackers can be in the same country as the DDOSers.

~~~
camhenlin
But then wouldn't they be better off doing DDOSing for hire? Just a thought

------
hannob
I think the answer is surprisingly simple: The attack was just huge.

The unfortunate truth is that with the Internet of Things the amount of
devices that can easily be taken over has grown so fast that we see DDoS
attacks of unprecedented size. Even more unfortunate is that there is no sign
whatsoever that this is going down again.

~~~
ericcholis
Does anybody have solid recommendations for secure IoT devices? Initial
searches lead me to believe that they are non-existent.

~~~
drather19
Where's the pain-free device with open source, easily upgradeable firmware,
that puts all of our IoT devices in their own private network but lets us
tunnel through to them? It needs to be easy enough that our (grand)parents
could pick one up on Amazon, Best Buy, or Home Depot and plug in and go...

~~~
fakir
If these are connected by cellular, they are given a private network that does
not connect to the public internet and are in-accessible from the public
internet unless the app provider explicitly chooses to do so

------
Silhouette
The real question here is whether there was anything they could realistically
have done to prevent it at all.

In order to defend against a DDoS attack, you really only have two options.
One is to have sufficient capacity to cope with the extra load without
undermining your normal service. The other is to reduce the amount of extra
load you have to handle, by identifying and blocking the hostile traffic at
some point before your main system deals with it fully.

In this case, the scale of the attack was huge thanks to all the woefully
insecure IoT devices out there. But worse, from the initial reports it appears
that the requests being sent were effectively indistinguishable from valid DNS
requests: they came from diverse sources, and asked DynDNS to do exactly what
it's normally supposed to do, just for random subdomains that don't actually
exist. Unless there is some pattern in those requests that allows for
identification of the hostile incoming traffic so it can be dropped early,
there's probably very little DynDNS could have done here. And of course the
attack is particularly effective because by taking out infrastructure rather
than attacking a specific site, it brings down large numbers of high profile
sites all at once.

It is disturbing, but apparently the reality we face, that there are now so
many hopelessly insecure devices on the public Internet that this is possible.
The best long term strategy for dealing with it seems to be trying to improve
the standards of Internet-connected devices and reduce the number of highly
vulnerable devices with access to the Internet, but this was always going to
be difficult with IoT products aimed at the general public. I suspect some
sort of remediation/recall scheme for manufacturers/vendors and some sort of
throttling of users' Internet connections to force them to respond to security
recall/update notices may be necessary if this kind of attack starts to become
a pattern.

------
45h34jh53k4j
I think this is a plausible theory of the attack - (first seen in from npr
report on incident):

NANOG 68 BackConnects Suspicious BGP Hijacks is shown 4ish days ago. Last talk
of the night, discusses BGP hijacking shenanigans and krebs; touches on MO of
possible attacker. Speaker is Director at Dyn. Attack in retaliation.

So far the targets have been organisations that have responded to or made
allegations of corrupt DDoS business.

Please don't buy into all this cyberwar bullshit, this may just be a well
resourced (its really not that hard to pop boxes with default passwords.....)
attacker doing criminal response to commentary.

~~~
ryanlol
This is likely, Backconnect hosted Mirai in the past right before attacks on
Krebs. (however not during them.)

There's also no small amounts of publicly available evidence that Backconnect
used insider information provided by their CEO (ex Staminus employee) to
compromise Staminus network earlier this year by hijacking a management range
of theirs.

------
beachstartup
i think there is a larger strategy at play. this is pure speculation and
anecdote.

recently there has been an aggressive uptick of dns ddos attacks against
smaller companies/service providers that run their own dns infrastructure.
this includes small/regional internet service providers and individual
sites/hosts that still run their own servers.

in almost all of these cases that i'm aware of, the smaller companies
immediately outsourced their dns services to a larger company, one that
ostensibly is able to either absorb, scrub, or otherwise defend against these
types of attacks.

extrapolating to a global scale, what's happening is a forced consolidation of
dns infrastructure into a handful of large players. even in the case of having
redundant providers, it's usually two very large providers. and as we just saw
today, a terabit-level attack is not something we can readily defend against.
what if there's even more in reserve?

in other words, we're putting all of our eggs into one basket. and someone is
aggregating enough attack capacity to take out nearly the entire internet at
once. it doesn't help that everyone is voluntarily consolidating their
infrastructure onto a small handful of public cloud providers.

we are setting ourselves up for a massive internet outage.

------
NelsonMinar
I've been wondering if the UDP nature of a DNS server makes it harder to
protect. Particularly coupled with the amplification attacks that DNS makes
possible.

~~~
Animats
That's part of the problem. DNS servers should probably reject queries that
require long answers when they come in over UDP. If you want a zone transfer,
use TCP. That prevents amplification attacks.

------
qaq
if the attack is sufficiently distributed and scale is very large it can knock
out even much bigger targets. I think there have been attacks at over 600 Gbps
scale.

~~~
matheweis
Indeed, flashpoint (1) confirmed that the botnet attacking Dyn was the same
one that attacked Krebs (2), and Krebs has more details as well (3). The
previous attack on Krebs was seen to exceed 620Gbps.

1\. [https://www.flashpoint-intel.com/mirai-botnet-linked-dyn-
dns...](https://www.flashpoint-intel.com/mirai-botnet-linked-dyn-dns-ddos-
attacks/)

2\. [https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-
with...](https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with-record-
ddos/)

3\. [https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-
powe...](https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powered-
todays-massive-internet-outage/)

~~~
nickthemagicman
Wow. That means the same culprits are still out there with their botnet? And
it's still growing?

~~~
idlewords
The code for it has been released on Github, so there are now likely to be
many botnets.

~~~
Matt3o12_
I'm not too sure. I have heard that the attack also fixed the security
vulnerability (changing the default root password) after installing the back
door so other people cannot use it.

Although the source code is out there, those will not be able to control all
those devices.

~~~
nerdy
I'm not sure. Maybe that's the case for the passwords which can be changed via
the administrative app but I read many of these are in firmware and not able
to be disabled or changed:

“The issue with these particular devices is that a user cannot feasibly change
this password,” Flashpoint’s Zach Wikholm told KrebsOnSecurity. “The password
is hardcoded into the firmware, and the tools necessary to disable it are not
present.

\- [https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-
powe...](https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powered-
todays-massive-internet-outage/)

That's not to say it couldn't flash the devices but I don't recall seeing that
capability in the Mirai source and haven't read about it doing so.

------
bklyn11201
I've been waiting for some announcement around the Gbps of the DDOS similar to
this Cloudflare announcement:

    
    
      https://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack/
    

Does DYN routinely deal with very large DDOS which would past this attack in a
new category? Can someone who attends security conferences with DYN personnel
comment?

~~~
beachstartup
last night the consensus was 1.2 tbps.

~~~
bklyn11201
For a DNS-only service provider, it seems like 1.2Tbps could be 1000x normal
traffic. But Akamai claims 30Tbps+ is their routine traffic[1]. Some have
commented that this DDOS questions consolidation around cloud providers, but I
think it will cause consolidation among service providers. You can no longer
be a critical service provider if you don't have the capacity to absorb
attacks like this.

[http://www.csoonline.com/article/3123797/security/some-
thoug...](http://www.csoonline.com/article/3123797/security/some-thoughts-on-
the-krebs-situation-akamai-made-a-painful-business-call.html)

~~~
zzzcpan
I suppose Akamai wasn't ready to deal with the attack that size. They only
recently bought Prolexic, but things move slowly on their scale.

------
inetsee
Hackers have started to use insecure Internet of Things devices, especially
internet connected video cameras, to produce DDoS attacks larger than have
ever been seen before. The KrebsonSecurity website was hit by a DDoS that was
twice as large as the previous largest attack seen by Akemai, and there have
been larger attacks since.

The problem will continue, and may get even worse, since many of the insecure
internet attached video cameras are insecure because of passwords hard-coded
into the devices; they can't be easily made more secure.

------
mrcabada
I wonder if there's any way to tell apart real-users-requests from fake-users-
requests.

If I'm not wrong, it's only preventable by increasing the resources of the
server, doing anti-bots things like CAPTCHAS (not feasible for stand-alone IoT
devices) or detecting weird patterns (which can be masked really easily).

How will DDoS attack be preventable in the future? There will be so many
things and nano-thing connected to the internet that can act as "attackers".
Is getting harder and harder everyday.

~~~
mabbo
What software is the piece that is answering the question "is this a real user
or fake?". Because that's the piece that will fall over during a DDoS, as it's
doing per-request processing.

------
gagan2020
Just thinking, Is there any chinese production of IOT involved? might be
firmware involved?

~~~
sn41
That's what the following blog claims:

[https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-
powe...](https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powered-
todays-massive-internet-outage/)

I don't know any other independent researcher who confirms this.

~~~
gagan2020
That's interesting since they faced the attack so they could have data to
analyze that. Apart from firmware, chinese companies also pushing UC browser
and WeChat like anything.

------
zzzcpan
Presumably it would take a lot of cooperation with ISPs they are peering with,
which is not something easily done. Or a google-sized network.

------
t3ra
I would also like to know what exactly are "a lot of resources to prevent."?

------
akulbe
I wonder how much of this would be mitigated/avoided if folks would just
change to something other than the default credentials on IoT devices?

Is it that simple? or am I missing something?

~~~
akulbe
A downvote for a legitimate question. NICE.

------
meira
Probably they got beaten because of orders of magnitude. They were prepared,
but not for cyber nuclear war.

