
Update Regarding DDoS Event Against Dyn Managed DNS on Oct 21 - sajal83
https://www.dynstatus.com/incidents/5r9mppc1kb77
======
TekMol
I am sure the DDoS problem is something that the free market will sort out.
The individual players will make it costly for the other players to send
problems their way. I expect a chain of "charge the next node for resource
usage" to evolve.

If this chain will go all the way to the end user, I don't know. If it will,
then end users will probably start using routers that feature restrictions /
monitoring / control of outbound traffic. So you know your coffee machine will
not be able to use up too much resources. Just like you have a fuse for the
power line.

~~~
clvx
I agree with you. IoT devices inside a SOHO should communicate externally
through a proxy gateway device. IoT devices should only have communications in
a p2p network in a LAN, and have strong restrictions or none access to WAN.
Any type of updates should be given from a proxy device having proper
hardening than a normal IoT device.

~~~
Pxtl
The router could provide password-protected web proxy to access the LAN IOT
webserver. Then you've reduced the attack surface to the router.

It almost seems like we need some protocol extensions:

1) Standard auth protocol (not just web-based) for the router to protect the
local computers. Some kind of user-and-software-friendly firewall. This could
even extend to game servers and whatnot - what if the "shared password" for
connecting to a hosted game server had the shared password implemented at
router protocol level?

2) DHCP registration on a network should require a _name_ , one that the user
was prompted to provide at some point. No more identifying devices on you
router by IP or MAC. You already need to provide a name for SMB or DNS, just
finish the job and name all DHCP clients. Possibly this should work with DNS
in some way.

This way user-friendly logging information can be presented to the user.
Without that, routers don't have the critical information needed to tell the
user which device is screwing up.

Edit: Google tells me this is already a thing... Sadly, good conformance on
providing meaningful DHCP client names won't happen unless the FCC et al start
testing IP-enabled devices for it.

------
philiphodgen
The free market will solve this problem with an existing and efficient tool.

Tort law.

A few wins in court will do the trick. Here is how.

A victim of a DDoS attack sues manufacturers, distributors, and retailers of
that product for selling a defective product--the IoT device used in the IoT
attack.

As soon as there is a win, the product disappears. Distributors and retailers
must now price in the externality of the risk created by the product. It is
far easier to take the product off the shelf. Therefore the manufacturer must
either make a better product or die.

We don't need government regulation. We just need time. The legal system will
do what it is designed to do: assign economic consequences to the right
parties.

~~~
natch
You say this as if it is just one device used in these attacks. And as if it
will be easy to stroll over to the owner's location and determine the supply
chain of that one device. And as if it will be easy to collect from the
Chinese manufacturer who probably folded last week and reopened under a
different name for completely different reasons.

~~~
philiphodgen
This is why tort lawyers sue distributors and retailers in the USA. They are
here and they have insurance.

Once there is a court decision, a rational seller (oh, Amazon hypothetically)
will understand that selling fly-by-night small manufacturer items is fraught
with peril. The offending items disappear from the marketplace.

~~~
AnthonyMouse
More likely the marketplace itself moves overseas. People use Amazon because
they sell the stuff they want. If they stop selling it, the buyers go
somewhere else. Amazon is a website. It could as easily be a website hosted
out of China.

Also, it sounds like you don't mind if the effect of your proposal is to
destroy things like Etsy. And eBay.

~~~
philiphodgen
The legal system is less brain-dead than you imagine. :-) This is not the
first time in history that a plague of imported items causes a problem.

Etsy and eBay will survive. Ford survived the Exploding Pinto. Firestone
survived its tire debacle. And we are all the safer for it.

~~~
AnthonyMouse
> Etsy and eBay will survive. Ford survived the Exploding Pinto. Firestone
> survived its tire debacle. And we are all the safer for it.

Ford and Firestone aren't retailers, they're manufacturers, and they were held
responsible for their own mistakes, not the mistakes of third parties.

There is no reasonable way for online retailers to evaluate product safety of
millions of small batch third party products. Either they sell them without
evaluating them or they don't sell them. Imposing liability on them is exactly
how you get them to not sell them, but then we can't have Etsy or eBay.

> This is not the first time in history that a plague of imported items causes
> a problem.

It seems like the first time the problem has happened in this particular way.
Historically importing was a large-scale operation done in bulk with
homogeneous products, so the importer knew what they were doing and had deep
pockets. Today you can cost-effectively get a 99 cent piece of electronics
shipped directly from a one-person shop in China. Either you shut down the
entire _idea_ of that, and then things are going to cost a lot more than they
do now, or we need a different approach.

------
mjec
Pardon my ignorance, but why don't companies run their own nameservers?

I get why you don't want to run email - it's highly reputation driven. But as
far as I can tell, running nameservers is no harder than running webservers or
DB servers. HA is potentially even easier, because the system was designed
that way from day zero.

I'm not suggesting I'd run one for my personal website, but twitter and github
are already managing distributed networks for this. What are the services Dyn
and others provide that are so invaluable?

~~~
neom
Getting good, consistent, well routed, fast and secure DNS Is harder than
you'd think. Dyn typically sing speed as the main selling point for their DNS
product, they do this through a large distribution of domain name servers
geographically and anycast. Many hosts (like say, DigitalOcean) run their own
DNS but use something like CloudFlare Virtual DNS on top. Personally I was
surprised so many large sites trusted Dyn, Route 53 is a more robust product
for production and scale. In the past, I've seen hosting providers switch to
Dyn, give them load, cripple them, and have to scramble to revert away. I'm
not at all surprised his happened, even given the uptick in botnet traffic
globally.

~~~
scurvy
Route 53 aint all that. We approached them about handling our customer's
domains, and they said no way. They didn't have the capacity. Granted, this
was 2 years ago, but Dyn has a much better reputation (still) than Route 53.

------
ComodoHacker
This attack looks like another probing into critical internet infrastructure
Bruce Schneier had talked about.

Who's next?

~~~
kukx
You suggest it's probing. Then how would the full scale attack look like?
Also, how does probing help the perpetrators, doesn't it lead to better
defenses in the future?

~~~
Taek
These attacks seem to be getting sophisticated faster than defenses are being
thrown up. It might lead to better defenses in 5 or 10 years, but the Internet
at large is built out of infrastructure is difficult and slow to upgrade. And
is everyone scrambling for a solution? Not really... Most are taking the
mentality "well sucks to by Dyn but it's not affecting me so I don't need to
respond".

Especially because the vulnerabilities being exploited right now seem to by
systemic. It's not like patching a zero day. "Uh... we've got 45,000,000 un-
upgradeable IOT devices from dozens of different manufacturers executing a DoS
attack". You can't fix that the same way you fix a privilege escalation bug in
the linux kernel.

If the attackers end up finding the Internet's equivalent to a jugular, there
might not be much we can do about it. BGP isn't going to be replaced in the
next 10 years, it's here whether we like it or not. DNS is also not going to
be replaced in the next 10 years. And neither are the major centralized
internet exchange points. Any vulnerabilities an attacker can find related to
the fundamental design of those things are going to remain vulnerabilities for
many years. If the attacker can get good at exploiting them, we are in
trouble.

------
peter303
The movie Fantasia Sorcerers Apprentice was right! All the the household
objects will rise and overthrow their masters!

------
nik736
Is there any info on what volume (how much bandwidth) the DDoS attacks were?

~~~
vonklaus
Curious as well. I didnt see any in the status here nor the original.

------
linsomniac
DNS seems ripe for revolution!

Yesterday it felt obvious that we are treating DNS data as too ephemeral. I am
not intimately familiar with the implementations in BIND and others, but it
seems like when we hit the TTL we just throw away the data. Usually, that
works fine. But, in the case of the origin servers not responding, yesterday I
was wishing that it would just give me back the stale data rather than giving
me nothing.

I'll admit the impact on me was somewhat limited. Around 10am Mountain I was
trying to install some Atom.io modules and couldn't reach that site or a
github download URL. I had some success with using 4.2.2.2 (8.8.8.8 was not
answering the names).

Using a stale cached result probably wouldn't have helped for atom.io though,
I hadn't been there in a while and this was querying my own local name
servers. Do I want my name servers keeping weeks old stale data around?
Probably not in RAM, but saving old names to disk sounds like it would require
a lot of IOPS for a big provider. But I do know I'd visited the sites I was
trying to hit within the last few weeks, since I couldn't reach the
authoritative servers it'd be nice to have tried the last IP I had for them.

Of course, I ran into this about an hour after I rebooted my entire
dev/staging infrastructure to fix the Linux kernel privilege escalation issue,
so my caches were cold.

Sure would be nice if my server could "ask around" if it can't talk to an
authoritative server. "Hey Google, hey Comcast, hey Level-3, do you know this
name?" That's effectively what I did by changing my resolv.conf. But if you
start asking around too widely, you ideally probably want to have some
signature to verify the data you are getting.

Seems like the new norm might be listing authoritative DNS servers from
multiple big providers (Dyn and Route53) and having to keep them in sync? Then
you lose some of the advance features...

Funny aside: One of the sites I run uses Distil in front of it to protect
against content scrapers. Months ago I was working with their support about
getting a health checker set up in Route53 to test the full paper path through
distil and fail over to our backup site if anything on the primary paper path
didn't work. Distil assured me that their services were so resilient that we
shouldn't worry about them being down. Guess what the only part of our
infrastructure was that was impacted by this? :-)

------
honksillet
So is it retaliation for cutting Assange's internet, retaliation for
threatening Russia with a cyber attack or both?

~~~
leephillips
[https://twitter.com/wikileaks/status/789574436219449345](https://twitter.com/wikileaks/status/789574436219449345)

------
zer0gravity
So I'm wondering, what are the implications of this ?

Someone is controlling a powerful enough botnet to do this.

How powerfull is it really ? What else can it do ? Was this just a message or
a test ? Or both ?

What would happen if they would point it to google's nameservers ?

What should we expect next ?

~~~
neom
It's complicated because the internet is a complicated(and I'm no expert)-
but, it's not good, but it's not insurmountable. The big issues today are:
lots of network attached compute, lots of types of traffic, highly highly
distributed network. There are numerous ways of mitigating DDoS, today a lot
of it is via BGP route announcement[1] - although we've seen folks using BGP
in questionable ways to mitigate DDoS recently[2]. As more and more of the
internet becomes software defined (like SD WAN sees large global rollout)[3]
more and more granular but non-disruptive control will be enabled. To answer
your question about google, their DNS is probably resilient enough to deal
with a pretty huge attack but who knows really -- In my dream world we get
some good neural networks built and deployed to the edge watching for unusual
traffic patterns and disrupt them. It used to be biggest pipes win, I don't
know that will continue to exclusively be the case.

[1]
[http://www.enterprisenetworkingplanet.com/netsp/article.php/...](http://www.enterprisenetworkingplanet.com/netsp/article.php/3615896/Networking-101-Understanding-
BGP-Routing.htm) [2]
[https://www.youtube.com/watch?v=LFJzu0AFDpU](https://www.youtube.com/watch?v=LFJzu0AFDpU)
(Dyn Engineer gave the talk) [3 ][http://www.rcrwireless.com/20160408/telecom-
software/using-s...](http://www.rcrwireless.com/20160408/telecom-
software/using-sd-wan-combat-ddos-aggressive-attacks-tag2)

~~~
zer0gravity
> we get some good neural networks built and deployed to the edge watching for
> unusual traffic patterns and disrupt them

I was thinking at this as well. It's probably under active development right
now, if not already live for some.

------
elcct
Did they mitigate the attack or attack has stopped? I am asking, because
update is lacking the details about said mitigation.

------
DyslexicAtheist
you can watch BGP routes changing (as we speak) here
[https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16](https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16)

------
djhworld
I think it's about time they updated
[http://dyn.com/ddos/](http://dyn.com/ddos/)

It was an unprecedented attack, sure, but I'm not sure how their sales guys
are going to spin this.

------
amq
Did Dyn mitigate the attack, or did it stop by itself?

------
nucotano
I'm so looking forward at IPv6, the death of NAT, and billions of IoT devices
with all ports exposed to the world :-)

~~~
acomjean
Arent most IOT devices behind a router and thus unexposed directly to the
internet (excepting routers)?

This part of these attacks confuses me.

~~~
sajal83
Compromised routers can be used to compromise devices behind it. Also many
devices (like IP cameras) usually have port forwarding to allow the users to
access it from outside.

