
Dyn Analysis Summary of Friday October 21 Attack - LVB
http://hub.dyn.com/dyn-blog/dyn-analysis-summary-of-friday-october-21-attack
======
edutechnion
Changes made by large companies that relied 100% on DYN:

* us-east-1.amazonaws.com: split between internal, UltraDNS, DYN

* spotify.com: all internal nameservers now

* reddit.com: all Route53 now

* github.com: all Route53 now

* netflix.com: all Route53 now

* paypal.com: split between UltraDNS and DYN

No changes made:

* twitter.com: 100% with DYN

~~~
blakesterz
* spotify.com: all internal nameservers now

I don't know much about running nameservers but moving to all internally
hosted seems like an odd choice to me, can anyone explain whey that's a good
move?

~~~
user5994461
DNS is a rather simple service that was always meant to be run internally.

The question you should ask is why did these companies used an external DNS in
the first place?

~~~
i__believe
So they don't waste cycles on something not part of their core business or
competency? Pretty standard reasons to pay someone to solve a problem. I think
what this really showed is Dyn was not as competent in mitigating as what
people thought.

~~~
scarmig
The implication of incompetence isn't really fair here. This attack was fairly
unique, in that it had a sufficient quantity to be a quality of its own. It's
unclear whether any DNS provider could have survived it, except by luck of not
being chosen as the target.

~~~
trhway
>This attack was fairly unique, in that it had a sufficient quantity to be a
quality of its own.

isn't that basically the definition of DoS?

------
bradleyjg
One thing I don't understand about this attack:

Virtually everyone is behind NAT these days, often multiple layers of NAT. So
how does the botnet manage to telnet or ssh into these set top boxes or
lightbulbs or whatever? When I _want_ to SSH into my home computer I have to
go through elaborate maneuvers to get it to work.

~~~
jackweirdy
I believe UPNP is used commonly by many of the IoT devices that end up in
botnets.

~~~
bradleyjg
As I understand it UPNP allows a device to negotiate a port map with a NAT
box. But the developer of the IoT device would have to specifically want to
map port 22 or 23. It's not something that would happen automatically.

So are you saying that these IoT device makers not only hard-coded root
usernames and passwords into their devices, but deliberately set up UPnP
mappings to those ports?

That looks malicious rather than negligent. Am I missing something?

~~~
SteveNuts
I've heard a lot of routers accidentally or negligently allow upnp on the WAN
interface

~~~
bradleyjg
That's pretty horrifying but if true it answers my question about how this can
affect so many devices. It sounds like whether because they were the directly
hacked devices or because of this UPnP flaw, home routers are the main villain
here. Not smart lightbulbs and webcams which the news media seemed to want to
blame -- though the latter may well have been secondary villains.

------
youeeeeeediot
Dyn aside, xTbps DDOS is the new norm.

Expect more of this in the near future, single-source infrastructure is
becoming a huge liability (not that it wasn't before). I wonder what impact on
SLAs it will have when cloud services providers are taken down - will they
honor their SLAs or inject DDOS clauses into them to shield themselves. You
won't see many standing up to multi-Tbps attacks, at least for the moment.

~~~
ben_jones
Doesn't that also provide a huge incentive for cloud offerings to sell DDOS-
resistance products and services? Isn't that a huge market already with
gigantic margins?

~~~
SysArchitect
OVH throws their DDoS mitigation system in for free when you host with them.

~~~
raarts
How does their system work and how effective is it?

~~~
Thaxll
It's built on FPGA and it's very effective.

[https://www.ovh.com/ca/en/anti-ddos/](https://www.ovh.com/ca/en/anti-ddos/)

~~~
dogma1138
Game servers that are hosted on OVH get DDOSed daily and they drop like flies.

Myself and a lot of friends had servers killed both when renting VPS/Dedicated
server or a dedicated "Game Server".

And all and all with considerably smaller botnets like the ones you rent for a
few $ per hour.

If you are running a public server you learn quite quickly that if you
permaban a cheater or just some annoying kid you should expect to be DDoSed
these days.

------
snowy
Can any one explain why they keep referring to this as a complex attack? From
the article it seems to be a simple volumetric attack. They mention that it
uses UDP port and TCP port 53, nothing complex about that...

Am I missing something here. It wasn't an L7 attack (or was it?) Why keep
referring to it as complex?

~~~
chjohasbrouck
I think the market's expectation is that a DNS provider is prepared for a DDoS
of any size, but not necessarily any level of complexity, so that's a lot of
incentive to talk up the complexity of the attack.

What's described in this incident report is totally within the capabilities of
a single individual with public knowledge, though. If they could have proven
otherwise, they probably would have (unless that somehow conflicted with their
criminal investigation).

------
bluedonuts
I wonder what 1.2Tb would do if pointed at a ELB or an AZ in AWS. Someone must
have tried at some point but i've never heard of any widespread outages caused
by DDOS on AWS. Is it just that bandwidth available to an AWS datacenter is
that much bigger than what dyn have?

~~~
bklyn11201
It's safe to assume that 1.2Tbps isn't a big deal for
Google/MSFT/AWS/Cloudflare/Akamai/Yahoo/Verizon/etc.

DNS is normally a low-bandwidth protocol so if you only provide DNS services,
needing to purchase 1000x your normal bandwidth to handle these bursts would
be miserable. If a DNS provider were also providing video services
(Vimeo/Twitch/etc), then a 1.2Tbps increase in traffic could be easily
absorbed.

~~~
rasz_pl
You would think that, yet Akamai dropped Krebs over half that.

------
eternalban
I've read a few comments elsewhere that the attack on 21st may have been an
element of a state actor MITM attack. What's the expert opinion?

[p.s.] The question is precisely this: is it possible that a DDoS attack on
DNS can be used to affect/mask a MITM attack.

~~~
takeda
"State actor" today is an abused term, because it helps the victim look less
bad, the government will back it up, because it paves way for new regulations
and there's no way to prove one way or the other. Especially when it comes to
DDoS.

------
papabearshoe
This is insane how big these things are now: "There have been some reports of
a magnitude in the 1.2 Tbps range; at this time we are unable to verify that
claim."

~~~
e12e
They mention 100k participating devices/IPs - that would mean an average of 12
Mbps upload. Sounds high, but plausible? [ed: I actually think it's kind of
sad that most users aren't on symmetric gigabit links yet... But in _this_
context I guess it's a blessing of sorts... ]

~~~
meowface
We already know Mirai has been able to reach over 1 Tbps, and we know Mirai
was at least one of the cannons hitting Dyn. So 1.2 Tbps is definitely
plausible. Mirai has decreased in size to a degree due to more public
awareness, but it's still massive.

~~~
ryanlol
Way more than 100k bots lol.

------
capkutay
Can anyone recommend a good place to learn more about this field of network
security? I'd really appreciate!

~~~
argio
maybe this is something for you ->
[https://cybersecuritybase.github.io/introduction/](https://cybersecuritybase.github.io/introduction/)

------
rasz_pl
How exactly was the attack mitigated when in the same sentence they confess it
STOPPED _on its own_.

~~~
dsp1234
Mitigated means "make less severe, serious, or painful.". In other words, they
took steps that stopped the attack from effecting legitimate traffic. Once the
attacker sees that the attack is no longer effective, the attack stops. After
all, it's a denial of service attack. If the service is no longer being
denied, then what is the point of continuing the attack.

~~~
hueving
To cost the company significant amounts of money. Bandwidth in those volumes
isn't cheap.

~~~
subie
Wouldn't it be costing the attacker some amount of money running the C&C
server?

I'd think at some point if the attacker isn't getting the same results from
the attack he'll wait for them to scale back then strike again.

Is my thinking right here?

~~~
alphex
No. each individual attacker is actually an infected "bot" of somekind. In
this case, Internet of Things devices all over the world. Your "smart TV" or
"Smart refridgerator" all contributed to this, over YOUR bandwdith.

The average broadband user has at the least, 10Mbps/UP. If your smart
appliances all started sending at least 10mbps up... you only need a million
smart TV's to start causing damage.

I'm guessing there's 100M Smart TV's in the USA? Each sending 10Mb/s up?
There's 1GB/s of traffic. Multiply this by the next 50 nations who have smart
TVs and bandwidth to spare...

Then make it multiuple devices in a home. Then make it every smart device on
the planet, using SOME bandwidth... it gets painful fast, and its free for the
attackers, since the poor sap with a hacked smart TV is doing the work.

~~~
subie
The attacker is still spending resources giving out instructions to the bot
army.

Unless they are using a setup like @tomschlick mentioned or some P2P thing.

~~~
pixl97
Sending out a message to 1 million compromised devices takes almost nothing.
Also, if you've compromised 1 million devices, chances are you also
compromised a linux server on a fast host for command and control.

------
1_2__3
Not smart of them to have a product guy write this. Even if he had it would
have been better delivered (if shadow written) by someone in engineering.
They're customers are not necessarily people who want to hear technical claims
from Product.

~~~
rasz_pl
There is a reason behind this, engineering had nothing positive to say, even
this announcement admits attack stopped when it stopped, there wasnt anything
technical they did that fixed the situation, unless by mitigation they meant
paying ransom and that was the cause attack stopped on its own.

------
R_haterade
Any word anywhere on motive? Or are we still left to speculate?

------
mnx
OT: Is my eyesight going away, or is the font unreadably light-coloured?

~~~
cbg0
No, the font weight and color are horrible.

~~~
ben_jones
My company only just noticed our font renders terribly for one of our newer
web apps - all the developers had Macs and didn't notice an egregious
difference when rendered on lower quality displays. We changed it immediately.
I feel empathy for this particular development oversight among small teams.

------
justinlardinois
> We began to see elevated bandwidth against our Managed DNS platform in the
> Asia Pacific, South America, Eastern Europe, and US-West regions that
> presented in a way typically associated with a DDoS attack. As we initiated
> our incident response protocols, the attack vector abruptly changed, honing
> in on our points of presence in the US-East region

Interesting. Much of the reporting on the day of suggested that the attack was
felt exclusively in the United States, but this says otherwise.

~~~
dx034
Sounds like someone found a way to target one server with all devices
worldwide. Shouldn't that be impossible with Anycast? Or did they reveal an ip
address that was just referring to US-EAST?

