Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: How did Dyn fail to fend off DDOS?
62 points by ruler88 on Oct 22, 2016 | hide | past | favorite | 69 comments
I'd imagine that DDoS attacks is something that DYN and other DNS providers would spend a lot of resources to prevent. Was there something specific about this DDoS attack that DYN was unprepared for? Or is there some reason that distributed natural of DNS makes it hard to prevent DDoS? Anyone know of any steps that DNS guys are taking to prevent another DDoS?



I would like to remind those that think all is lost with this:

A serious conversation with vendors about default passwords and backdoors post this incident will help prevent recurrence. This has forced this talk and we are better for it.

There was a time when your windows box would get popped from being online for more than 4 minutes. We recovered from this. Conficker in 2008. Blaster in 2003. It was a 'BIG BOTNETS OH NO', but we cleaned up, recovered, hardened. Microsoft went from being botnet enabler to an active force in dismantling bots and crime rings. It sucks, and some of us have a bad day, but we recover ever stronger.

XiongMai Technologies may well find themselves in some international hot water over this incident, and I think they deserve it. They sold a faulty product that caused billions of dollars in lost revenue to some very large internet properties for a day in October 2016. I would encourage vendors look at these incidents from last decade and how these were turning points for upping their security game. I would encourage its victims to investigate legal recourse.

Specifically the current vulnerable nodes of Mirai, i am sure these will be removed from the internet pretty soon. One only gets to fire something like this a few times before the feds are on the door.

Your regularly scheduled program will commence shortly.


Security is a process. We might be able to browbeat (insert clueless-about-security manufacturer here) into making an investment in secure firmware. Maybe they'll even get it right. But our experience is that additional security holes are always found, even in software written by knowledgable and motivated teams.

These devices need to have an update mechanism. The manufacturer needs to have an ongoing security effort, across their whole device line (probably a significant investment in development resources and process -- consider that right now, the firmware for a device is probably coming off of a firmware dev's laptop; I've seen this happen at a big company). And devices will have to be sunset, to control the ongoing cost. Consumers will love that.

I don't think we're doomed, exactly, but it's probably always going to be a problem. And there's probably a market for embedded firmware application layers that don't suck, for starters.


I would encourage its victims to investigate legal recourse.

It's all well and good saying that, and yes, if manufacturers are repeatedly/grossly negligent then maybe they should pay compensation and/or punitive financial penalties. However, unless you know something the rest of us don't about how to guarantee Internet-connected devices are perfectly secure, that sort of financial pressure can't be the whole solution, or even the main part of the solution. Ultimately, it may just mean that smaller players can't afford to risk participating in the industry any more, and no-one will be better off if reduced competition is the main result of this. We must be able to handle this more constructively than just demanding perfection and punishing those who inevitably fail to deliver it.


if you draw the line between attacks of the past through this one you see that the scale of DDoS attacks continues to get worse. It's all well and good to say that the enablers of the past learn and improve their products. The problem is continually the enablers of the future.

It's been said before but I will repeat it here; manufacturers have no reason to expend any resources on security until they are held liable for the damage they facilitate. We must make selling insecure devices a liability just like selling unsafe devices is in meatspace.


Just eliminate default passwords completely. The first person that opens the box, or applies a license key, sets the password and it must be strong.


A better solution is for each physical instance of a device to have a default password that is strong and unique (and encoded in the firmware, such that a factory reset of the device doesn't make it default to a non-unique PW).

There are a few other ways to handle the problem of securing endpoint devices. For example, for devices that are intended to use a local aggregator, gateway, or proxy of some sort you can get around the issue (and improve the UX) by avoiding passwords entirely, and requiring that the device instead be paired with a base station through a physical action the user performs (pressing a button on both, knocking them together, etc.) instead.


Yes. But keep in mind that these where in part maintenance accounts not visible to users. The areas that also need to be kept in mind are manufacturing, maintenance, repair and upgrades.


It's time to apply some serious pain to the junk IoT manufacturers, retailers, distributors, and importers. A nice big billion-dollar lawsuit against Amazon for gross negligence would be a good way to start. US consumer law allows suing everybody in the supply chain. (They can then sue each other and try to sort out who pays, but that's not the victim's problem.)

We also need some big recalls. If Homeland Security tells the Consumer Product Safety Commission this is a national safety issue, the CPSC can order a recall. Something like this worked with those exploding "hoverboards". CPSC ordered recalls, Amazon took the junk back, and Amazon refused to pay manufactures in Shentzen. The manufacturers were furious, but hoverboards with crap batteries disappeared from the market very fast.


I think the more realistic solution is that a vigilante group of hackers continuously scan and take over vulnerable IOT boxes with the intention of bricking and/or disabling their network access would be the most feasible.


The problem with this idea is that it is illegal, and federal agents are much better at tracking people down on the Internet than they were even 5 years ago. So while I think a lot of us would cheer the vigilantes on, they would be taking a serious personal risk.


If they were so good at that then this wouldn't be a problem in the first place. The hackers can be in the same country as the DDOSers.


But then wouldn't they be better off doing DDOSing for hire? Just a thought


There is shodan.io which is pretty good.

The vigilante hackers is for for comic books IMHO. I would trust a 3-letter gov org. Maybe the NSA would be a lot more useful if instead of breaking the internet, trying to fix it.


> US consumer law allows suing everybody in the supply chain

IIRC, US consumer law requires the consumer to be the victim. (IAAL/NY, but not practicing) This restriction is called privity – the exceptions to privity are narrow, and no exception comes to mind here.

In this case the primary victims, the online services, are third parties, with any consumer recourse blocked by privity.

These third parties arguably have a couple options, though. The first and perhaps most theoretically interesting is the "class defence", the procedural complement of a "class action", where a few people (the third party online services) can sue multitudes (owner-operators responsible for malicious devices on the Internet) in a single process. Were such a case brought forward, these consumers could sue the manufacturers for indemnity. While as a litigator this makes the most theoretical sense, and this procedure exists in at least one jurisdiction I know of, I have never seen it tested.

Arguably a better option would be for the third parties to sue the manufacturers for negligence, based on the obligation that the manufacturers have to the public.

Any litigation is fraught with uncertainty though, not least of which is having a member of the judicial bench who is capable of properly evaluating the facts (which is not to say they are not out there, but they remain rare).

Like most externalized costs, the recourses of affected individuals are slim and ineffective.

> If Homeland Security tells the Consumer Product Safety Commission this is a national safety issue, the CPSC can order a recall

Proper regulation is a better choice, IMHO, though I don't know what the best process might be.


>"A nice big billion-dollar lawsuit against Amazon for gross negligence would be a good way to start."

Wait why is Amazon responsible? Why should they be sued?


They sell lots of the garbage, and have a track record of ignoring supply chain issues.


It sounds pretty harsh but I agree. Companies aren't going to take this stuff seriously until it really starts to hurt the bottom line. Now that politicians are starting to wake up to "the cyber" there may finally be the public will to take security seriously.


Feels wrong to blame this on Amazon of all parties.


That's a game of whack a mole, and even if you whack them down, the devices are already out there and are going to stay online for years.

The only thing that will make a dent at the problem quickly, is wholesale filtering of all Internet traffic by all network providers originating from the IP addresses identified for being part of these botnets.


Unfortunately, unless either you can get that sort of result across a substantial part of the developed world or it happens that most of the insecure devices used here were sold to US-based customers, the US legal system alone isn't necessarily going to help much.


You can't litigate your way to a fix. Lots of those devices will be in parts of the world where your lawsuit can't reach. Asia, Eastern Europe, Africa. We must engineer better solutions.


I think the answer is surprisingly simple: The attack was just huge.

The unfortunate truth is that with the Internet of Things the amount of devices that can easily be taken over has grown so fast that we see DDoS attacks of unprecedented size. Even more unfortunate is that there is no sign whatsoever that this is going down again.


>The unfortunate truth is that with the Internet of Things the amount of devices that can easily be taken over has grown so fast that we see DDoS attacks of unprecedented size.

Not quite, the "IoT" botnets are particularly small in the great scheme of things. Google "conficker" for example.

Edit: Interesting how this is getting downvoted so much. Conficker had up to 15 million nodes, far bigger than any "IoT" net (when did home routers become IoT anyway?). It's far easier to build such huge windows nets because you get millions of insecure computers with relatively standard hardware and software, not so much with "IoT".

In the past decently sized botnets simply weren't used to send DDoS attacks as much, that's all that's changed.


Does anybody have solid recommendations for secure IoT devices? Initial searches lead me to believe that they are non-existent.


Where's the pain-free device with open source, easily upgradeable firmware, that puts all of our IoT devices in their own private network but lets us tunnel through to them? It needs to be easy enough that our (grand)parents could pick one up on Amazon, Best Buy, or Home Depot and plug in and go...


If these are connected by cellular, they are given a private network that does not connect to the public internet and are in-accessible from the public internet unless the app provider explicitly chooses to do so


Most better home routers can restrict devices connecting to the internet (either through the Firewall or more comfortably configured through family filters) and offer VPNs to the internal network?


It's called PLAN (short for physical LAN). It doesn't need a managed switch, like VLAN, because you just use one switch for each network. Careful: Don't connect them.


Change the default admin password.

The original Mirai program tried a little over 60 passwords and it would just brute force into an IoT device.[1]

From what I read, it seems that one specific manufacturer in China is the owner of a lot of devices used in the Mirai botnet attacks.[2]

1: https://github.com/jgamblin/Mirai-Source-Code/blob/master/mi... 2: (I cannot find the link, but it was an article from yesterday)

EDIT:

Found this when googling the strange '7ujMko0admin' password in Mirai: http://www.cam-it.org/index.php?topic=9396.0 So it looks like the Chinese manufacturer that they target is Dahua.


Brian Krebs pegged a company called XiongMai: https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe...


That article also mentions the credentials are in the firmware.


Apple's HomeKit supports Bluetooth-only devices. Seems like a good design choice right about now.


Until Web Bluetooth opens those devices to exploitation from internet websites. It would be best if Bluetooth remained isolated from web browsers, but the powers that be want websites to be able to talk to them.


Well, a good initial step is usually changing the default password.


A good initial step is not to have a default password. There was a time when all routers came with a default password and people were told to change it. They didn't. Now most new routers come with a randomly generated unique password printed on a sticker under the router. IoT devices should follow the same practice.


The real question here is whether there was anything they could realistically have done to prevent it at all.

In order to defend against a DDoS attack, you really only have two options. One is to have sufficient capacity to cope with the extra load without undermining your normal service. The other is to reduce the amount of extra load you have to handle, by identifying and blocking the hostile traffic at some point before your main system deals with it fully.

In this case, the scale of the attack was huge thanks to all the woefully insecure IoT devices out there. But worse, from the initial reports it appears that the requests being sent were effectively indistinguishable from valid DNS requests: they came from diverse sources, and asked DynDNS to do exactly what it's normally supposed to do, just for random subdomains that don't actually exist. Unless there is some pattern in those requests that allows for identification of the hostile incoming traffic so it can be dropped early, there's probably very little DynDNS could have done here. And of course the attack is particularly effective because by taking out infrastructure rather than attacking a specific site, it brings down large numbers of high profile sites all at once.

It is disturbing, but apparently the reality we face, that there are now so many hopelessly insecure devices on the public Internet that this is possible. The best long term strategy for dealing with it seems to be trying to improve the standards of Internet-connected devices and reduce the number of highly vulnerable devices with access to the Internet, but this was always going to be difficult with IoT products aimed at the general public. I suspect some sort of remediation/recall scheme for manufacturers/vendors and some sort of throttling of users' Internet connections to force them to respond to security recall/update notices may be necessary if this kind of attack starts to become a pattern.


I think this is a plausible theory of the attack - (first seen in from npr report on incident):

NANOG 68 BackConnects Suspicious BGP Hijacks is shown 4ish days ago. Last talk of the night, discusses BGP hijacking shenanigans and krebs; touches on MO of possible attacker. Speaker is Director at Dyn. Attack in retaliation.

So far the targets have been organisations that have responded to or made allegations of corrupt DDoS business.

Please don't buy into all this cyberwar bullshit, this may just be a well resourced (its really not that hard to pop boxes with default passwords.....) attacker doing criminal response to commentary.


This is likely, Backconnect hosted Mirai in the past right before attacks on Krebs. (however not during them.)

There's also no small amounts of publicly available evidence that Backconnect used insider information provided by their CEO (ex Staminus employee) to compromise Staminus network earlier this year by hijacking a management range of theirs.


i think there is a larger strategy at play. this is pure speculation and anecdote.

recently there has been an aggressive uptick of dns ddos attacks against smaller companies/service providers that run their own dns infrastructure. this includes small/regional internet service providers and individual sites/hosts that still run their own servers.

in almost all of these cases that i'm aware of, the smaller companies immediately outsourced their dns services to a larger company, one that ostensibly is able to either absorb, scrub, or otherwise defend against these types of attacks.

extrapolating to a global scale, what's happening is a forced consolidation of dns infrastructure into a handful of large players. even in the case of having redundant providers, it's usually two very large providers. and as we just saw today, a terabit-level attack is not something we can readily defend against. what if there's even more in reserve?

in other words, we're putting all of our eggs into one basket. and someone is aggregating enough attack capacity to take out nearly the entire internet at once. it doesn't help that everyone is voluntarily consolidating their infrastructure onto a small handful of public cloud providers.

we are setting ourselves up for a massive internet outage.


I've been wondering if the UDP nature of a DNS server makes it harder to protect. Particularly coupled with the amplification attacks that DNS makes possible.


That's part of the problem. DNS servers should probably reject queries that require long answers when they come in over UDP. If you want a zone transfer, use TCP. That prevents amplification attacks.


Yes, it does. But no, it does not seem to make any difference this one time.

In a DNS based amplification attack, you use several DNS servers to take down some other unrelated service, this time it's just a lot of devices in a botnet attacking the DNS servers directly.


if the attack is sufficiently distributed and scale is very large it can knock out even much bigger targets. I think there have been attacks at over 600 Gbps scale.


Indeed, flashpoint (1) confirmed that the botnet attacking Dyn was the same one that attacked Krebs (2), and Krebs has more details as well (3). The previous attack on Krebs was seen to exceed 620Gbps.

1. https://www.flashpoint-intel.com/mirai-botnet-linked-dyn-dns...

2. https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with...

3. https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe...


> While Flashpoint has confirmed that Mirai botnets were used in the October 21, 2016 attack against Dyn, they were separate and distinct botnets from those used to execute the DDoS attacks against “Krebs on Security” and OVH.

So not quite.

> Dale Drew, chief security officer at Level 3, an internet service provider, found evidence that roughly 10 percent of all devices co-opted by Mirai were being used to attack Dyn’s servers. Just one week ago, Level 3 found that 493,000 devices had been infected with Mirai malware, nearly double the number infected last month.

http://www.nytimes.com/2016/10/22/business/internet-problems...

If they aren't significantly underestimating the number of devices participating in this attack, it paints an ugly picture of things to come. My understanding is these botnets are almost impossible to eradicate due to how fast/easy it is to re-compromise the devices, so traditional methods of taking out C2s do almost nothing. Bonus - Mirai source code is freely and easily available for skids to use now, so there's no single threat actor for attribution/retaliation/arrest/etc.


Wow. That means the same culprits are still out there with their botnet? And it's still growing?


The code for it has been released on Github, so there are now likely to be many botnets.


I'm not too sure. I have heard that the attack also fixed the security vulnerability (changing the default root password) after installing the back door so other people cannot use it.

Although the source code is out there, those will not be able to control all those devices.


I'm not sure. Maybe that's the case for the passwords which can be changed via the administrative app but I read many of these are in firmware and not able to be disabled or changed:

“The issue with these particular devices is that a user cannot feasibly change this password,” Flashpoint’s Zach Wikholm told KrebsOnSecurity. “The password is hardcoded into the firmware, and the tools necessary to disable it are not present.

- https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe...

That's not to say it couldn't flash the devices but I don't recall seeing that capability in the Mirai source and haven't read about it doing so.


Lol. Thats insane!


OVH DDoS late last month was over 1.5Tbps: https://twitter.com/olesovhcom/status/779297257199964160

I believe the Dyn attack was via Mirai also.


I've been waiting for some announcement around the Gbps of the DDOS similar to this Cloudflare announcement:

  https://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack/
Does DYN routinely deal with very large DDOS which would past this attack in a new category? Can someone who attends security conferences with DYN personnel comment?


last night the consensus was 1.2 tbps.


For a DNS-only service provider, it seems like 1.2Tbps could be 1000x normal traffic. But Akamai claims 30Tbps+ is their routine traffic[1]. Some have commented that this DDOS questions consolidation around cloud providers, but I think it will cause consolidation among service providers. You can no longer be a critical service provider if you don't have the capacity to absorb attacks like this.

http://www.csoonline.com/article/3123797/security/some-thoug...


I suppose Akamai wasn't ready to deal with the attack that size. They only recently bought Prolexic, but things move slowly on their scale.


or 2x krebs, the 2nd? previously largest in history; we could use that or this incident as the future benchmark of ddos capacity. Attacker may have been involved with the 1.5Tb against OVH.


Is Brian Krebs going to become a unit of measurement for ddos attacks? Because that would be awesome.

Ex. "I can't believe our network can't handle that traffic, it is only 20 milliKrebs!"


Hackers have started to use insecure Internet of Things devices, especially internet connected video cameras, to produce DDoS attacks larger than have ever been seen before. The KrebsonSecurity website was hit by a DDoS that was twice as large as the previous largest attack seen by Akemai, and there have been larger attacks since.

The problem will continue, and may get even worse, since many of the insecure internet attached video cameras are insecure because of passwords hard-coded into the devices; they can't be easily made more secure.


I wonder if there's any way to tell apart real-users-requests from fake-users-requests.

If I'm not wrong, it's only preventable by increasing the resources of the server, doing anti-bots things like CAPTCHAS (not feasible for stand-alone IoT devices) or detecting weird patterns (which can be masked really easily).

How will DDoS attack be preventable in the future? There will be so many things and nano-thing connected to the internet that can act as "attackers". Is getting harder and harder everyday.


What software is the piece that is answering the question "is this a real user or fake?". Because that's the piece that will fall over during a DDoS, as it's doing per-request processing.


Just thinking, Is there any chinese production of IOT involved? might be firmware involved?


That's what the following blog claims:

https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe...

I don't know any other independent researcher who confirms this.


That's interesting since they faced the attack so they could have data to analyze that. Apart from firmware, chinese companies also pushing UC browser and WeChat like anything.


Here's a pretty fun 2 year old talk from Blackhat, re surveillance camera firmware and how s*tty they are: https://www.youtube.com/watch?v=B8DjTcANBx0


Presumably it would take a lot of cooperation with ISPs they are peering with, which is not something easily done. Or a google-sized network.


I would also like to know what exactly are "a lot of resources to prevent."?


I wonder how much of this would be mitigated/avoided if folks would just change to something other than the default credentials on IoT devices?

Is it that simple? or am I missing something?


A downvote for a legitimate question. NICE.


Probably they got beaten because of orders of magnitude. They were prepared, but not for cyber nuclear war.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: