
Possible Vendetta Behind the East Coast Web Slowdown - whiskypeters
https://www.bloomberg.com/news/articles/2016-10-21/internet-service-disrupted-in-large-parts-of-eastern-u-s
======
jerf
For a long time, I've wondered what would finally be the Securitypocalypse,
the thing that finally caused our industry as a whole to take security
seriously. These IoT DDoS attacks are as good a candidate as any I've seen in
a long time. They are fundamentally very difficult to fix in light of the non-
updateability of many of these devices, and this is only the beginning,
because the IoT has hardly begun to develop. And in the short-term, I'm not
sure I see any hope, because the forces that make people throw out cheap
devices with broken firmwares with no update capability aren't going away.

If we could somehow mandate that these devices were supported with firmware
updates for the indefinite future, that would simply destroy the entire
market. And you can't do that, because even the devices created by an entity
that no longer exists and didn't sell its IP to anybody else will eventually
be enough to do these DDoSes, if they aren't already.

~~~
NKCSS
It's easy to fix; back in the day when a machine was infected; an ISP would
just block outgoing traffic, contact line owner and re-enable when the issue
is resolved.

~~~
rev_bird
"Fix" is a relative term, especially if IoT devices are in play – yes, turning
off the internet to customers stops the attack, but then (at least?) thousands
of people lose internet connectivity because of a vulnerability that they
could very well be powerless to fix. I'm not saying it's ok with me that an
army of smart refrigerators could be taking out big chunks of the web, but
it's a lot easier to tell someone, "Hey, either get the infection off your
computer or re-format" than it is to make someone buy new lightbulbs and
appliances.

~~~
zzleeper
Not powerless, just unplug their toaster and they get their internet back.

What is powerless is that many people _today_ couldn't get twitter, github,
reddit, spotify, box, etc. because many people don't care about securing their
webcam.

------
egypturnash
I am a non-programmer who reads HN and keeps up with tech news in general.

And every time I read about the IoT botnet, my immediate response is to look
around my apartment at my Internet-connected lights, and wonder if they're
part of it.

How can I find this out?

Is anyone making a tool that a non-technical user can run to squint at their
network and look for evidence of Mirai, or anything else trying to take
advantage of this niche?

There are plenty of tools with a reasonably simple interface that will tell me
if my laptop/desktop computer is infected with something. But what can I use
to diagnose the health of all of the _other_ computers proliferating around my
house?

How can a non-technical user easily monitor the overall health of their
connected household? Is this a project anyone is building? Because I think
it's definitely something that needs to exist now.

~~~
egypturnash
Looking over all the replies this comment received, I think my plan for seeing
if my apartment's Internet Things are on any botnet is going to be "bribe that
security researcher I flirt with sometimes to visit my place and run some
tests". Which is not really a solution that scales, either for that friend, or
for people who don't happen to run in the kinds of circles where that's
someone they could conceivably trade favors with.

And it's probably not gonna get any better any time soon, either. Because I'm
not sure there's a money stream in making this something a non-programmer can
do. And maybe there _shouldn 't_ even be a money stream in this - maybe there
should just be huge-ass fines to motivate as many people as possible along the
chain from "my Internet Thing" to "the Internet" to include a white/grey hat
or three on their team very early in the design process of making their
camera/light bulb/pacemaker/router/modem/whatever. Although if someone reading
this can figure out a way to get a money stream out of making it a lot easier
to see the health of your home's devices, and keep them safe, that might be a
decent YC app for you.

How do we add an immune system to the Internet Of Things? Because we sure as
hell need one.

~~~
jnbiche
> bribe that security researcher I flirt with sometimes to visit my place and
> run some tests...[w]hich is not really a solution that scales...

Assuming the flirting displayed is sincere, that security researcher may prove
much more scalable than you'd imagine.

~~~
egypturnash
There's not really enough of a size difference between us to make "scaling"
come into play.

------
gorbachev
Here's a better article from Mr. Krebs:

[https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
twit...](https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-
spotify-reddit/)

Personally I think his case is pretty convincing.

~~~
brightball
From the article:

"Last month, a hacker by the name of Anna_Senpai released the source code for
Mirai, a crime machine that enslaves IoT devices for use in large DDoS
attacks. The 620 Gbps attack that hit my site last month was launched by a
botnet built on Mirai, for example."

I repeatedly hear people refer to IoT devices that are notoriously difficult
to update...yet this Mirai code is technically able to access millions of
devices and bend them to its will.

So what I'm wondering is just, what prevents the good guys from using Mirai to
slurp down every available device to patch the vulnerability that allowed
Mirai to work in the first place?

It seems like if vulnerabilities in these devices can destabilize the entire
internet that it should be perfectly viable as a response to actively look for
those vulnerabilities, patch/minimize them and notify their creators of the
issue.

~~~
tw04
The problem is you're reading the situation wrong. Mirai isn't about an
exploit, it's IoT devices that haven't had the default username/password
changed.

Now, you might say "why doesn't a good samritan just login to all of those
devices and change the password to something random?"

OK - ignoring the fact that THEY would be committing felonies in several
countries... what happens when the device manufacturer wakes up and decides to
patch these devices via that remote access? Suddenly the password doesn't
work, and the end-user can't change it because... what's the procedure for
changing the default ssh password on a light bulb?

Technically you could make the situation better by writing a worm that changes
the passwords, but at this point even that is a lost cause since mirai has a
command that will change the pw on all infected hosts.

~~~
brightball
I guess that's what I'm getting at though. If we were to scan for the affected
devices, change the passwords and notify the manufacturer of the change and
that it was made because their carelessness essentially endangered the
internet it would make it possible for them to fix it.

You're plugging a leak and letting the owner know, hey this was leaking and I
stopped it but you're going to need to address that.

------
bcheung
I know the TTL is set really low for a lot of DNS entries but this recent
outage got me wondering if it makes sense for servers further down the chain
to hold onto it for longer than the TTL, honor it when they are able to get a
new DNS entry within a reasonable amount of time, but fall back to the
"expired" version if the authoritative server is not reachable.

I'm wondering what would be the negative consequences of this and if they
outweigh the benefit of being more resilient to these types of attacks.

~~~
idlewords
There was a good discussion on this in a sibling thread earlier today:
[https://news.ycombinator.com/item?id=12762110](https://news.ycombinator.com/item?id=12762110)

------
elmigranto
No luck with Google DNS for me, but Yandex seems to work:

    
    
        77.88.8.8
        77.88.8.1
    

[https://dns.yandex.ru](https://dns.yandex.ru)

~~~
drinchev
Probably, but I would definitely avoid giving all my DNS resolutions to a *.ru
domain.

The reputation of the government - shutting down access to websites that hurt
them is kind-a no-go for me.

~~~
Jerry2
US government is not much better... or have you forgotten all those hundreds
of FBI/ICE domain seizures. How many have Russians taken down? If your'e gonna
use DNS servers and you don't want someone to track you, use the DNS server
based somewhere where your government cannot access them. If you're in the US,
it's easy to assume that US DoJ/FBI will not be able to subpoena Yandex or
some Chinese internet provider.

------
jpeg_hero
bloomberg was down for me.

I had disabled adblock at their insistence...

i re-enabled adblock and I could get the article. hmmmm. maybe something about
the 50 unrelated js calls?? perhaps?

------
inostia
More specifics about Mirai bots and their numbers:

[https://threatpost.com/mirai-bots-more-than-double-since-
sou...](https://threatpost.com/mirai-bots-more-than-double-since-source-code-
release/121368/)

------
kakarot
Unfortunately, forced firmware updating is an area our governments should not
be mandating. That puts unnecessary strain on small companies and creates a
larger gap that companies must cross to become commercially viable

~~~
ams6110
Liability should be on the people who connect these things to the public
internet. The owners of the devices. Like with cars, you have certain
responsibilities and liabilities when you operate a potential dangerous
machine on the public roads.

In the case of ISPs providing cable modems and routers and DVRs and other
boxes to their customers, they should be responsible for keeping those secure.

If people start getting fines or sued over what their internet-connected
devices are doing, they might stop connecting them to the internet, or shop
more carefully for devices or providers that are secure.

~~~
Florin_Andrei
So grandpa goes to Home Depot, buys a fancy new thermostat and installs it at
his home, the device gets hijacked by the archetypal 400 lb hacker, and is
used to take down a major commercial site, and then grandpa is liable for the
whole thing?

I don't think so.

You make a little gizmo with shitty security, you are liable. Full stop.

~~~
ams6110
So grampa doesn't take care of his car, the brakes fail and he kills a family
with four kids. Is he liable? Yes. He may not know the first thing about
brakes or car repair but owns the car, and he took it out on the road without
being sure it was in safe operating condition.

But to steal an idea from another comment, make the ISPs liable also for
routing the malicious traffic onto the internet. They will then have incentive
to monitor their networks and they can take homes offline until their
customers fix or disconnect their hacked devices.

~~~
jlgaddis
I'm the "head fred" networking/infrastructure guy at an ISP. I want to _avoid_
, as much as possible, peeking at my customer's traffic.

In my personal opinion, an ISP should be a dumb pipe. I'm providing you with
the ability to send/receive "n" bits per second; I don't care whether you use
it to participate in e-mail discussions with your church group or stream
pornography and play online poker.

Are you _certain_ you want ISPs to be responsible for monitoring all of your
traffic and what you're doing online? Do you really want somebody else
deciding -- at their own discretion -- what is "acceptable" for you to do
online?

I'm very pro-privacy, pro-encryption, "pro-Internet freedom", etc., but the
next guy may not be.

------
thesteverichey
Any evidence this is using the IoT botnet that was reported on earlier this
year?

~~~
micaksica
Mirai? With the source of that being public, there are probably quite a few
Mirai botnets now.

------
rrggrr
These attacks are possible because the US Congress hasn't extended tort
liability to manufacturers of software and network hardware. The full weight
of the US products liability bar will quickly and rapidly motivate
manufacturers to ship secure devices. The lack of accountability is enabling
vulnerability.

~~~
nradov
Who is the "manufacturer" in the case of FOSS?

~~~
niftich
Whoever puts the FOSS on the device

------
davidf18
The failing here as in many cases such as a number of security breaches was a
lack of investment. As someone with an engineering degree that worked as a
VLSI design engineer, good engineering requires __* backup systems __*. This
costs money that people don 't want to spend. In some cases such as a startup
they might be cash short, but many firms have the money but don't want to
spend it ensuring that they have well engineered software that includes
backups, up-to-date software and security upgrades, hiring (expensive) highly
competent software engineers and consulting firms.

The mistake in this case was relying on one vendor for DNS. Amazon Route 53
would be a good alternate vendor for DNS, for example.

------
patrickg_zill
I think even basic home routers these days, have enough cpu power to handle
egress filtering.

If you have an iot device, by its nature it only needs to connect to a few
services and hosts.

The manufacturer can provide this in their docs, and give an automatic config
url that the router uses to load its egress rules.

The rules to load are displayed and the user checks they are legit by
comparing to the printed version in the manual, then clicks ok. Or something
like that.

Rate limits in terms of packets per second, total bandwidth both instantaneous
and over time, are set also.

------
raverbashing
Not only East Coast, Twitter can't be resolved in Ireland/UK right now (I
assume the mobile app uses some kind of 'dns pinning' as that is working)

~~~
profmonocle
> (I assume the mobile app uses some kind of 'dns pinning' as that is working)

The app was down for me until I switched my WiFi network to use OpenDNS. It's
possible your phone has the DNS record cached, or it's using a different DNS
server. (Is it on cellular?)

Hardcoding IPs into a mobile app typically isn't done because it makes
changing your infrastructure extremely painful.

------
woliveirajr
I love those comments about IoT and who should be responsible for error-proof
products, or ISP monitoring traffic, or ...

Internet, in the beginning, was even more insecure. Including the computers
and OSes. There were less abuse because few had resources and knowledge. Read
some old software and you'll find all bad designs in it. Software didn't
become worst, it's just targeted with more knowledge and intensity.

------
ilaksh
DNS is actually fairly centralized the way it is actually used.

We need protocols and systems that are designed to be distributed from the
outset.

------
pc2g4d
I always thought DNS had enough redundancy built-in that this sort of thing
wouldn't really have much effect. But here I am unable to access websites,
simply because name resolution isn't working. If my local DNS server were
caching things longer there would largely be no issue.

~~~
Falkon1313
Yeah, DNS entries are usually (or at least used to be) cached for what would
seem like long enough, but I guess it doesn't really work the way it sounds.
"a hierarchical decentralized naming system [that] provides distributed and
fault tolerant service and was designed to avoid a single large central
database" doesn't sound like it should be so fragile. Having single
'authoritative' servers for the sort of thing that should be inherently
distributed sounds more like an Achilles heel.

------
reacharavindh
Perhaps a naive question, but Why can't a DNS provider identify such
participants in a DDOS and ban their IPs forever?

~~~
JayNeely
Because IP addresses are often shared resources. Your ISP gives each customer
an IP address (often a temporary one), and then that customer's router system
handles assigning private, local-network-only IP addresses to any devices
connecting through the network.

So if a DNS provider starts banning public IPs (which are the only IPs it
sees), you could end up with an entire college getting banned because of one
hacked webcam in one student's dorm room.

Or someone in an apartment somewhere with (unknowingly) a hacked thermostat
finds their internet no longer works (DNS provider has banned them), so they
reboot their modem, which causes their ISP to provide them with a new IP
address. Guess what happens to their old IP address? It goes back into the
pool of available IPs that that ISP can assign to other customers, and more
and more banned-from-DNS addresses keep getting passed along to innocent, un-
hacked customers.

~~~
reacharavindh
Ah, (inter)networking 101. Thanks! Then, is there a way for the DNS providers
to know the ultimate recepient at all? MAC address? (or does it get truncated
at the lower levels and not passed over IP protocol?)

~~~
jlgaddis
Nope. Assuming you have a router connecting your home network to your ISP, for
example, the MAC addresses of your "internal" devices are not visible to the
ISP. The only MAC address they see is the MAC address of your router's "WAN"
interface.

The source/destination MAC addresses in an Ethernet frame (layer 2) are
rewritten at every router (layer 3) hop. The original IP source/destination
addresses in the IP packet, however, do not change (exception: NAT, which does
exactly that).

Another problem -- in many (most?) DDoS attacks where UDP traffic is involved
-- is that the source IP addresses are "spoofed". That is, IP packet that the
victim receives _says_ that it's coming from Alice but it really came from
Bob. There are also "amplification" attacks, where an "innocent third-party"
is used, unknowingly, to "help" perform the attack.

------
anotherevan
Did any one else find the style of writing in this article really annoying?
Things like using prefacing statements with "so-called" or putting terms in
quotes to make them seem suspect.

e.g.s:

a so-called distributed denial-of-service (DDoS) attack

York said Dyn was “actively” dealing with a “third wave” of the attack.

~~~
GrinningFool
I tend to assume that the larger publications use it in the underlying sense
of "..as it is so called".

------
meira
Not working, is bloomberg down too?

------
trendia
If you are unable to connect because of DNS problems, switch your DNS server
to 8.8.8.8 (Google).

Edit: sorry there, this worked for me but apparently it's not guaranteed.

~~~
losvedir
I switched temporarily from those to Open DNS's 208.67.222.222 and things are
working for now.

But, just to be clear, it's not Google's fault: 8.8.8.8 are not the
authoritative name servers for the sites that are down. Rather, Dyn, the
provider of the NS is down, and I presume Google (8.8.8.8) is _correctly_ not
returning any IP address because the underlying authoritative name server is
not.

Presumably Open DNS is working because it's not abiding by the TTL it's
supposed to? It's caching the underlying authoritative name server longer than
it was told?

~~~
detaro
Yes, the call it "SmartCache": [https://www.opendns.com/about/press-
releases/opendns-introdu...](https://www.opendns.com/about/press-
releases/opendns-introduces-smartcache-new-feature-enables-web-sites-to-load-
successfully-with-opendns-while-offline-for-the-rest-of-the-internet/)

------
nastyasiwannabe
I'm suggesting this just so someone more knowledgeable can debunk it. Suppose
FBI or someone up there had a meeting and said "in three weeks, there could be
millions of armed Americans who believe that democracy was just stolen from
them by some evil dictator in a massive globalist conspiracy. These people
love twitter. Is there a way to make twitter go down without making it look
like we're suddenly pulling the plug?" The answer was yes, we'll do a test run
Friday.

~~~
galdosdi
I'll bite.

It would take a lot longer than a couple of hours of twitter being down for
that to have a useful effect. For something as major as the presidential
election result, it would probably take minimum a week before people got bored
and moved on to a different topic.

So this kind of attack that only takes something out for a few hours would
have no useful effect for an actor that wants to prevent people from
discussing a recent event.

IMHO, it would be hard in general to take out a service run by a serious IT
organization (of which there are admittedly few, by my definition of serious)
for more than a few days unless the attacker carried out non-trivial physical
damage (eg, bombing multiple datacenters, murdering multiple system
administrators, etc) or managed to somehow destroy enough backups (which in a
serious IT shop, should be hard, as there should be some offline cold backups
that require physical human activity to destroy)

~~~
nastyasiwannabe
Sure it wouldnt work for an extended time. I'm just thinking that in an
unpredictable situation, a few hours might be all you need to diffuse it. For
example suppose someone claims they have evidence of some crazy shit happening
at the polling places, and the only thing that can be done is to for Patriots
to seize the equipment at the polling places before the globalists can cover
their tracks.

