

DoS attacks that took down big game sites abused Web’s time-synch protocol - RougeFemme
http://arstechnica.com/security/2014/01/dos-attacks-that-took-down-big-game-sites-abused-webs-time-synch-protocol/

======
mirashii
As someone who was considering Black Lotus as a potential service for DDOS
protection, they had a number of failures during this whole episode that make
it hard to take them quite seriously.

First, and most importantly, they suffered many hours of downtime to many
sites hosted on their service, including their own web page, and still to this
day have not tweeted or given any public statement about the incident. A lot
of Googling got me some responses to customers who complained in private with
weak justification about it being their upstream provider's problem and not
theirs. The fact that they now estimate that it was 28GBit/s in traffic makes
me even less trusting of their service, given that much larger DDOSes are
flying around.

If anyone has any other services that are competitive, other than Cloudflare,
I'd be interested in hearing about them out of curiosity. It seems there
should be more than one company that can do this well.

~~~
scurvy
defense.net and Prolexic come to mind before Black Lotus or CloudFlare. Black
Lotus and CloudFlare are bottom of the bucket options compared to defense and
Prolexic.

Keep in mind that Prolexic was recently acquired by Akamai, so prices will
skyrocket soon.

------
ghshephard
Easy DOS to block upstream, thankfully. Time seems like the sort of thing that
a CoLo should provided to it's tenants. GPS time is likely good enough for 99%
of sites, and, the final 1% who really require super reliable time, and need
to be concerned about GPS spoofing/outages, can install a local atomic clock
in their cabinet.

~~~
sargun
How would you block it? Any packet with the source port of NTP should be
dropped?

No. ISPs should not get into the business of default block rules. This gets
into all sorts of problems. At what point does a block become unreasonable?
What if a tenant needs multiple sites in sync -- they must run NTP between
them (or PTP) in order to determine their drift from one another.

If you really want upstream blocking of traffic, the best thing to look
towards is BGP Flowspec (RFC5575)

~~~
ghshephard
"How would you block it? "

The same way all Upstream providers blocks a DDOS - by characterizing the DDOS
flow, scrubbing, and dropping any packet which matches that pattern.

The idea here, is that during a DDOS, where you might get a bit more NTP drops
than normal, none of the local users should be impacted if they had a quality
local time source.

"What if a tenant needs multiple sites in sync"

That's why you have a quality time source. By definition, they are "correct"
to within any tolerance that all but CERN physics experiments care about, and
they don't use NTP anyways.

See:
[http://static.googleusercontent.com/media/research.google.co...](http://static.googleusercontent.com/media/research.google.com/en/us/archive/spanner-
osdi2012.pdf) for an example of using synchronized time globally, without
reliance on running NTP between sites.

In particular, check out section 3, "TrueTime"

~~~
scurvy
It's trivial to add a UDP policer to ntp traffic to your ntp hosts. Also, how
many people are exposing NTP externally on the same IP's they're running
regular content (http/https) on? If you're allowing UDP/123 to your web
servers, you need to fire yourself.

I guess what I'm trying to say is, you should already have this ACL'd on your
own.

~~~
ghshephard
A UDP Policer doesn't help in a DDOS, because you need to block the traffic
before it gets onto your circuit. By the time you start inspecting traffic,
it's too late, you've already passed the traffic on your (presumably limited
size) circuit.

The NTP DDOS doesn't require that the web server (or, indeed, any server) at
the target be listening to UDP/123 in order to cripple them. You just need to
use up their circuit capacity.

What I was trying (and failing) to suggest, is that when the Upstream starts
scrubbing out the NTP DDOS, there is a reasonable chance, that _during the
event_ the downstream customers will start to see more NTP drops than they
normally would. And that one way that customers who _care_ about NTP greatly
could mitigate that possibility, would be by having multiple stratum 0 sources
locally - I.E. A GPS Antenna and Atomic Clock. I then took it a step further,
and suggested that this would be a great service for the CoLo to provide,
because they are downstream of any packet scrubbing, and would therefore be
able to provide a reliable time source during a DDOS attack involving rogue
NTP traffic, which might end up with some NTP packets being dropped by the
upstream ISP who was scrubbing the DDOS off the circuit.

~~~
scurvy
I was suggesting ways to prevent being part of the DDoS, not protect yourself
from the DDoS. In other words, lock down your hosts so that they don't
participate in the attack.

------
chris
We (Weebly) had 18Gbps of UDP/123 (NTP) traffic sent our way on New Years eve
-- definitely one of the larger attacks we've seen recently.

~~~
midas007
This sound like what brought down the GnuPG main site. Sounds like an
Anonymous(TM)-type thing.

------
skymt
Why are ISPs letting spoofed traffic leave their network? I don't work at that
level, but I thought this was a solved problem.

~~~
sargun
I imagine you're suggesting most ISPs implement BCP38? A lot of organizations
implement BCP38, but the overhead of BCP38 can actually be a lot, especially
once you get far above the DSLAM / CMTS.

Not only is it not free, it is not straightforward. In some more complex
network architectures, you don't even know what the sources of downstream
packets might be (case: downstream has two transits, and they don't want to
bring ingress traffic via transit B, because transit B has ridiculous
pricing).

~~~
cma
Its value is way higher than its overhead, but the overhead is paid by the guy
who essentially doesn't benefit from it.

~~~
_delirium
I agree there's a payer/benefit mismatch, since filtering your own egress only
protects other people's networks. But I would've thought it would still end up
being incentivized somehow through things like peering or transit contracts. I
don't have a good insight into how such contracts work, so I could be way off.
My assumption was that in at least some such situations the other party would
care if you're feeding them bogus stuff through the link, even if you don't
care, so they would be motivated to require that you do egress filtering on
your end as a condition of getting the link.

------
nradov
So both DNS and NTP can be abused in amplification DoS attacks. The obvious
question is: which other protocols can potentially be exploited the same way?

~~~
rwg
Any protocol that rides in UDP and that will reply to an unsolicited n byte
packet with a >n byte response. (But if the amplification factor is close to
1, then you might as well have your botnet just attack the target directly.)

Devices that speak SNMP and use "public" for their community could be used in
an amplification attack. I don't know what the amplification factor would be,
though, and I imagine there are loads more Internet-facing DNS and NTP servers
than Internet-facing SNMP agents configured with a "public" community...

~~~
devicenull
SNMP is already being abused for this

~~~
midas007
SNMP has a history of being horribly insecure, because it was assumed people
wouldn't put their mgmt hardware on the public IPs. It should only be deployed
in trusted networks. It's about as secure as nonkrb5, unencrypted telnet.

------
PaulRobinson
The Web != The Internet.

NTP is not "the Web's time protocol".

Really fed up with tech journalism these days assuming that nobody has quite
finished the first few chapters of the "Networking for Dummies" book. Massive
gap in the market for informed quality tech journalism.

------
p1mrx
The Internet is not the Web, and NTP has nothing to do with the latter.

