
Technical Details Behind a 400Gbps NTP Amplification DDoS Attack - jgrahamc
http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack
======
spindritf
_Somewhat ironically, the large French hosting provider OVH was one of the
largest sources of our attack and also a victim of a large scale NTP
amplification attack around the same time._

And their own semi-official ntp server supports monlist with a hefty response

    
    
        $ ntpdc -c monlist ntp0.ovh.net
    
        remote address          port local address      count m ver rstr avgint  lstint
        ===============================================================================
        10.x.x.246             123 213.251.128.249      515 3 2      0     12       0
        10.x.x.248             123 213.251.128.249      396 3 2      0      7       0
        10.x.x.245             123 213.251.128.249      104 3 2      0     10       0
        212-x-x-101.rev.pon   123 213.251.128.249   178326 3 4      0      0       0
        sw.178.x.x.248-n5.f   123 213.251.128.249       12 3 2      0     12       0
        proxy.ovh.net          46863 213.251.128.249   252113 3 3      0      0       0
        v1.ovh.net             50733 213.251.128.249     2443 3 3      0      0       0
        a2.ovh.net             44965 213.251.128.249  3394192 3 3      0      0       0
        cross.rfid.ovh.net     33352 213.251.128.249    11823 3 3      0      0       0
        10.x.x.176             123 213.251.128.249     1865 3 2      0      4       0
        gw6.ovh.net              123 213.251.128.249     1476 3 4      0      3       0
        sw.5.x.x.248-n5.fr   123 213.251.128.249      361 3 2      0      2       0
        b6.ovh.net             40862 213.251.128.249     1095 3 3      0      4       0
        10.x.x.245             123 213.251.128.249      164 3 2      0      6       0
        10.x.x.211            123 213.251.128.249   314567 3 2      0      4       0
        .
        .
        .

~~~
vidarh
I have a server with OVH, but frankly I'm considering moving elsewhere after
we've now been repeatedly hit by DOS from servers at OVH. It's fairly low
grade, primitive SYN-flood attack that we easily knock back within minutes
each time the attacker moves elsewhere (clearly he does not have access to
many server resources, or he might have actually managed to muster enough
simultaneous resources to do some damage; he's right this minutes wasting
resources getting a SYN-flood from some no-name Russian hosting provider
dropped by our firewall at a low enough rate that I can keep an eye on it live
with tcpdump).

But while our colo provider was extremely responsive and started calling OVH
and the other providers right away, and I also emailed evidence to OVH
repeatedly, we were met with total silence. The other providers used reacted
quickly. OVH let the servers continue to hammer us for days.

I'm seriously considering just dropping all their net blocks in our firewalls.
We have next to no legitimate traffic originating there anyway.

~~~
thaumaturgy
FWIW large portions of ovh.com have been blackholed on my web server for
persistent Wordpress comment spam to hosted sites.

Unfortunately, I can't blackhole their traffic on mail services, because IIRC
some open source mailing lists use them.

OVH is not my favorite network.

------
bhauer
Great write-up and very helpful for those of us who, despite doing so for
years, remain amateurs at running our own servers. I am among those who think
the Internet would be better as a whole if more people did in fact run
servers—server software would gradually become easier for us amateurs to
install and run without leaving it in a state that is open to nefarious
exploits. But for the time being, I appreciate it when experts take the time
to explain simple counter-measures as you have done. Thank you!

As far as I am aware, I am not responsible for any Internet-facing NTP servers
(I certainly never set one up willingly), but it's good to have this in the
back of my mind now in the off-chance that I ever do set one up.

I did have one of my Windows machines used for DNS amplification. I wrote
about the incident [1] at my blog because I had been a bit surprised that it
was not sufficient to simply disable recursion. That much had seemed like
common sense, and I thought I had been so clever and thorough in turning it
off. But later I found attackers were leveraging my server's willingness to
provide a list of root DNS servers in response, even with recursion disabled.
I ended up deleting the list of root servers and the problem went away.
(Though, to be clear, I never ran the incident by any DNS experts, so I may
have misdiagnosed the whole thing.)

I don't know what else I don't know about amplification attacks, so reports
such as yours are helpful for people like myself who find it fun to run our
own servers, but don't consider it an area of expertise.

[1] [http://tiamat.tsotech.com/dns-
amplification](http://tiamat.tsotech.com/dns-amplification)

~~~
rhizome
Not knowing about this particular vulnerability doesn't make one a pejorative
amateur.

~~~
bhauer
I am part of the group I was referring to as _amateurs_ , and I was not using
the word in a pejorative manner. I mean it literally. We, the amateurs I am
speaking of, are not experts at server administration and do it either out of
necessity, for fun, or out of principal. For me, it's a bit of fun and
principle—I feel it's a good thing for me to more or less know how to manage a
server even as I acknowledge that experts know much more about it than I.

I would like to see servers demystified in general, and that's why I applaud
articles such as this one that make the necessary counter-measures/workarounds
plain and clear for those like myself.

~~~
rhizome
That's all well and good, and I wish you luck in your ambitions, but you're
still making a category error.

~~~
bhauer
I'm reaching to understand the point you're making.

I assume you mean that you interpret what I've said as implying anyone who
does not know about NTP amplification attacks is an amateur server
administrator. I do not mean to imply that. I simply mean that for people like
myself, who are in fact amateur server administrators, this sort of down-to-
earth style of article (here is what the problem is; here is how to deal with
it) is very welcome.

Although it's only academic at this point, I am curious why you think I used
the word "amateur" pejoratively and what category you believe I am describing
in error. I ask simply because I'd like to avoid making similar errors in the
future.

~~~
rhizome
Because you more-than-implied that nobody who got hit with it is a
professional.

------
aidos
As far as I can tell these attacks always rely on amplification using IP
Spoofing. I take it there's no way of mitigating that in a lower layer without
adding some leaky abstraction or general overhead to the network? So, for
example, (speaking as someone who knows nothing about these things) you could
add some sort of handshake along the lines of:

    
    
        ntp server sees request from 1.1.1.1 (spoofed by attacker)
        ntp server goes to 1.1.1.1 to check that they really sent the request (sort of ack type thing)
        1.1.1.1 comes back to say that it's an uninitiated request
        ntp server discards similar future requests for some time
    

Obviously that would require more toing and froing, along with more white /
black list tracking etc. Then again, can't all machines have sensible defaults
in their firewalls to stop them from participating in such attacks?

Is this not an issue for TCP?

EDIT: I'm assuming it's because UDP doesn't do any checking / acknowledge
stuff by default?

~~~
devicenull
Yes. And that's exactly what's been implemented in modern versions of ntpd (>
4.2.7p26, 2010/04/24... yes, 2010).

The problem is: 1) No one has upgraded NTPD (and often can't, for embedded
devices like IPMI controllers) 2) This can be fixed by basic configuration in
older NTPD versions, but up until recently many linux distributions were
shipping vulnerable configs.

This particular command (monlist) is a management query, it's in no way
related to serving up accurate time.

~~~
michaelmior
As far as I can tell, the default Ubuntu ntpd config still allows monlist.
(That is, /etc/ntp.conf in saucy doesn't have disable monitor.)

~~~
makomk
The Debian default config has noquery set for everywhere but localhost which
is I believe safer (disables all ntpdc queries not just the one used here).
Ubuntu's likely the same.

~~~
michaelmior
The default Ubuntu config has the following

    
    
      restrict -4 default kod notrap nomodify nopeer noquery                          
      restrict -6 default kod notrap nomodify nopeer noquery
    

Edit: Note that this also exists in Ubuntu 12.04, so the latest LTS is fine as
well.

------
yread
Details on the SNMP amplification

[http://www.nothink.org/misc/snmp_reflected.php](http://www.nothink.org/misc/snmp_reflected.php)

------
voltagex_
Sigh. [https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=733940](https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=733940)

FWIW, if you install the ntp package and do ntpdc -n -c monlist localhost
you'll get a response but I haven't checked if it's configured by default to
reject non-LAN requests.

~~~
alex1
FWIW, here's what I got on an Ubuntu 12.04.3 server running on my LAN. It
looks like we should be fine with the defaults on Ubuntu at least. (Obviously
always a good idea to use ufw/iptables to block everything you don't need
exposed so you don't have to worry about stuff like this).

Before installing ntp (from another host on my LAN):

    
    
      $ ntpdc -n -c monlist 192.168.1.50
      ntpdc: read: Connection refused
    

After installing ntp (from another host on my LAN):

    
    
      $ ntpdc -n -c monlist 192.168.1.50
      192.168.1.50: timed out, nothing received
      ***Request timed out
    

After installing ntp (from the server itself):

    
    
      $ ntpdc -n -c monlist localhost
      remote address          port local address      count m ver rstr avgint  lstint
      ===============================================================================
      91.189.94.4              123 192.168.1.50         1 4 4    1d0     54      54
      ...

~~~
voltagex_
Thanks, I should have checked this.

------
powertower
> it returns a list of up to the last 600 IP addresses that last accessed the
> NTP server

It just gives up that data to anyone that asks? Seems like a huge privacy
issue.

Imaging Apache or Nginx giving up the last 600 IPs it served and maybe the
URLs they went to.

 _edit: there is always the occasionally open Apache /server-status handler
that leaks this type of data._

~~~
Karunamon
>It just gives up that data to anyone that asks? Seems like a huge privacy
issue.

I could do a nmap on the public internet and probably get a similar amount of
addresses. An IP is about as "private information" as a phone number nowadays
(You know, those things that get sent out en masse in yellow and white books
for public consumption with _real-life names_ next to them).

------
NelsonMinar
Some historical context on NTP monlist: it's an old debugging interface from
the days of the Friendly Internet, when services were more open and people
were much less worried about this kind of security. NTP daemons give up a
whole lot of information if you ask them; see also the "peers" and "sysinfo"
commands, for instance.

Back in 1999 I used these monitoring commands to spider the NTP network,
surveying some 175,000 hosts from a desktop workstation. Lots of fun! This
kind of survey is much harder to do now because so many systems are locked
down. [http://alumni.media.mit.edu/~nelson/research/ntp-
survey99/](http://alumni.media.mit.edu/~nelson/research/ntp-survey99/)

------
tinco
This is offtopic, but this post was deleted some time ago, and now it's back,
it was submitted by jgrahamc then also.

What happened to it? Did the algorithm snip it, but did jgrahamc undelete it
somehow, or a mod? Just curious about the way those things work, not
complaining.

~~~
jgrahamc
I deleted it and resubmitted it later.

------
rahimnathwani
Is the response to MONLIST also sent as UDP? If so, why does CloudFlare even
accept those packets to IP addresses used for web hosting? Shouldn't all
legitimate traffic be TCP on ports 80 and 443?

~~~
mnw21cam
The packets have to actually reach you in order for you to filter them out. If
you have a 300Gbps incoming pipe, and you're getting 300Gbps of attack
traffic, then there isn't any space left in the pipe for your legitimate
traffic. It doesn't matter that your router is throwing away the packets as
soon as it receives them.

Also, web servers might want to consult NTP servers now and again.

~~~
alex1
> _Also, web servers might want to consult NTP servers now and again._

CloudFlare doesn't host web servers for their customers. They forward
HTTP/HTTPS requests to origin servers outside of their network (or serve from
their cache). I don't think the DDoS traffic actually hit any of their
customers' origin servers (assuming origin server IPs are not known by the
attackers). But yeah, it still means CloudFlare's incoming pipes being hit
with 400Gbps of traffic before they're able to filter anything.

------
poobrains
How do you test a server for this attack? I want to make sure my servers don't
participate.

~~~
biot
The fine article states:

    
    
      You can check whether there are open NTP servers that
      support the MONLIST command running on your network by
      visiting the Open NTP Project[0]. Even if you don't think
      you're running an NTP server, you should check your
      network because you may be running one inadvertently.
    

[0] links to [http://openntpproject.org/](http://openntpproject.org/)

~~~
e12e
As I happen to have openntpd installed on a box I attempted to test this from
(in Debian that package conflicts with ntp -- which includes the ntpdc client)
-- I also found this:

[https://github.com/sensepost/ntp_monlist](https://github.com/sensepost/ntp_monlist)

It at least correctly identifies ntp0.ovh.net as responding -- and seems to
match up with what openntpproject.org thinks...

[edit: apparently this (partly) also illustrates why more people should heed
the advice to "run only what you need, listen only where you must" \-- or in
other words, make sure that:

    
    
        netstat -lnutp # listening, numerical, udp, tcp, program
    

gives essentially no output, at the very least not a lot of 0.0.0.0:x
(listening on all interfaces). I'm always a little sad when people don't check
that, and just throw up some complicated iptables-rules -- before checking if
they're actually running some daemons that should be removed, or pointed at
less public interfaces.]

------
unethical_ban
What about a mesh network? How would they implement BCP38?

If a bunch of computers are hooked together in a mesh relaying traffic in all
directions, how can one do ingress filtering?

------
mturmon
OT, but: Tasteful choice of album cover art for the pic at the top.

------
junto
I'm not much of a network guy, but is it possible for Cloudflare to just
redirect that DDOS traffic back to the NTP server that sent it?

This would have two benefits. Firstly the owner of the insecure NTP server is
going to get a nasty message to fix their damn server, and secondly, the
insecure NTP server gets taken out of the attack and becomes useless to the
attacker.

As a visual reference, it would be a bit like in Star Wars when Mace Windu
fights Palpatine on Coruscant [1]:
[http://www.youtube.com/watch?v=Pk4AiCnMqpg#t=2m35s](http://www.youtube.com/watch?v=Pk4AiCnMqpg#t=2m35s)

Eventually these server provider who have left their servers wide open will
get the message when there NTP servers no longer respond?

1\.
[http://starwars.wikia.com/wiki/Showdown_on_Coruscant](http://starwars.wikia.com/wiki/Showdown_on_Coruscant)

~~~
jgrahamc
No, we are not going to do that.

Two reasons: it's a bad idea and it wouldn't help very much. Each individual
NTP server is only generating a modest amount of traffic and such a response
might well go unnoticed by the NTP server. Also, it would mean we'd have to
generate 400Gbps back to the NTP servers creating an enormous amount of
traffic.

------
petermonsson
This is probably a stupid question, why can't tier 1 providers (whom I suppose
there are relatively few of and who I would expect to incorporate best
practices) just decide to kill any NTP monlist UDP that ever crosses any of
their NPUs?

Why would that not solve a large part of the problem?

Thank you in advance.

~~~
jlgaddis
I don't work for a Tier 1 ISP but I do work for an ISP.

As a customer, I don't want my ISP screwing with my traffic. As a provider, I
don't want any customers complaining because we screw with their traffic.

To block monlist and only monlist queries, we'd have to be looking into the
layer 7 payload of IP traffic. I'd rather not do that.

The brute-force method would be to block traffic to/from 123/UDP but that's
gonna mess up a lot of stuff (including my own).

------
peterwwillis
So again we find the problem is protocols that are designed for convenience
and not security. Sure, network providers could filter out bogus routes, but
that's a band-aid more than a fix; the protocol is still broken from a
security perspective. Nobody would stand for using rsh with host-based
authentication in today's age, but for other protocols it's fine? And public
services for the internet at large are great, until they become tools for the
public to abuse other people. These protocols either need to be fixed to
prevent abuse, or switch to using tcp (which nobody wants - so fix the
protocols!)

------
TA-bye
All NTP seriousness aside, this new "record-breaking" DDoS attack was only
possible because CloudFlare -- after the Spamhaus attack -- upgraded and
expanded their network endpoints all over the world. When the next attack hits
and they have again upgraded their connections with 100Gb/s combined, they'll
be able to say that there was again a new record, this time it was 500Gb/s.

------
JulianMorrison
I wonder, is QUIC vulnerable?

UDP? [✓]

Amplification? [✓]

Spoofable? [?]

~~~
oasisbob
QUIC datagrams should be as spoofable as anything else using UDP.

The _QUIC Crypto_ design doc contains a section that covers spoofing [1], and
seems to push responsibility for DDoS mitigation to the server implementation:

"[...] servers may decide to relax source address restrictions dynamically.
One can imagine a server that tracks the number of requests coming from
different IP addresses and only demands source-address tokens when the count
of “unrequited” connections exceeds a limit globally, or for a certain IP
range. This may well be effective but it’s unclear whether this is globally
stable. If a large number of QUIC servers implemented this strategy then a
substantial mirror DDoS attack may be split across them such that the attack
threshold wasn’t reached by any one server."

[1]
[https://docs.google.com/document/d/1g5nIXAIkN_Y-7XJW5K45IblH...](https://docs.google.com/document/d/1g5nIXAIkN_Y-7XJW5K45IblHd_L2f5LTaDUDwvZ5L6g/edit#)

