
NTP Pool Bad Actors: The Rising Sophistication of Network Scanning - tshtf
http://netpatterns.blogspot.com/2016/01/the-rising-sophistication-of-network.html
======
tyingq
It seems like many aren't quite getting what's going on here...here's a brief
overview:

Shodan.io fancies itself as a search engine where you can search for IOT
things (webcams, refrigerators, etc) that are on the internet.

They have no technical issues doing this in the IPV4 space, because it's easy
enough to scan every single address in the space.

This isn't practical for IPV6. So, it seems they wanted a way to identify
every IOT device in the IPV6 space without having to scan all of it.

At least one approach they found was to join the ntp.org pool[1], and
effectively donate server time. Since pool.ntp.org is the default NTP server
listed for many linux distros (and thus, probably IOT devices), they now are
getting live connections from the exact devices they want to index on their
search engine.

Once you connect, they scan you back on 100 ports on so (ports unrelated to
NTP) to see if you are a webcam, router, or whatever else they want to put in
their index.

Pretty shady. Kind of like volunteering for a charity so that you can raid
their internal mailing list for spam purposes.

[1][http://www.pool.ntp.org/join.html](http://www.pool.ntp.org/join.html)

~~~
morgante
I don't really see what's shady about it. If you connect to a server, it's
accepted and expected that they'll get your IP address.

~~~
tyingq
They run a port scan against your ip and publish the discovered info publicly.
You may find that okay, others don't.

~~~
therein
Well then I guess people should stop relying on their IP address being one of
many in a vast space for security and implement proper authentication instead.

It's not like your service has to expose all its secrets to an unauthenticated
client sending you a GET / HTTP/1.1.

~~~
tyingq
That's not the irk. Rude behavior isn't suddenly polite just because I have
great security. The spirit of donating server time to the ntp.org pool is not
to use the data for your own self promotion.

------
Lazare
The lesson here, I think, is that security intuitions developed in an IPv4
world with NAT everywhere may need re-examining in an IPv6 world.

If your device has a publicly routable IP address, it should probably be
behind a firewall. If it's not behind a firewall, it's going to get scanned.
Relying on security through address space size is stupid.

> What was most puzzling was the fact that the devices that were targeted had
> randomized IPv6 addresses and were not published in DNS or any public
> record. For all intents and purposes they were hidden safely within my lab
> network.

That is just wrong, and it's wrong in a very glaring, obvious way: The devices
were not "hidden safely within [his] lab network", because he was not using
NAT. They were connected to the internet with publicly routable IP addresses,
which they were using to communicate with other hosts on the internet.

Edit: That being said, it's a great writeup, and some good technical work was
done to figure out what was going on, and I enjoyed reading it. But the
starting premise seemed to be "how could this be happening, my devices are
hidden!" and I feel like it should have been "oh, it makes sense that someone
would do this since my devices are no longer hidden".

~~~
mentat
Even if they were NATed I'd assert they weren't safely hidden but that's a
more obscure discussion.

~~~
Lazare
You're quite right; people significantly overestimate the safety of NAT in my
experience.

But what I was getting at is that however illusory the safety of NAT is, the
author seemed to be instinctively assuming that his IPv6 devices had it,
because we've spent 20 years training ourselves to assume that everything is
behind NAT. It's a new world.

------
devicenull
I'm not really seeing a vulnerability here. The entire IPv4 internet is
scanned probably hundreds of times a day. Was anyone really counting on IPv6
addresses being longer to add security?

[http://www.internetsociety.org/deploy360/blog/2015/02/ipv6-s...](http://www.internetsociety.org/deploy360/blog/2015/02/ipv6-security-
myth-4-ipv6-networks-are-too-big-to-scan/)

~~~
dspillett
A machine with a random address that makes no (direct) contact with the
outside world will be a lot harder to find in a 64-bit address space then a
32. As soon as it talks to the outside world though (by talking to public NTP
servers in this case) that difference is rendered moot.

~~~
clinta
Unless it's using Privacy Extensions in which case it will change it's
outbound IP in a matter of hours and once again be lost in the huge address
space.

Unfortunately privacy extensions are not enabled by default on most linux
distributions and server operating systems.

------
ChuckMcM
Very interesting post, I've suggested to Brad that he consider probes with a
decreasing ttl in the packet to see if the harvesting is happening at an
interstitial node or directly on the ntp server. If you had access to a
compromised Juniper router it would be straight forward to add a rule to
mirror packets which were headed to the NTP pool addresses.

Oddly after being an unwitting participant in one of the NTP amplification
attacks I set up my own stratum 0 NTP server based on the beaglebone black,
the adafruit GPS module, and the PPS time keeper code. So all of my machines
only talk internally for time, although initial installs still use what ever
the distro packed into them. I brought up Debian on a Dragonboard 410c and it
sets the time via an NTP call during the initial startup process. (or fails to
set the time as NTP is blocked egress/ingress from the firewall)

~~~
arca_vorago
I have been interested in using the dragonboard with linux as a small
controller for some embedded things, how is that working for you?

On subject, I have always though having a master internal ntp under strict
controls should be used, so good on you for that, I dont see ntp payed
attention to far too often in the enterprise.

~~~
Rebelgecko
I have a Dragonboard that I haven't really had a chance to do anything with
yet. However I've noticed two main drawbacks:

1\. You have to use some awful hacks if you want to ssh into it (IIRC it
ignores ARP requests for some reason) 2\. The GPS only works on Android

That said it's still a nice upgrade hardware-wise from something like a
Raspberry Pi. The software support is just spotty in a few areas

------
dspillett
If someone is detecting which IPv6 addresses are in use in your range from NTP
requests, that implies they are all talking to the external NTP servers
directly.

Surely if all your internal hosts are talking directly to the external NTP
servers you are doing it wrong? My gateway box sets itself by pool.ntp.org and
the internal ones set themselves by it. I thought that was how you _should_ do
things (even if it isn't a rule, it is only polite to try not overuse a public
resource).

 _> These addresses are 128 bits in length_

Only 64 are relevant here: once you make outgoing connections from any address
a scanner knows there is at least one active host in that /64 and there may be
more. Though of course a 64 bit address space is still impractical to scan.

~~~
toast0
> Surely if all your internal hosts are talking directly to the external NTP
> servers you are doing it wrong? My gateway box sets itself by pool.ntp.org
> and the internal ones set themselves by it. I thought that was how you
> should do things (even if it isn't a rule, it is only polite to try not
> overuse a public resource).

Depends on how many internal hosts you have, and ease of configuring them
(there is a DHCP option for ntp server, but not everything will use it), and
available hardware to setup ntp servers: ideally you would have each client
syncing from three servers so the clients drop a broken server.

~~~
dspillett
_> ideally you would have each client syncing from three servers _

For a small network (a home or small office network for instance) you likely
have one incoming router anyway, so if that dies clock setting is the least of
your worries until it comes back up, meaning setting the internal hosts by
that one clock is as fine as anything else.

For a larger network you have redundant everything, including edge servers
that can act as your multiple NTP sources for internal hosts rather than every
internal host poking the public pool.

~~~
toast0
> For a small network (a home or small office network for instance) you likely
> have one incoming router anyway, so if that dies clock setting is the least
> of your worries until it comes back up, meaning setting the internal hosts
> by that one clock is as fine as anything else.

I'm not so much worried about the router dieing and losing sync with the outer
world; but about the router's clock going wonky: I've seen computers where
something got initialized wrong and they were 30 minutes slow after 10 hours
(reboot helped in that case, but sometimes it's the oscillator is just too far
off)

------
fl0wenol
While it is pretty freaky, I think the point of Shodan scanning IPV6 devices
that use the default pool is to detect linux-based IoTs that use the default
NTP settings, which is kind of the reason why Shodan exists.

It is unlikely that Shodan is going to hack your Raspberry Pi specifically
because then people will go after them. But hackers of many hat colors will
use the freely available information it gathers for their own purposes, so act
with this in mind.

------
Johnny555
Isn't this article just saying that "security through obscurity" doesn't work,
and you can't count an obscure IPv6 address to keep you hidden?

If your server isn't meant to be contacted by the world, then put it behind a
firewall. Don't count on an obscure address to keep you hidden.

~~~
__jal
For folks only looking for a cautionary tale, who don't worry about larger
security trends and only want to keep their own machine safe, that's a fine
takeaway.

For people who do care about network security, it is a very interesting post
for a few reasons.

If someone is harvesting IPs from NTP queries sent to Debian infrastructure
for intelligence gathering, that in itself is a big deal.

It is also a somewhat novel technique - if an attacker is placed such that
they can see the first NTP queries sent by a new install, they are well placed
to target that device before it is fully hardened/patched, because they likely
see some of the first packets sent by that install. It is really rather
clever.

It also points to the sort of correlation we need to be getting better at -
orchestration isn't just for devops weenies anymore, and this sort of thing is
only going to become more sophisticated.

I've been noodling about with making a tool for making this sort of analysis
easier, but there are a number of problems to fix, not least of which is the
sheer volume of data generated watching the wire, even on my home network
(which is a bit absurd for your average home, but tiny compared to a business
of any size).

~~~
devicenull
> If someone is harvesting IPs from NTP queries sent to Debian infrastructure
> for intelligence gathering, that in itself is a big deal.

Debian doesn't run the NTP pool.

~~~
jlgaddis
Debian does use [0-3].debian.pool.ntp.org as the default NTP servers, however.
Debian (and Ubuntu) also start up the NTP daemon as soon as the client is
installed -- even before one has a chance to change those default servers.

------
jlgaddis
There is (starting to be) discussion on a few mailing lists as well, including
pool@lists.ntp.org [0].

[0]:
[http://lists.ntp.org/pipermail/pool/2016-January/007758.html](http://lists.ntp.org/pipermail/pool/2016-January/007758.html)

------
vetrom
Nice evidence for why site operators should be running their own NTP servers.
If you run 1 server set per site which doesn't even need a strong local clock,
you mitigate this information leakage.

I'd think this is something that most network designers/engineers would get.
That said, I'd be a rich man if I had a nickel for every time I saw NTP
misconfigured.

------
imglorp
[https://www.ntpsec.org/](https://www.ntpsec.org/)

~~~
brandmeyer
Improving the security of the Internet's go-to time transfer protocol is
certainly useful. However, switching to a different protocol would not
mitigate this port scanning attack. According to this researcher, just sending
an empty packet to an NTP server's port is sufficient to result in a port scan
back.

~~~
imglorp
... Which is the sort of bugs ntpsec effort is fixing.

------
w8rbt
Unless I missed something in the article, I'm not sure this is odd or
malicious behavior. The NTP servers know your IPs because you sent packets to
them from those IPs. So you basically told them what your big random 128-bit
IPv6 address is.

And the fact that they sent packets back to you (after you sent them packets)
is not surprising either. However, if you can show that a full-blown port scan
occurred after you sent them packets, then that would be odd. I did not see
evidence of that in the article... did I miss that?

~~~
linuxguy2
Yes, I think you missed that. From the article:

"It takes less than five seconds for your address to be harvested and scanned.
The entire scan takes less than one second and scans over 100 common TCP and
UDP ports. "

~~~
w8rbt
OK. I'd like to see a full packet capture. And technically, the address was
not 'harvested'. He gave it to them when he sent them packets. The story
strikes me as very dramatic and over the top because of statements such as
that.

~~~
throwaway2048
Look in to what shodan is, its a service dedicated to portscanning the
internet. With ipv4 you can easily scan every ip, with ipv6 you need some
mechanism to discover addresses. This very much looks like they joined the ntp
pool soley to harvest and portscan addresses.

------
adekok
We've gone from anonymous people scanning you, to semi-trusted people scanning
you.

The mantra of the new Internet is "Trust no one". :(

~~~
dspillett
I don't see the people running hosts in pool.ntp.org's pool as any less
anonymous than the general internet at large.

~~~
adekok
You trust them to give you a correct time. That's at least a bit more trust
than you give any other random IP address.

~~~
Johnny555
Technically, you're trusting that collectively they are giving you the correct
time, so you're not putting full trust in every single server. A rogue server
with incorrect time will be ignored.

~~~
adekok
Does it violate your trust to scan anyone who does NTP queries?

a) I never trusted them, so this is fine and to be expected

b) I sort of trusted them, so this is surprising and not polite.

I lean towards (b).

------
yAnonymous
So I finally found an NTP server where I can set the update interval to a few
seconds.

------
CrankyBear
This is interesting, but really, why are you not running a firewall on
incoming requests to the NTP port? No fuss, no muss, no scan.

~~~
dspillett
This isn't talking about incoming request for the NTP service - it is fuller
scans to valid IPv6 addresses that have been harvested because hosts using
those addresses talked to an NTP service that logged them.

