
Excessive load on NTP servers - BCM43
https://news.ntppool.org/2016/12/load/
======
easytiger
Wait.. they are saying the app itself is making NTP requests?

> _Confirmed - starting up the iOS Snapchat app does a lookup to the domains
> you listed, and then sends NTP to every unique IP. Around 35-60 different
> IPs._

Hmm. Is that a fraud prevention thing or something? No way on earth a user app
should be getting its own time

~~~
cuonic
Snapchat have a lot of "fraud protection" in the form of time sensitive tokens
hashed with secret keys generated by strange .so libraries. This is used to
keep third party apps from using their API, obviously a lot of user's devices
have incorrect clocks, so when they reduced the secret token time frame lots
of users probably started getting API errors, so this is their attempt at a
solution.

~~~
matt_wulfeck
Thank you! This is the most reasonable answer I've seen in the thread so far.

------
sschueller
Why on earth would you do that?

If you want to prevent users from altering their time use your server and do a
time compare with your server.

NTP can be easily intercepted and altered so it would make a lot more sense to
do this via a encrypted certificate pinned communication path increasing my
work load drastically to alter the time.

I snapchat going to pay for the DDOS they created?

~~~
CaptSpify
Stupid decision by SC for sure, but is there a reason there is no rate-
limiting on the NTP servers? I'm not up to date on their structure. Maybe it's
just not feasible because they don't synchronize clients?

~~~
dfox
Even without thinking about storing some context, replying to NTP request is
probably similarly expensive as evaluating whether it should be rate limited.

~~~
coderholic
Yeah exactly this. Rate limiting isn't free. It's almost certainly more
expensive to rate limit than to statelessly respond with the time.

~~~
paulgear
NTP does store context for a limited number of clients (600 by default).
[https://www.eecis.udel.edu/~mills/ntp/html/miscopt.html](https://www.eecis.udel.edu/~mills/ntp/html/miscopt.html)

------
Declanomous
For whatever reason, ntppool.org is blocked at my work.

And of course, you don't get the page that states why when the website is
served via https. Not that I need to see the page to know it was either
blocked for "hacking" or "entertainment", and I'm guessing it's not
entertainment.

Edit: This probably explains why our clocks have been off by 45 minutes since
Monday. I guess it will be entertaining to see how long it takes for IT to
figure this one out.

~~~
Twirrim
> I guess it will be entertaining to see how long it takes for IT to figure
> this one out.

Why not just tell them. What have you got to lose? Hell, blame your charitable
spirit on the holiday season if you must.

~~~
Declanomous
I got told off for diagnosing issues in the past. The IT director is a
megalomaniac and interprets it as a challenge to his power. The only time I
offer suggestions now is when one of his employees specifically asks me for
help.

Edit: I realize "got told off" didn't really capture what happened. I came in
early one day and noticed we were having a dns issue. I manually refreshed my
DNS cache and it started to work. I sent him an email to let him know that the
DNS cache was expired. He told me I was out of line and complained to HR.

I had to go meet with HR, which was pointless since they think he is on a
power trip as well. Anyways he added a line to the IT policy that specifically
prohibits "performing a diagnosis on the network or any of IT managed
systems."

~~~
leesalminen
I encountered a similar IT manager in high school. I kept telling him that
netsend wasn't locked down and that any user could run a .bat.

He told me I was wrong. So, I wrote a .bat with a netsend command and emailed
it to all staff. Multiple staff clicked on the attachment.

Once they figured out it was me, they made me start a computer club with the
IT manager as the supervisor of the club. First order of business was locking
down .bat execution.

~~~
SomeStupidPoint
Im endlessly glad that my high school was extremely tolerant of us exploring
the system and messing with things as long as we didn't try to cause harm (eg,
deleting one file is okay, trashing a whole network drive is not) and reported
what we found to the school IT manager. Most of us ended up as techs for the
school and district as a student job. Several of those students went on to be
whitehats. (Who knows where careers would've gone if they'd been discouraged
and come to view the system as the enemy.)

They also were way cooler than they had to be about the several times we took
down the network or broke the porn filters, or the time we port-scanned a
district tech's machine, or had a whole collection of malware on the network
drive, or....

I just worry about how students like me would fare in schools these days.

~~~
JChase2
Me and my buddy in like 2010 ended up being questioned by police, then he
caught some charges. No fun. All we did was spoof some emails.

------
acqq
According to the forum, the pattern matched this third-party library:

[https://github.com/jbenet/ios-ntp](https://github.com/jbenet/ios-ntp)

Specifically, all the servers(!) from here are contacted:
[https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-
lib/Ne...](https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-
lib/NetworkClock.m#L121)

Note that the library author wrote:

"ios-ntp is often (mostly?) used to make sure someone hasn't fiddled with the
system clock. The complications involved in using multiple servers and
averaging time offsets is overkill for this purpose. The following skeleton
code is all that is needed to check the time."

And that "skeleton" contacts just "time.apple.com"

But the library really has the default possibility of contacting a lot of the
ntp.org servers from a big list ("createAssociations" with no parameters!) and
it's bad.

As we know, the developers like to just "copy-paste" whatever is where. Or use
any defaults. "Hey it works."

~~~
pavel_lishin
This is a really weird comment format. It almost suggests a lack of syntax
highlighting.

    
    
        /*┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
          │ Prepare a sort-descriptor to sort associations based on their dispersion, and then create an     │
          │ empty array for associations to fill ..                                                          │
          └──────────────────────────────────────────────────────────────────────────────────────────────────┘*/

~~~
TeMPOraL
Nope, I guess it's just that the developer _really_ likes comment boxes. See
e.g. here:

[https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-
lib/Ne...](https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-
lib/NetworkClock.m)

for use of two different styles of boxes, + some additional typographic
experiments.

I actually kind of like it. Not enough to start using it yet, but then again,
in Lisp code I make a judicious use[0] of ^L characters and form-feed-mode.
Form feed character seems to be a forgotten but pretty neat invention.

[0] - [http://i.imgur.com/5pnDZmJ.png](http://i.imgur.com/5pnDZmJ.png)

------
coleca
FWIW my teenage daughter has been complaining about this latest Snapchat
update for iOS the past couple days. It constantly crashes and causes the
phone to reboot itself. Looking at Twitter, there's tons and tons of people
reporting the same issue, so it seems pretty widespread. Wonder if it's
related to this NTP issue.

~~~
skeptic2718
Can apps cause iOS to _reboot_? That's a bit shocking.

I don't own an iPhone.

~~~
ayuvar
It's not isolated to iOS. Snapchat does something funky in userland.

I think on the Nexus 4, Snapchat _still_ ships with a warning that it doesn't
work properly.

Mine would reboot about every second time I took a picture.

~~~
ansible
I haven't investigated the issue, but I've heard that it was indeed a bug with
the Nexus 4 drivers that caused the crash.

Still, I've got to wonder, what are they doing that's so different than other
camera apps that seem to work fine?

~~~
snuxoll
Not using the Camera API for starters, last I checked they still took a
screenshot of the viewfinder on Android.

~~~
mi100hael
lolwut

------
sateesh
It is interesting to read through the whole thread in a chronological order
starting from the first message:
[http://mailman.nanog.org/pipermail/nanog/2016-December/08952...](http://mailman.nanog.org/pipermail/nanog/2016-December/089525.html)

It took 4 days, to zero on the root cause. As is usual in a complex scenario
like this there are a few false positives, some suspects abusing the protocol
and alas final redemption. Amazing work by a dedicated group of technical
folks in coordinating (just via emails, I suppose) and tracing the root cause.

------
lima
Worst part is that they did not bother to use a vendor zone.

~~~
profmonocle
Indeed. This is a pretty clear misuse of the NTP pool.

> You __must absolutely not use the default pool.ntp.org zone names __as the
> default configuration in your application or appliance.

\- [http://www.pool.ntp.org/en/vendors.html#vendor-
zone](http://www.pool.ntp.org/en/vendors.html#vendor-zone)

Hopefully they were just unaware of the vendor zone policy.

~~~
acqq
> Hopefully they were just unaware of the vendor zone policy.

It seems they didn't know, or didn't care, how both the third party iOS
library they used and the NTP worked, see my other posts here.

They surely didn't need ntp.org pool at all.

------
mark-r
This happens often enough that Wikipedia has a page devoted to it:
[https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse](https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse)

The first one I had heard of was Netgear vs. UW-Madison.

------
gbrown_
For all of Apple's App Store vetting one would think this kind of behavior
would have thrown up a flag at some point no?

~~~
AlphaWeaver
App Store vetting varies wildly and tends to trend towards more close
introspection on smaller apps. Well known apps such as Facebook Messenger and
Snapchat for example can get an update reviewed and pushed out faster than a
standalone developer.

~~~
pooper
I think Apple should require app vendors to just submit source code and build
instructions and have Apple just build it.

It'd be harder to pull off on Google Play Store but I think Apple could make
this happen if they wanted to.

~~~
codazoda
As a developer, I'd rather they build it and sign it for me. The key signing
stuff always seems difficult.

~~~
dan1234
I would hate that. How could ever be sure Apple (or anyone else) hasn’t
added/modified my code without consent?

I wouldn’t ever stake my reputation on signed code which hasn’t been signed by
myself!

~~~
algesten
Apple owns the hardware, OS and distribution, they can do whatever they want.
Your signature is neither here nor there.

The only thing it can do do is show Apple you compiled the code.

There's no way you as the dev or the end user can verify the installed
software really originated from you.

------
_RPM
And to think that SC's engineering is praised among college kids is laughable.

~~~
nxtrafalgar
On my device, at various points, Snapchat and Uber have both been completely
nonfunctional for days on end. I didn't think building phone apps could be so
difficult for these large companies.

------
Faaak
I wondered why I was seeing so much packet loss on my IP: [http://mrtg.vi-
di.fr/krootservers.ping.html](http://mrtg.vi-di.fr/krootservers.ping.html)

Guess I know why now..

------
thejosh
Yeah, it's been really hit and miss here in AU for a few people I know.

------
sstevo66
I do some work for the Network Time Foundation and we were not contacted by
snapchat as far as I know. Anyone have a contact there, they probably need our
help.

~~~
askbjoernhansen
People from the NTP Pool community were talking to them (including myself,
briefly). Given the available information I'm not sure why you think they need
help from NTF ...

------
1_2__3
I for one am shocked - shocked! - that Snapchat would be the kind of company
to be cavalier about this kind of thing.

------
known
Captcha should fix it

