

NTP's Fate Hinges on 'Father Time' - doughj3
http://www.informationweek.com/it-life/ntps-fate-hinges-on-father-time/d/d-id/1319432?_mc=RSS_IWK_EDT

======
rythie
From the article:

"Apple Macintosh computers and servers running OSX use NTP, and Stenn said
Apple developers have called him for help on several NTP issues. In the last
such incident, he said he delayed a patch to give Apple more time to prepare
OS X for it. When they were ready, he applied the patch and asked "whether
Apple could send a donation to the Network Time Foundation," Stenn recalled.
"They said they would do their best to see that Apple throws some money our
way." But it hasn't happened yet."

Surely, he needs to say upfront that there is a consultancy fee. I'm sure most
big companies can't make donations easily, what they can do is pay for
services and products which they do all day, everyday.

~~~
pnathan
It's becoming entirely clear to me that the vast bulk of nerds running open
source projects do not have the requisite skills to operate their projects as
a sustainable business adequate to pay their own bills.

I am, broadly, of the opinion that a non-profit either needs to take up (or
form), an "infrastructure consultancy" firm with financial structure and
incentives to ensure that projects like LibreSSL, GPG, NTP, etc are funded and
maintained; some of that will involve consulting work for large firms for
large piles of money.

Anyway, I don't have a lot of swing in that field, but... it's my conclusion.
:-)

~~~
rythie
I've come to the same conclusion. It needs a set of junior consultants to do
more basic stuff for a decent amount, leaving the main person/people as the
highest cost per day - and only called in if necessary. Someone to setup
funding or support structures with companies that need/want it.

------
acqq
The article is split on 5 pages and due to the ads not easy to navigate
through. The current situation of Harlan Stenn (59), who maintains NTP alone
is on the fourth page:

"With the Linux Foundation's $7,000 in monthly cash flow, Stenn finances his
movement between his home lab, in Talent, Ore., and the NTP servers located in
San Jose, Calif. In Oregon, Stenn lives with his wife and does most of his
patch inspection, code writing, and release building three weeks a month. The
fourth week, he stays in San Jose, close to two co-location data center
providers that host NTP computers. He rents a room there to work on server and
network administration, maintain the email list, and check on server backups.

Much of the travel, room, replacement hardware such as disk drives, or needed
commercial software such as the Coverity system for security checking on code,
must come out of the $7,000 monthly stipend or be charged to his consulting
business.

Most of his 17-20 servers came out of a one-time, $10,000 grant in 2010 from
the Internet Society, a policy and technology infrastructure advisory body for
the Internet founded in 1991. Those servers are running at ISC.org in Redwood
City, Calif., which hosts BIND and several other open source pieces of
Internet infrastructure. For 15 years, it has provided space, electricity and
some management "smart hands" to host NTP operations, without charging, said
Stenn. "They would love for us to pay them," he said, and he once totaled the
monthly bill at $1,400. But ISC.org also knows the NTP project can't pay and
continues to host it, Stenn added."

And on the first page:

"Its ongoing development and maintenance now rest mostly on the shoulders of
Stenn, and that's why NTP faces a turning point. Stenn, who also works
sporadically on his own consulting business, has given himself a deadline:
Garner more financial support by April, "or look for regular work.”"

Now although he has a donate link on [http://www.ntp.org](http://www.ntp.org)
to:

[http://networktimefoundation.org/donate/](http://networktimefoundation.org/donate/)

he doesn't have a clear goal set like GnuPG had. GnuPG doesn't state the goal
because it reached it. But until then you were able to actually see how your
support moves the bar. Maybe setting the goal would help here?

~~~
mikeash
What would happen if he got fed up and quit, or was hit by a bus? I assume the
article covers that too, but as you mention it's a bit painful to navigate,
and since you seem willing to summarize/quote a bit, I thought I'd take
advantage in a most selfish manner.

~~~
gpvos
The article doesn't really say. As far as I know, basically, there is no
contingency plan. We'll just have to hope someone steps up.

------
kapsel
Its also worth checking out Poul Henning-Kamps (FreeBSD comitter) work on
Ntimed, check out:
[https://github.com/bsdphk/Ntimed](https://github.com/bsdphk/Ntimed)

 _The overall architectural goals are the same as every other FOSS project
claims to follow: Simplicity, Quality, Security etc. etc. but I tend to think
that we stick a little bit more closely to them.

This work is sponsored by Linux Foundation, partly in response to the
HeartBleed fiasco, and after studying the 300,000+ lines of source-code in
NTPD. I concluded that while it could be salvaged, it would be more
economical, much faster and far more efficient to start from scratch.

Ntimed is the result._

~~~
acqq
His post announcing his "from the scratch" effort:

[http://phk.freebsd.dk/time/20140926.html](http://phk.freebsd.dk/time/20140926.html)

The problem with all such initiatives ("we start clean") is that they
typically never even reach the feature set of the previous software
(implementing features takes orders of magnitude more time than some simple
"proof of concept") and that they often make the same errors the older
software already solved. JWZ wrote nicely somewhere about that effect
(something like "it's not 'hard to read code,' those are the actual features
implemented") but I can't find the link.

~~~
spectre256
Joel Spolsky has one of the most convincing discussions on this topic:
[http://www.joelonsoftware.com/articles/fog0000000069.html](http://www.joelonsoftware.com/articles/fog0000000069.html)

"Back to that two page function. Yes, I know, it's just a simple function to
display a window, but it has grown little hairs and stuff on it and nobody
knows why. Well, I'll tell you why: those are bug fixes. ...

Each of these bugs took weeks of real-world usage before they were found. The
programmer might have spent a couple of days reproducing the bug in the lab
and fixing it. If it's like a lot of bugs, the fix might be one line of code,
or it might even be a couple of characters, but a lot of work and time went
into those two characters.

When you throw away code and start from scratch, you are throwing away all
that knowledge. All those collected bug fixes. Years of programming work."

~~~
acqq
The short (but perfect) JWZ text is here:

[http://www.jwz.org/doc/cadt.html](http://www.jwz.org/doc/cadt.html)

"that's what happens when there is no incentive for people to do the parts of
programming that aren't fun. Fixing bugs isn't fun; going through the bug list
isn't fun; but rewriting everything from scratch is fun (because "this time it
will be done right", ha ha) and so that's what happens, over and over again."

But the longer one, containing more or less the quote I first approximated, I
just can't find. If I remeber he wrote about Netscape, the code for FTP and
how long it took to get it right in all edge cases, and then it was thrown
away.

~~~
wglb
I hardly think PHK is ADT at all. There are often good engineering reasons to
rewrite software. I think Joel's point is that it is more often undertaken for
wrong reasons.

~~~
acqq
There's the software that _works,_ he gets the money to find the bugs and fix
them, he decides to write from the scratch _something_ that certainly isn't
the replacement of the existing software except for some specific users. That
_is_ very ADT, exactly to JWZ's definition:

"This is, I think, the most common way for my bug reports to open source
software projects to ever become closed. I report bugs; they go unread for a
year, sometimes two; and then (surprise!) that module is rewritten from
scratch -- and the new maintainer can't be bothered to check whether his new
version has actually solved any of the known problems that existed in the
previous version."

[http://www.jwz.org/doc/cadt.html](http://www.jwz.org/doc/cadt.html)

He got the money to look for the bugs and to fix them (hard). He instead goes
to make the fully new undiscoverd bugs in fully new (his own) code (easy).

------
zb
_…commercial software such as the Coverity system for security checking on
code, must come out of the $7,000 monthly stipend or be charged to his
consulting business._

Coverity is free for Open Source projects, why is he paying for it at all?

------
danbruc
Can somebody shed some light on how maintaining one network protocol
implementation and possibly improving it every couple of years could consume
three million Dollars every year? We all know how easy it is to underestimate
the required effort and how quickly tiny things can blow up into major issues,
but it is completely beyond my imagination in the case of NTP.

~~~
brohee
You need drivers for time sources for a start, there are enough of these to
use full time devs for years...

~~~
danbruc
This sounds like a bad and unsustainable idea, with capital B and U. Shouldn't
it be the responsibility of the vendor to provide a driver? Why would you put
the burden to deal with all kinds of clock hardware onto someone maintaining a
network protocol implementation? I could see reasons to have one or three or
so generic drivers, but if the vendor comes up with something fancy not
covered by one of the few generic drivers it should be their responsibility to
provide a suitable driver.

~~~
gpvos
Most time sources have other uses than serving NTP time. Those are the uses
that people want to pay lots of money for. Serving NTP time isn't very
lucrative.

------
moe
So this guy gets $7000/mo from the Linux foundation for maintaining NTPd
and... that's not enough?

~~~
rxt
$7000/mo for his expenses, data center expenses and all the other associated
costs...

~~~
dingaling
He spends one week per month staying in San Jose ( where the servers are
hosted for free ) to be... closer to the servers?

It's hard to sympathize when there are such frivolous expenses. There's what,
a thousand bucks at least on travel and accommodation per month.

Going on-site is understandable when updates need to be performed that are too
risky to do remotely. On-site on a fixed schedule, however, would quickly be
over-ruled in any commercial operation.

~~~
vidarh
You couldn't get most people capable of doing the work to take on that
responsibility for a flat out salary twice that amount, so how he spends the
money is frankly relatively uninteresting - consider it a salary, and that
even if you see it as a salary and ignore his costs it's _way_ below market.

There does not exactly appear to be a queue of people wanting to take over the
reigns.

~~~
gambiter
I wouldn't call $84k/year "well below" market. Especially considering that the
project isn't under heavy development. Also, I have to echo what other
comments have said... he seems to be spending it oddly. He's choosing to live
somewhere else, even though he seems to think monthly physical proximity to
the servers is important. I currently live near work... it would be like me
choosing to move 1000 miles away and then complaining that my salary won't
cover my travel expenses.

There are at least 4 simple fixes I can think of:

1) Move him and his family closer to the data center (Perhaps difficult)

2) Move the servers to a data center that's closer to his home (Not as
difficult)

3) He could start using VMs like the rest of the world... physical access
really isn't needed except for physical security (making sure the rack is
locked and access is controlled), which data center employees can manage
(Fairly easy)

4) Fire him and find someone else who would like a $7k/month stipend to do
bugfixes (Fairly easy)

~~~
brohee

      3) He could start using VMs like the rest of the world... physical access really isn't needed except for physical security (making sure the rack is locked and access is controlled), which data center employees can manage (Fairly easy)
    

You don't seem to get that stratum 1 servers require connecting to actual
hardware as the time source (stratum 0, typically GPS). I don't know of any VM
seller with provision to hook actual hardware... And even if they did, I'm not
sure they could offer the low latency needed.

For some stuff, you just need your own rack.

~~~
moe
_For some stuff, you just need your own rack._

Well, the article does a terrible job at explaining why that rack is needed
and why it can't be in a datacenter in the same city as the NTPd maintainer.

Honestly, I have a hard time believing there is new NTP hardware coming out
every month that the maintainer then personally has to write drivers for.

And even if that was the case, what prevents that hardware from working when
it's located, say, on a desk in the house of the maintainer (I think most
timeservers are fanless and don't consume very much energy)?

Perhaps there's valid reasons and explanations for his monthly trips to that
datacenter, but the article only left me with questions.

~~~
vidarh
It would be nice if people questioning what he's paid would ask questions
first, before making claims about what he _should_ be doing...

> Well, the article does a terrible job at explaining why that rack is needed
> and why it can't be in a datacenter in the same city as the NTPd maintainer.

For starters, the majority of the servers are currently hosted for free at
isc.org. You'd save a tiny amount on his monthly travel (I don't know what his
costs are, but I used to stay in Palo Alto for weeks at a time in my last job,
and it should be doable for him to travel to, and stay in, San Jose for a week
for <$1k/month), in return for a as much or more in hosting costs unless you
can find someone willing to give him free space.

That's assuming you can find a nearby suitable data centre.

> I wonder what prevents that hardware from working from, say, a desk in the
> house of the maintainer

Nothing. If you're willing to fund sufficient low latency, high capacity,
redundant internet connections to his house. The price would almost certainly
be far higher than flying him to San Jose once a month to maintain the servers
there.

~~~
moe
_If you 're willing to fund sufficient low latency, high capacity, redundant
internet connections to his house_

What for?

Does he run public timeservers himself?

------
pbhjpbhj
Not wanting to be too blunt but how're the Linux Foundation ensuring that if
this guy dies [or decides to quit] then the situation is readily recoverable -
do they have a system where they keep credentials recorded and have access to
the data-centres and such?

Wouldn't it be considerably cheaper to hire someone at the data-centre to do
the reboots or whatever it is he's in situ to do? [What do they call that
"remote hands" or something.]

------
thoughtsimple
Apple is touting a 50 ms accuracy on their new Apple Watch. That sounds
suspiciously like the accuracy of NTP. If Apple is relying on NTP for the
watch, they would be crazy to not help out financially for its support.

~~~
nate_meurer
No watch gets it's time using NTP, and I don't even know how that would work.
Almost all radio timepieces take their time from WWVB (or international
equivalents). There may be a few that use GPS like cell phones do.

EDIT: The apple watch uses GPS.

~~~
thoughtsimple
No GPS hardware in the Apple Watch so it would be more accurate to say it uses
GPS in the iPhone. Any reference for this? It does make sense though.

~~~
nate_meurer
Hmmm, you're right, the Apple site says it uses GPS and WiFi in the iphone,
whatever that means. So I guess it's possible the watch syncs with the phone
using NTP. Good catch.

------
tech-no-logical
how does this relate to alternative implementations like openNTPd ? this guy
maintains the reference implementation of the protocol I guess ?

~~~
slasaus
From [1] I get that PHK decided not to use privilege separation because it's
not portable enough. AFAIK that's one of the main differences between OpenNTPD
and Ntimed. Well, that and precision of course. OpenNTPDs accuracy is "only"
around milliseconds [2] (which is probably good enough for anyone not using
dedicated time hardware like a stratum server). The portable version missed
the frequency adjustment code for years, but recently this has been added to
the portable version as well.

I've also read a discussion with PHK somewhere that he didn't like OpenNTPD
because it had no auth (sorry, can't find the source again). I guess this
complaint has been mitigated recently now that TLS auth is supported in
OpenNTPD [3].

/edit: not sure if it's clear but I was comparing OpenNTPD to Ntimed. But the
named differences still exist when compared to ntp4.

[1]
[http://phk.freebsd.dk/_downloads/FOSDEM_2015.pdf](http://phk.freebsd.dk/_downloads/FOSDEM_2015.pdf)
page 13 and 14

[2]
[http://www.bsdnow.tv/episodes/2015_02_11-time_for_a_change](http://www.bsdnow.tv/episodes/2015_02_11-time_for_a_change)
at 30:30 and 34:00

[3]
[http://undeadly.org/cgi?action=article&sid=20150210103656](http://undeadly.org/cgi?action=article&sid=20150210103656)

~~~
floatboth
ntimed is a weird project IMO, because it's not different enough from
OpenNTPD: both are small NTP daemons written in C. He could've created a fork
of OpenNTPD with more precision and without privilege separation instead.

If ntimed was written in Rust though... THAT would've been excellent.

~~~
gpvos
OpenNTPD is only a client. ntimed is currently only a client, but is intended
to grow to a complete replacement of ntpd.

~~~
slasaus
That is not true. OpenNTPD is a server as well. Just put the following line in
your /etc/ntpd.conf:

    
    
      listen on *
    

See ntpd.conf(5) for all options: [http://www.openbsd.org/cgi-
bin/man.cgi/OpenBSD-current/man5/...](http://www.openbsd.org/cgi-
bin/man.cgi/OpenBSD-
current/man5/ntpd.conf.5?query=ntpd.conf/man5/ntpd.conf.5?query=ntpd.conf)

------
smackfu
Maybe I missed it, but I don't understand why one of the companies that
creates NTP devices doesn't just hire him. They surely already have plenty of
engineers on staff, what's another?

Is it that he doesn't want to work for a company?

------
jbb555
Too many pop up adverts on that page. I gave up on reading the article, it
wasn;t worth the utter spam

~~~
doughj3
Sorry, I tried to find an alternate article but this seems to be the original
and the only other one I found was blogspam based on this. I'm a bit annoyed
with the ads and the fact it's split over 5 pages, also (I even checked the
"Printer Friendly" link to see if it would display all on one page, but it
didn't). Still, I thought the content itself would be interesting here.

------
imaginator
tl;dr: Timelord needs financial support.

------
ChrisAntaki
When I knock on someone's door, and then hand them a hot box of pizza, they
smile and often hand me money: $5's, $10's, sometimes a $100. I'm just doing
my job, just like this Harlan "Father Time" Stenn. I suppose we take for
granted, that which we can't see.

