
One second per second is harder than it sounds - luu
https://rachelbythebay.com/w/2014/06/14/time/
======
ChuckMcM
For a long time synchronized clocks was the networking code's equivalent joke
to Regular Expressions ("Now you've got two problems ...") That said, having
implemented a 100ns accurate clock using the PPS on a $30 GPS module
(Adafruit) and a Beaglebone Black (multiple) these days there are some
interesting alternatives. It would be interesting if the Open Compute folks
would specify a 1PPS GPIO input as part of the spec, you could easily deliver
rack level 1PPS accurate signals throughout the data center pretty cost
effectively (a few wires, a transmitter, a TDR tool. Not that it will ever be
a 'solved' problem but it certainly makes some transactional protocols safer.

~~~
rdtsc
I've used PTP with special timestamping hardware (network cards) and could
synchronize machines on a LAN pretty well. Now that doesn't guarantee global
sync just sync between machines.

GPS NTP server can be obtained for as low as $600 or so. So those are becoming
not as exotic now.

~~~
ChuckMcM
So FWIW my beagle bone _is_ a "GPS NTP server" and it was built out of the
following:

1) BeagleBone $55
([http://www.adafruit.com/product/1876](http://www.adafruit.com/product/1876))

2) Proto cape $10
([http://www.adafruit.com/products/572](http://www.adafruit.com/products/572))

3) Power supply $8
([http://www.adafruit.com/products/276](http://www.adafruit.com/products/276))

4) Enclosure $7 ([http://www.makershed.com/products/beaglebone-black-
enclosure](http://www.makershed.com/products/beaglebone-black-enclosure))

5) GPS board $40 w/ PPS
([http://www.adafruit.com/products/746](http://www.adafruit.com/products/746))

6) Ext Antenna ($13)
([http://www.adafruit.com/products/960](http://www.adafruit.com/products/960))

7) SMA to Antenna ($4)
([http://www.adafruit.com/products/851](http://www.adafruit.com/products/851))

So what $137 + tax and shipping, and the Robert Nelson Ubuntu pack with
Openntpd running on it. The trick though in data centers is always getting
visibility to GPS satellites. I wonder when the data center guys will figure
the can add a 'cross connect' to the GPS on the roof for $75/month or
something :-)

~~~
rdtsc
That is pretty neat. I got my Beaglebone but haven't quite figured out what I
am going to do with it. A GPS time server is a neat idea.

The price sounds about right. But, you also used your expertise, time and
labor. Then add an enclosure box, documentation, support line, CE+FCC
certifications and it can quickly get up there in price.

~~~
ChuckMcM
Oh absolutely, I completely get why folks sell them for $600 each. (They were
$25,000 each at one time which I found to be rather extortionate). I used to
run ntpd on my home server to keep the machines in my house synced until the
reflection attack using it came out and folks were dragging down my cheezy
home net using it to attack others. Now I only run ntp inside the firewall :-)

------
kiram9
There is a standard to keep time on LANs now: IEEE-1588 which is built into a
lot of new network cards. It enables the NIC to time stamp incoming packets
which can be used to achieve sub microsecond timing.

Normally it can be implemented with a single GPS receiver on the network that
then feeds all servers/devices. It is starting to become very common in
broadcast applications as a reference. Furthermore most manufactures already
offer some application notes/libraries to make use of it. And linux offers
PTPd which implements the standard.

[http://ptpd.sourceforge.net/](http://ptpd.sourceforge.net/)

[http://en.wikipedia.org/wiki/Precision_Time_Protocol](http://en.wikipedia.org/wiki/Precision_Time_Protocol)

[http://www.nist.gov/el/isd/ieee/ieee1588.cfm](http://www.nist.gov/el/isd/ieee/ieee1588.cfm)

~~~
bradknowles
PTP is a good choice for synchronizing clocks on a per-segment basis, but it
doesn't work across a WAN and there are lots and lots of devices out there
that don't support it.

NTP is good for doing time sync across the WAN, but to get the most out of it
for certain applications you need to hook it up to reference clocks.

So, NTP can use PTP as a reference clock, and Bob's your uncle.

------
dm2
28 people killed because a clock was off by .34 seconds (system hadn't been
restarted in 100 hours)

[http://fas.org/spp/starwars/gao/im92026.htm](http://fas.org/spp/starwars/gao/im92026.htm)

~~~
manojlds
Isn't it actually "28 lives couldn't be saved" because of the clock issue.
Also, the article states that the defense was never tried against SCUD and
could have failed nevertheless.

~~~
spingsprong
It probably would have failed.

The Gulf War Patriot was designed to shoot down Mach 2 jets, not Mach 5 Scuds.

Detonating slightly late against an aircraft is actually a good thing, since
it makes the Patriot explode near the aircraft's middle instead of its nose.

A Scud is a short range ballistic missile. Once it's burnt out its fuel, it
just carries on under the influence of gravity. From that point on, the only
important part is the warhead in the very tip of the nose.

Detonating by the mid section of a Scud means you're just shredding empty fuel
tanks. And the huge speeds involved might actually make Gulf War Patriot
missiles detonate against the rear of the Scud and just shred the non-
functioning engine, or even detonate behind the Scud entirely.

~~~
umeshunni
"...the delay in distributing the software from the United States to all
Patriot locations was due to the time it took to arrange for air and ground
transportation in a wartime environment."

Fascinating to think that just 23 years ago, the only way to transmit the
updated software was by physically transporting it from one location to
another.

~~~
pjc50
I suspect you still can't do OTA updates of this kind of mission-critical
system. If you can, I'd be very concerned about security.

~~~
YokoZar
I don't see any reason the software update process needs to be more secure
than the firing process, and presumably these batteries can be fired remotely.

------
chubot
I think the question is: when does it matter? What applications rely on an
accurate clock? I guess it would matter if you are running some algorithm on
logs that uses time as an input. Interested to hear any other examples.

DJB has a clockspeed package that addresses this problem. Anyone used it?

[http://thedjbway.b0llix.net/clockspeed.html](http://thedjbway.b0llix.net/clockspeed.html)

It seems interesting but I haven't (yet) run into the need for super accurate
clocks (or perhaps I am not running enough machines).

~~~
keeperofdakeys
As soon as you have any kind of highly-distributed system, accurate clocks are
usually very important. When events can start on one machine, and end on
another, it's important to have some kind of consistency (knowing that one
event occurred before another). Probably the most famous paper on this is
Lamport's "Time, clocks, and the ordering of events in a distributed system"
\-
[http://dl.acm.org/citation.cfm?id=359563](http://dl.acm.org/citation.cfm?id=359563).

But to give a practical example. On something like facebook, if you receive
two comments on a post within a millisecond, how do you determine the order
that they appear in the list? Each request may hit a different datacentre in
the world, yet somehow the machines determine the ordering, and every request
sees that ordering from then on. You can't just have a single server, because
that couldn't handle the load for the entire world. These kinds of things
require very strict temporal ordering, which is why time is such an important
thing. This leads to the CAP theorem, and distributed systems in general.

~~~
chubot
So that's exactly why you don't depend on hardware clocks -- you use software
techniques when your algorithm depends on some notion of time.

The point of Lamport's paper is that time in distributed systems is a partial
ordering, not a total ordering. If you are taking values from hardware clocks,
you are using a total ordering.

~~~
tlarkworthy
Google spanner is a DB that uses atomic clocks distributed world wide. They
seem to think the pain involved dealing with partial ordering is not worth the
hassle given there genuinely is a total ordering on operations.

[http://en.wikipedia.org/wiki/Spanner_(database)](http://en.wikipedia.org/wiki/Spanner_\(database\))

~~~
dllthomas
_" given there genuinely is a total ordering on operations."_

I thought relativity says there genuinely _isn 't_ (but it's close enough we
can fudge it...)

~~~
tlarkworthy
ha! I think its ok as long as all parties involved are travelling on the earth
and not around it near the speed of light

[http://en.wikipedia.org/wiki/Relativity_of_simultaneity](http://en.wikipedia.org/wiki/Relativity_of_simultaneity)

~~~
dllthomas
Right. Like I said, we can (presently) fudge it. When we get moving faster
that ceases to be the case.

------
spydum
Since the advent of virtualization (most notoriously VMware ESX) time keeping
has gotten to be much more difficult. I know back in my unix admin days, I
spent more than a few days on problematic Linux kernels on VMs under ESX (made
far worse when hosts were over provisioned and VMs were starved for CPU
readiness). I know later enhancements like tick-less kernels may have helped,
but I do not envy anyone who has to wrestle these beasts.

~~~
MichaelGG
Windows time sync only cares about being accurate to 5 !minutes, for Kerberos.
HyperV and makes Windows not even able to be that accurate. Oddly enough,
Linux guests have no trouble with subsecond accuracy.

I'm told the w32time codebase is not something MS employees like to look at.

~~~
72deluxe
I don't think that w32time is something that users like to look at! I recall a
situation where you run w32time on a server and it runs fine and doesn't
report an error, but still silently ignore the option you set as it is using
another time source from another machine that it has designated a master.

I could be wrong, but I was disappointed that the utility didn't return any
errors when it knew that it was going to ignore you.

------
lamontcg
Run a cronjob every hour which:

a. ensures ntpd is running and starts it if its not.

b. checks `nptq -rv` and looks for the self-reported state of ntp and bounces
if it is not sync'd.

c. 'manually' checks ntpdate for the drift against a known stratum 1 and
bounces the daemon if the drift is out of bounds (something like 50ms is a
pretty good cutoff since ntpd normally does much better than that).

You can run this out of chef/puppet/whatever but it needs to run with a
frequency of about an hour (faster than that and ntpd often won't settle well,
and slower than that and you can be needlessly waiting too long to fix
problems). Running it as a cronjob uncouples it from how often your config
management runs.

This will catch all kinds of issues -- crashing ntpds, ntdps that randomly
lose sync, kernel problems that keep them from keeping sync, ntpds that lie
about their sync status, etc.

I set it up to e-mail me as a form of monitoring (with heavy-duty procmail
filters).

Had 30,000 servers with this script running against it. I think 6 of them had
shitty clocks that were getting reset every single hour, but were still
keeping time correctly to under a second. Every couple days there would be a
server that'd flip out and send a few e-mails until it quieted down and
synch'd. At one point we had a shipment of several hundred new servers that
all had issues with kernel drivers and the cronjob was bouncing ntpd
constantly on them until a kernel upgrade fixed the problem.

I also used to bounce ntpd once a night, but had to take that out because it
would cause non-monotonic slew in the clock which timers around service calls
would turn up as negative seconds that due to unsigned int conversion would
turn into 4 billion second p100 times.

You do want to be careful about things like network outages, if you can't ping
your upstream stratum1/2 its probably better to leave things as it is. And
again, you really want to limit the frequency that you bounce it at, and you
want to be aware of servers where it gets into a state where you're bouncing
ntpd constantly.

~~~
easytiger
This is a terrible idea when dealing with an application other than "users
checking the time on their system clock".

Performing hard resets of the clock continually is a catastrophic
infrastructure failure. A high performance 24/7 application that relies on sub
millisecond let alone sub microsecond accuracy across multiple physical
systems will be utterly destroyed by this approach.

~~~
jacquesm
Not just that, monotonicity could be impaired.

~~~
wlesieutre
To elaborate, monotonicity means that a value is either always increasing or
always decreasing, and never changes direction. If a program is built on the
assumption that time always goes forward, it may not like you setting the
clock back 0.000000001sec every hour.

~~~
shadowfox
Many license management software are especially bothered by time changes of
this sort.

~~~
wlesieutre
Very true, a lot of software with time limited trials or features like "check
out a network license for 30 days of offline use" used to be easy to bypass by
turning your clock back so that it never expired.

This is pretty widely fixed now, but often overzealously. Clock changed
unexpectedly? User is a dirty pirate and the license should deactivate.

------
kijin
In the age of mobile phones that can get the time directly from GPS satellites
with sub-100ns accuracy, the isolated hardware clock of a typical server
strikes me as an arcane method of keeping time.

Is there an expansion card or USB device that can get the time from GPS and
feed it to something like ntpd (and adjust for leap seconds from data
downloded from other networks)? Or would that be impractical because of all
the walls and metal cages that surround the typical server? What about an
antenna on the roof of the datacenter that supplies GPS time to every server
in the building?

~~~
rachelbythebay
Datacenter GPS-backed time sources exist. You can get little boxes which sit
in a rack and hook to an antenna and feed nice time to the rest of the place.
Those other machines pick it up with ntpd.

The problem is when your system clock is so far out of whack (and/or
unpredictable) that even ntpd won't help you. Get enough supposedly-identical
machines running and this will probably bite you too.

~~~
kastnerkyle
This is 100% true. If you have ever watched ntpd ignore a stratum 0 clock (and
every other time source) because some stratum 6 device reports the wrong time,
you know the pain. Yes, this isn't supposed to happen, and yes it has happened
to me many times.

------
rdtsc
You can buy GPS receivers and install antennas. Those are pretty good but
expensive.

For S2, it can be configured to never jump. Instead you can just the time
manually (run the equivalent of 'ntpdate <server>') at boot perhaps. Then
don't start your software until ntp has been synchronized. Then maintain a
watchdog that will alert you if you drop to a too low of a stratum.

Don't have an answer for S4 and S5. But I have heard that chrony is a good
replacement for ntp (same network protocol but different discipline, faster
conversion).

[http://chrony.tuxfamily.org/](http://chrony.tuxfamily.org/)

For a whole new other thing there is PTP
([https://en.wikipedia.org/wiki/Precision_Time_Protocol](https://en.wikipedia.org/wiki/Precision_Time_Protocol)).
This is good if you want to synchronize machines to each other to withing a
couple of hundred micros (maybe more with special network cards and switches).

~~~
kastnerkyle
The question is - how often do you reboot? For long running high uptime
applications "ntpdate at boot" doesn't cut it... and the other solutions are
nasty. In my experience, ntpd will ignore clocks almost at a whim even if they
are stratum 0! Some stratum 6 clock decides to report a different time, and
suddenly all your other sources are ignored until an ntpd restart. If your
clock drifted too much... even ntpd restart with options to "force the time"
can fail if the clock is too far out of whack. The you are back to forcing
ntpdate - hope none of your processes are time dependent when system time
jumps by 3 or 4 seconds.

Half of the GPS PCI devices require you to hack ntpd directly to get anything
working, and the Ethernet sources have a strange tendency to go belly up after
a year or 2 of running on non-120V sources. Not to mention that if you run the
antenna cable too long, you might not notice that your GPS amplitudes are a
little low until the next cloudy day. Or if someone adds a new hunk of metal
on your roof, or blocks line of sight to some portion of the sky...

Long story short, time is a nightmare and I am glad other people think so too.

~~~
onli
That is the nice thing about chrony. It tries to synchronize clocks even when
offline, it works not only on a reboot. If it works as advertized (and I tried
it some years ago at home and it worked there, for what it's worth), it should
fix all of the issues described in the article.

~~~
kastnerkyle
Very nice - I will have to check it out! Thanks for the tip.

------
socialist_coder
I have a problem in my AWS Windows cluster with the clocks as well. We have a
mobile game with a tournaments feature, and tournaments can be as short as 5
minutes. So, it's important that all the clients get the same "end time".

Normally, at least 10% of the VMs are between 5 and 20 seconds off from the
average.

By default, the VMs are syncing their time to an NTP server, but it just
doesn't seem to be enough. I haven't changed any settings as of yet.

What are some of the things I could do on Windows box to sync up the clocks in
my cluster? Scheduled shell scripts to force more clock syncs? Or maybe
something on the application level?

~~~
bradknowles
First off, Windows isn't the greatest client for trying to get good timesync.
The way the OS usually handles interrupts isn't sufficiently precise and
accurate. That said, it's not uncommon for real-hardware Windows clients to
get decent timesync down into the double-digit millisecond range -- usually
about ~40-50ms or so.

Second, you're trying to run them in a virtualization environment, and that
plays holy hell with whatever good timesync you might otherwise have been able
to get. And AWS is probably about as bad as you could reasonably get in this
regard -- VMWare has had a long time to get to where they are with regards to
timesync for client OSes, and that's not particularly good.

So, your best bet may be to run NTP clients on the Windows machines, but to
set them up to run at an artificially high rate of updates. Or maybe even run
ntpdate periodically against good real-hardware time servers that are
available to you.

You're starting off with both hands tied behind your back, and that's a hard
place to come back from.

------
easytiger
Err no mention of PTP at all?

There are hardly any oscillators which can be plugged into a host which have
any decent holdover. One should never expect them to run in reasonable time.
Standard TCXO prob losses 100 ms a day. Rubidium loses 0.01 ms a day. Time
providers include the perceived accuracy of the clock in the protocol so that
the client can make a decision about which time source to use.

Relying on the oscillator is never done in time critical applications except
in emergency states where GPS is out of service.

Of course a lot of people run relative clock domains, which is just a pretty
bad idea for most applications.

------
ars
I had an S5 motherboard (ASRock P55 Delux), after pulling out my hair I
finally gave up on it and bought a new motherboard.

And I made a resolution to never ever buy the V1 of anything. (It was a brand
new Intel chipset, and a brand new motherboard design to go with it.)

If everyone did like me and never bought V1 then there would never be a V2, I
don't have a solution for that, but I refuse to be the guinea pig.

~~~
makomk
I had a desktop PC a few years back that exhibited S5. Can't remember the
exact details, but I think it turned out that Linux was using a time source
which wasn't stable on that particular hardware and switching to a different
time source fixed it. There's been a lot of issues like that in the past, most
of which are thankfully no more.

------
feld
FreeBSD timecounter in detail:

[http://phk.freebsd.dk/pubs/timecounter.pdf](http://phk.freebsd.dk/pubs/timecounter.pdf)

------
jchonphoenix
Um... In large scale systems... vector clocks? Or am I not reading this
correctly (I skimmed)

~~~
MrBuddyCasino
Not sure why downvoted, vector clocks are an interesting alternative. Its just
that they won't fix your log timestamps or kerberos clock skew errors.

