

1 second per second is harder than it sounds (2014) - signa11
http://rachelbythebay.com/w/2014/06/14/time/

======
rwg
At ${PREVIOUS_JOB}, I ran a stratum-1 NTP server that got a timecode and PPS
signal from a GPS clock. I could tell if someone had left the door to the lab
open by what PLL frequency "ntpdc -c kerninfo" reported on that server — the
various oscillators in the server would drift above/below their ideal
frequencies depending on how hot/cold they were, and the room would get a few
degrees colder with the door open.

~~~
cpach
Cool!

I’m curious what it was that made you need that level of precision? :)

~~~
rwg
There was no need, really. The university's networking/telecommunications
group ran their own pair of stratum-1 NTP servers, plus four stratum-2 NTP
servers, so my stratum-1 wasn't really needed. I ran my stratum-1 NTP server
simply because the hardware was available and I had an interest in it. You
see, the GPS clock (and its predecessors, a WWVB clock and a pair of GOES
clocks) were relics of a time long past...

[insert wavy flashback transition here]

The lab I mentioned was a seismological observatory. In the days before cheap,
high-resolution A/D converters and computers with massive amounts of storage
existed, almost everything was analog. Gloriously, unashamedly analog.

Seismograph stations sent data from their seismometers back to the lab over a
"dry loop" — a leased line with no dial tone or voltage on it — from AT&T. To
do this, the seismometer's output signal was greatly amplified, then frequency
modulated onto a relatively low frequency carrier (1–2 kHz). The signal then
traveled through AT&T's network all the way to the lab, where the signal was
demodulated.

Okay, great, we're getting the signals back at the lab, but how do we store
these waveforms? The answer is a giant drum, a motor, some paper, and a pen or
stylus — a drum recorder.

[http://www.aeic.alaska.edu/input/west/proj/ASRA/2007/picture...](http://www.aeic.alaska.edu/input/west/proj/ASRA/2007/pictures/large/lbgoldschool.JPG)

"Helicorder" was Teledyne Geotech's brand name for their line of drum
recorders, but it was so popular that "helicorder" has become a generic name
for "drum recorder" in the seismological community, much like "Xerox" became
synonymous with "photocopier." (If you ever look at earthquake records on-
line, check for "heli" in the URL.)

A piece of paper is wrapped around the drum. A pen/stylus rests on the paper
and deflects side-to-side depending on the polarity and magnitude of the input
signal. A big positive voltage makes the pen move really far to one side, and
a small negative voltage makes the pen move not so far to the other side. The
pen is also attached to a threaded shaft that rotates, slowly moving the pen
from one side of the paper to the other. The drum itself also rotates, and the
rotation speed of the drum and the shaft was usually selectable — most people
had recorders with small drums set to record ~24 hours of data per piece of
paper, with wider drums set to record for a proportionately longer amount of
time. (The higher the drum speed, the better the record quality, but then you
had to change the paper more often.)

So we can record the signal onto paper, but we're missing a very important
thing — time. We need to know _exactly when_ stations saw ground motion in
order to locate earthquakes and other seismic events. Enter the GPS clock (and
the WWVB clock and GOES clocks before it). The GPS clock received a very
accurate time signal and was configured to output a very simple timecode known
as "slow code." Slow code works as such:

• At exactly the start of the 0'th second of every minute, generate a voltage
pulse for some amount of time, usually 2 seconds.

• At exactly the start of the 0'th second of the 0'th minute of every hour,
generate a voltage pulse for some longer amount of time, usually 4 seconds.

• At exactly the start of the 0'th second of the 0'th minute of the 0'th hour
of every day, generate a voltage pulse for some even longer amount of time,
usually 6 seconds.

This slow code would be added to the signal being recorded by the drum
recorder, adding precisely timed "bumps" to the record. When the paper was
changed on the drum every 24 hours, someone would write or stamp several
pieces of information on the paper: the seismograph station's name, the date,
and the time of the first time mark:

[http://www.hilo.hawaii.edu/~nat_haz/earthquakes/media/SeisDE...](http://www.hilo.hawaii.edu/~nat_haz/earthquakes/media/SeisDES.tif.jpg)

Note the column of time marks between the stamped dates. The narrow marks are
minute marks, the slightly wider marks are hour marks, and the widest mark
(five lines above the little earthquake) is the day mark.

[insert wavy flash-forward transition here]

Eventually, everything at the seismological observatory went digital, and the
seismograph stations were upgraded with digitizers that had their own GPS
clocks for timestamping data. The WWVB clock and GPS clock sat unused until I
cleaned them up and reconfigured them to serve up time for ntpd to consume.

~~~
cpach
Fascinating. This was a very interesting insight into the world of seismology.
Thank you for taking the time to write it up and sharing it!

------
tomkinstinch
Quartz clocks can be inaccurate and can vary greatly with temperature. Beyond
temperature-compensated crystal oscillators (TCXO) which are designed to be
reasonably stable over a prescribed temperature range, there are oven-
controlled crystal oscillators (OCXO)[1], that are held at a stable
temperature in a small can with a heating element controlled by something like
PID control. For anything requiring affordable stable timekeeping (this
includes high data rate/quantization multimeters like the HP 3458A), an OCXO
is a pretty good way to go. GPS is a fine reference to start one from and with
specialized hardware GPS itself can be used as a MHz clock. For everything
else there are TCXOs, and if you really don't care on-die RC oscillators can
be used (this is the internal oscillator on an Atmega, for example, the
default until you add an external crystal). If money is no object, atomic
clocks are of course also an option. Past press mentioned that Google uses
GPS-referenced atomic clocks to keep their data enter clocks in sync [2].

As an amusing aside, my kitchen oven clock has a poorly-chosen TCXO, such that
it is not stable over the temperature range normally experienced by an over
control panel. It's a great example of not designing to the application.
Whenever I bake something, time accelerates on the oven clock. It's like a
cookie-making time machine.

If you are interested in learning more about the hardware of crystal
oscillators, this application note from HP is terrific:

[http://www.leapsecond.com/pdf/an200-2.pdf](http://www.leapsecond.com/pdf/an200-2.pdf)

1\.
[http://www.vectron.com/products/literature_library/ocxo.pdf](http://www.vectron.com/products/literature_library/ocxo.pdf)

2\. [http://www.wired.co.uk/news/archive/2012-09/20/google-gps-
po...](http://www.wired.co.uk/news/archive/2012-09/20/google-gps-powered-
database)

~~~
comboy
> First of all, to be clear, I'm not talking about the hardware clock. That
> really only matters when you read from it, and you generally do that once:
> at boot.

~~~
tomkinstinch
That's absolute time that's read at boot, from a battery-backed real-time
clock (RTC). Relative time can still drift. Crystal resonators with
multipliers and PLL are used to supply the CPU with the signal distributed by
its clock network. The crystal is a voltage-controlled oscillator (VCO), so it
is tightly controlled (could be by voltage controlled by propagation delay
within the chip, for example), but it can still drift over time and with
temperature. As the article mentions, this can be responsible for system time
error, "Your clock's pace will always vary a tiny little bit for different
reasons". Operation of the computer shouldn't be impacted since things sync to
the clock, but absolute timekeeping can be.

~~~
comboy
My bad, I was before my coffee.

One additional thing that wasn't yet mentioned is that freqency depends also
on gravitation, so orientation of your motherboard could matter (for the
curious -
[https://www.youtube.com/watch?v=zILwgQhjC_Q](https://www.youtube.com/watch?v=zILwgQhjC_Q)
). I'm not really sure it's big enough to matter though.

------
jacquesm
You missed one: Your clock is not monotonically increasing at all times. That
one doesn't have to happen often to _totally_ screw you up when it matters.
Monotonicity is a super important property of a clock.

~~~
chris_wot
Am I missing something? Wasn't this covered by Rachel in the following:

 _S5: your clock ticks fast, slow, normally, and everything else in-between,
and changes constantly. Sometimes it gets more than 1 second per second, other
times it gets less. Even then, the degree by which it runs too quickly or too
slowly changes unpredictably._

~~~
imglorp
Yeah. The ticks just determine when the system increments its time to the next
fraction of a second.

There absolutely exist bugs that will hop the clock backwards in time. This
does wonders for certain daemon's health and will invoke colorful language
from anyone reading system logs.

I encountered a bug like this at $work; it was due to one kernel thread
reading the RTC while another thread wrote to it. But most OS's have had them.
[http://www.linuxmisc.com/3-solaris/2351bb216cda257d.htm](http://www.linuxmisc.com/3-solaris/2351bb216cda257d.htm)

~~~
chris_wot
P.S. That's quite an interesting bug report that was filed at Sun: "4514730
dosynctodr code has structure similar to game of fizzbin"

------
jamespitts
Glad to see that people appreciate time, one of my favorite engineering
challenges. We took it on at rapt.fm in order to have the beat and other
events start in sync.

A distributed tick was used on game servers and in web clients via websockets.
Max Seiden deserves credit for the work, it involved subscribing to a custom
tick emitter on the server-side, and in each subscriber, an adjustment
mechanism and local failover.

I did the javascript client work that ended up in the final release, basically
a mechanism wherein events could be scheduled a few seconds ahead, and these
would get triggered on the same tick. This way, even with the obvious latency
in the video feed, at least countdowns and music cues happened simultaneously
for everyone participating (well, when it wasn't buggy).

rapt.fm does not use this, but Precision Time Protocol is the ultimate way to
make time sync work. I intend to get to know it better for future projects.

[http://en.wikipedia.org/wiki/Precision_Time_Protocol](http://en.wikipedia.org/wiki/Precision_Time_Protocol)

[http://ptpd.sourceforge.net/](http://ptpd.sourceforge.net/)

~~~
angersock
Very cool!

At work I implemented a sort of NTP-over-ajax to help guarantee browser/server
time synchronization for displaying signals properly in realtime. I'd much
rather use websockets, but for various reasons I'm holding off on doing that
until that part of the codebase is cleaned up more.

It's proven invaluable especially with wireless devices and also people
wanting to hold up two displays next to each other and not notice a
difference.

------
davidw
Reminds me a bit of this:

[http://www.erlang.org/doc/apps/erts/time_correction.html](http://www.erlang.org/doc/apps/erts/time_correction.html)

------
panic
How expensive would it be to add a very accurate hardware clock to a CPU
instead of relying on the OS to keep time based on interrupts?

~~~
cnvogel
The most straightforward thing is to just run the CPU with an accurate clock
frequency. You'll have to replace the Quartz next to the clock-generation chip
next to your CPU/Chipset by something which is very accurate.

[http://phk.freebsd.dk/soekris/pps/](http://phk.freebsd.dk/soekris/pps/)

[https://deltafabri.wordpress.com/2015/03/08/under-60-ns-
an-a...](https://deltafabri.wordpress.com/2015/03/08/under-60-ns-an-atomic-
stratum-1-ntp-server-hardware/)

For the cost, it's about $150 or so for a used GPS with precise 10 MHz
reference frequency output, there's a different PLL IC (different from the
ClockBlock module) from Silabs which is available from Adafruit for <$10, but
I don't know if the xtal there can be exchanged for a 10 MHz frequency. The
C-revision of the used IC can be, though...
[https://learn.adafruit.com/adafruit-si5351-clock-
generator-b...](https://learn.adafruit.com/adafruit-si5351-clock-generator-
breakout/overview)

~~~
jacquesm
That's straightforward if you're a hardware hacker, it wouldn't slow me down
much but I think that for most software oriented people building your own
time-server and verifying that it works is out-of-scope.

GP probably meant to hook this straight into their PC and the CPU you refer to
is actually the CPU of a separate board rather than the main CPU (where
thermal drift will usually make the oscillator less than reliable over longer
periods of time).

~~~
cnvogel
Ok, if you want something off the shelf, then there are things like this:

[https://www.meinbergglobal.com/english/products/pci-
express-...](https://www.meinbergglobal.com/english/products/pci-express-
clocks.htm)

But I don't know what functionality they actually provide available over the
pci-express port, if one is unlucky, it's only the serial port to the GPS
receiver ;-).

------
ignoramous
Related:
[https://news.ycombinator.com/item?id=4128208](https://news.ycombinator.com/item?id=4128208)
(Falsehoods programmers believe about time)

------
lloeki
Reminded me of [0]. Whatever aspect of time you're concerned with, it's always
a thorny issue, however easy it may look like at first. And we're not even
talking about fleet of relativistically affected nodes such as with GPS!

[0]: [http://infiniteundo.com/post/25326999628/falsehoods-
programm...](http://infiniteundo.com/post/25326999628/falsehoods-programmers-
believe-about-time)

------
kokey
Reminds me of old versions of Xen with RHEL6 guests. That's a combination that
teaches you very quickly about clocksources, ntp and adjtimex.

