
Living Without Atomic Clocks (2016) - bobbiechen
https://www.cockroachlabs.com/blog/living-without-atomic-clocks/
======
hinkley
My first long-term job involved collecting telemetry data from equipment and
displaying it.

The display was technically challenging, but in the meaty way that developers
often relish. The clock skew, however, was not.

For liability purposes it was sometimes necessary to know if Event A happens
before Event B, which means you have to normalize all of the events across
time zones and then correct for drift too.

That experience and a bunch of others (including statistics classes, and
studying the Java Memory Model, which other languages have borrowed or stolen)
have left me with a lingering doubt about how we record distributed
activities.

I really kind of feel like we [all] need a new data model [like the one they
mention here] where dependent events are recorded in that way. I don't know
exactly what that would look like, but I think it would help a great deal in
consensus situations where you have to resolve a conflict, or even just for
displaying a sequence of events in proper order.

It feels like we keep trying to get the exact nanosecond when something
happens, but the only thing I ever see humans use that information for is to
reverse engineer a sequence of events that resulted in a peculiar state in the
system.

[edit: tie-in to article]

~~~
ottumm
I think you might find this seminal paper interesting and relevant:
[https://amturing.acm.org/p558-lamport.pdf](https://amturing.acm.org/p558-lamport.pdf)

From the abstract:

> The concept of one event happening before another in a distributed system is
> examined, and is shown to define a partial ordering of the events. A
> distributed algorithm is given for synchronizing a system of logical clocks
> which can be used to totally order the events. The use of the total ordering
> is illustrated with a method for solving synchronization problems. The
> algorithm is then specializedfor synchronizing physical clocks, and a bound
> is derived on how far out of synchrony the clocks can become.

~~~
hinkley
I will read that, thank you.

You know, it seems like most of my quality of life improvements over the past
20 years has been due to my peers and I finally acting on much older
information. The future is here, it's just unevenly distributed.

It is only infrequently I encounter something that still feels properly new
under any kind of scrutiny, instead of revealing itself to be a refinement of
something that already was known. Off the top of my head, I can think of
escape analysis, Burrows Wheeler transform, and the object ownership semantics
in Rust. I'll throw Raft on there since the joke is that only 12 people
understood Paxos.

------
bitwize
> In essence, it provides a means to absolutely order events, regardless of
> which distributed node an event originated at.

Einstein said you _can 't_ absolutely order events -- and with widely enough
distributed systems and small enough time quanta, sooner or later you're going
to run into relativistic implications.

Probably not a problem for most applications we currently have to deal with,
but one day -- soon enough that we're already giving new protocols names like
"Interplanetary File System" \-- our databases will spread out among the
stars, and how will we handle time and event ordering then?

~~~
Dylan16807
Everyone on Earth is in basically the same reference frame. They can easily
agree on a standard event ordering.

Even between stars the difference isn't that big, but offsets don't matter at
that scale anyway. When it takes a decade to send an email from one system to
another, it doesn't matter if their timelines are offset by a week.

~~~
JdeBP
You are some several decades in the past. The fact that everyone on Earth is
_not_ in the same reference frame affects your everyday life nowadays, because
it affects Atomic Time and that in turn affects everything that is based upon
Atomic Time.

Since the 1970s, TAI construction has had to compensate for the differences
between the physical locations of the atomic clocks in laboratories around the
world and an ideal surface of equal gravitational potential around the world.

Nowadays, BIPM and other laboratories routinely talk about general
relativistic corrections _across the width of the measuring devices_. To quote
Appendix 2 of the _SI Brochure_ :

> _In 2013, the best of these primary standards produces the SI second with a
> relative standard uncertainty of some parts in 10^16. Note that at such a
> level of accuracy the effect of the non-uniformity of the gravitational
> field over the size of the device cannot be ignored. The standard should
> then be considered in the framework of general relativity in order to
> provide the proper time at a specified point, for instance a connector._

\-- [https://www.bipm.org/en/publications/si-
brochure/](https://www.bipm.org/en/publications/si-brochure/)

~~~
Dylan16807
And despite that, we already have a single unified clock standard that we
follow!

Being able to measure the drift at a specific point doesn't mean it's relevant
to computers timestamping their calculations. If a computer is ten nanoseconds
off, that's basically the same as it being one rack to the left, or having
some slack in the cable. There's no real effect.

Just syncing your clock once a day is enough to let you completely ignore the
effects of relativity.

------
kens
CuriousMarc made a video last week where he uses a vintage HP 5061A atomic
clock and explains how it works. It's interesting to see that atomic clocks
aren't just some giant laboratory thing, but a product you could buy, even in
the 1960s.

[https://www.youtube.com/watch?v=eOti3kKWX-c](https://www.youtube.com/watch?v=eOti3kKWX-c)

~~~
tyingq
CDW has a Cesium reference clock for $92k.
[https://m.cdw.com/product/microsemi-5071a-high-
performance-t...](https://m.cdw.com/product/microsemi-5071a-high-performance-
tube-cesium-clock-with-48-vdc-psu/5654336?enkwrd=Cesium)

Pretty similar form factor.

~~~
vibrolax
I believe this is the direct successor to the HP 5061A. HP spun off Agilent,
who sold the time/frequency product division to Symmetricom, who was then
acquired by Microsemi. It seems that the inflation-adjusted price of the
Microsemi 5071A is about the same as the ~1970 catalog price of the HP 5061.

------
bogomipz
A long time ago I worked in a finance/trading environment and we had GPS clock
receivers in our datacenters that synced from satellites. While they weren't
that expensive I would imagine at Google's scale they would might be. Is there
a reason Google wouldn't have used this same technique if it achieves the same
accuracy?

~~~
scottlamb
iirc, one big reason is leap second handling. The closed-source GPS receivers'
firmware tend to behave strangely around leap seconds, in a way that
contradicts their documentation. And their behavior is untestable in advance:
a global satellite network for time synchronization is the ultimate un-
mockable time bomb input.

~~~
bogomipz
Thanks but isn't that a reason to forgo using GPS receivers? Google obviously
decided to use them however.

~~~
scottlamb
Oh, sorry. I was distracted and misread your comment. I was trying to answer
the question of "why an atomic clock rather than a GPS receiver" (and Google's
"Armageddon masters" have their own atomic clocks), but I see now you were
asking "why not use Cockroach's technique rather than any specialized
hardware".

I think one answer is in this blog post:

> A simple statement of the contrast between Spanner and CockroachDB would be:
> Spanner always waits on writes for a short interval, whereas CockroachDB
> sometimes waits on reads for a longer interval. How long is that interval?
> Well it depends on how clocks on CockroachDB nodes are being synchronized.
> Using NTP, it’s likely to be up to 250ms. Not great, but the kind of
> transaction that would restart for the full interval would have to read
> constantly updated values across many nodes. In practice, these kinds of use
> cases exist but are the exception.

~~~
bogomipz
Thanks for the clarification, somehow I missed this tradeoff summary. Cheers.

------
aidenn0
I'm curious what the distribution of time offsets between servers running NTP
is. Properly functioning NTP should maintain an offset well under 100ms, but
what fraction of the time servers are "properly functioning" is a question I'm
interested in knowing.

~~~
sethammons
Talking with ops folks at work, they have seen system clocks in our fleet
(which run ntpd) suddenly report times off by _years_ and then go back to
"accurate." We have thousands of nodes for the software I work on. We
sometimes get negative times when comparing events, if only by a few seconds.
You can't fully trust time in a distributed system.

~~~
aidenn0
Sure, but it's also possible to write a word out to memory and then read
something different back. There are probably many DBs that fail to meet their
requirements when that happens. My sense is that time errors are much more
common, but I'm interested in actual data surrounding that.

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=11120744](https://news.ycombinator.com/item?id=11120744)

------
ckocagil
It shouldn't be too expensive to build and deploy GPS disciplined oscillators
based on commodity crystal oscillators (temperature compensated or ovenized).
Only one would be needed for a whole datacenter, then local NTP would provide
it to the LAN. No need for fancy telecoms grade Rubidium/Cesium stuff.

~~~
jrockway
The fancy oscillators are for longer "holdover" time. It's essential that
frequency not go out of spec for CDMA (etc.) networks even if the GPS signal
is degraded for whatever reason, so they use oscillators with large holdover
times to buy themselves an extra layer of redundancy.

I don't know if you need that for distributed transactions or not. The logic
as to whether or not the time is "good" is probably a large part of the
complexity of this scheme. Better hardware makes the software simpler and
vulnerable to less failure modes.

------
thedance
The latest entry in the surprising crowded genre: How can we copy Spanner, but
without all that pesky correctness?

~~~
jordanlewis
The post is from 2016.

~~~
thedance
You're right, this is the one that started it all. Sorry for being so jaded;
I've been through two distinct efforts of "Spanner, but without thinking" in
my recent career.

~~~
dang
Since you obviously know about this, it would be good to share some of what
you know, so the rest of us can learn. Dismissive comments without information
lower discussion quality, not to mention frustrate curious users.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

