
Clock Skew Exists - charlieegan3
http://jvns.ca/blog/2016/02/09/til-clock-skew-exists/
======
antirez
I posted this comment in Julia's blog 5 hours ago, currently pending. I'm not
sure if it will be accepted, but I'm reporting it here:

Hello Julia, the kind of skew that is a problem for Redlock, or other
algorithms that would rely on the same clock assumptions, is not when the
computer "logical" clocks skew, for example by getting a wrong time from NTP.
By using the monotonically increasing clock API we only need to care about
failures that cause the clock to go faster / slower even if never modified.
Also note that there is no issues with clocks of different processes to be at
a totally different absolute times, as long as they continue to count the time
approximately at the same speed. If you restrict the failures to this class of
issues, then to see it in the practice becomes not impossible but extremely
rare. Most real world systems out there using proven algorithms, have to do
the same assumptions on the fact that the system works in a mostly expected
way. For example disk corruptions or network packets corruptions can cause the
same issues, even if they are rarely checksummed at application level, or
checked with a checksum length that does not make collisions impossible in the
long run. Btw kudos for trying to reason with your head instead of trusting
other people's or my words.

~~~
alblue
"If you restrict the failures to this class of issues" is exactly where the
assumption that clocks behave is hiding in this paragraph. It's a bit like
saying "as long as you ignore false, all booleans can be assumed to be true".

~~~
antirez
If you use the monotonic time API the issues _are_ restricted to this subclass
of hardware failures. All the others will be irrelevant, like ntpd setting a
random time or alike.

~~~
querulous
how does using the monotonic api protect against time discontinuity as seen
when a vm is migrated from one hypervisor to another for example?

~~~
jschwartzi
Monotonic clocks are guaranteed never to go backwards, for one. In POSIX it
would be a major violation of the API, although implementations are free to
substitute clock sources if the monotonic source is not provided.

~~~
derefr
Or, to put it more bluntly: a hypervisor that allows its VMs to be paused
SHOULD NOT expose a monotonic time source to said VMs. (In the RFC meaning of
"SHOULD NOT".) Enabling monotonic time should disable the ability to
pause/migrate/snapshot the VMs, and vice-versa.

------
packetized
Clock skew is a bad enough thing that it's accepted wisdom to not run timing-
sensitive authentication authorities (read: Kerberos, mostly) in a hypervisor
or VM container.

~~~
josh2600
This has a few of reasons:

1) all abstraction necessarily adds a performance penalty.

2) Hypervisors or VMs have core affinity, but, often, multiple VMs and
Hypervisors are pinned to the same core. Since one cpu cannot expose time to
two different vm requests at precisely the same time, there is always some
variance.

3) Process isolation is pretty good, but total performance isolation is not
quite the same.

There are a TON of papers written about VM and hypervisor time management.
It's all fascinating and hard.

~~~
packetized
Absolutely. I'm just surprised that this many people seem to be in the dark
about clock synchronicity (or lack thereof) effects on distributed systems.

------
danielhlockard
Servers, even ones that run NTP, aren't always on time. I had one the other
day that somehow got 30 minutes behind.

~~~
existencebox
It's the more subtle instances of this that always are the most worrying to
me, to be honest.

Imagine you have a data pipeline that spans multiple hosts, and you want
telemetry on when an entity passes through each step? Suddenly you have to
start caring about clock alignment in an order of magnitude comparable with
network transit/computation time (microseconds) and that level of skew happens
VERY often. There are certainly ways around this in how you build your system,
but it's something that _must_ be considered and I find often isn't.

~~~
toomuchtodo
If you're monitoring other system metrics, and your application is time-
dependent, you should be monitoring system time and ntpd stats as well.

/u/devnull42 is doing it right:
[https://news.ycombinator.com/item?id=11075883](https://news.ycombinator.com/item?id=11075883)

~~~
existencebox
For critical time dependent systems, I absolutely agree. I was unclear in that
I was referring to far more mundane issues where slight skew can be
problematic. I've seen telemetry systems in which the aformentioned deltas
would be computed by doing a timedelta() on the recorded timestamp from each
machine. With even subsecond skew, you can get some really interesting
artifacts; negative durations, that sort of thing.

