
Keeping Time with Amazon Time Sync Service - bretthoerner
https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/
======
NelsonMinar
Details worth highlighting.. They recommend chrony over ntpd, which makes
particular sense given that chrony is designed to work well on virtual
machine. They are doing leap second smearing like Google does internally. They
don't say anything about what sort of accuracy you can expect but if the clock
source is truly local it could be very good.

It's kinda neat they're using a link local address.

~~~
ams6110
What is leap second smearing? Spreading it out over some other period, vs. a
sudden jump of one second?

~~~
irhd
> Spreading it out over some other period

Yes.

They make a second last slightly long for some time period.

~~~
zaxomi
Or shorter... Leap seconds can be negative.

~~~
duskwuff
Theoretically, yes, but it's never happened since leap seconds were
established in 1972, and it seems very unlikely to happen for quite some time.
Chances are that a lot of software doesn't actually support negative leap
seconds at all.

------
CaliforniaKarl
Argh! I wish the NTP servers were usable from outside of AWS.

Outside of North America and Europe, there aren't as many public NTP servers
as there should be. For example, according to
[http://www.pool.ntp.org/zone/south-
america](http://www.pool.ntp.org/zone/south-america), South America has less
than 50 NTP servers.

It would be awesome if the large, widely-distributed providers (not just AWS
with CloudFront, but also Akami, Cloudflare, etc.) provided public, Stratum-1
NTP servers at their POPs.

~~~
LeoPanthera
time.windows.com, time.apple.com, and time1.google.com are all publicly
accessible.

Apple's one is also available at time.asia.apple.com and time.euro.apple.com

~~~
1010011010
windows and apple tend to keep terribly inconsistent time, though, and windows
time goes nutso at leap smears.

~~~
LeoPanthera
Do you have a citation for this? I've been monitoring NTP servers for years
and Apple's are among the closest to my GPS reference.

------
carterschonwald
I’m amidst digging into getting some gps clock hardware to make sure an open
source db we're developing at work can work in some interesting situations.
(Hit me up if you wanna join the team ;) )

High accuracy time is a fascinating domain. It’s this weird mix of general
relativity, hardware, and anthropology. Cesium atoms and satellites vs
calendars vs astronomy: who wins? :)

Point being: more organizations making high accuracy time available to more
folks is a good thing!

~~~
dfc
What trouble are you having with the GPSs? Getting them hooked up to ntpd is
fairly easy after you get used to some of the common hiccups.

As far as who wins for a long time now it has been astronomers. The fact that
Google and Amazon are doing leap smearing is a nice sign that the tide may be
turning. It is very rare that you see organizations supporting leap seconds
during big standards discussions that are not astronomers.

~~~
carterschonwald
Getting a rig for some servers is easy. Figuring out a range of hardware to
test and support for a range of different types of possible users with
different budgets, from some tiny hobby user at home to the entire global
economy... have very different needs and hardware budgets. :)

~~~
dfc
The entire global economy? Maybe you should narrow your market focus a little.
NTPD and or GPSD support anything with NMEA and PPS. You should be able to
satisfy a large range of users with any ublox 6, 7 or 8T.

~~~
Veratyr
Yep, you can buy a Neo-6M with breakout board and antenna for $5 and hook it
up to a Raspberry Pi without much trouble.

Software can be fiddly but I found this guide to be helpful:
[http://www.satsignal.eu/ntp/Raspberry-Pi-
NTP.html](http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html)

Hell, you can hook it up to a NodeMCU as well:
[https://blog.squix.org/2016/05/esp8266-peripherals-ublox-
gps...](https://blog.squix.org/2016/05/esp8266-peripherals-ublox-gps-
module.html)

Bam, $10 wifi enabled GPS.

------
dfc
> uses a fleet of redundant satellite-connected and atomic clocks in each
> region

I am really curious to know what atomic frequency standards they are using.
There is something a little suspicious about the wording. I have a feeling
they mean "we use GPSDOs" and there are no atomic frequency standards
physically located in each regional datacenter.

I'm also a little surprised to see that the remote clock is stratum 4. It
would be interesting to see how they are distributing the time between their
clocks.

If someone from amazon reads this it would be neat if you posted some more
details about your setup.

------
BillinghamJ
I have been hoping for this for a long time, with Google’s TrueTime system
being so unique. Quite a lot more work would have to go into guaranteeing the
results all the way into your application, but this is still a great start.

------
anubhavmishra
When is Google announcing TrueTime as a service on GCP? I see them doing this
really soon since aws has it now

~~~
NelsonMinar
TrueTime is a lot more than just the NTP service Amazon just announced.

~~~
elvinyung
Can you expand on what you meant by that?

If you take it at face value, the TrueTime API is implementable on top of any
clock synchronization protocol. It's just coincidentally true that if you're
close to a stratum 0-equivalent clock, your skew will be low. In terms of
implementation, TrueTime daemons essentially also use Marzullo's algorithm to
estimate timestamps, just like NTP.

What am I missing?

------
decebalus1
Interesting to know if they have an SLA for this and how it's formulated.

Time is important and hard. I doubt that customers who really need precise
time would walk away from their ghetto solutions with no SLA because when
something like this craps out, it's almost always a circus.

But hey, when time precision and accuracy is a bonus not a requirement for
your AWS instance then 'free local NTP server!'

------
ajessup
I'd be curious to know how this compares to Google's (internal ) TrueTime
service[1], and if it allows for development of globally consistent but highly
available storage systems like Spanner.

[1]
[https://research.google.com/pubs/pub45855.html](https://research.google.com/pubs/pub45855.html)

~~~
wahnfrieden
Considering this coincided with announcements of multi-region multi-master DBs
(Aurora, DynamoDB) it seems likely.

------
ninjakeyboard
Does this actually work? We had horrible issues with time drifting between
services in AWS at scale in distributed systems.

Something like eg Google Spanner relies on atomic clocks - it's a combination
of hardware and software that enable the use of wallclock time.

------
benwilber0
System clock synchronization is a really hard problem but especially important
for things like live video stream encoding where the frame timestamps need to
correspond to the real wall-clock (within milliseconds) for things like sub-
second play-by-play synchronization.

~~~
SomeStupidPoint
How accurate does that need to be...?

100ms? 10ms? 1ms? 100us?

~~~
Daviey
Finance centric:

Finra (Financial Industry Regulatory Authority) requires <50ms

MiFID II (EU centric) HFT requirement can be as low as 100 microseconds
accuracy, or 0.1ms.

------
sudhirj
The amount of dog fooding that happens at AWS is impressive - looks like they
needed and built this for DynamoDB global tables (assumes clock sync across
all regions) and made it available to all instances.

------
mwarkentin
Curious if I’ll be able to replace the NTP servers inside our PCI-compliant
environment with this.

------
elvinyung
Are there SLAs for this out? Can I get TrueTime-style assumptions about my
clock skew?

------
jacquesm
Is it going to be monotonically increasing?

~~~
etskinner
I see no reason that it would ever go backward, so yes. Especially with leap
second smearing, meaning it's not only monotonically increasing, but also
never stops (never a slope of zero).

~~~
chowells
Double leap-seconds are allowed, though I can't find any evidence one has ever
happened. In such an occurrence, POSIX time would actually go backwards
afterwards.

~~~
etskinner
Is there any reason to believe they don't just smear both leap seconds?

------
falcolas
Somewhat meta, but this is a bit ironic of an offering considering the time on
their display boards at re:Invent are noticeably out of sync.

