
With 61 Seconds in a Minute, Markets Brace for Trouble - rsgoheen
http://www.bloomberg.com/news/articles/2015-06-28/with-61-seconds-in-a-minute-markets-brace-for-trouble
======
cnvogel
Wow, that's a pretty bad illustration of leap smearing. I think a much nicer
illustration would have been roughly like this (with obvious limitations
caused by the ASCII spacing):

    
    
        Real: true time, showing the 61st second
        Comp: computer time, only counting 60 seconds
    
         not smeared,
         time counted at regular speed
          ...--->|                        |<---...
        Real 54  55  56  57  58  59  60  00  01  02  03
        Comp 54  55   56   57  58   59   00  01  02  03
                 |<- smearing ----------->| 
                  time counting slowed down
                  by 20%

~~~
MereInterest
Also, who the heck decided to call it a "leap smear"? There isn't anything
"leap"ing about it anymore. If you are changing from "leap second" to
something indicating smearing, "smear second" is the logical choice.

~~~
pdpi
Well, you're turning a leap into a smear, so "leap smearing" seems like a
perfectly apt description.

------
mikeash
Why aren't leap seconds treated as a time zone change, like a daylight saving
time transition? Keep the underlying clock counting the number of actual
seconds since midnight, January 1st, 1970 (or whatever your epoch of choice
is) and just add one more second into the offset to local time. Is there some
big problem with this approach I haven't thought of, or is there just too much
history with doing it the other way to change now?

~~~
Someone1234
I don't understand why either. Counting up from 1/1/1970 and then making
adjustments for rendering only seems like a simple, reliable, and time tested
solution.

I guess people don't want a database of leap seconds and a lookup each time
time is rendered, but ultimately it seems like the best solution which is
least likely to break things.

~~~
danielweber
The nice thing about time standards is that there are so many to choose from!

One drawback of your approach is that I can't just see if a time = 0 mod 60 to
check if we are on a minute boundary.

Your way is what is used by the GPS network, IIRC, and it works for them, so
there you go.

~~~
mikeash
If your dates and times and locations vary enough, you already can't do a mod
60 check on UTC to detect a minute boundary, as various historical time zones
include seconds-level offsets from UTC. For example, US cities were on local
solar time until the latter part of the 19th century, and the changeover to
uniform time zones involved lots of seconds-level offsets. Interesting stories
here:

[http://historymatters.gmu.edu/d/5748](http://historymatters.gmu.edu/d/5748)

~~~
Dylan16807
I don't think anything to that resolution is in any time zone database, so you
would be using a custom system anyway.

~~~
mikeash
I just downloaded the standard ones from: [https://www.iana.org/time-
zones](https://www.iana.org/time-zones)

It appears to have this stuff down to the second. For example:

    
    
        # Zone	NAME		GMTOFF	RULES	FORMAT	[UNTIL]
        Zone America/New_York	-4:56:02 -	LMT	1883 Nov 18 12:03:58
    
        # Zone	NAME		GMTOFF	RULES	FORMAT	[UNTIL]
        Zone America/Chicago	-5:50:36 -	LMT	1883 Nov 18 12:09:24
    

Although this won't really be accurate, because it doesn't have every single
city. The New_York zone covers the whole eastern time zone even though all the
cities in that zone would have been different before 1883. But it is in the
database, even if it's not quite right for anything outside of the named
cities.

~~~
Dylan16807
Well that's a mess. Wrong data is worse than missing data for something like
this...

------
dijit
Last time this happened I was the only sysadmin in a fairly small ecommerce
company dependant on mysql.

The leap second caused our databases to all peg at 100% CPU/memory
consumption..

Given the wide adoption and commercial nature of mysql, I can't begin to
imagine how systems designed to be super precise will handle this..

~~~
valleyer
What was the root cause of the problem?

~~~
soup10
if(seconds>60) while(1) malloc(1);

------
Balgair
My brother used to work for an atomic clock company. Their array of clocks was
the 2nd most precise next to NIST in Colorado, he said, and when small
fluctuations in the earth occurred, their's was another point of contact for
GPS timekeeping (allegedly). Reading through the OP, I remember that this sort
of mid year leap second happened a few years back with the Japanese major
earthquake. My brother said that they had to reset all the clocks about a
millisecond due to the shifts in the rotation of the earth because of the
quake. One would think that the programmers would have written into these
market timing devices a simpler way to adjust these calculations, seeing as
these timing updates keep on happening.

------
h43k3r
If 1 second can cause so much of havoc, I am wondering, to what precision are
the servers in markets synchronized?

Are they made in a distributed fashion where it is assumed that two clocks can
never be synchronized with full precision.

PS - I am a noob in markets and trading.

~~~
kasey_junk
Clock sync is one of the areas where the markets and their participants
actually deserve their reputation for sophistication.

That said, I'd be surprised if keeping the clocks together is what is causing
the worry. Much more troubling are third party bugs (kernel, jvm, db, etc) and
incorrect assumptions in their code.

This same story can be equally applied to any industry honestly, but the
markets make for better headlines.

------
lucb1e
This sounds rather exaggerated to me. 10% of the systems will have trouble
with it? Perhaps _during_ the leap second something will bug or look weird,
but outside of that second I doubt it's even 1% that will have trouble
recovering. Also, many systems don't even support the leap second, nothing
will happen at all until they sync with time servers the next time and correct
for the second. Many systems are off a second or so anyway.

As a quick example, Javascript is not designed to handle leap seconds
correctly, so I doubt we'll ever see the value 60 out of _new
Date().getSeconds()_. Things will just be off by 1 second for a second.

Edit: Another example from the Windows Time service:

> The Windows Time service does not indicate the value of the Leap Indicator
> when the Windows Time service receives a packet that includes a leap second.
> [...] Therefore, after the leap second occurs, the NTP client that is
> running Windows Time service is one second faster than the actual time. This
> time difference is resolved at the next time synchronization.

It will just be a second off and resynchronize next time.

~~~
spacefight
If your system is one of those 10% and if your trading algo is executing
fucked up trades because of timing issues and you lose real money - I wouldn't
call that exaggerated.

Also, not sure if any of the systems mentioned in the article run JS for
critical server side code.

~~~
lucb1e
High-frequency trading and other important realtime processes that handle
money should of course be tested for this scenario (either by simply not
trading for a few seconds before and after, or by properly handling it). The
vast majority of the computer systems around the world are however not HFT
systems and the world will continue to work normally. No Y2K-like doom
scenarios needed like the article suggests.

We've been here before, June 30th 2012 also contained a leap second (see
Wikipedia). The article only forecasts doom without looking back at previous
leap seconds and evaluating its effect.

~~~
ars
> We've been here before, June 30th 2012 also contained a leap second (see
> Wikipedia). The article only forecasts doom without looking back at previous
> leap seconds and evaluating its effect.

Clearly you didn't read the article.

If you had you would have seen that they noted that last time it was on a
weekend when all the markets were closed.

~~~
lucb1e
I did know that. I just don't consider HFT to be so important compared to,
say, traffic lights and apache servers continuing to function.

~~~
Zuikaku
Unless you are someone who keeps all your money in physical cash and drives
cars for a living, there is no way HFT is less important than traffic lights;
one keeps the economy of the world flowing, the other stops people getting
driving tickets.

"Being a second off and resynchronizing next time" could be a million dollar
difference, there is no way that you can shrug that off.

As a developer for a financial firm, this leap second is an extremely big
deal; the fact that this is the first time in history that financial markets
(which are normally millisecond accurate) have had to deal with this
phenomenon will absolutely result in something somewhere going wrong.

This article isn't exaggerated; it's there to highlight what's going on for
the traders and brokers who aren't technologically advanced.

~~~
michaelt

      there is no way HFT is less important than traffic lights
      one keeps the economy of the world flowing, the other 
      stops people getting driving tickets.
    

The New York Stock Exchange is closed for 17.5 hours each day, and the entire
weekend. I doubt the world economy is going to stop flowing if they're closed
for an extra second - or even if a few people take their systems offline for
several minutes to restart them.

~~~
brobinson
Stocks are a very, very small part of the financial markets. Forex and futures
markets are open most of the week and a lot more money flows through them.
Forex alone accounts for about five trillion USD per DAY of money movement.
Stocks are barely a blip on the radar in comparison.

~~~
kasey_junk
And yet all of the major exchanges have mandated down times that even small
e-retail operators would kill for.

This thread has reached sort of silly proportions. Of course electronic
trading impacts most people's lives even if they don't want it to, but in
reality the markets are largely self correcting at this level, so even if
there is some oddity associated with the leap second it likely won't have long
term (or maybe even observable) impact on the wider world.

~~~
brobinson
Agreed on all points!

------
alkonaut
Another reason to just have artificial delays in trading. If every exchange
had a minimum of, say, a minute after an order is submitted until it is
executed (during which time it couldn't be amended or cancelled of course) it
seems you would greatly reduce the risk of "flash crashes"and unnatural
manipulation of the market.

I know "market liquidity" is an argument for allowing HFT, but haven't hard a
good explanation about why the economy would suffer so much if the ultra high
frequency trading just wouldn't exist.

Note: I have about as much knowledge about this as people who suggest how
SpaceX should improve. Just sayin.

Of course computer systems everywhere will have issues with an extra seconds
(deltas being negative that can't be etc) but most of those systems aren't
trading systems. Why is it that "markets" are in the focus of this? At y2k we
were worried about planes falling out of the sky. Surely there must be worse
things than stock market computer systems being confused?

~~~
kasey_junk
> but haven't hard a good explanation about why the economy would suffer so
> much if the ultra high frequency trading just wouldn't exist.

The short explanation is that the speed is a tool for managing risk (like most
other market tools) and by removing it, you remove that ability. That risk
must be priced into the market somewhere and that will be in the spread that
everyone pays.

Will that be worse than what we currently have? Who knows, but what we have is
working pretty well with regard to providing liquidity (it is cheaper and
easier to get in more markets than ever before and the margins are as low as
they have ever been on providing it), so why mess with something that has so
few down sides?

> Surely there must be worse things than stock market computer systems being
> confused?

There are, but Bloomberg doesn't specialize in them, and they don't garner
nearly the HN upvotes.

~~~
rectang
> why mess with something that has so few down sides?

When I look at high frequency trading, it says to me that the game is rigged.
It's like going on Jeopardy up against a machine contestant that always buzzes
in first.

How on earth can I possibly succeed as an individual investor in a trading
environment where institutional investors are so privileged? How many other
ways is the ostensibly neutral marketplace overseer profiting by offering
those who pay-to-play opportunities to shave pennies, nickels, dimes, and
dollars from me?

~~~
kasey_junk
> When I look at high frequency trading, it says to me that the game is
> rigged.

And when I hear this complaint, I don't understand the rationale. Why
shouldn't participants that do something professionally, invest in
infrastructure, invest in research or other kinds of IP have an advantage. If
they didn't that wouldn't be a market, that would be a lottery.

Quite simply, the market dynamics that make it such that you might want to be
involved in it as an individual are created by the sophisticated market
participants engaging in individual zero sum trades that add up to a
beneficial whole and they have been for hundreds of years.

Computers have made this process more efficient and you should be celebrating
your ability to trade for cheaper than ever. That you don't is a sign that you
don't understand how the markets work now nor how they have ever worked.

 _Edit: the more I thought about this, the more I decided it was overly harsh.
I think well informed participants can disagree about the values of HFT and
needn 't celebrate it. I do think that the idea that markets don't (or
shouldn't) reward more engaged participants is fundamentally broken._

~~~
sirseal
HFT is a tool of oppression. Who can afford it? Who has the time, capital, and
personnel to make and deploy HFT algos? And ultimately, who is _really_
benefitting the most from this technological advance?

~~~
djfm
Could "High Frequency Trading As A Service" be a thing? Someone provides the
infrastructure and a nice API to talk with the market and runs your algorithms
on their hyperfast computers at the heart of the stock exchange for a fee.

------
lappa
This seems to be something that would benefit from standardization and
modularity...

Pausing for a second, going back a second in time or changing how long a
second is all are hacks that are bound to cause some edge case failure
somewhere. I'd like to see a minute have a 60th second just as February has a
29th day.

~~~
mikeash
The problem is that the effect being compensated for is so tiny that it's
really noisy. It's not like leap years, where all you have to do is measure
the actual period of the Earth's orbit very precisely and come up with a nice
way to fit the appropriate correction factor into the calendar.

The Earth's rotation is slowing, but it's not a steady process. For example,
the Boxing Day earthquake in 2004 permanently decreased the length of the day
by about 3 microseconds. Here is a graph of the changes over the past few
decades:

[https://upload.wikimedia.org/wikipedia/commons/5/5b/Deviatio...](https://upload.wikimedia.org/wikipedia/commons/5/5b/Deviation_of_day_length_from_SI_day.svg)

~~~
lappa
As long as we have a standardized way of determining when leap seconds are
going to happen, some authority determining and announcing, there shouldn't be
problem. Leap years are predictable, but leap seconds can be accounted for a
decade after an event disrupts the pattern.

Also, given all that variance and the fact that we only seem to be shifting
around 1 minute per century at the current rate, we probably could get by
having a leap minute every few decades when necessary.

This works fine for any system trying to interact with other computers and
humans while keeping consensus on the time. Any system that is trying to
determine earths yaw would no longer be able to rely on F(time), but that is
only a minor inconvenience.

~~~
mikeash
How is that different from what's being done now? There is a standardized way
of determining when leap seconds are going to happen (they always happen on
June 30th or December 31st when the deviation reaches about 0.6s), there is an
authority that determines and announces it (the International Earth Rotation
and Reference Systems Service) and it is, in fact, done by causing the last
minute on the day in question to have 61 seconds instead of the usual 60.

The complication is just that there are a whole lot of systems out there that
assume that every minute has exactly 60 seconds with no exceptions, so you end
up with crazy workarounds like smearing to hide that 61st second from such
systems.

~~~
lappa
>How is that different from what's being done now? There is a standardized way
of determining when leap seconds are going to happen (they always happen on
June 30th or December 31st when the deviation reaches about 0.6s)

It's different because you don't have to keep track of what an authority is
saying or deal with the actual time adjustment except once in a century. Once
in a century is acceptable since we will only deviate by around a minute at
our current rate.

------
Lawtonfogle
Can someone please explain why this keeps being explained as 'earth's rotation
is slowing down'?

Having a scheduled addition of the leap second only corrects an issue
resulting from our timing being slightly off (by about 1 second every couple
decades). Now, if the second had to be add in changing frequency... the
changing in frequency would be a result of the earth's rotation changing...
but even still that wouldn't be why the second itself is being added, is it?

~~~
jerf
"One second every couple of decades" grossly, _grossly_ overestimates our
timing inaccuracies. Multiple orders of magnitude.

The problem is in some sense, the opposite... our timing is now so good that
we can notice the natural variations in Earth's rotation to astonishing
accuracy. Thus, _if_ we want to keep the clock in sync with what the Earth is
actually doing, we need to add the leap second.

(There's quite a bit of debate about that "if", and IMHO right now industry
after industry seems to be experiencing major, expensive disruptions and
repeatedly incurring risk, to save astronomers from having to consult a lookup
table that they of course would build into computer programs and pretty much
forget about ever after. But I digress.)

The Earth's rotation really does wobble by some amount due to natural shifts
in density, temperature changes, and any number of other things. It's why we
don't have a schedule of leap seconds for the future. If it were merely a
matter of accurate timing we'd have a schedule, because our timing is accurate
enough by orders of magnitude to make a schedule for a hypothetical perfectly-
steady or perfectly-slowing Earth for millennia in advance. The problem is
that we can't predict what the _Earth_ is going to do.

~~~
NickNameNick
Astronomers use sidereal time instead of solar time anyway, so they're used to
making time corrections, the sidereal day is only 23 hrs, 56m, 4 seconds and
change. Equivalent to the length of a day if the light from the sun was coming
from an infinitely distant source. That way at a given sidereal time, anything
far enough away will always be in the same place in the sky.

------
lifeisstillgood
I would suggest the whole thing can be avoided by the exchanges setting their
own ticks. Any and all Bid/offers are queued during the tick and then applied,
in order, during the next tick.

before I did some research I thought this would be a good "anti-HFT abuse"
idea but I still think it has considerable merits.

------
jimmcslim
The article states that the planet's speed of rotation is slowing down, albeit
very slowly. Is the implication that at some point in the (distant) future the
rotation will stop completely, or will it just reach a new equilibrium? Could
it in fact speed up again?

~~~
bad_user
Our sun will turn into a red giant and destroy earth long before earth has a
chance to stop its rotation, as we're talking about an increase in
milliseconds per century. Since 1820 the earth's rotation slowed down by about
2.5 milliseconds.

~~~
valleyer
That sounds like it contradicts with the fact that we've had dozens of leap
seconds in the last several decades. Explain?

~~~
jug
The Earth's rotation speed can be influenced by many things and can both be
sped up and slowed down. For example, the Sumatran earthquake in 2004
shortened the rotation period by 3 milliseconds. In theory negative leap
seconds could be added but none have yet.

This gives rise to problems for computers since they are irregular and
unpredictable, and can't be made part of algorithms. In extent, to properly
calculate a precise time difference (actual elapsed time) over several years,
you actually need to consult a table of added, historic leap seconds and take
these into account too...

------
nadocrew
Are there any security implications for leap seconds? Do pseudo random number
generators have any issues with 61 seconds in a minute?

~~~
delinka
I have yet to meet a [CS]PRNG that cares about the passage of time.

~~~
jessaustin
Some protocols that use prngs might make assumptions about the number of
seconds in a minute, however.

------
InclinedPlane
Why are leap seconds still considered a good idea? They have almost zero
practical value and a huge practical cost.

------
xedarius
Wonder why they don't add an average weight to the time rather than this
troublesome extra second.

~~~
bad_user
What do you mean?

~~~
to3m
You could add an extra second by having the previous 1,000 seconds (16 hours)
last 1,001ms each, say. Then the scope for confusion is more likely to be
limited to software that reads human time, and counts milliseconds, and
requires the two to match over time. Probably a much smaller percentage of all
programs.

When you know what the problems are likely to be, you can of course pick the
approach that's least likely to cause you problems! But if you're going in
blind, then this seems like it could be a safer way to do it.

(You might use some kind of a curve to do this, so it's smooth like. I imagine
this is what the comment about weighting refers to.)

~~~
ucho
That's how Google does that:
[http://googlecloudplatform.blogspot.com/2015/05/Got-a-
second...](http://googlecloudplatform.blogspot.com/2015/05/Got-a-second-A-
leap-second-that-is-Be-ready-for-June-30th.html) . The drawback is that their
clock will be "wrong" for most of that day.

------
rm_-rf_slash
People who were losing their shit over Y2K had no idea what normal life would
become.

------
tempodox
Ah, this is one high-profile example of the CAP theorem in full action (not
counting Linux kernel bugs).

------
spacecowboy_lon
As opposed to the situation in Greece I have lost over £1k since the Friday
close and its only Lunch time :-)

