I'm still tryin to understand why all my servers seem to be ok even if they have kernel that should be affected and some of them are running mysql... For example one of them is a debian kernel 2.6.32 running mysql and ntpd, and i see in dmesg Clock: inserting leap second 23:59:60 UTC but the cpu load is ok...
The leap second was scheduled in January. That event is so unusual you might get worried. So you do a simple google search and find out that there was a critical bug in Linux kernel last time leap second was inserted. People got worried rightfully. I don't know about debian, if it was known prior, if it is the same bug as before, ... But I don't run Debian, you do.
Don't get me wrong, I wouldn't too by default. But do you remember Azure crashing on February 29th? And checking for that date is a matter of three conditions. Leap second is much more complex. I'm not trying to be a smart-ass.. I'm just saying it's something I would worry about and would try to find something about it. And perhaps it wouldn't lead anywhere with Debian.
And still, something in your app stack could crash on this as well, leaving the kernel patching pointless.
Google uses a "leap smear" and slowly accounts for the leap second before it happens. As long as you are not doing any astronomical calculations or constrained by regulatory requirements I think google has the right idea.
As part of Google Compute Engine we provide an NTP server to the guest which is based on Google Production time. As such our VMs get to take advantage of this leap second smearing implementation. I was going to mention this at my talk at IO but forgot.
Not surprising. In spite of all press that Y2K was just a silly waste of money, its events like these that makes me suspect it would have been a much bigger deal if everyone had ignored it and fixed it after things where shown to break.
A lot of engineers spent a lot of time successfully fixing Y2K bugs.
Because nothing well known blew up, many people wrongly assumed that Y2K was never a real problem to begin with.
 I moved a Fortune 100 manufacturing company's database off an ancient mainframe that would've been disastrous come Y2K. It went smoothly and was thus a thankless job. They paid well though (mid six figures - those were the days).
Why does everyone always say Y2K wasn't an issue ? I'm sure there were a lot of consultant making too much money with little work - however _alot_ of bug fixes were done, that would have caused problems. So because it was taken seriously , stuff were fixed and issues didn't happen because of that.
Personally, I fixed 3 Y2K bugs back then, 2 of them would have brought down a rather critical business support to simply crash every time new data arrived.
"Why does everyone always say Y2K wasn't an issue?"
From the outsider's perspective, it is indistinguishable from any number of other putative disasters that required lots of money to fix, yet didn't come to pass... in some cases including putative disasters in which the money wasn't spent and the disaster didn't happen anyway.
I have the insider's perspective and I agree that it is the more accurate, that Y2K was, if not necessarily going to end the world, certainly a bad thing and was largely averted through effective engineering. But I can still see how from the outside it sure doesn't look that way.
Many, if not a majority, of my non-technical friends and acquaintances have expressed at one time or another a reference to the "Y2K disaster" and rolled their eyes to suggest it was somehow not an issue. I was the 'Y2K compliance officer' at my startup at the time (we even got certified, and that actually may have been a scam (the certifying part)) but we identified and fixed a number of issues our box would have suffered had we not done the work.
"In spite of all press that Y2K was just a silly waste of money, its events like these that makes me suspect it would have been a much bigger deal if everyone had ignored it and fixed it after things where shown to break."
It seems to be the unique class of bug that not only is it easy to forget to test, and won't ever show up until a particular date... but then affects everyone!
I can't think of any other kind of bug that never shows up ever, but then affects everyone. Rare bugs tend to stay rare, common bugs tend to get caught before they affect everyone... this is the exception.
From discussion of this same issue in prior threads, my takeaway was
(a) it's really not at all difficult to handle leap seconds, but
(b) the POSIX standard specifically disallows them, by specifying that a day must contain exactly 86400 seconds. (Analogously, imagine if leap days occurred as normal, but a "year" by definition contained exactly 365 days.)
The existence of leap seconds means that it's not possible to simultaneously have (1) system time representing the number of seconds since the epoch, and (2) system time equal to (86400 * number_of_days_since_epoch) + seconds_elapsed_today, and all the proposed methods of dealing with the problem involve preserving (2), which seems worthless to me, and throwing away (1), which I would have thought was a better model.
edit: actual system times may be in units other than seconds, but the point remains
It's harder than leap days, because leap seconds aren't inserted on a regular schedule. Leap days follow a predictable pattern of insertion. Leap seconds are inserted whenever the IERS decides to insert them.
The problem of leap seconds is therefore closer to that of time zone definitions -- which are a total mess, because they depend on keeping rapidly changing system tables up to date. I can see why people don't relish the idea of requiring similar tables just to keep system time accurate.
How are systems being notified of the leap seconds now, that wouldn't immediately enable them to update their hypothetical leap second table?
It seems like we already have a much bigger lead time for notification than we could possibly need.
> I can see why people don't relish the idea of requiring similar tables just to keep system time accurate.
But the 'solution' we're using now is to make system time less accurate, not more accurate. Accurate would be if leap seconds incremented the system clock like normal seconds do. If the accuracy you're worried about is displaying a clock time rather than time since the epoch, you already need a time zone to do that.
"How are systems being notified of the leap seconds now, that wouldn't immediately enable them to update their hypothetical leap second table?"
I am not an expert, but as far as I know the most automated solutions are doing it via NTP, which just resets the second, then relies on clock drift to bring everything back into synch. Otherwise, I think your only option is to keep the timezone packages up-to-date (which is a non-trivial task for large deployments). A quick search found this:
"But the 'solution' we're using now is to make system time less accurate, not more accurate."
Yeah, I'm not disputing this. I'm just saying that preserving the assumption that "day == 86400 seconds" probably breaks less code than the alternative. NTP messes with the notion of seconds-since-epoch anyway, so we know that single-second variations in unix time aren't automatically deadly to most unix software.
NTP sends a special message to the kernel (using adjtimex), that boils down to: today you will insert a leap second. This isn't the same as clock drift, which gets smoothed out, it means a minute with a 60th second (in UTC) or with the 59th second happening twice (in POSIX). NTP servers need a leap second table ( http://support.ntp.org/bin/view/Support/ConfiguringNTP#Secti.... ), but most other systems only need to know the current delta between POSIX and TAI, and manage without a leap table.
Interesting analysis. Preserving (2) means that systems that don't handle leap seconds (or don't have a leap seconds table updated in the last six months) have a POSIX time at most 1 second away from systems that do. Dropping (2) would mean a more complicated formula to convert system times to times of day, but it would also make the failures and missing leap tables immediately noticeable, rather than a one-in-a-few-years event. Which would have made it the better choice; bugs encountered during development are easily handled, bugs that happen years later on production systems are inconvenient.
That's not really a definition of time, just a way to postulate its direction. Time as a physical unit has been decreed to be the rate of decay of a certain Cesium atom; i.e., a fixed regular interval, whereas the increase of entropy varies from moment to moment. E.g. right now I'm typing and before I wasn't, that doesn't mean time is going ever so slightly faster because of that ...
The second is the duration of 9 192 631 770 periods
of the radiation corresponding to the transition
between the two hyperfine levels of the ground state
of the caesium 133 atom.
The details of this effect are a little more complicated, but it boils down to the fact that you can measure the "angular momentum" of your nuclei when you have them pass through a non-uniform magnetic field. Particles in the one state are deflected differently than particles in the other state. And when you irradiate a particle beam in the F=0 state with the right frequency (9.2 GHz) you can very efficiently swap many particles over to the F=1 state. By adjusting your frequency sufficiently to find the maximum rate of flips, you can can tune for the exact 9'192'631'770 Hz.
The cesium particles have not decayed, in principle you could run forever on a certain supply of atoms... even though in practice a Cs-beam is produced on one side, at a hot filament, and dumped to the other side of the clock after passed through the apparatus that performs the steps described above. They will be disposed of when the lifetime of the beam-tube is reached (typically 10 years or so with a few grams of cesium inside).
I wasn't making a philosophical statement, I specifically qualified it by saying "as a physical unit." That's still nowhere near precise enough, but I only meant to point out that we typically don't conceive of time as an emergent property of physical processes (i.e., entropy change), but as a local property of a very specific atom, such that time passes at a constant rate, if only because we choose to define it that way. "Telling something about time itself" sounds like a philosophical question which I fear has no definite answer.
In this case, i believe we created a problem we did not have. Leap seconds is a dubious construct from the start, problematic with computers or space travel. We have added only 25 since 1972. Their unpredictability means they will be forever a problem with computing. We should either quit the whole idea or in the worst case allow them only every 25 years or so.
It's 2038, not 2037. (Specifically, January 19th, 2038 at 3:14:08am.) And while lots of systems will be 64-bit, many programs still won't be -- and it seems highly likely that this will be a significantly more serious and widespread problem than, say, Y2K or DST. (And certainly more serious than leap seconds, which happen relatively frequently.) Then again, I might be biased: perhaps I'm secretly hoping to spend the years leading up to 2038 paying for my retirement with high-priced consulting gigs to fix it...
If you think that being 64-bit protects you, then you do not understand the problem.
The problem is that 32-bit time is embedded in filesystem representations and related protocols. (eg the POSIX specification for file times in tar.) Therefore even if your machine is 64-bit, it still needs to use 32-bit time for many purposes.
To name a random example, the POSIX specification for times in the tar format is 32-bit. GNU tar has a non-standard extension that already takes care of it. But will everything else that expects to read/write tar files that a GNU tar program implement the same non-standard extension to the format in the same non-standard way? Almost certainly not. And there will be no sign of disaster until the second that we need to start relying on that more precise representation.
SLE9 (kernel 2.6.5-7.325): NOT AFFECTED
SLE10-SP1 (kernel 22.214.171.124-0.2.12): NOT AFFECTED
SLE10-SP2 (kernel 126.96.36.199-0.42.54.1): NOT AFFECTED
SLE10-SP3 (kernel 188.8.131.52-0.83.2): NOT AFFECTED
SLE10-SP4 (kernel 184.108.40.206-0.97.1): NOT AFFECTED
SLE11-GA (kernel 220.127.116.11-0.2.1): VERY UNLIKELY
SLE11-SP1 (kernel 18.104.22.168-0.3.1): VERY UNLIKELY
SLE11-SP2 (kernel 3.0.31-0.9.1): VERY UNLIKELY
Update (06/26/2012): after thorough code review -> SLE9 and SLE10 not affected at all.
Pardon the ignorance if this is a stupid question. I've been looking at some of my hosts and have noticed a message "Clock: inserting leap second 23:59:60 UTC" in dmesg output but each of the hosts is in the EDT timezone so the I was under the impression that the leap second hadn't been applied yet. So what does that mean? That the systems have applied the leap second successfully or have only received it from their NTP servers?
You are a lifesaver. All morning my desktop's load has been pegged at 20. I upgraded FF, Chrome, etc. and no impact. I was dreading a full re-start, as I have lots of windows, tabs, etc. open. The above command knocked the load down to almost nothing in seconds.
> Just set the clock to today (or any day with a leap second), and watch what happens.
That won't work. The bug is only triggered when an upstream NTP server reports that a leap second was scheduled. Since leap seconds aren't predictable (and aren't even scheduled very far in advance), just setting the time back to the date of a previous leap second won't do anything.
The leap second is added at the same point in time regardless the timezone your server is configured to use. So if you're GMT+3, the leap second will be inserted at 03:00 local time.
From the answer: "The reason this is occurring before the leap second is actually scheduled to occur is that ntpd lets the kernel handle the leap second at midnight, but needs to alert the kernel to insert the leap second before midnight. ntpd therefore calls adjtimex sometime during the day of the leap second, at which point this bug is triggered."
Implementation specific. It looks like it is a bug in the Linux kernel with how it adjusts the time. It is possible that Windows, OS X, and other BSDs will be affected by a similar bug, but that would be coincidental as the bug is not due to ntpd but rather how the kernel handles a request that ntpd generates.
More specifically, there is a condition in which the kernel tries to insert a leap second and, in doing so, attempts to acquire the same lock twice causing the spinlock lockup and (effectively) halting the kernel.
I found mysqld (5.5.24) running at 159% CPU on a Ubuntu 11.04 (64-bit) box this morning. ntpd had drifted in the leap second between 1am and 2am (GMT) this morning (NTP drift info is one thing I graph with MRTG).
[EDIT] Ah, covered elsewhere. Fixed by manually setting the date on the box; stopping/restarting mysqld or ntpd doesn't make any difference.
This is evidently a kernel bug. The fact that both operating systems rely on time isn't particularly relevant. Could there be time bugs in OS X? Certainly. But it wouldn't be this one. Windows relies on time too, so I don't see why you bring up the fact that OS X is a BSD variant.