

Out of control Java processes when dealing with a leap second? - wzm
https://bugzilla.mozilla.org/show_bug.cgi?id=769972

======
silviosantoz
Leap second + Java + Linux fix (without reboot):

/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;

\-- then restart java

~~~
_thekev
A lot simpler:

date --set "`date`"

~~~
larelli
On a localized box, I had to run "unset LANG" first, because otherwise date
doesn't seem to understand its own output.

------
rasur
I had a lovely morning (2am-4.30) rebooting about 50 boxes by hand. All
wearing some shade of java.

------
Maxious
Reddit is also down due to Java
<https://twitter.com/redditstatus/status/219244389044731904>

------
cmorrisrsg
We saw this on every Ubuntu Lucid machine we run. Restarting the JVM was
insufficient - we ended up fully rebooting the machines.

------
matthewlmcclure
Would love to read a root cause analysis of the relevant source code. Anyone
have a link?

~~~
_thekev
<https://lkml.org/lkml/2012/6/30/122>

------
nickzoic
I didn't really think about it at the time, but right about then (I'm in
GMT+10) a Minecraft server (OpenJDK 64-bit, Ubuntu 11.10) I had running got
rather upset ...

    
    
         2012-07-01 09:59:59 [WARNING] Time ran backwards! Did the system time change?
         2012-07-01 10:01:14 [WARNING] Can't keep up! Did the system time change, or is the server overloaded?
    

... and stuck on 100% CPU until I noticed it and restarted it this morning.
Thankfully no other issues that I've noticed.

------
aSig
I saw the same across all of our Ubuntu servers. Some running Tomcat others
running SmartFox Server (a java based game server). A reboot + app restart
fixed it.

------
burningout
I was just wondering why everything stopped working at night. System.exit()
didn't work anymore in any of the java processes, or just random hangs.

wow.

------
stevencorona
Even if you're not running java in your stack, it's worth a quick check on
your servers. Our SoftLayer servers use an Adaptec RAID card and the
monitoring software uses the JVM and will suck up all of your CPU. All of our
MySQL/Cassandra/Redis/Kestrel servers needed to be bounced because they had
RAID cards. Frustrating.

~~~
Negitivefrags
I also use Softlayer and have a pair of servers with Adaptec RAID cards.
Checking up on them they were not affected.

Having looked in to it, it seems our server admin decided to install OpenNTP
to replace the standard NTP daemon. I'm not sure why he did that, but it seems
OpenNTP just ignores leap seconds and compensates after the fact just as if
your clock had drifted a second out of sync in the usual way.

------
_joe
Same problem here. I am supposedly on vacation in Barcelona and was summoned
to help :/ Gotta love your java.

~~~
Sharlin
Except that it's not a Java problem, it's a Linux futex problem.

------
gouranga
At the risk of shooting myself, all our jvms were fine on windows and Ubuntu.

------
DEinspanjer
Fun that a bug for my team happens to be the thing linked in this article. :)

~~~
matthewlmcclure
Hey Daniel,

Funny to see your name there!

------
ryanpers
its the futex() call that is bugged, so therefore things that are highly
multi-threaded... for example mysql and java!

Many unix things are not as multithreaded as those examples.

------
_thekev
I was able to restore my java app boxen (and all other boxen) by only setting
the date to the current date. No restarts necessary.

------
matthewlmcclure
What's the explanation of that wacky date format?

minute day hour month year . second

~~~
emmelaich
It's actually month day hour minute.second

Originally (and perhaps still) the month day is optional; allowing you to set
the date with just hour and minute.

------
rbanffy
I experienced no problem today. Is this Java-specific?

~~~
NDizzle
Yeah this is what is slaying linkedin at the moment.

~~~
cedrichurst
MySQL is also apparently affected, according to the #mysql IRC channel.

~~~
codeka
Yes, this happened on my development MySQL instance and also one of my
production instances of MySQL -- but not another one that I have. I'm not sure
why it only affected one, maybe different versions of MySQL?

------
spullara
Has anyone seen this not on Ubuntu?

~~~
jerdfelt
I saw this on my Fedora 17 laptop. Chrome started eating up 100% CPU and my
laptop started getting really hot. I tried restarting Chrome, updating Chrome,
etc. Nothing worked. Rebooting fixed the problem.

Surprisingly, none of my servers have had problems, but I don't run Java
anywhere.

