
Linux gets frozen, what do you do? - 300
http://www.jovicailic.org/2013/05/linux-gets-frozen-what-do-you-do/
======
TacticalCoder
I'm using Linux since 20 years and used to do this all the time, back when
Linux --or more specifically X Window system-- was less stable than today. I'd
make sure to always compile the kernel with MagicSysRQ (you need it for the
combo mentioned in TFA).

It wasn't even mandatory to restart the system: the trick was to use first
MagicSysRQ and then issue a "vgareset" (IIRC I couldn't even see what I was
typing, but the command was taken into account) and then, miracle: I could
unlock frozen X Window sessions (more specifically: kill X, "reset" the GPU
and then restart a new X Window session).

Note that very often X is fine: it's just X which is frozen. Heck, if you have
another machine on your LAN and allow SSH in, you can SSH and kill X /
vgareset without needing MagicSysRQ.

But since quite a few years X is so stable that such hackery ain't needed
anymore. Moreover if I recall correctly "vgareset" did only exist for 32 bits
system (at least at one point). Nowadays my Linux workstation regularly
reaches 6 months of uptime (there are only very rarely know remote root
exploits mandating a kernel upgrade) so I've kinda "forgot" how MagicSysRQ
works ^ ^

~~~
arethuza
When I read that I thought of "Stop A" on Sun workstations - and indeed that
is mentioned on the wikipedia page:

[http://en.wikipedia.org/wiki/Magic_SysRq_key](http://en.wikipedia.org/wiki/Magic_SysRq_key)

Mind you I don't think I ever used "Stop A" on a Sun for anything
constructive...

~~~
gaius
I thought Stop-A launched the Forth interpreter ;-)

~~~
jmj42
Sort of. Stop-A drops you into the firmware prompt (OpenBoot), suspending the
OS in the process. OpenBoot, of course implements a forth interpreter shell.

Ahh, the good old days... I once implemented a nice little boot device
selecting utility in for for some of my Suns. All written in Fourth, and
executed by the firmware on every boot.

------
nly
It's a lot easier just to reboot and let fsck patch up your filesystem...
flushing buffers isn't going to help you get back the last paragraph of your
PhD thesis if Libre Office hasn't called write().

In any case, in ~8 years of using Linux, I don't think I've had a freeze that
was the kernel and not just X.

~~~
spaznode
"sync; sync; sync; shutdown -r now"

I remember some sort of magic invocation like that years ago for our medical
"device" product we had to manage hundreds of remote node instances of.
Something along those lines. I don't remember why the developer (Gabe) came up
with sync three times being magic number. Maybe three times was just paranoia.
=)

~~~
mikeash
Back in the bad old days of pre-UNIX Macs, it was a common troubleshooting
step to reset your PRAM. This is battery backed up RAM that holds some basic
settings, and if it got corrupted somehow it could cause weird problems. You'd
reboot while holding down command, option, P, and R, then wait for the boot
chime to sound a _second_ time indicating that it had been reset, then release
the keys and boot normally.

Somehow this advice got mutated so that you'd keep holding the keys until you
heard _two_ boot chimes (thus resetting the stuff _twice_ ). And then it
started to grow. Three was common. Some people would advise more. I'm pretty
sure that doing it more than once never helped anything, but there we are.

(The cmd-opt-P-R sequence still works on modern Macs and I actually used it to
resurrect a machine that wouldn't start up just a month ago, but it's far less
frequently needed now.)

~~~
celebril
It's the same with the battery stats resetting and the Dalvik cache wiping
these days in Android land. You do it three+ times.

Or the "Repair permissions" thing in OS X. You do it several times as well.

It's like whenever there's this one-step fix thing that a system utility does,
the Common Man will interpret it as needing to repeat 3+ times in order for it
to be effective.

------
freiheit
Or, if you're on anything approaching a modern system:

1\. Tap the power button (one single brief tap)

2\. Wait a minute

3\. If the system hasn't already shut down or isn't obviously in the process
of shutting down or you just get impatient: hold the power button down

4\. Now that the system is off: tap the power button.

Modern systems (whether server, desktop tower or laptop) have a "soft" power
button, that when tapped briefly sends a signal to the OS. Most flavors of
linux are configured so that receiving that signal initiates a proper
shutdown, just like "shutdown", "poweroff" or a GUI shutdown. All of this has
been true for quite a few years now.

If the system is so locked up that the power button initiated shutdown doesn't
work, you might as well just pull the power, which holding the button down for
a second or two will do. Even if this happens, you're probably using a
journaling filesystem that comes back up cleanly.

None of that will help with unsaved data in an application, because you almost
certainly have to tell the application to save, and if X has locked up there
is no way to do that. A clean shutdown might signal the app with a terminate
signal and allow for data to be saved before doing a hard kill.

------
Nursie
This is not for the situation in the article, but a slightly different one.
You have a remote server, there's an ssh session still open to it but it's
somehow lost access to _all_ its mount points including / You have nothing to
work with but bash builtins and /proc, you're 100 miles away and you need to
get it up and running again NOW. Emergency reboot -

    
    
        echo 1 > /proc/sys/kernel/sysrq
        echo b > /proc/sysrq-trigger

~~~
peterwaller
Do a `sync` first! Also This is quite like yanking the power chord and
plugging it back in.

[https://en.wikipedia.org/wiki/Sync_(Unix)](https://en.wikipedia.org/wiki/Sync_\(Unix\))

~~~
joncp
[http://www.folkpeople.com/guitar-
lesson/pic/power_chord_tabl...](http://www.folkpeople.com/guitar-
lesson/pic/power_chord_table.gif)

~~~
pit
At first, I wasn't sure why you posted this, but I was too busy rocking out to
mind.

------
eikenberry
Another way to deal with X freezing.

Use acpi to set your power button to switch you to console. Just edit your
/etc/acpi/power.sh and change it to...

    
    
        #/bin/sh
        /bin/chvt 1
    

The acpi system is separate from the X/keyboard interface and still works. So
this way you can switch to the console even if your keyboard is locked.

Note you can use other keys as well. On my laptop I use my "thinkvantage" key
for this. Any key that has an associated acpi event will work.

~~~
ingenium
Which script in /etc/acpi/ corresponds to the thinkvantage key?

~~~
eikenberry

      $ cd /etc/acpi
      $ cat events/thinkvantage-button 
      # Thinkpad 'ThinkVantage' button
      event=button/prog1
      action=/etc/acpi/thinkvantage.sh
      $ cat thinkvantage.sh
      #!/bin/sh            
      /bin/chvt 1
    

Hope this helps.

------
danparsonson
Handy that 'reisub' is 'busier' backwards

~~~
chris_wot
You rock.

------
Yuioup
This sounds like like a feature that needs to be automated, probably with a
very handy key combination. I don't know ... maybe with Ctrl+Alt+Delete?

~~~
i386
Agreed. Why isn't this default behaviour? Is there some negative aspect that's
not obvious?

~~~
marcosdumay
That's because the default behaviour of CTRL + ALT + DEL is to issue a proper
reboot.

That, and because X consumes those keys, making them useless when X goes bad
(what is about all the times that Linux freezes and it's not hardware fault).

~~~
Yuioup
Okay maybe another key combination?

------
tbrock
It's possible that their computer is frozen enough to need this but it's more
probable that this person hasn't figured out how to disable DontZap in newer
versions of xorg. You need to disable it in your xorg.conf in order to have
ctrl-alt-bksp work again:

    
    
        Section "ServerFlags"
            Option "DontZap" "false"
        EndSection

~~~
jcastro
Alt + SysRq + k will also do this if you don't feel like writing a xorg.conf.

~~~
mpyne
That was one of my favorite Magic SysRq keys. It's the Linux analog to
Windows's Ctrl-Alt-Del.

It's the "Secure Access Key" (SAK): You press that key and it kills _all_
programs hooked to the TTY (incl. X, in your case) and displays a proper login
prompt so that you can know what you're about to login to was run by the
system and not a clever malware trying to steal your password.

------
Stolpe
Raising Elephants Is So Utterly Boring

~~~
namarkiv
Raising Skinny Elephants Is Utterly Boring

~~~
jeffcox
Reboot Even If System Utterly Broken

------
jedanbik
Linux gets frozen, what do you do? I remember to press nine buttons on my
keyboard, duh. and why? Because a hard reset will "make you a lot of
problems?" I am not convinced, and I am not impressed.

This is a laughably awful shortcut for anything. Why make the supposedly safe
stuff so out of reach? I would have to use a separate computer to look up how
to type this command. Even if I knew the command off the top of my head, what
if my keyboard lacks a PrtSc (SysRq) button?

When folks are pointing to Windows for design cues, the open source community
ought to reconsider some aspects of how it has implemented certain features.

~~~
clarry
How do you perform the equivalent operations on Windows?

~~~
jedanbik
It isn't equivalent, but it is approximate; folks were mentioning ctrl-alt-
dlt. Three keys instead of nine keys represents a 66% increase in efficiency.

~~~
clarry
I would've said it was approximate enough in whe Windows 3.1 days when it
could perform a soft reboot. But since Windows 2000, it has had completely
different functionality (sometimes allowing you to log in, sometimes giving
you a process manager oslt), not even remotely approximate to the magic sysrq
sequence presented here. Do tell me if this has changed since Windows Vista.

Linux, by the way, can also respond to Ctrl-alt-del, and depending on the
desktop it might perform functions similar to Windows.

------
TorKlingberg
Before this try Ctrl+Alt+F1. You might get a full-screen terminal where you
can log in and kill the offending process. Ctrl+Alt+F7 to get back to the
desktop.

I had to use this just today when ddd stole all other mouse and keyboard
input.

~~~
gnarbarian
I used to use Ctrl+alt+backspace but it doesn't work anymore on many desktop
environments.

~~~
anon4
That's because it's turned off by default on modern X.

You can enable them in the keyboard options of your DE, or directly with
setxkbmap on startup (sorry can't look up the exact options right now) if
you're not using one.

------
markhahn
if your X is hung (ie, NOT "linux"), why not switch to a VC and fix it? yes,
it's good to know about magic-sysrq, but most people don't even understand the
layering of X on a VC, and the fact that other VCs are available.

of course, fixing why your X hung would be wise too, since for any normal
distro and mainstream hardware, that's just not going to happen. if you really
want to be fubar, have that f@cking POS systemd die on you... (yes, on-topic,
since only sysrq saves the day.)

~~~
stonogo
Because X11 has trashed your video interface. You _can 't_ switch to a virtual
console. X11 has ring0 access to the video card, to take advantage of DRM.

~~~
laumars
More often than not, it's your WM, DE or a GUI application eating up your RAM
(Chromium used to be deadly for this) that's causing the freeze.

In more than a decade of running Linux on my desktop (and at work), I
genuinely can't think of a single instance when I've not been able to pull a
virtual console from a frozen desktop (albeit it often performs laggy).

~~~
stonogo
Thanks for trying to diagnose my computer over a web forum, but I am competent
enough to identify when and how my computer has failed. I'm not interested
anecdotes from users. I develop video drivers. Am I allowed to experience
these lockups now?

~~~
laumars
Firstly, I'm not just a Linux user. Like you, I'm a developer too. Given the
demographic of this forum, it would pay for you not to assume that you're the
only one on here that works in the industry (in fact I even hinted at that
when I said I use Linux at work - but never mind)

Secondly, I was making a general comment about peoples desktops rather than
talking specifically about your example (given the lack of details you posted,
it would be insane of me to assume I could diagnose your fault with any
precision). My point was that generally when people think their computer has
locked up / X has crashed, it's actually one of the items I mentioned earlier
that's at fault.

The snappy reply was appreciated though </sarcasm>. But given just how unusual
your circumstances are (assuming what you said is true) and how much you seem
to hate it when others discuss these topics with you; it might be an idea if
you clarify your position a little better the next time such a topic arises.
Like maybe saying _" my crashes aren't typical because I'm a kernel developer,
but....."_. This way people don't accidentally post something that hits one of
those raw nerves you have and it saves us all from a lot of unnecessary
condescension.

------
mikeweiss
If its a Linux Desktop alot of the time its just the X session that is frozen.
Ctrl ALT F2 and sudo killall X will simply restart X

------
colechristensen
Something very important to note which is missing in the comments and the
original article: waiting!

Each one of these commands (r, e, i, s, u) takes or can take a few seconds to
complete successfully, so let them do their thing.

~~~
ttsiodras
Indeed!

In particular, if you have HD-activity LEDs, watch and wait until they stop
blinking - after the 's' (sync) key.

------
blcknight
Only if you have magic SysRq keys enabled in the kernel... are kernels
shipping these days with this by default?

~~~
mistercow
I just tried it on my Ubuntu 12.04 install, and it worked. I definitely didn't
change any kernel config options.

------
Aardwolf
When visiting the site I get:

unused The server encountered an internal error or misconfiguration and was
unable to complete your request. Please contact the server administrator,
(etc...)

Did their Linux get frozen?

[edit] it works again

~~~
manca
Yeah, being first on Hackers News is not good for your box :)

------
AaronFriel
What about when an SSH terminal gets locked up? If I'm running something and I
want to abort it, I often find I can't because of the kernel thrashing. Often
I see a nontrivial amount of CPU time given to kswapd but sometimes not, and
more often I don't have any monitoring up to see what's going on, or the
terminal I'm monitoring is also nonresponsive.

This is especially a problem with Docker, I've found, because when I've got a
container thrashing the system like this, I can't access the host or any other
containers.

Is this because I'm running single-core instances and the problem goes away
once I have more than one thread available? Or should I be restricting
workloads to a block device not used for caching? Does anyone else experience
this?

~~~
derefr
Docker can set CPU, memory, and network-bandwidth quotas for containers. If
you can guarantee that the aggregate total of each quota is less than the
amount of that resource available to the host, then in the worst case you'll
always have some of each left for a "rescue" container.

This throws away one of the neat advantages of containers compared to VMs,
though, which is that you can overcommit (or "thin-provision", if you want to
be charitable) the host's resource allocation, since it's extremely unlikely
that all your services will balloon in resource-consumption simultaneously.

A less guaranteed, but more economical strategy, is to just set quotas on each
container such that if _it individually_ starts using more than, say, 80% of
the host's resources for an extended period, it'll be terminated. This doesn't
save you from a bad interaction between containers that makes everything
explode in parallel (e.g. all your containers attempting to reconnect to a
stuck service with no backoff) but it should save you in the majority of cases
where each host has a heterogeneous container-load, and horizontal scale
happens between hosts rather than internal to them.

(The dev-side solution to this problem, though, is to just set up your
software architecture such that any host that gets into such a state _can_
hard-reboot, without you losing anything of consequence.
[http://12factor.net](http://12factor.net) -type containers excel at this, and
it looks to be the strategy [https://coreos.com](https://coreos.com) embraces
as well.)

~~~
AaronFriel
Docker supports (supported?) what LXC supported, which was relative CPU
shares. I am not sure it would help if it supported something else (like quota
minimum/maximum). When the IO subsystem is hit with a memory intensive
workload that thrashes swap, all your guarantees go out the window. From my
perspective (staring at top while an interactive terminal is not responding),
it's the kernel that owns the CPU shares being used, and the IO volume, and it
looks like the container is "almost idle" because most of the CPU time is
being spent by the kernel.

I don't know if it's that something Docker can fix. It might be something
kernel devs can fix, though.

------
deathanatos
The most common reason I've had to do this is that I've exhausted RAM and
swap, and the system is disk thrashing for spare pages. In some setups, Linux
does not like to OOM a process when it really should. (That's adjustable, by
the way.)

Also, you don't need to go all the way through it. In my above example,
MagicSysReq+RE or MagicSysReq+REI is enough usually. E and I are for sending
SIGTERM and SIGKILL, so all the processes are now dead, and we have plenty of
RAM. Sometimes it does take a bit of waiting: the system is disk thrashing.
That said, you can then just resume, though you'll likely need to switch to a
VT and restart X and other things. (But you get to keep your uptime.)

There's also F, which invokes the OOM killer, though I often forget about it.

------
mikecarlton
What do you mean 'linux gets frozen'? I literally can't remember the last time
a linux box froze on me. As others said, I would just reboot it now and let
fsck pick up the pieces.

That's going to be faster than the time spent to search google for the magic
incantation.

------
thanatropism
This would be where keyboards with hardware-level macro support (like the
Kinesis Advantage) come to be useful.

... to be honest, I can't think of another scenario. I have an Advantage for
every place I spend extended computer time in -- it's too big and clunky to
carry around...

~~~
jerf
You do not necessarily want to type that _too_ quickly; sync may take a
moment.

------
blueblob
The only way my linux machine has ever locked up is the usb driver seemed to
crash in which case I couldn't do this anyways. The only other way I could
have possibly restarted is to ssh to my machine from my phone, I opted for the
power button instead.

------
cordite
Hmm, I wonder if something like this extends to mac. Sometimes the login
manager crashes and I can't do anything. Music still playing, but no mouse or
keyboard reaction. I can ssh in, but I don't know what to run.

Any ideas?

------
girvo
I learnt about this a few months back... and never had to use it. Seriously,
on my AMD APU netbook running Mint, I've had no freezes whatsoever, despite
the hardware being a bit odd. A neat trick though.

------
donniezazen
Can't get it to work on Fedora 20. Are you suppose to press and hold Ctrl,
Alt, PrtSc(SysRq), r, e, i, s, u, and b all together? Does it have to be in
that order?

~~~
pbhjpbhj
Switch to a terminal (Ctrl+Alt+F2) and do AltGr+PrtScr+H - if you've got Magic
SysReq enabled that will print the help information (a list of commands with
their (k)eycodes) to the terminal. If it doesn't print anything then you need
to RTFM to find out how to enable it. AFAIR I had to do this shortly after I
switched to Kubuntu as it was disabled by default - I often had to use
Ctrl+Alt+Backspace or AltGr+SysRq+K to kill X as there's a hardware bug on my
Nvidia graphics card.

My 8yo has a mnemonic for it something about Elephants and Umbrellas but I've
always done reissub (now with extra sync'ing power). For a time the process
for him was login, Alt+F2, konsole, Ctrl+R, mine, Enter, AltGr+SysRq+R, E, I,
S, U, B; then repeat the first part and you're finally ready to play
Minecraft!

~~~
donniezazen
If AltGr is right Alt, it does print a bunch of stuff. On my system, SysRq is
under PrtSc. Does that mean I have to do Fn+PrtSc to get SysRq or just PrtSc
is enough.

~~~
pbhjpbhj
If it prints out without presssing Fn then you're doing it right. Laptop
keyboards are weird but usually it's just the PrtScr key as it acts as SysRq
when using the Alt[Gr] modifier.

It should print something like:

"SysRq : HELP : loglevel(0-9) reboot(b) [...]"

------
mavroprovato
> You need to press and hold Ctrl, Alt and PrtSc(SysRq) buttons, and while
> holding them, you need to press r, e, i, s, u, b

I think I need an extra hand to do this

~~~
DaCapoo
Control isn't necessary - I'm not sure why it would be specified. On a laptop
you might have to hold the Fn key to hit PrtScr, but aside from that it's two
fingers to hit 'Right Alt' and 'PrtScr' and the other hand can mash
r,e,i,s,u,b.

Source:
[http://en.wikipedia.org/wiki/Magic_SysRq_key](http://en.wikipedia.org/wiki/Magic_SysRq_key)

~~~
mavroprovato
In my workplace I have a Dell Keyboard that has the SysRq/Scroll Lock/Pause
Keys on the top right, above the numpad. Holding Alt+SysRq with one hand is
just impossible.

Maybe it's easier in saner keyboards.

~~~
DaCapoo
Yeah, with a typical ANSI layout it's a small stretch but definitely doable.

This is the layout that I have (and the only keyboard layout I'll ever buy,
because every other throws the pipe key and backslash in a random spot along
with randomly sizing the enter key)

[http://commons.wikimedia.org/wiki/File:ANSI_Keyboard_Layout_...](http://commons.wikimedia.org/wiki/File:ANSI_Keyboard_Layout_Diagram_with_Form_Factor.svg)

------
samsaga2
"You need to press and hold Ctrl, Alt and PrtSc(SysRq) buttons, and while
holding them, you need to press r, e, i, s, u, b"

I'll need three hands.

------
iwwr
Bind xkill to a global hotkey. If it's just fullscreen or screen-grabbing app
that crashed, xkill will take care of it.

------
jsemrau
Let it go, let it go! Can't hold it back any more. Let it go, let it go! Turn
away and slam the door. I don't care what they're going to say. Let the storm
rage on. The server never bothered me anyway.

------
jack57
Sorry for the double posts. An app I was using caused it.

------
otikik
If you only have one hand, just press the power button.

------
squigs25
If only they had made this more ergonomically possible

------
tomrod
Using Ubuntu minimal + xmonad + dmenu. Does not work.

~~~
plorg
Certain Magic SysRq sequences are now disabled by default in Ubuntu. You can
re-enable them by editing /etc/sysctl.d/10-magic-sysrq.conf. Alternately, you
can enable them temporarily by # echo 1 > /proc/sys/kernel/sysrq (Or
substitute "1" by the number described in magic-sysrq.conf, above)

------
sgtnasty
I have not had to do this in a while now.

------
jack57
This complexity, my technically oriented friends, is part of the reason normal
people do not use Linux.

~~~
NotHereNotThere
How do you even come up to this conclusion? Knowing the magic key sequence is
not required to operate a Linux system at all.

If a Windows box froze, and you had a (somewhat slim) chance of gracefully
shutting it down, would you not use it?

Would you call pressing F8 to access magical boot options in Windows, a reason
non-technical people wouldn't use Windows?

------
ragsagar
The site is blocked in UAE.

~~~
013
What's UAE?

~~~
LoneWolf
I would say United Arab Emirates

~~~
chris_wot
Censor's not happy with magic?

