The kernel OOM is just plain broken. I can't understand how can it be possible that, on Windows, whenever I run out of RAM the OS kills whatever process is consuming too much and the computer keeps running flawlessly. However, on Linux, my computer just... freezes. It freezes and stops responding. Not even the mouse moves. Having to use a userspace OOM is the most inelegant thing I've seen. So I need to have 2 OOMs so that the good one can beat the bad one? It's so redundant, it's plain stupid. How come NOBODY is doing anything. If I knew c++ I would for sure send a patch.
The reason the OOM killer never kicks in is because you actually never run into an OOM, or almost never.
What usually happens is that in near-OOM conditions, the kernel starts reclaiming memory pages backed by file (sometimes called "trashing"). This operation manages to keep some extra memory available, but it makes the system almost unresponsive because it's constantly copying memory back and forth from the disk. It may take anywhere between minutes to hours before the system finally OOMs and the OOM killer is invoked.
This problem has been there forever but has been made worse by the improved speeds of modern storage technologies: with slower disk I/O, the OOM condition was reached sooner.
There are several solutions:
- Buy more RAM: if your system routinely goes nearly OOM something is not right.
- Add a (small) swap. It doesn't have to be a partition: nowadays most filesystems support swap file. Just create an empty file and mark it as swap.
- Limit the amount of thrashing or protect some pages from being reclaimed. This has been proposed by Google first and several other people since then, but AFAIK it has never been implemented in the mainline kernel.
Regarding the latter solution, there is a patchset called le9-patch[1] that is included in some alternative Linux kernels and it should be relatively safe to use.
I’m not sure if you misread the previous comment, but their point is quite commonly experienced by people who use Linux vs MacOS/Windows.
All hardware being the same (RAM, SSD, CPU) as OOM is reached, Linux will freeze, whereas Windows continues to run smoothly. All OSes try to reclaim memory pages, just Linux seems to hang the user space while doing so.
As someone who has dual-booted Windows and Linux for a decade, I can 100% attest to this glaring problem.
I am sure that this distinction between a near-OOM condition and an actual OOM condition matters to someone familiar with the current kernel implementation. You seem confident describing what happens when the memory gets closer to full, so I believe you. However, the user experiences the PC freeze during certain conditions, however you choose to name them, and it is during that freeze period the user needs a program to be killed to free some memory and prevent the freeze. I would take one crashed program over power cycling the entire PC any day of the week.
> I am sure that this distinction between a near-OOM condition and an actual OOM condition matters to someone familiar with the current kernel implementation. You seem confident describing what happens when the memory gets closer to full, so I believe you.
I'm not a kernel developer or anything like that, I've just spent some time investigating why this issue happens and has been happening for more than 10 years now.
> the user experiences the PC freeze during certain conditions, however you choose to name them
I'm not trying to defent the Linux kernel, I just described how it works. In particular it's not true that the OOM killer "takes too long" or doesn't work: it's just not invoked at all. If you invoke it manually (enable the magic SysRq with`sysctl kernel.sysrq=1` and press `alt-sysrq-f`) it does its job and solves the OOM instantly.
So, if you don't want to deal lockups and don't like an OOM userspace daemon (I don't), these are the possible solutions.
> I would take one crashed program over power cycling the entire PC any day of the week.
On a laptop or desktop PC, you don't need to power cycle in a near-OOM: use the magic SysRq key.
>On a laptop or desktop PC, you don't need to power cycle in a near-OOM: use the magic SysRq key.
Thanks for the tip! If my Linux were ever to start locking up regularly, I will apply it.
But right now (so I don't have to give up the use to which I currently put my SysRq key) I would prefer some method for determining after I forcefully powered down the computer, then powered up again, whether the lockup or slow-down that motivated the force-power-down was caused by a near-OOM condition.
I don't think so, sorry. The kernel emits a few messages when an OOM is detected, including the tasks killed to free memory, but in a near-OOM probably nothing: the system is technically still working normally, though very slowly.
> Add a (small) swap. It doesn't have to be a partition: nowadays most filesystems support swap file. Just create an empty file and mark it as swap.
My experiences with swap on Linux have been similarly bad. If even brief memory pressure forces the kernel to move things to swap, the only way to revert that in any reasonable timeframe is to unmount the swap partition or to restart the machine.
Meanwhile using Windows with a swap file of twice the size of physical RAM runs smooth as butter. I have a 200GB swap file right now and no problems.
>I can't understand how can it be possible that, on Windows, whenever I run out of RAM the OS kills whatever process is consuming too much and the computer keeps running flawlessly.
I've long wondered this too. How does Windows handle memory pressure differently?
I avoid swap since it needs to be encrypted to protect sensitive data written out from memory to disk. Instead I reserve more memory for the kernel vm.min_free_kbytes based on the installed ram and also based and some redhat suggestions, reserve more memory in vm.admin_reserve_kbytes and vm.user_reserve_kbytes, adjust vm.vfs_cache_pressure based on server role and finally set vm.overcommit_ratio to 0. This worked well on over 50k bare metal servers with no swap. OOM was extremely rare outside of dev. OOM basically only happened with automation had human induced bugs that deployed too many java instances to a server. All of the servers had anywhere from 512GB to 3TB ram and nearly all the memory was in use at all times.
The kernel OOM killer is only concerned about kernel survival. It isn't designed to care about user perception of system responsiveness.
That's what resource control via cgroups is about. Fedora desktop folks (both GNOME and KDE) are working on ensuring minimum resources are available for the desktop experience, via cgroups, which then applies CPU, memory, and IO isolation when needed to achieve that. Also, systemd-oomd is enabled by default. The resource control picture isn't completely in place yet, but things are much improved.
cgroups often make the situation worse, not better, by insisting that a small memcg drop caches because that control group is full while the system overall has plenty of resources. This can lead to a system severely swapping for no apparent reason.
Putting desktop apps into individual cgroups is one of the more counter-productive ideas that has cropped up lately.
Huh? I have never seen desktop Windows killing process due to out of memory -- does it even do that?
It does thrash much more gracefully than Linux, though. In fact the "your computer is low on memory" prompt actually can show up even when severely thrashing, something utmost impossible in Linux (even starting something like zenity may take hours..).
You can already disable the linux memory overcommit feature if you want linux to never allow more memory to be allocated than exists. However, you may run into problems with programs which rely on the ability to allocate more memory than they need, or if you computer has low amounts of memory.
The reason is that Windows doesn’t have fork(), and therefore doesn’t have to promise huge multiples of the available memory only to be left holding the bag when that fiction failed. Look up “overcommit” if you’re interested.
