
Linux ate my RAM  - wallzz
http://www.linuxatemyram.com/
======
makomk
This is actually wrong. For some reason (probably because of the way it's
implemented internally) Linux includes shared memory allocations in the
"cached" figure. If you calculate the amount of free memory using "free -m" in
the way this web page advises, you'll think that memory is free when it really
isn't - unlike true cache, shared memory cannot be discarded when the space is
needed for something else.

I've actually had buggy desktop applications use several gigabytes of RAM as
shared memory, so this can make a big difference.

~~~
gue5t
Linux also counts files on tmpfs as part of the cache number--it's _really_
annoying that no number in top can serve to show you a decent approximation of
how many page-size malloc()s you can do before OOM (with overhead and
fragmentation and so on, of course it's infeasible to get an exact number).

~~~
marcosdumay
It's worse. No line of /proc/meminfo has that value, and since not all of the
cache can be instantly discarded (I didn't know that), I don't see any
combination of them that would help.

That sentence:

> There are no downsides, except for confusing newbies.

Just isn't true. There is the downside that we can't know how much memory is
available.

~~~
makomk
Yeah, and obviously some (though not all) forms of shared memory are actually
implemented using tmpfs behind the scenes which makes life even more
interesting. I think the problem I encountered involved SysV shared memory
which isn't backed by tmpfs and does show up in /proc/meminfo, whereas other
types may not.

------
jared314
Previous Discussions:

[https://news.ycombinator.com/item?id=3698652](https://news.ycombinator.com/item?id=3698652)
(1 year ago)

[https://news.ycombinator.com/item?id=6368562](https://news.ycombinator.com/item?id=6368562)
(2 months ago)

[https://news.ycombinator.com/item?id=1831322](https://news.ycombinator.com/item?id=1831322)
(3 years ago, no comments)

------
bediger4000
How bizarre. In the 1993-94 time frame, one of the big talking points about
Windows NT was that it was the only OS with an integrated file system cache.
That wasn't true even at that time, as Solaris also had such a cache, but such
is the nature of propaganda.

I'm tickled that what once was a big "technical" advantage becomes a major
source of confusion. Presumably, OSes of Windows NT heritage have retained
file system caching, which offers a huge performance benefit. Why don't
Windows users have confusion over this issue? Doesn't Windows offer an easy
way to check or is the whole thing presented clearly, or is Windows so
complicated not many people get this far?

I'll also draw a parallel to DUI - without a breath-o-meter, DUI is a lot like
driving while distracted. It's impossible to quantify, and therefore not that
much of a crime. With a breath-o-meter, or with "top", it's possible to put a
number on drunkeness or lack of free memory, and the whole thing becomes a
problem.

~~~
bjourne
Because there are a lot of things in linux/unix-land that is completely
braindamaged. But they have been with us for so long that they become
impossible to change. Like dotfiles, the ps command and Linux memory handling.

Windows 8 has a nice graph which shows memory usage over time in which the
filesystem cache is not visible. If it was included, memory usage would
constantly hover at about 90% which wouldn't be very useful and very
confusing.

When Windows XP was introduced famous tech journalist John Dvorak complained
that "System Idle Process" was using 99% of his cpu and slowing down his
computer. In later Windowses, System Idle Process is no more.

Also even when you subtract the fs cache and get number of megabytes free
memory you have available. That number has almost no bearing on how many
number of megabytes an application of yours can allocate. Linux memory
management is much more complicated than that.

~~~
ris
> Linux memory management is much more complicated than that.

I don't know if I'm being trolled, but memory management in any modern OS that
makes proper use of shared memory is fundamentally very complicated. I suspect
what is happening is Windows is lying to you to give you a simple (though
essentially wrong) answer.

~~~
lmm
NT was able to use a simpler (maybe not simple, but simpler) design because it
was written from scratch once we knew that caches and buffers were desirable.
So it's got a unification of swap and cache that would be difficult to
retrofit onto *nix.

~~~
AnthonyMouse
The Linux kernel and the NT kernel are approximately the same age.

------
rdtsc
A quick practical answer is to use htop and look at the memory bar.

* Green = resident memory used (probably what you want most of the time)

* Blue = buffers

* Yellow = cache

~~~
hisham_hm
htop author here. It was nice seeing the table in the article with the memory
split in three colors and thinking "well, but that's how I already split it in
htop, isn't it?".

The "Linux calls it" column should probably be called "top calls it"...

~~~
rdtsc
Thank you for writing htop, it is very useful tool.

I just about fell off my chair when I realized I could click with my mouse on
the columns to sort them!

~~~
gregf
I always forget the key bindings for this, you just changed my life!

~~~
rdtsc
It was accidental too. I was just showing my coworker something and I have a
bad habit of highlighting stuff I am reading and randomly clicking on text,
and I noticed that column was selected to be sorted. It blew my mind, this was
in an SSH session logged into a remote server.

------
hippich
That's how I check true free memory available:

    
    
      root@joe-pc:~# free -m
                   total       used       free     shared      buffers     cached
      Mem:         32057      30347       1710          0         1312      21568
      -/+ buffers/cache:       7466      24591
      Swap:            0          0          0
      root@joe-pc:~# sync; echo 3 > /proc/sys/vm/drop_caches
      root@joe-pc:~# free -m
                   total       used       free     shared      buffers     cached
      Mem:         32057       5628      26429          0            4        891
      -/+ buffers/cache:       4732      27325
      Swap:            0          0          0
      root@joe-pc:~#

~~~
dice
Yikes! Hopefully not on production systems. I've had fileservers which needed
60+GB of dentry cache to serve up files in a reasonable time, something like
that could bring the site to a crawl for 10 minutes or more.

~~~
hippich
Good point. I am doing it on my laptop and server without load. So this is not
a big problem in my cases.

------
pearjuice
Then you should [http://downloadmoreram.com](http://downloadmoreram.com)

~~~
unimpressive
When my circle of friends found that site it became an in-joke. I'd never
heard of it, and eventually they came up to me and asked "Can you download
more RAM?".

I don't think they were expecting my answer:

"Sure you can, just use plan 9."

------
daGrevis
The page seems to be down.

Here's how it looked:
[http://web.archive.org/web/20131011053519/http://www.linuxat...](http://web.archive.org/web/20131011053519/http://www.linuxatemyram.com/)

------
nly
Forget the kernel, someone really needs to do chromeatemyram.com

------
andmarios
This is the worst piece of Linux advice ever. I use Linux for many years and
every time I had a memory leak or a process using too much memory, every
google search about the problem would lead to some "guru" giving this advice.

This explanation which has been replicated thousands of times on every linux
forum on the planet, prevents people from solving real problems. It started as
a useful piece of information but nowadays it is spam.

~~~
jlgreco
How does this understanding prevent people from solving real problems, simply
by virtue of it being commonly cited? I don't understand your complaint.

~~~
marcosdumay
I understand this is a complaint about Google. It won't help you find pages
that cite a problem when a diffrent, but similar problem is too popular.

I've seen that happening several times, but I'm not the GP, so I may still be
missunderstanding him.

~~~
andmarios
Yes, this is exactly what I mean. Maybe I should've describe it as noise
rather than spam.

But it isn't only a Google search issue. Every time you will ask a question
about a memory problem in Linux, everybody will give you this answer and
classify you as newbie. When you explain them that this isn't the case, they
will simply leave the thread/question and nobody would ever bother reading
your problem again.

People just go in denial mode because of this common knowledge, as Linux can't
have memory leaks or any memory related problem.

~~~
jff
I think what you're running into is the shallow knowledge of people on
forums/mailing lists. With a few exceptions, the people who actually know the
most about the Linux kernel, glibc internals, etc. do not frequently post on
StackOverflow. They're too busy, you know, working on the kernel and shit.

So you get Joe Chucklefuck, he installed Ubuntu three months ago and posted on
the Ubuntu forums about how he has no free RAM and Linux sucks. Someone showed
him this page, and now when he sees your question about why malloc isn't
working right, he thinks, "Aha! A memory problem! I shall post that linux ate
my ram site!" Meanwhile, Ulrich Drepper isn't reading the list, even though he
could tell you that the version of glibc you're using has a known bug in
malloc. (He'll also tell you that you're stupid, but that's just because he's
an asshole)

~~~
andmarios
Exactly. But I believe that developers have their hands full solving real
problems, no need to ask them play level 1 support too.

So when I find an issue that I can't be sure if it isn't my own fault, or I
don't know the root cause of it, I try to gather some information before
heading to the bugzilla. Memory issues are the only thing it is absolutely
impossible to gather any useful information. Your only option is to learn C
and debugging tools.

~~~
jlgreco
A suspected memory-leak is definitely the sort of thing that you should report
to the authors/maintainers. Even if you think it is triggered by something
that you are doing wrong, their program should not respond to that incorrect
input by leaking memory.

Places like stackoverflow and IRC support channels are really only useful if
the answer you are looking for can probably be found in some form of
documentation somewhere. For things like suspected bugs or issues with
specific implementation, it is best to go to the source (either the actual
source, or the responsible devs (which almost always means going to a mailing
list).

Some particular projects are infamously dismissive specifically of memory
leaks, really no matter how much detail you put into your report _(Mozilla, I
am looking squarely at you)_ , but those should be the minority. Typically if
you see run-away memory usage, and are able to provide enough details about
what you are seeing, the developers should be receptive. That's been my
experience anyway.

------
caw
Wow, I can't believe how many times I've had to explain this to people as they
log on to shared VNC servers. "I thought this had 16G of RAM free, I asked for
a 16G machine!". "It does, stop trying to read top!"

Now I'll just point them to this website.

------
jbogp
Also HN ate your server apparently.

------
FlyingAvatar
I hate the invention of "burstable" RAM on VPSes for this reason. Eventually
your IO cache grows until it has most of your guaranteed RAM. Now every new
process is playing a game of Russian roulette with your hypervisor to see if
it can actually allocate into your "burstable" RAM what the kernel is
reporting is freely available.

~~~
vajrabum
Actually this is a general issue when running multiple Linux VMs under a
hypervisor. In that case you don't want them filling up RAM with prefetch data
from disk because that's likely to cause contention and reduce page sharing
between VMs.

------
v-yadli
[https://www.kernel.org/doc/Documentation/sysctl/vm.txt](https://www.kernel.org/doc/Documentation/sysctl/vm.txt)

You can drop cache by sending a byte into drop_cache.

~~~
mprovost
But why would you do that? (The only time I've ever used that is when
benchmarking and you need a clean cache between runs). Otherwise the kernel
will just empty it as needed when you run a program.

~~~
jrkatz
I can't answer for everyone, but I can tell you why I have a cron job dropping
my cache every several minutes:

The application I develop consumes most of the ram on my machine when running,
and takes several minutes to rebuild and start up to begin with. When most of
my memory is being used for the cache, this process takes several minutes
longer, because I'm making millions upon millions of calls for more memory --
and each one has to get some of that cached memory back for itself. If I
simply drop the cache all at once, every minute, it takes a split second. If I
shrink it over a million increments, it takes around a minute.

Even typing that I feel I must be doing something wrong; and yet, it worked.

~~~
deathanatos
Your app is making more calls to malloc when there is memory being used by
disk cache, or you mean something else by "millions upon millions of calls for
more memory"?

You control hope many times you call malloc. Just call it once with what you
need.

------
nraynaud
that's funny, like avery complex or "smart" algorithm, if it works it's cool,
but when something doesn't work, it makes it very hard to pinpoint a culprit.

I'm more and more inclined to either use stupider algorithm that at least I
can explain to the user, incremental stuff where they can check intermediate
results or taking into account the fact that this algorithm has to be conveyed
to users in one way or another since inception.

I guess in this case a measure of the swap activity would be a good indicator
that the current limiting factor is the RAM, but it also has to be credible.
Somehow having 16G written on the box of the ram and 7G written in the
"available ram" of a software makes everything more credible that saying "I
have a pressure of 1.5madeupunit on the ram front". Human factors, again.

~~~
lamontcg
I've been doing performance analysis and debugging as part of being a
unix/linux sysadmin now for something like 20 years. And even though I have
many well-cherished anecdotes about explaining the VM free list to software
architects who were the go-to guys for the vision of our entire global
platform, complete with PhDs, I still don't want to go back to the bad old
days without a shared buffer cache. There's a reason why that's there, and you
don't want to go back to 1992-era memory management.

It also makes perfect sense from the perspective of writing the software. The
free list really is the list of VM pages that have been free'd and that has a
specific meaning, and changing that name internally would be terrible. You
just need to understand how the machine actually works.

------
fit2rule
Another thing, you know how sometimes a USB stick needs some time to un-mount?
This non-sync'ed data (say, a copy of a big file) eventually has to be written
to the physical disk, and through the little pipe, takes time. Find out how
much of this "dirty" buffer you have yet to physically write to the device,
while giving the kernel 'hints' to sync the dirty blocks quickly, with:

    
    
        while true; do sync && grep Dirty /proc/meminfo; done
    

Lots of gold to be mined from /proc/meminfo .. bonus points for wiring it up
to gnuplot so you can get a real picture of the dynamics of the Linux kernel
.. ! :)

~~~
voltagex_
This seems a bit magic, what does it do? (stuck in Windows-land at the moment)

~~~
lcrs
I do this a lot, on machines which are annoyingly vendor-tweaked to allow a
lot of dirty pages. Copying files to USB sticks happens instantly, but the
unmount then takes an age - like, 20 minutes. This is a way to get a measure
of the progress of the unmount. I generally have a sleep in there as well
though ;)

------
Morgawr
Best way to check how much ram is actually free is to use a higher level tool
like a task manager (from various DEs) or a command line tool like htop, they
report the actual memory usage of processes

------
richo
Not strictly accurate, in the "Why would you want to do this" regard.

Ages ago I was hacking linux onto an ARM soc that claimed to have memory it
didn't (it wasn't aware of the display's DMA segment). It was near the top of
the address space, so generally speaking it didn't cause too many problems,
but until I could debug it I needed to keep memory usage low, wchih meant
disabling the disk cache. What a shitshow that was :/

------
dmourati
I always run Ganglia on my servers. You can see memory used, memory shared,
memory cached, memory buffered, and memory swapped all over time as well as
total in-core memory. This has saved me (and my team) from ever having to
misinterpret command line output. Takeaway, Linux is smart about using memory,
probably smarter than you.
[http://ganglia.sourceforge.net/](http://ganglia.sourceforge.net/)

------
Sami_Lehtinen
I think it's still disinformative. Disk cache also uses RAM which is allocted
and reserved by applications, of course data is swapped out in this situation.
o you'll need more swap, but it isn't a bad thing at all. Even OS/2 did this a
long time ago, nothing new at all.

------
jvdh
So basically this is the inverse of swap. Using memory to make disk
reads/writes faster, instead of using disk to make more memory.

If Linux is consistently doing this, why would you still need a swap
partition?

~~~
ableal
> why would you still need a swap partition?

You don't, but if something decides to grab memory, the OOM-killer will kill
processes abruptly. Swap may allow the system to limp along until there's some
solution.

It's not like 10 or 20GB HD these days matter much either.

~~~
rosser
Would you rather consume physical pages of RAM by storing less recently used
programs and data, or by caching more recently used disk pages?

Myself, I'd generally prefer to evict the older stuff to swap in order to
cache the newer stuff. But maybe that's just me.

~~~
ableal
Quite a while ago, HP-UX (v9, if memory serves) got this balance wrong - the
system was keeping files in RAM, and kicking active processes to swap.

Much gnashing of teeth ensued until HP fixed that ...

------
Damogran6
FWIW, it's relevant to OS X too (even Mavericks)...'sudo purge' will reclaim
some of the resources.

~~~
sounds
OS X does not behave _exactly_ in this manner, but is indeed confusing.

Linux never needs a 'purge' or equivalent. Doing a 'drop_cache' only slows the
system slightly because nothing is cached. 'sudo purge' can prevent OS X from
using swap – i.e. running out of RAM even though there is RAM it could
reclaim. This behavior has been tweaked but not fixed in Mavericks.

~~~
Zr40
Actually, purge is completely unrelated to swap. Purge forces disk cache to be
flushed to disk, and can be used to approximate a cold disk buffer cache
scenario for performance analysis.

Some small amount of swap usage (compared to total RAM) is perfectly normal,
by the way. More cache, for instance, is a better use of RAM than avoiding
swap usage by keeping never-accessed pages present.

------
talles
Or you can just [http://downloadmoreram.com/](http://downloadmoreram.com/)

(joke)

------
usamec
Ehm. But sometimes stupid linux kernel prefers moving something usefull to
swap and caching some shit instead of keeping stuff in ram and not doing
caching.

So linux sometimes eats your RAM (but in different way).

~~~
Amadou
For situations where you have a time-critical app, you can pin the app's
memory into RAM with the mlock() family of functions.

There may be a command-line utility that lets you pin an arbitrary running a
process too, I haven't looked for it on linux.

------
xacaxulu
I just love the website design :-) It makes me happy.

------
AsymetricCom
ITT tech startups learn about Lunix.

------
jheriko
right. so there is a bug in linux. gotcha.

~~~
annnnd
No. Well, not technically... There is a UX bug in Linux. Not the only one at
that, but they don't count much in Linux-land.

~~~
hansbogert
Just a difference in semantics -- I mean, c'mon UX in a utility like top?
which is probably specified in some Unix convention anyway. Pure BSD Unix do
this as well in their own way.

------
jaseemabid
Lovely! This domain could iframe that SO question :)

~~~
daGrevis
What question?

~~~
Schiphol
Why, _that_ one, of course.

~~~
lucasnemeth
I've thought the answer to that one was 42.

~~~
Samuel_Michon
ITT: reddit-style comments that don’t get downvoted. I’m all for it.

[http://i.imgur.com/lrlFq0y.jpg](http://i.imgur.com/lrlFq0y.jpg)

------
free652
Windows loves to cache and "prefetch" apps, I still need to figure out how to
disable prefetching since I have a very fast SSD.

~~~
stephencanon
Memory is still an order of magnitude faster than your “very fast SSD”.

~~~
dingaling
And memory is still an order of magnitude more expensive than SSD, per GB.

I just bought 4 GB of PC-6400 for a laptop at £40. A fast 128GB SSD is £100.

Summary: use RAM judiciously

~~~
sp332
The prefetched data is managed by the OS. If a program needs the RAM, the
prefetched data is immediately discarded. Prefetching never creates RAM
pressure.

~~~
voidlogic
This. It blows my mind how many people who write software don't understand how
OS caching and virtual memory work. They are great filter interview questions
;)

