
Help, Linux ate my RAM - ez77
http://www.linuxatemyram.com/
======
illumin8
I've tried to educate Oracle DBAs on why top is wrong and their memory really
isn't being used. It's painful and they often refuse to believe that I know
what I'm talking about, and that they should use the free -m command to see
what memory is actually available for use.

Is there any particular reason why Oracle DBAs are less likely to believe
this? Perhaps it's because most of them grew up in legacy UNIX environments
rather than Linux.

~~~
krobertson
I think this is a pretty common misconception overall. Working at startups,
often find devs with multiple hats sometimes doing ops tasks. Seen many who
hop on a system trying to diagnose some issue, fire up top and proclaim "OMG,
the problem is we're running out of memory!"

2nd is explaining virtual/resident set size.

~~~
francoisdevlin
Okay, I'm one of those devs. What is this virtual/resident set size thing
you're talking about?

EDIT: Thank you for all the helpful responses!

~~~
maushu
I'm a developer too and this is what I understand (might be wrong):

\- Virtual memory is basically "abstract memory" that is linked (mapped) to
RAM or HDD, this means that by accessing this "memory" you might actually be
accessing the HDD and, because of this, larger than physical ram.

\- Virtual set size is the allocated virtual memory (the above) to the
process.

\- Resident set size is the allocated physical (RAM) memory to the process.

\- Shared memory is memory that is shared on multiple processes, meaning that
if you have 10 processes using 10mb of resident memory each and have 2mb of
shared memory each, the total of resident memory used is not 100mb but 82mb.

~~~
scott_s
Virtual memory is actually more subtle than that.

At a mechanical level, virtual memory is permission from the operating system
to use addresses in your address space. It is so called because, as you point
out, it allows us to separate the concept of "memory for a process" from
"physical memory on a chip." The reason I further refine the concept is that
_allocating virtual memory does not allocate actual memory_. Let's look at an
example:

    
    
      void* addr = mmap(NULL, 10 * 1024 * 1024, PROT_READ | PROT_WRITE, 
                        MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
    

That call allocates 10 MB of virtual memory. But there is no physical memory
backing any of it. All that has happened is that the operating system has now
said "Okay, starting at the address I return to you, you can now access 10 MB
of memory. I will do all of the work of making sure physical memory backs the
virtual memory when you access it." That is, once I try to access the memory,
it will trigger a _page fault_ , and the OS will find a page in physical
memory to back my virtual memory. But until that happens, no memory - not in
RAM, not on disk - backs the virtual memory.

------
suboptical
Looks like their site went down, have a mirror:

[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://www.linuxatemyram.com/)
[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://www.linuxatemyram.com/play.html)

~~~
JshWright
Looks like HN ate all their RAM...

------
click170
It's cute that this pops up every few years, and I think it points to a steady
(if slow) attraction of new users to Linux.

~~~
scott_s
People say the same things on Mac:
<http://news.ycombinator.com/item?id=3584609>

It represents a fundamental misunderstanding of how modern OSes work. That
misunderstanding is not the problem; modern OSes are complex pieces of
software, and most people shouldn't have to understand them. OSes should just
work. The problem comes in when people who don't understand how they work get
the itch to "improve" their system.

~~~
llambda
I'm actually continually amazed that even some technically inclined people
misunderstand how memory works in /all/ modern OSes: recently I had a
discussion on Twitter that went something like this: "OS X is horrible, it
uses nearly all of my 8 gigs of RAM and my browser is horribly slow!" to which
I replied, "it's perfectly normal for the OS to appropriate the RAM in the
fashion you see in Activity Monitor; this doesn't mean your machine is slow
due to lack of RAM." Unfortunately, even after a lengthy discussion and
several forwarded links, I seem to have failed to make the case.

It would be interesting if someone more knowledgeable than I am were to do a
write up explaining memory usage in OS X, Windows, and Linux; would be an
awesome resource to share with curious tinkers who may be slightly misguided
in their understandings of the inner-workers of their computers.

~~~
flomo
On a few occasions, Safari has claimed gigabytes of memory and caused my
machine to start thrashing the swap. In that case, the correct answer is to
restart Safari, not take a refresher course on virtual memory allocation.

Windows Vista had an issue where copying large files would set off such a huge
swap-storm that OS became completely unresponsive for several minutes. People
gave all the same VM excuses then as well, but there obviously was something
wrong.

~~~
scott_s
Those aren't excuses, those are reasons: copying large files are the
pathological case for just about all caching techniques. You've now kicked out
everything useful from your cache with something that you will never use
again. It has little to do with virtual memory itself.

------
plaes
The cache can be cleared via `/proc/sys/vm/drop_caches`.

<http://linux-mm.org/Drop_Caches>

~~~
mjb
Yes it can, but you probably don't want to do that. You definitely don't want
to do it in an automated way when the machine is experiencing memory pressure.

There are very good reasons that Linux (and most other modern operating
systems) makes aggressive use of page caches and buffers. For the vast
majority of applications dropping these caches is going to reduce performance
considerably (disk is really really slow) and most applications for which this
isn't true are probably using O_DIRECT anyway.

The arguments in favor of page caching are: (a) disks have very high latency
(b) disks have relatively low bandwidth (c) for hot data RAM is cheaper disk
IO both in dollars and in watts [1] and (d) it's basically free because the
memory would have been unused anyway.

The arguments against page caching are: (a) occasionally the kernel will make
poor choices and do something sub-optimal and (b) high numbers in 'free' make
me feel better.

Too many inexperienced operators (or those experienced on other OSs) confuse
disadvantage (a) for disadvantage (b) and decide to drop caches using a cron
job.

[1] Old but good: ftp://ftp.research.microsoft.com/pub/tr/tr-97-33.pdf

~~~
plaes
Yes, it was actually sort of a response to that webpage that said it was not
possible to free this cached memory.

The cache dropping is actually useful when you are doing benchmarking...

~~~
mjb
> The cache dropping is actually useful when you are doing benchmarking...

Agreed.

My response was more to the "Let's Put 'echo 3 > /proc/sys/vm/drop_caches' In
Cron and Get Free RAM!!!!" thinking, which sadly seems to be widespread.

------
mwexler
firefoxatemyram.com is still available. Perhaps we can put a site up there as
well.

~~~
jff
Yep, firefox is currently 555 MB resident and using _1.5 GIGABYTES_ of virtual
memory space. Goddamn, firefox, you are a _pig_. Saddest thing? I've got
gmail, github, and maybe 10 static pages open.

Linux just _looks_ like it ate your RAM. Firefox straight up _does_ eat it.

~~~
ineedtosleep
Why is Firefox always mentioned in this context? Sure they have had a history
of it, but lately they've been fine and if one compares the combined spawned
processes of Chrome, Chrome typically has more memory consumption.

~~~
calloc
Remember that when you are accounting for those you also need to remove all of
the stuff that is paged into every process and is the same across them, such
as shared libraries and the like. Most likely Chrome is using less ram than
you think.

~~~
makomk
Chrome has a built-in task manager that takes this into account, as well as a
slightly more hidden "Stats for nerds" link in the task manager that gives
total memory usage. It's just genuinely very memory hungry once you add up all
its processes.

------
bshep
Is there any way to have top display this information?

~~~
viraptor
Use `htop` instead. It displays the information in a bit more verbose way.
You'll get each section of "used" memory colour-coded, so that the last yellow
area can be ignored as cache.

~~~
datagramm
yep +1 for htop.

------
ez77
I found this page because I was pretty confused about these issues, and still
am...

Question for the crowd: In this site the example given says that in reality
there are 869MB of used RAM. I'm comparing this with my VPS values, and would
like to know if this is the sum of some column in top. Is it? It looks like
it's pretty close to the sum of the SHR column. Does this make sense? Thanks
in advance.

~~~
mgedmin
You can't really do sums of top columns, because some memory is shared and
you'll end up double-counting it.

And you can't just subtract the shared memory numbers, because different sets
of pages are shared between different sets of processes, and top doesn't give
enough information to figure out what's actually happening where.

Running the pmaps tool on all pids and summing the Pss number is perhaps the
closest you can get to the actual memory use.

------
gghh
I found a good introduction on unix memory caching in chapter 3 (The Buffer
Cache) of 'Design of the UNIX Operating System',
[http://www.amazon.com/Design-Operating-System-Prentice-
Hall-...](http://www.amazon.com/Design-Operating-System-Prentice-Hall-
Software/dp/0132017997) . At least it was good for me (mathematician by
training, programmer by profession)

------
mark-r
Does the Linux disk cache push out pages that are used by running
applications? I believe Windows does it, though I can't state that for a fact.

------
glfomfn
And Hacker News ate your website :-/

