This is actually wrong. For some reason (probably because of the way it's implemented internally) Linux includes shared memory allocations in the "cached" figure. If you calculate the amount of free memory using "free -m" in the way this web page advises, you'll think that memory is free when it really isn't - unlike true cache, shared memory cannot be discarded when the space is needed for something else.
I've actually had buggy desktop applications use several gigabytes of RAM as shared memory, so this can make a big difference.
Linux also counts files on tmpfs as part of the cache number--it's really annoying that no number in top can serve to show you a decent approximation of how many page-size malloc()s you can do before OOM (with overhead and fragmentation and so on, of course it's infeasible to get an exact number).
It's worse. No line of /proc/meminfo has that value, and since not all of the cache can be instantly discarded (I didn't know that), I don't see any combination of them that would help.
That sentence:
> There are no downsides, except for confusing newbies.
Just isn't true. There is the downside that we can't know how much memory is available.
Yeah, and obviously some (though not all) forms of shared memory are actually implemented using tmpfs behind the scenes which makes life even more interesting. I think the problem I encountered involved SysV shared memory which isn't backed by tmpfs and does show up in /proc/meminfo, whereas other types may not.
he means the rationale behind developing it that way was due to the way other stuff is laid out internally, not "the number shows up because the existing code codes for it and causes it to show up".
I used to have a similar problem on OS X. When I was working on a 32-bit app that could eat tons of memory, I ended up using a bunch of weird heuristics and allocation tracking just to get a probable assurance that the program could recover properly from running out of address space.
How bizarre. In the 1993-94 time frame, one of the big talking points about Windows NT was that it was the only OS with an integrated file system cache. That wasn't true even at that time, as Solaris also had such a cache, but such is the nature of propaganda.
I'm tickled that what once was a big "technical" advantage becomes a major source of confusion. Presumably, OSes of Windows NT heritage have retained file system caching, which offers a huge performance benefit. Why don't Windows users have confusion over this issue? Doesn't Windows offer an easy way to check or is the whole thing presented clearly, or is Windows so complicated not many people get this far?
I'll also draw a parallel to DUI - without a breath-o-meter, DUI is a lot like driving while distracted. It's impossible to quantify, and therefore not that much of a crime. With a breath-o-meter, or with "top", it's possible to put a number on drunkeness or lack of free memory, and the whole thing becomes a problem.
Because there are a lot of things in linux/unix-land that is completely braindamaged. But they have been with us for so long that they become impossible to change. Like dotfiles, the ps command and Linux memory handling.
Windows 8 has a nice graph which shows memory usage over time in which the filesystem cache is not visible. If it was included, memory usage would constantly hover at about 90% which wouldn't be very useful and very confusing.
When Windows XP was introduced famous tech journalist John Dvorak complained that "System Idle Process" was using 99% of his cpu and slowing down his computer. In later Windowses, System Idle Process is no more.
Also even when you subtract the fs cache and get number of megabytes free memory you have available. That number has almost no bearing on how many number of megabytes an application of yours can allocate. Linux memory management is much more complicated than that.
> Linux memory management is much more complicated than that.
I don't know if I'm being trolled, but memory management in any modern OS that makes proper use of shared memory is fundamentally very complicated. I suspect what is happening is Windows is lying to you to give you a simple (though essentially wrong) answer.
NT was able to use a simpler (maybe not simple, but simpler) design because it was written from scratch once we knew that caches and buffers were desirable. So it's got a unification of swap and cache that would be difficult to retrofit onto *nix.
It was at least changed to lie to avoid complaints from users.
I remember that running taskmgr.exe from an XP machine on Vista gave a more accurate info, ie. no lying to keep ignorant users happy.
Even htop doesn't show you everything it can by default. See the (default off) option "Detailed CPU time (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)"
And Dvorak's complaint about "System Idle Process" had a little bit more context. He was complaining that XP would become non-responsive while still being mostly idle (something I remember well from that era too). He could definitely have done a much better job of expressing it though:
IDLE-TIME PROCESS. Once in a while the system will go into an idle mode, requiring from five minutes to half an hour to unwind. It's weird, and I almost always have to reboot. When I hit Ctrl-Alt-Delete, I see that the System Idle Process is hogging all the resources and chewing up 95 percent of the processor's cycles. Doing what? Doing nothing? Once in a while, after you've clicked all over the screen trying to get the system to do something other than idle, all your clicks suddenly ignite and the screen goes crazy with activity. This is not right.
Well writing "hogging all the resources and chewing up cycles" shows he doesn't really get it at all. Not to mention that there's many reasons things can become non-responsive without using the CPU. Like a shitty driver blocking on some timeout.
Why don't Windows users have confusion over this issue?
Because windows users don't care about how much memory a particular application is using. New Windows developers get confused by it all the time. There are plenty of questions on stack overflow where someone is using task manager to try and measure their application's working set.
Not sure what you mean. There is a general lack of sophistication among windows users, so they believe what the system is telling them if they go so far as to even ask the question. Among developers it is well known that, that number is at best wrong. If you actually want to know that number use a memory profiler. If I wanted to know about memory consumption on Linux I wouldn't be looking at top I would be looking at valgrind.
Top and task man are like stepping on the scale to keep track of your weight the exact number doesn't mean much, and it regularly varies in an unmeaningful way vs a memory profiler which would be like a high precision body fat measurement. Right tool for the job and all that crap.
From what I can tell, Windows does not show the disk cache allocation in Task Manager. So even though all the RAM is likely in use, that is hidden from the user. You can probably see it in PerfMon, as that is less "typical user" oriented.
In windows 7, take "available" and subtract "free", that's your cache usage.
Windows Vista started using RAM very aggressively for caching, including pre-filling cache with frequently used pages (apps/data) from disk... ReadyBoost was basically an extension of this...
> Windows' Task Manager has generally counted memory occupied by the disk cache as "available."
Windows Vista got lots of complaints about memory usage, because Task Manager listed memory as Total, Cached, and Free. People saw that "Free" was close to 0, and complained that Windows Vista ate all their RAM.
Windows 7 changed this to: Total, Cached, Available, and Free. People saw that they had a lot of "Available" memory, so they didn't panic. Well, most of them didn't panic.
Windows 8 removed the "Free" line entirely, and now lists memory as: Total, Available, and Cached. If you care about the actual amount of free memory, you can run Resource Monitor or Process Explorer.
htop author here. It was nice seeing the table in the article with the memory split in three colors and thinking "well, but that's how I already split it in htop, isn't it?".
The "Linux calls it" column should probably be called "top calls it"...
It was accidental too. I was just showing my coworker something and I have a bad habit of highlighting stuff I am reading and randomly clicking on text, and I noticed that column was selected to be sorted. It blew my mind, this was in an SSH session logged into a remote server.
+1 for htop, it not only shows excellent memory stats in an easy to absorb way but it also shows CPU cores usage. Definitely one of my "wtf is going on?" tools.
Yikes! Hopefully not on production systems. I've had fileservers which needed 60+GB of dentry cache to serve up files in a reasonable time, something like that could bring the site to a crawl for 10 minutes or more.
When my circle of friends found that site it became an in-joke. I'd never heard of it, and eventually they came up to me and asked "Can you download more RAM?".
This is the worst piece of Linux advice ever. I use Linux for many years and every time I had a memory leak or a process using too much memory, every google search about the problem would lead to some "guru" giving this advice.
This explanation which has been replicated thousands of times on every linux forum on the planet, prevents people from solving real problems. It started as a useful piece of information but nowadays it is spam.
Yes, this is exactly what I mean. Maybe I should've describe it as noise rather than spam.
But it isn't only a Google search issue. Every time you will ask a question about a memory problem in Linux, everybody will give you this answer and classify you as newbie. When you explain them that this isn't the case, they will simply leave the thread/question and nobody would ever bother reading your problem again.
People just go in denial mode because of this common knowledge, as Linux can't have memory leaks or any memory related problem.
I think what you're running into is the shallow knowledge of people on forums/mailing lists. With a few exceptions, the people who actually know the most about the Linux kernel, glibc internals, etc. do not frequently post on StackOverflow. They're too busy, you know, working on the kernel and shit.
So you get Joe Chucklefuck, he installed Ubuntu three months ago and posted on the Ubuntu forums about how he has no free RAM and Linux sucks. Someone showed him this page, and now when he sees your question about why malloc isn't working right, he thinks, "Aha! A memory problem! I shall post that linux ate my ram site!" Meanwhile, Ulrich Drepper isn't reading the list, even though he could tell you that the version of glibc you're using has a known bug in malloc. (He'll also tell you that you're stupid, but that's just because he's an asshole)
Exactly. But I believe that developers have their hands full solving real problems, no need to ask them play level 1 support too.
So when I find an issue that I can't be sure if it isn't my own fault, or I don't know the root cause of it, I try to gather some information before heading to the bugzilla. Memory issues are the only thing it is absolutely impossible to gather any useful information. Your only option is to learn C and debugging tools.
A suspected memory-leak is definitely the sort of thing that you should report to the authors/maintainers. Even if you think it is triggered by something that you are doing wrong, their program should not respond to that incorrect input by leaking memory.
Places like stackoverflow and IRC support channels are really only useful if the answer you are looking for can probably be found in some form of documentation somewhere. For things like suspected bugs or issues with specific implementation, it is best to go to the source (either the actual source, or the responsible devs (which almost always means going to a mailing list).
Some particular projects are infamously dismissive specifically of memory leaks, really no matter how much detail you put into your report (Mozilla, I am looking squarely at you), but those should be the minority. Typically if you see run-away memory usage, and are able to provide enough details about what you are seeing, the developers should be receptive. That's been my experience anyway.
The gap between 'Joe Chucklefuck' and the lead developer for glibc is wide enough to fit a whole lot of people who can triage a bug without necessarily being skilled (or interested) enough to submit patches.
I think describing the phenomenon as 'noise' is reasonable. You find a moderate number of answers with a reasonable degree of certainty as to their accuracy and a large number of answers that are just whatever came to mind first (these aren't necessarily provided by completely distinct groups of people). The result is a lot of almost-but-not-quite-related nonsense that drowns out the answers with any relevant content.
This probably isn't purely a problem of numbers, either. Even attempting to be helpful takes a certain amount of effort that naturally limits the impact of the former group at any particular instant. It's also easy (as evidenced by this very article) to take helpful information and render it non-helpful by repeating it without context.
Just give more information right off the bat. Don't say "XYZ is chewing up lots of RAM"; the best case scenario if you say that is they have to respond back asking for more information. Give that information to them in the first place "XYZ starts at XXXXMB shared and XXMB resident. Over five minutes it steadily climbs to XXXXMB resident. Shared remains constant at XXXMB."
Doing this, I can't say I've ever been brushed off using this page.
I had similar experience with MacOS memory issues, and all the answers I was getting: "MacOS is superior to Windows, its just Cache, the memory is actually free". Yet my applications could not allocate memory and were hogging whole OS as a consequence.
Back on topic, never had similar 'cache' problems on Linux though.
Wow, I can't believe how many times I've had to explain this to people as they log on to shared VNC servers. "I thought this had 16G of RAM free, I asked for a 16G machine!". "It does, stop trying to read top!"
I hate the invention of "burstable" RAM on VPSes for this reason. Eventually your IO cache grows until it has most of your guaranteed RAM. Now every new process is playing a game of Russian roulette with your hypervisor to see if it can actually allocate into your "burstable" RAM what the kernel is reporting is freely available.
Actually this is a general issue when running multiple Linux VMs under a hypervisor. In that case you don't want them filling up RAM with prefetch data from disk because that's likely to cause contention and reduce page sharing between VMs.
But why would you do that? (The only time I've ever used that is when benchmarking and you need a clean cache between runs). Otherwise the kernel will just empty it as needed when you run a program.
First, it's not why. It's that this is possible. Please don't say "this is impossible because this doesn't make sense".
Second, I agree with you. Normally people don't need to do this unless they hate their hard drives. :-) But there's some special cases, for example when the disk is "volatile", or when you want a huge chunk of continuous physical address and you want to tell the OS to stay away and don't touch any pages inside.
PS: An interesting thing is that Windows people always seek for a way to enlarge disk cache. It's fine on server versions, but on desktop versions such as Windows 7, if I don't remember wrong, disk cache is capped at ~4GB(not sure if globally or per file). Oh well.
I can't answer for everyone, but I can tell you why I have a cron job dropping my cache every several minutes:
The application I develop consumes most of the ram on my machine when running, and takes several minutes to rebuild and start up to begin with. When most of my memory is being used for the cache, this process takes several minutes longer, because I'm making millions upon millions of calls for more memory -- and each one has to get some of that cached memory back for itself. If I simply drop the cache all at once, every minute, it takes a split second. If I shrink it over a million increments, it takes around a minute.
Even typing that I feel I must be doing something wrong; and yet, it worked.
Your app is making more calls to malloc when there is memory being used by disk cache, or you mean something else by "millions upon millions of calls for more memory"?
You control hope many times you call malloc. Just call it once with what you need.
that's funny, like avery complex or "smart" algorithm, if it works it's cool, but when something doesn't work, it makes it very hard to pinpoint a culprit.
I'm more and more inclined to either use stupider algorithm that at least I can explain to the user, incremental stuff where they can check intermediate results or taking into account the fact that this algorithm has to be conveyed to users in one way or another since inception.
I guess in this case a measure of the swap activity would be a good indicator that the current limiting factor is the RAM, but it also has to be credible. Somehow having 16G written on the box of the ram and 7G written in the "available ram" of a software makes everything more credible that saying "I have a pressure of 1.5madeupunit on the ram front". Human factors, again.
I've been doing performance analysis and debugging as part of being a unix/linux sysadmin now for something like 20 years. And even though I have many well-cherished anecdotes about explaining the VM free list to software architects who were the go-to guys for the vision of our entire global platform, complete with PhDs, I still don't want to go back to the bad old days without a shared buffer cache. There's a reason why that's there, and you don't want to go back to 1992-era memory management.
It also makes perfect sense from the perspective of writing the software. The free list really is the list of VM pages that have been free'd and that has a specific meaning, and changing that name internally would be terrible. You just need to understand how the machine actually works.
Another thing, you know how sometimes a USB stick needs some time to un-mount? This non-sync'ed data (say, a copy of a big file) eventually has to be written to the physical disk, and through the little pipe, takes time. Find out how much of this "dirty" buffer you have yet to physically write to the device, while giving the kernel 'hints' to sync the dirty blocks quickly, with:
while true; do sync && grep Dirty /proc/meminfo; done
Lots of gold to be mined from /proc/meminfo .. bonus points for wiring it up to gnuplot so you can get a real picture of the dynamics of the Linux kernel .. ! :)
I do this a lot, on machines which are annoyingly vendor-tweaked to allow a lot of dirty pages. Copying files to USB sticks happens instantly, but the unmount then takes an age - like, 20 minutes. This is a way to get a measure of the progress of the unmount. I generally have a sleep in there as well though ;)
Best way to check how much ram is actually free is to use a higher level tool like a task manager (from various DEs) or a command line tool like htop, they report the actual memory usage of processes
Not strictly accurate, in the "Why would you want to do this" regard.
Ages ago I was hacking linux onto an ARM soc that claimed to have memory it didn't (it wasn't aware of the display's DMA segment). It was near the top of the address space, so generally speaking it didn't cause too many problems, but until I could debug it I needed to keep memory usage low, wchih meant disabling the disk cache. What a shitshow that was :/
I always run Ganglia on my servers. You can see memory used, memory shared, memory cached, memory buffered, and memory swapped all over time as well as total in-core memory. This has saved me (and my team) from ever having to misinterpret command line output. Takeaway, Linux is smart about using memory, probably smarter than you. http://ganglia.sourceforge.net/
I think it's still disinformative. Disk cache also uses RAM which is allocted and reserved by applications, of course data is swapped out in this situation. o you'll need more swap, but it isn't a bad thing at all. Even OS/2 did this a long time ago, nothing new at all.
You don't, but if something decides to grab memory, the OOM-killer will kill processes abruptly. Swap may allow the system to limp along until there's some solution.
It's not like 10 or 20GB HD these days matter much either.
Because sometimes you (read applications) need to use all/most of your memory. Ability to swap out all the not being used memory to disk maximizes amount of memory available to applications.
Because you can (and usually do) have areas of memory and disk that are more or less active, meaning it can make sense to evict some ram pages to disk while still caching other disk pages.
OS X does not behave exactly in this manner, but is indeed confusing.
Linux never needs a 'purge' or equivalent. Doing a 'drop_cache' only slows the system slightly because nothing is cached. 'sudo purge' can prevent OS X from using swap – i.e. running out of RAM even though there is RAM it could reclaim. This behavior has been tweaked but not fixed in Mavericks.
Actually, purge is completely unrelated to swap. Purge forces disk cache to be flushed to disk, and can be used to approximate a cold disk buffer cache scenario for performance analysis.
Some small amount of swap usage (compared to total RAM) is perfectly normal, by the way. More cache, for instance, is a better use of RAM than avoiding swap usage by keeping never-accessed pages present.
Ehm. But sometimes stupid linux kernel prefers moving something usefull to swap and caching some shit instead of keeping stuff in ram and not doing caching.
So linux sometimes eats your RAM (but in different way).
Leaving your memory with no use at all would be the bug. It is a free resource that could be used to improve system performance at basically no cost (the disk is already read, the memory is still available immediately for any app that request it, etc), so it is a good default to let linux do it.
But could be cases where it could be side effects of it (i.e. running in a VM could make less memory available for the host, and it could be caching disk already) so you could consider to disable it sometimes.
Just a difference in semantics -- I mean, c'mon UX in a utility like top? which is probably specified in some Unix convention anyway. Pure BSD Unix do this as well in their own way.
The prefetched data is managed by the OS. If a program needs the RAM, the prefetched data is immediately discarded. Prefetching never creates RAM pressure.
This. It blows my mind how many people who write software don't understand how OS caching and virtual memory work. They are great filter interview questions ;)
At least one. Potentially more. The actual multiple depends strongly on the specific SSD and memory configurations in question, and on the benchmark of interest.
Do you have any proof of that? "Free" memory uses as much energy as "used" memory, there is no distinction in SD|DDR-RAM. And if the OS is smart, it only has to keep a simple index of memory pages still relevant. I'd argue that this has a measurable battery-hit.
I've actually had buggy desktop applications use several gigabytes of RAM as shared memory, so this can make a big difference.