

Exo Research CTO still does not understand how OS file caching works in Windows - apower
http://exo-blog.blogspot.com/2010/02/rebutting-ars-technica.html

======
apower
Ok, I made the comment in <http://news.ycombinator.com/item?id=1138840>, that
"it's surprising that the CTO of a company specialized on system performance
doesn't know how OS cache memory works." It turns out that he really has no
clue.

To the people who said the Windows source code is needed to understand how it
works, hogwash. The file caching in Linux has been doing it this way for
years. Windows does it for years as well. It's fairly standard OS caching
technique.

~~~
barrkel
The file system cache doesn't contribute to committed bytes, the metric that
the Exo CTO is appealing to - in this respect the Ars Technica article is
mistaken. (Actually, the fact that the Ars Technica article used Task Manager
made me inclined to dismiss it as clueless.)

Estimating real memory pressure is tough. There's the working set; but the
working set may expand when you have sufficient physical memory, even though
the marginal performance gains may be small. Similarly, the file system cache
may expand when you have sufficient physical memory, but the marginal gains
become small there too.

Probably the best unified metric would be the amount of time applications
spend blocked on I/O, where that I/O is either paging committed bytes in from
disk, or data that would be cacheable - and would have been cached - on a
system with more memory. But that's extremely hard to predict.

~~~
sid0
If you've spent any amount of time at the Ars forums, you wouldn't refer to
Peter as "clueless".

Committed bytes is completely the wrong metric to appeal to, since it includes
pages that have been committed but have never been touched. Windows reserves
space for such pages in the page file (it never overcommits), but saying that
it leads to "memory pressure" is ridiculous.

~~~
barrkel
I wrote off the article, rather than the author. If I indeed read more of the
author, perhaps my opinion would be updated, but the article on its own
doesn't add much weight of authority. The trouble with Task Manager is that it
doesn't have sufficient per-process memory data to be useful. Process Explorer
is better, because you can view Working Set (Private|Shareable|Shared) and
Private Bytes. I'd view both private working set and private bytes as the most
important metrics for a Windows app, but depending on the circumstances (such
as how likely multiple instances are) other things may be considered.

Committed Bytes would be a measure of potential maximum physical memory usage
if every process were touching every page that it had available to it. Yes, no
process ever does that, but statistically there will be an average fraction of
that memory which is touched, so it can be interpreted as a proxy for physical
memory pressure. A poor one though, as different processes have different
patterns.

As to the Ars forums, with a brief perusal it seems similar to the programming
Reddit - i.e. a lot of opinionated and frequently misinformed noise,
interleaved with a small amount of signal. It looks like a time sink, to be
frank.

But to give Peter his due, I registered with the forums and scanned through
his past posts (DrPizza), and I can't find much that I disagree with; I
particularly agree with him on the consolification of PC games, and the lack
of MW2's appeal to a mature gamer, etc.

~~~
sid0
Peter did admit his mistake about superfetch (he thought the xpnet thing was
counting that too as used memory), but he's still right about the overall
metric being wrong -- see the comments to [http://exo-
blog.blogspot.com/2010/02/editorial-what-took-you...](http://exo-
blog.blogspot.com/2010/02/editorial-what-took-you-so-long.html) .

edit: and do you think he doesn't know about procexp/perfmon? :)

~~~
barrkel
Why do you insist on making this personal? I discounted an _article_ , not a
_person_.

By bringing the _person_ into this, you turn it into an irrational, emotional
thing, where _facts don't matter_. That's not where you want to be.

And at the risk of extending this thing far beyond its worthy lifetime, that
comment you link to proves very little. As I already said:

> _Committed Bytes would be a measure of potential maximum physical memory
> usage if every process were touching every page that it had available to it.
> Yes, no process ever does that, but statistically there will be an average
> fraction of that memory which is touched, so it can be interpreted as a
> proxy for physical memory pressure. A poor one though, as different
> processes have different patterns._

Allocating a huge bunch of memory and not touching it is one statistical
pattern, _but it's not particularly common_ , which is key. It's an outlier,
so it isn't strong evidence. On _average_ , committed bytes can be interpreted
as a proxy for memory pressure, but like I said, there's likely to be a fair
amount of variance there, so it's a poor proxy. There certainly wouldn't be a
1:1 relationship between committed memory and physical memory pressure; if I
had to guess, I would guess somewhere like 60% of committed memory should
ideally be in physical ram.

For example, my machine has 12GB of RAM, 3.5GB committed bytes total and 2.5GB
working set total. But that working set includes both private and shared
working set pages - calculating the active working set properly would require
dividing shared page sizes by the number of copies in different processes.

But just for fun, I plotted the log of working set values in MB along the X
axis, and the log of private bytes in MB along the Y axis, for each process,
and this is what I got:

<http://imgurl.filetac.com/img/68343980.png>

Looks like a pretty close correlation to me - yes, there are some spikes in
private bytes, but _on average_ , private bytes is a proxy for committed bytes
(I excluded shared pages for confounding reasons, as they are usually things
like EXEs and DLLs - not counted against commit charge), committed bytes is a
proxy for physical memory usage, which in turn is a proxy for physical memory
pressure.

------
ghshephard
Actually, there was one nugget of truth in the article, and that was the
_best_ way to determine if your operating system is low on memory, is to watch
it's swapping activity. This is probably true across all operating systems,
Linux, Solaris, etc..

An operating system will typically not swap out (or in) cache memory to disk,
so, as long as you are exhibiting no swapping behavior, (And you have swap
enabled) then you have enough memory. I think it was Adrian Cockroft, in fact,
in "Sun Performance and Tuning: Sparc & Solaris" who in fact said that was
_the_ way to determine if you have memory pressure on your operating system -
run vmstat and watch the si and so column for activity.

~~~
barrkel
An OS can game that kind of statistic, though, by spending more memory on what
Windows would call working set (the "hot" set of pages for a process which are
preferentially kept in physical memory) at the expense of the file system
cache.

Also, predictive prefetching may increase paging in, but in an async fashion,
so the application is never actually blocked. This may cause spurious
increases in the stats. You need to examine the amount of time threads are
actually blocked by the paging behaviour.

------
itgoon
Heh. I've been reading the various discussions about all of this, and having
considerable experience digging into internals to find performance problems,
it still seems to come down to one thing.

Does it _feel_ fast?

------
VBprogrammer
Regardless of whether the CTO is correct or not, the tone and poor attitude of
his posts have convinced me that he is not someone I would want to do business
with.

------
kevingadd
I'm not even sure the facts in this case matter if the CTO is willing to make
such a fool of himself in public. Even if I agreed with his point of view,
it's impossible for me to take him seriously when he writes such needlessly
incendiary rants on his company's official blog. It reminds me of conspiracy
theorists.

