

An experiment: Shared memory vs. files - HerraBRE
http://bre.klaki.net/blog/2011/12/27/

======
rbehrends
Using read()/write() to access shared files is a bad idea because you have no
control over buffering. Just because you've written something to a file does
not mean it actually has been written somewhere where other processes can see
it when they try to.

However, using shared files for IPC is very practical (I've done it before),
but you want to use mmap() in conjunction with at least memory barriers or
mutexes/semaphores/condition variables that work between processes. If you
just need memory barriers, then the atomic_ops lib
(<http://www.hpl.hp.com/research/linux/atomic_ops/>,
<https://github.com/ivmai/libatomic_ops/>) will include those. For more
advanced interprocess synchronization, you can use named semaphores on
Linux/OS X (sem_open() and friends); on Linux, mutexes also work in shared
memory.

Also, channels created with pipe() are very efficient on modern UNIXes; if
that type of IPC is all you need, you may not require anything else (I've
benchmarked them myself vs. handwritten implementations, and they're difficult
to beat if there's any measurable degree of contention).

Incidentally, despite the author's father having misgivings about shared
memory models, there's nothing inherently problematic about them; it's the
near complete lack of programming language support to make it safe that makes
it so frustrating to use. If you have a language that does not allow you
read/write access to data without first having acquired the corresponding
read/write lock, it's not any more error prone than interprocess
communication.

The other element that makes shared memory concurrency hard is again a
programming language issue; the absence of efficient thread-local variables.
If you have efficient and convenient thread-local storage, you have all the
benefits of separate processes (you can just make all your data thread-local),
but can gradually share those data structures that can safely be shared, and
generally more easily than in a multi-process model (try keeping anything that
contains pointers in an mmap()ed area without tearing your hair out).

~~~
HerraBRE
If you are using write(2)/read(2) on a local file, is there any other buffer
that comes into play aside from the memory in buffer cache itself?

You are correct that obviously we don't really know when the data will hit
disk, but that doesn't matter because the readers and writers are all hitting
the same in-memory cache.

~~~
rbehrends
Well, technically, the POSIX standard specifies that "After a write() to a
regular file has successfully returned: ... Any successful read() from each
byte position in the file that was modified by that write shall return the
data specified by the write() for that position until such byte positions are
again modified."

How well any given operating system implements this and whether it properly
considers all the aspects of multiple processes writing/reading to the same
file is anybody's guess. Linux /should/ do it correctly, but as I recall,
SQLite had some related issues with shared files on OS X a few years ago.

------
jstclair
I thought that was what mmap was for: <http://en.wikipedia.org/wiki/Mmap>.

And compare <http://beej.us/guide/bgipc/output/html/multipage/mmap.html> to
<http://beej.us/guide/bgipc/output/html/multipage/shm.html>

~~~
HerraBRE
Mmap just gives an alternate API - access the file contents as a memory
buffer, not a file. Turns out the underlying mechanism is the same, and if you
are happy with fseek(), read() and write(), you don't really need mmap at all.

~~~
lobster_johnson
That's true on Linux, but is it also true about other OSes?

Also, this will map the file privately (MAP_PRIVATE), meaning every process
gets its own copy in its address space, and any coordination will have to go
through the IO subsystem to reach the other processes.

Sharing with MAP_SHARED will save space by sharing the memory between
processes. It's also possible to share memory not backed by a file, which is
even more efficient.

------
willvarfar
Related reading that may be interesting:
[http://williamedwardscoder.tumblr.com/post/13363076806/buffc...](http://williamedwardscoder.tumblr.com/post/13363076806/buffcacher)

