Hacker News new | comments | show | ask | jobs | submit login
An experiment: Shared memory vs. files (klaki.net)
13 points by HerraBRE on Dec 27, 2011 | hide | past | web | favorite | 8 comments

Using read()/write() to access shared files is a bad idea because you have no control over buffering. Just because you've written something to a file does not mean it actually has been written somewhere where other processes can see it when they try to.

However, using shared files for IPC is very practical (I've done it before), but you want to use mmap() in conjunction with at least memory barriers or mutexes/semaphores/condition variables that work between processes. If you just need memory barriers, then the atomic_ops lib (http://www.hpl.hp.com/research/linux/atomic_ops/, https://github.com/ivmai/libatomic_ops/) will include those. For more advanced interprocess synchronization, you can use named semaphores on Linux/OS X (sem_open() and friends); on Linux, mutexes also work in shared memory.

Also, channels created with pipe() are very efficient on modern UNIXes; if that type of IPC is all you need, you may not require anything else (I've benchmarked them myself vs. handwritten implementations, and they're difficult to beat if there's any measurable degree of contention).

Incidentally, despite the author's father having misgivings about shared memory models, there's nothing inherently problematic about them; it's the near complete lack of programming language support to make it safe that makes it so frustrating to use. If you have a language that does not allow you read/write access to data without first having acquired the corresponding read/write lock, it's not any more error prone than interprocess communication.

The other element that makes shared memory concurrency hard is again a programming language issue; the absence of efficient thread-local variables. If you have efficient and convenient thread-local storage, you have all the benefits of separate processes (you can just make all your data thread-local), but can gradually share those data structures that can safely be shared, and generally more easily than in a multi-process model (try keeping anything that contains pointers in an mmap()ed area without tearing your hair out).

If you are using write(2)/read(2) on a local file, is there any other buffer that comes into play aside from the memory in buffer cache itself?

You are correct that obviously we don't really know when the data will hit disk, but that doesn't matter because the readers and writers are all hitting the same in-memory cache.

Well, technically, the POSIX standard specifies that "After a write() to a regular file has successfully returned: ... Any successful read() from each byte position in the file that was modified by that write shall return the data specified by the write() for that position until such byte positions are again modified."

How well any given operating system implements this and whether it properly considers all the aspects of multiple processes writing/reading to the same file is anybody's guess. Linux /should/ do it correctly, but as I recall, SQLite had some related issues with shared files on OS X a few years ago.

Mmap just gives an alternate API - access the file contents as a memory buffer, not a file. Turns out the underlying mechanism is the same, and if you are happy with fseek(), read() and write(), you don't really need mmap at all.

That's true on Linux, but is it also true about other OSes?

Also, this will map the file privately (MAP_PRIVATE), meaning every process gets its own copy in its address space, and any coordination will have to go through the IO subsystem to reach the other processes.

Sharing with MAP_SHARED will save space by sharing the memory between processes. It's also possible to share memory not backed by a file, which is even more efficient.

Mmapped files are typically quicker than lots of fseeking and read()ing and write()ing because they avoid the non-trivial syscall overhead. This has been a very big driver in the adoption of mmapped files, even those not backed by disk.

Related reading that may be interesting: http://williamedwardscoder.tumblr.com/post/13363076806/buffc...

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact