

Atomic I/O operations - kamaal
http://lwn.net/Articles/552095/

======
a-priori
_Atomic I /O operation support can be taken a step further, though, by adding
asynchronous I/O (AIO) support. The nice thing about the Linux AIO interface
(which is not generally acknowledged to have many nice aspects) is that it
allows an application to enqueue multiple I/O operations with a single system
call._

I'm not familiar with the AIO interface, but it sounds fairly handy. Could
someone comment on what he means by it not having 'many nice aspects'?

~~~
jandrewrogers
The Linux AIO interface is excellent for high-performance disk I/O but using
it requires a much bigger code investment than is immediately implied by the
AIO API by itself. The complexity of using it is that for it to work properly,
you need to bypass most blocking and buffering operations in the kernel
related to file system I/O. Specifically:

\- You should always use O_DIRECT or it defeats the purpose. Consequently, the
caching and buffering you normally get for "free" using more conventional
approaches is not available. For good performance you have to write your own
disk buffer cache implementation which is not trivial if you have never done
it before. This buffer code will exceed the size of the AIO code if you are
doing it well. On the upside, for many purposes you can design a buffer cache
that greatly outperforms the standard kernel implementation.

\- You need to minimize the number of file system metadata operations that can
block and/or bypass the file system altogether. I recommend the former. In
practical systems, you end up doing very coarse file system operations, like
allocating a huge chunks of disk, and then building your own lightweight disk
management system on top of that. Again, this is a lot of extra work for the
sake of using AIO effectively.

\- You need to figure out the best way to schedule all of the disk I/O
operations you are now responsible for. Background writing, prefetching, etc
that are used to optimize disk performance are now part of your
implementation.

Linux AIO is simple and pretty nice in the abstract. Using it well requires
some complex domain expertise around scheduling, file organization, and cache
replacement algorithms and techniques. There are no libraries that provide
quality implementations of the supporting infrastructure that I am aware of so
you have to roll your own. You can't do just a little bit of AIO and see much
benefit in most cases.

There are big benefits though which is why advanced database engines are
implemented this way. If you build out an entire userspace disk I/O subsystem
on top Linux AIO you can easily double your disk I/O performance or more
relative to using the conventional POSIX, mmap(), etc interfaces. Furthermore,
it often can be designed to degrade performance more gracefully during periods
of high I/O pressure.

