
Intel Storage Performance Development Kit - luu
https://github.com/spdk/spdk
======
notacoward
"operating in a polled mode instead of relying on interrupts, which avoids
kernel context switches and eliminates interrupt handling overhead"

Yeah, that's great if all you're running is a benchmark. As soon as you need
to combine this with a network stack, also polling its own devices at fast as
it can, it becomes a lot harder to avoid those context switches. If you're
running an actual _application_ it becomes even harder. Likewise if you have
more devices than you have cores to spin waiting for them.

In an extremely latency-sensitive and resource-rich environment this kind of
thing can yield great results, but otherwise it's almost a form of cheating.
Yes, that's what I was accused of when I wrote network drivers at Dolphin and
again at SiCortex that some people felt polled too much. Oddly enough, users
felt that their CPUs should be running their applications, and were more
interested in maximum throughput _per cycle consumed_. Hardware designers
don't build interrupt-based interfaces just for fun, and it's worth
remembering that Intel makes most of its money from selling CPUs. Think about
it.

~~~
drv
It's certainly true that the polled-mode driver model doesn't interact well
when the application needs to use other APIs that don't provide a polled mode.
However, SPDK can be used in conjunction with a polled user-mode network stack
so that a storage application can operate fully in user mode without any user-
to-kernel context switches or hardware interrupts.

It's definitely not a drop-in replacement for a kernel storage stack in the
general case, but rather an optimization for specific applications (e.g.
storage appliances) that can be structured to take advantage of the polled/no-
interrupts model.

~~~
notacoward
Thank you. That's kind of the point I was trying to make. For a storage
appliance - and that's the most common deployment model for the software I
work on - it's great. I just don't want to see every "full stack" halfwit
trying to use a specialized tool for general-purpose stuff. That often ends up
wasting everyone's time - _especially_ that of the people providing the tool.

~~~
vetrom
That said, there are cases where it would make sense in a multicore
environment to pin one core to storage polling, and another to network
polling.

You will then, of course, have contention in transitioning data between the
two, but there are existing and useful models for eliding much of the locking
load there.

------
drv
I am an engineer working at Intel on SPDK, and I can answer any technical
questions you might have.

Currently SPDK consists of a usermode NVMe (PCIe-attached SSD) driver. We will
soon be releasing a usermode driver for the Intel I/OAT DMA engine (copy
offload hardware) that is available on some server platforms.

~~~
n00b101
Will combining SPDK with the usermode driver for the Intel I/OAT DMA engine
provide the necessary building blocks for a complete solution
(network+storage)?

And can this be combined with PCIe-attached accelerators (e.g. Xeon Phi or
GPUs)?

~~~
drv
The SPDK libraries are mostly storage-specific components (the I/OAT DMA
engine can be used for generic copy offload, but it is particularly useful for
copying between network and storage buffers). SPDK itself does not provide any
network functionality.

I am not familiar enough with the Xeon Phi or GPU programming model to say for
sure, but they could possibly be used to offload tasks like hashing/dedup or
other storage-related functions.

~~~
n00b101
> they could possibly be used to offload tasks like hashing/dedup or other
> storage-related functions

Sorry, I was not referring to accelerating storage-related functions, I was
wondering about efficient DMA copy from one PCIe device (Intel NVM storage) to
another (Xeon Phi accelerator) which would be for useful many different
functions, if the NVM storage device capacity is much larger than the
accelerator device memory.

~~~
drv
Ah, I see. The I/OAT DMA copy offload is essentially equivalent to an
asynchronous memcpy(), so anything addressable on the memory bus could be a
source or destination (with some caveats about alignment requirements and
pinned pages if copying to/from RAM).

------
eloff
This is a beautiful thing. I'm eagerly awaiting when NVM arrives in AWS to see
if I can put this to work.

Intro here [https://software.intel.com/en-us/articles/introduction-to-
th...](https://software.intel.com/en-us/articles/introduction-to-the-storage-
performance-development-kit-spdk)

------
jpgvm
Damn this is sweet. User-mode networking has been fairly widespread for a
while but user-mode storage stacks have been pretty rare outside MirageOS and
a few other places.

I wish I had an NVMe device to play with this.

~~~
justincormack
The Intel consumer one, the SSD 750, is relatively cheap (around £300 or so).

