
Bringing SSD Performance to the DIMM form factor - djoldman
http://www.sandisk.com.br/enterprise/ulltradimm-ssd/
======
NamTaf
I can't wait to see the programming paradigm of memory vs permanent storage
begin to blur in the next 5 or so years. It's going to make some major
assumptions about how you program stuff change quite significantly and it's
really exciting.

~~~
petercooper
Memristors (long promised and still not quite here) also bring together the
ideas of long-term and short-term memory into one and, in future incarnations,
even promise to bring computation and storage together.

~~~
valarauca1
When they work (there is to much cash on the table for them not too), they'll
completely change the way we think of volatile and non-volatile memory. Since
all memory will be non-volatile.

It'll be a huge paradigm shift in storage and IO speed.

~~~
nacnud
.. and possibly give viruses more places to hide?

~~~
valarauca1
Unlikely. Unless we start making file systems different.

~~~
sillysaurus3
Spectators may want to archive this HN thread for posterity, because I suspect
nacnud will have the last laugh. (Nacnud's comment is currently sitting at
about -3 points.)

It's unlikely that future technological advancements will give malware authors
more tools to work with? That seems unlikely, if history is any indication.

Imagine when you have 1TB of non-volatile storage which operates at speeds
close to those we currently enjoy for DDR3 memory. Most average consumers
won't use all of that space. It seems entirely plausible that a virus will
write itself to some area that is never overwritten. It becomes a latent piece
of malware, invisible to any filesystem, but activatable on demand by some
other hard-to-detect malware. This two-step scheme would be difficult to
detect since the second component does nothing except activate the first
component (stored in nonvolatile storage) under some specified criteria.

Or, how about key generation or password input? Remember cperciva's post about
how it was extremely difficult or even impossible to zero a memory buffer
correctly? Now add in the idea that suddenly "everything is nonvolatile" and
you get a perfect storm for security concerns: if a program segfaults or is
terminated mid-operation, then suddenly any sensitive data it had spit out
into memory might be capturable by other programs. Or, since it's nonvolatile,
those secrets will persist "forever" (until they're overwritten, which may be
a long, long time if storage is plentiful) and you may inadvertently leak
secrets when you sell your memristor-backed storage. Whoever buys it can do
some digging and uncover whatever was written into that buffer, if it's truly
non-volatile storage.

So, while the future is exciting, underestimating the security implications
probably isn't a good idea. We can develop new methodologies to combat the new
security concerns, but new security concerns will almost certainly come to
fruition. Of course, calling the concerns "new" is somewhat unfair, since new
concerns are almost always very old concerns manifesting themselves in new
domains, but new domains usually enable malware writers with new toolsets to
exploit.

~~~
j_s
I will archive this thread to save this:

    
    
      >  1TB of non-volatile storage [...] Most average consumers won't use all of that

~~~
pbhjpbhj
I'm with you I got my first 1TB HDD last year thinking it would last a few
years. Nearly full now. Why? Because with all that space available I've
changed how I use disk space - why throw stuff away. Also that I've moved to a
higher grade camera and moved to consuming video stored on HDD rather than
from discs.

When we're creating fully immersive 4D environments in place of holiday snaps
then I imagine we're going to be using quite a lot of memory waving through
our holiday review with friends.

~~~
StavrosK
> Because with all that space available I've changed how I use disk space -
> why throw stuff away.

Funny, I'm the opposite. With all this bandwidth available, why keep stuff?

~~~
jacquesm
Because storage availability is under your control but bandwidth+remote
storage availability is not. In other words, you could lose your remote stuff
+ access to it.

~~~
StavrosK
I could lose my local stuff too, by having a disk fail. I trust S3's
reliability more than my local disk, so it doesn't make sense to keep things
locally.

~~~
toyg
It's not just S3, it's that _and_ your local ISP _and_ the backbone it
connects to. All the planets have to be aligned for your remote storage to be
available.

Locally, if you're using RAID like everyone else, you just have to trust your
local electricity supply and your ability not to _rm -rf /_.

Remote is fine for backups, though.

------
IgorPartola
This is sort of the opposite of the RAM-based battery backed drives [1]. I can
see this being immediately useful for things like large database servers:
instead of re-priming caches on reboots you just have them already warmed up.
You can also suddenly have a whole lot more "RAM" at the cost of its speed.

I do have a hard time picturing what this will look like if it was as fast as
traditional RAM. If I can store everything in RAM, from the OS binaries, to
the running processes, it certainly has a kind of elegance to it. Lots of
microcontrollers already act this way: all your hardware has an address in a
single address space, be at a hardware port, ROM, RAM, NVRAM, etc. However, I
wonder how the modern UNIX OS will work with a system like this without block
devices at all. I wonder if what it'd do is actually just use RAM disks, and
years from now we'll still be doing this the way we do double emulation of the
TTY. That'd be kind of sad since it means we can never get away from the
concept of old block devices.

On the other hand it's damn convenient to be able to just append to a file and
not have to worry about reallocating it. This means that perhaps all we'd need
is a filesystem that's designed to work over RAM rather than over block
devices.

[1] [http://en.wikipedia.org/wiki/I-RAM](http://en.wikipedia.org/wiki/I-RAM)

~~~
itchyouch
While block devices can go away, the notion of a file will probably continue
to remain.

We run clusters of ephemeral linux hosts, and while we use disk, we also have
plenty of hosts that run diskless. The disk mount point just becomes a tmpfs
mount and the required minimal installation is made on a tmpfs partition which
amounts to a trivial amount for hosting busybox and some conf files in /etc.
Linux does appear to understand that tmpfs backed filestores are already in
memory and do not need to synchronize pages and flush out dirty pages. That
whole caching/memory layer for this fake block device is just not used at all.

~~~
bkeroack
Why bother?

I see two possibilities:

\- Two separate address spaces, one volatile (RAM) and one persistent (SSD).
"Writing" to persistent storage is as simple as "mov [src],[dst]".

\- One global address space, with a page attribute of "volatile" (true/false).
Writing is just as simple as above.

At that point the OS and userspace are free to create whatever abstractions
are necessary on top. They don't necessarily need to be filesystems or files
per se.

------
mrb
You can tell SanDisk's performance numbers do not add up and that they are
likely misrepresenting the true performance of their device. (Those red
asterisks next to the numbers correspond to a footnote that is conveniently
missing from the page...) A "read latency of 150usec" translates to maximum
possible read IOPS rate of 1/150e-6 = 6.67K (with one outstanding I/O). But
they quote a "random read IOPS of 140K". That would only be possible if their
DDR3-based DIMM could process 21 concurrent read I/O operations. But to the
best of my knowledge, DDR3 is limited to 8 banks/DIMM, so there could not
possibly be more than 8 concurrent read I/O at any one time. Far from 21.

So SanDisk is likely quoting a _worst case_ read latency and/or a _best case_
read IOPS. Customers are left to themselves to figure out which of these
numbers is most likely to represent the average performance...

PS: here is a paper with some more details but that still fails to explain
this discrepancy:
[http://www.snia.org/sites/default/files/SanDisk%20ULLtraDIMM...](http://www.snia.org/sites/default/files/SanDisk%20ULLtraDIMM_0.pdf)

~~~
nitrogen
_But to the best of my knowledge, DDR3 is limited to 8 banks /DIMM, so there
could not possibly be more than 8 concurrent read I/O at any one time._

Maybe you can submit more than 8 queued requests to the controller and have
them stream back at the full DDR3 data rate. Maybe it's not actually
addressable as RAM, it just uses the DDR3 interface as a communication bus.

------
petercooper
I appreciate this might not be the right way to look at it, but from a trivia
POV, in terms of raw performance, what era of regular memory would be
comparable with it? (i.e. "typical desktop memory in 2004", say.)

~~~
throwaway_yy2Di
The read latency is about as slow as the earliest RAM ever used -- tubes of
liquid mercury used as acoustic delay lines. (150 us vs. 222 us
(microseconds))

[https://en.wikipedia.org/wiki/Delay_line_memory#Mercury_dela...](https://en.wikipedia.org/wiki/Delay_line_memory#Mercury_delay_lines)

------
fpp
More info on the UlltraDimms at:
[http://www.anandtech.com/show/8396/fms-2014-sandisk-
ulltradi...](http://www.anandtech.com/show/8396/fms-2014-sandisk-ulltradimm-
to-ship-in-supermicro-servers) (developed together with Diablo, some benchmark
links).

and a video interview and demo with a SanDisk manager at a computer fair -
[http://www.youtube.com/watch?v=jarsTLGXx9c](http://www.youtube.com/watch?v=jarsTLGXx9c)
(currently only available for OEM, requires special bios setup)

------
ctz
'reliability rate of one unrecoverable error in 1017 (sic) bits read'

So about 4 unrecoverable errors per sector. Seems legit.

Do people not read their own copy?!

~~~
ccozan
it's 10^17. The article was probably copied and the <<sup>> got lost.

------
bitL
What would be the advantage of this comparing to PCIe/M.2 PCIe SSDs? Is it all
about reduced latency, i.e. very small reads/writes would benefit?

~~~
valarauca1
>The benefit of using a DDR3 interface instead of SATA/SAS or PCIe is lower
latency because the SSDs sit closer to the CPU. The memory interface has also
been designed with parallelism in mind and can thus take greater advantage of
multiple drives without sacrificing performance or latency. SanDisk claims
write latency of less then five microseconds, which is lower than what even
PCIe SSDs offer (e.g. Intel SSD DC P3700 is rated at 20µs).

Source: [http://www.anandtech.com/show/8396/fms-2014-sandisk-
ulltradi...](http://www.anandtech.com/show/8396/fms-2014-sandisk-ulltradimm-
to-ship-in-supermicro-servers)

 __TL;DR __about 4x faster then PCIe SSD 's

~~~
digikata
But raw flash writes on the order of 1-2 milliseconds, so the 5 microseconds
must be the latency to write to a ram cache inside the stick. I wonder what
the latency would be for a write larger than the buffer. For that matter, how
large is the buffer?

~~~
valarauca1
If you read the source you will see that no independent benchmarks have been
done yet.

~~~
digikata
The article did update with a benchmark link comparing a couple of models to a
fusion io card.

[http://www.percona.com/blog/2014/08/12/benchmarking-
exflash-...](http://www.percona.com/blog/2014/08/12/benchmarking-exflash-with-
sysbench-fileio/)

So the 95%, write latency is 1-1.5 msec for one model, and 2.5-3.5 msec for
another model. Maybe representing SLC/MLC flash?

95% read latencies are maybe 350 or 250 usec.

------
zrail
I don't quite understand. Does this act like a stick of memory or an SSD? Is
it just using the memory controller as a super fast parallel bus? If it does
act like normal memory, what happens at reboot?

~~~
gnufied
Answer is there in FAQ. Written data is not lost on reboot, so basically you
can mount it as a `ramdisk` partition and use that as data directory for your
database. I have used this approach in past to speed up large test suite of a
web application, except it required restoring from some sort of backup on
reboot. Which won't be necessary in this case.

~~~
achille2
Does this mean you won't have traditional memory available at all (outside of
L1-3 cache)? Or is it able to mix and match between real DDR3 memory and SSD?

------
kcarnold
For all of us who are thinking "it's just like RAM, just persistent!", here's
some perspectives about what persistent RAM means for OSes and applications:
[http://lwn.net/Articles/610174/](http://lwn.net/Articles/610174/)

------
snake_plissken
This is pretty cool, especially since the data is persistent. You could put an
entire data warehouse onto one or of a couple of these things, update it once
a week and get amazing response time.

But I thought the main drawback of SSDs was that eventually the individual
memory cells will degrade and lose the ability to write new data. I don't see
anything about endurance other than a MTBF of 2.5 million hours and the disk-
writes-per-day (DWPD), which I had to look up. I have no idea if these are
good or bad. I feel like these things will be great if you didn't write new
data a lot, or you have deep pockets to replace them when you need to.

~~~
Filligree
2.5 Mhours MTBF equates to 285 years. That's a long time.

On any kind of large-scale installation you need redundancy, but if those
numbers pan out then you could expect any _single_ SSD to outlast your
organisation.

Take backups anyway.

~~~
tw04
At what duty cycle? The MTBF rating means literally nothing without that
detail.

------
nine_k
How come read latency is 150 µs, while _write_ latency is 5µs, 30 times
shorter? Do they mean the latency to start a write operation? IIRC, flash
memory is written block by block, with a pretty significant time to write one
block.

~~~
aristus
At a guess, they are using MLC (or even TLC) transistor configurations, which
effectively packs more bits into on transistor, but is slower to read. Good
SLC has read cycles of 20-25 usec.

------
userbinator
The capacities are very odd - you'd expect something in a DIMM format to have
a power-of-2 size.

I think the market for this could be _much_ bigger if it behaved like a
regular RAM DIMM, only slower and nonvolatile; it somewhat reminds me of old
machines that used magnetic core RAM. This could be useful for laptops, like a
zero-power suspend-to-(NV)RAM. The only thing that is worrying is the
endurance of the flash - especially if it's being treated almost like RAM in
this application.

~~~
vidarh
It's still _way too slow_ to be treated as RAM. The performance they quote is
equivalent to a mid to lower range PCIe SSD.

For comparison, the stated read performance is 800MB/sec or so. Last time I
checked - on ca. 8 year old hardware, we could read files from a RAM disk
under Linux at a rate of about 6GB/sec. That's with the added overhead of the
benchmark software running and going through a filesystem.

~~~
com2kid
I wouldn't say that 800MB/s is slow.

[http://en.wikipedia.org/wiki/PC133](http://en.wikipedia.org/wiki/PC133)

I also wonder if that 880 is for just one slot filled up. If it is, then I
wonder if you can parallelize to 4 slots for ~3.4GB/s.

At that point, I imagine many types of data sets would work A-OK.

Though I wonder how that's scaled for cores, given that in the situation I
imagined above, you'd be streaming all that data to just one CPU to get that
aggregate bandwidth.

Hmm.

OK what about some other scenarios? Being able to have RAM and Storage on a
sliding partition is sort of cool. I wonder if it'd make provisioning VMs any
easier. "How much RAM do you want? Set the slider!"

~~~
vidarh
Yes, it is slow. PC133 is "ancient". As I pointed out, even in 8 year old
hardware it was trivial to get more than 6 times that performance out of a RAM
disk device using filesystem benchmarking tools.

> If it is, then I wonder if you can parallelize to 4 slots for ~3.4GB/s

You probably can, but then you can't _run_ code straight out of it, or
read/write data directly as if it is normal RAM. Putting a filesystem on it,
RAID'ed over multiple slots will likely work great (especially so since it's
easy to get servers with a ton of DIMM slots, and it'll allow much better
server density than trying to fit in more drive slots). But treating it as RAM
will be _slow_.

------
jgrodziski
I look forward to the point where 3/4 of business application code will go the
trash when we'll have a persistant memory with the latency of actual RAM disk
(Memristor !!). Exit all that "copy from RAM to Disk/Network" code. It will
definitely change the way we code and look at code. That's why nowadays I
think a good interface to your entities, like the Repository pattern, is a
must have.

~~~
itry

        It will definitely change the way we code
    

Only on a low level, right? I have not been thinking about ram and disk in
ages. I calculate with variables, I store stuff in a DB or in a file, I access
the net via http. Nothing would change if ram increases I think.

~~~
tokenizerrr
Well, if you have plentiful persistent RAM an application could easily
"suspend" itself (on shutdown for example), and then resume running at a later
point in time. The code would not have to think about this, since the entire
state is just snapshotted. It's basically what machines do now when they
"hibernate", except more effecient and could even happen if the power got cut.

Additionally I suppose your variables could also be persistent automatically.
Though the biggest change I see is that software would never really need to
completely close anymore when RAM == Disk. For RAM usage it wouldn't really
matter if they were running or not, unless they delete data upon shutdown. To
save CPU or other IO usage the application could simply be suspended.

~~~
realharo
Can't wait for the persistent memory leaks!

~~~
silon5
Or losing data on crash because load/save cycle is untested.

------
zitterbewegung
Is the advantage the low latency because the rest of the specifications seem
to be pretty standard for an SSD. Does this require a BIOS patch of some sort?

~~~
gnufied
I suppose the main usage will be a RAMDISK mounted partition which will not
lose its data on Server reboot, power loss or when program is not running.

I don't think this SSD can be accessed through regular SCSI, IDE, AHCI
controllers. [http://www.sandisk.com.br/enterprise/ulltradimm-
ssd/](http://www.sandisk.com.br/enterprise/ulltradimm-ssd/)

~~~
diydsp
Yes it would be very cool to have the entire operating system in SSD.
Literally flick the computer on and it's running ;)

Or flick rapidly between Linux and Windows. That makes me think it might be
wise to have a manager to do this. E.g. leave the disks accessible between
three+ operating systems.

~~~
icebraining
_Yes it would be very cool to have the entire operating system in SSD.
Literally flick the computer on and it 's running ;)_

Isn't that what you already get with suspension to memory (S3)? At least on my
Debian laptop, it's ready before the lid is fully up.

~~~
sliverstorm
Right, except:

\- S3 uses battery power (albeit not much)

\- NVM provides protection from power faults in datacenters

------
imaginenore
We have PCIe SSDs approaching 7.2 GB/s. While that's lower than DDR3 speeds,
it's not that far off.

[http://hothardware.com/Reviews/OCZ-CES-2012-Product-Tour-
ZDr...](http://hothardware.com/Reviews/OCZ-CES-2012-Product-Tour-
ZDrive-R5-Everest-2-Step-Out/)

------
Wildgoose
I would still say that the real performance bottleneck is ultimately the
bandwidth between the CPU and this memory. This suggests that the next stage
will be to incorporate heterogenous processors alongside that memory - thus
upgrading your computer could then be as simple as plugging in an another
combined non-volatile memory/CPU block into a fast inter-connector. Rather
reminds me of the the old S100 bus where everything just plugged into the same
channel, (which probably dates me quite well).

------
jpgvm
This technology is actually developed by a company called Diablo Technologies.
They seem to have licensed it to Samsung.

~~~
Igglyboo
Do you mean SanDisk, or does Samsung also play into this somehow?

~~~
jpgvm
My bad, I meant SanDisk.

------
unwind
Cool! And weird. I wonder what the OS support looks like.

Also available in a non-Brazilian page at
[http://www.sandisk.com/enterprise/ulltradimm-
ssd/](http://www.sandisk.com/enterprise/ulltradimm-ssd/), of course. :)

------
robmccoll
I honestly can't imagine having something in my DIMM slots with this bad of
latency and throughput. 150 microsecond reads? Doing a full POST would take
forever. What OS and software use - case does this serve?

~~~
vidarh
You'd put a filesystem on it and mount it as a "ramdisk", with the result that
you can cut out expensive RAID controllers and/or PCIe SSDs, and can
potentially reduce form factor.

E.g. I can get 1U servers with 32 DDR3 slots and 64 cores, but only space for
3x 3.5" drives, or if you double up (but few chassis will then let you use hot
swap caddies), 6x 2.5" drives.

You could easily fit 256GB RAM and still have 16 slots free for RAID arrays
over those DIMMs.

------
fastest963
Apparently once you hit 3.2TB the IOPs falls to only 200K. Typo?
[http://cl.ly/image/2J0u182D3229](http://cl.ly/image/2J0u182D3229)

------
egillie
SSDs as RAM + projects like Tachyon are the future of big data processing --
current workflows are still way too slow. I wonder how this will affect Spark.

~~~
sixdimensional
I am not a silver bullet kind of person, but I think this kind of technology
plus things like Spark are potentially the holy grail. Ok, let me rephrase -
I've often believed that there was a sort of "ultimate architecture" with
distributed cluster computing, distributed programming frameworks, high-speed
interconnect, and flash based storage. I think we are getting closer and
closer each day. I wish I worked in R&D at any of the companies doing this
fundamental research now - I have a feeling they have some amazing things
cooked up behind closed doors.

------
joelthelion
Do OSes already support this? Do they support it as memory, a storage device,
or both?

------
rlpb
What OS support exists for this, and how does it work?

~~~
jpgvm
You need not just OS support but specific motherboard too. Currently AFAIK
this is only supported on SuperMicro dev kits.

~~~
Artemis2
Wouldn't mounting the RAM stick as a ramdisk be enough though?

~~~
rlpb
You can't address a specific RAM stick, typically.

Try and boot an OS without knowledge of this, and it'll start using the stick
like it uses RAM. It'll take orders of magnitude longer to boot, to the point
of impracticality, probably (I haven't calculated).

The OS needs to understand the memory map, and at the minimum exclude use of
the stick for "normal" operations.

Then, it needs to expose the stick for access - preferably in a way that is
more usable than /proc/mem.

------
Fando
What implications does this have for us simple folk?

------
mohap
What's the catch here?

~~~
peter303
It still costs 10x disk, even though both are fantastically cheap compared to
decade ago.

Flash still has limited write cycles. Its for data you probably wont write
more than once an hour, but not virtual memory.

~~~
JohnBooty

      > Flash still has limited write cycles. Its for data you probably wont write more than once an hour, but not virtual memory.
    

You can write data to SSDs a lot more than once an hour, even virtual memory.
The drive's firmware (assuming it does its job) will do wear-leveling and
spread the logical writes across different physical locations on the drive.

If you write 20GB of data per day to a modern 480GB SSD you can expect 25
years of service. (Realistically, something else will probably fail before
then)

Example - [http://www.anandtech.com/show/8520/sandisk-ultra-ii-240gb-
ss...](http://www.anandtech.com/show/8520/sandisk-ultra-ii-240gb-ssd-review)

------
gr3yh47
FTA:

"Reduces total processing time, compared to hard-drive drives (HDDs)."

i loled.

~~~
consum3dbyfire
I couldn't believe that either... it's kind of sad.

------
lowlevel
Mind blown.

