
How to Read 100s of Millions of Records per Second from a Single Disk - cwinter
https://clemenswinter.com/2018/08/13/how-read-100s-of-millions-of-records-per-second-from-a-single-disk/
======
dang
[https://news.ycombinator.com/item?id=17492234](https://news.ycombinator.com/item?id=17492234)

------
gopalv
Hard-disks aren't slouches when it comes to sequential throughput, but
building around disks has some serious non-future-proof consequences.

Building things for spinning disks is a cul-de-sac, which has no exit over
onto the faster characteristics of SSDs.

A significant part of Hadoop is built around the idea that "random = bad,
sequential = good" and this is still arguably true when it comes to writes.

However, building around sequential reads gives you the wrong idea when the
hardware changes underneath you.

I see a tiny sign of that in this post, where it talks about the smarter data
partitioning.

For example, when dealing something like a join, a spilling sequential write
join like Grace hash-join works much better than a memory mapped hashtable for
spinning disks - however a grace join will be crushed by a better random IOPS
driven SSD.

Also, if the author is reading this - Zlib isn't a slouch either (just like a
spinning disk), the trick is to use the right combination of Zlib parameters.
Compression will be slower with Zlib, but the read throughput of Zlib can
exceed the performance of memcpy of the same data depending on your data
conditions (look for inflate_fast and inflate_slow).

And if you're getting into the business of improving sequential reads, the
next big trick for systems is vertical row partitioning of data (think of
TPC-H, except now you stored L_COMMENT in a separate file).

~~~
sitkack
Streaming is better for every technology, of course this might not be always
true ... but RAM isn't really random access, that is an abstraction, it is
much faster at streaming. Same goes for SSDs.

> However, building around sequential reads gives you the wrong idea when the
> hardware changes underneath you.

If one is going to build a system tuned to the hardware, they can't ignore the
hardware. What abstraction should the author be using?

------
canhascodez
This seems like a very clever way to accomplish this feat. I think I'm happy
that to date I haven't the sort of problems that might require reading 100M
records per second. However, I'd be all ears if someone wanted to contribute
to my education by telling me what sorts of interesting things that this might
enable.

~~~
BitcoinCash
Loading past unspent outputs for Bitcoin Cash; to validate new blocks, and
when queried by lightweight mobile wallets.

~~~
tomrod
This would be helpful to increase transaction speed.

