
Quickly Loading Things from Disk - PVS-Studio
http://probablydance.com/2015/12/19/quickly-loading-things-from-disk/
======
rdtsc
Depending on your platform (os, hardware, compiler) you can even do something
silly like write packed structs and then write those to disk (or send them
over sockets) and cast back.

Mostly it works if you know the hardware and compiler will stay the same (as
in packed struct representation will stay the same). It is basically a poor
man's Cap'n'Proto or Flatbuffers.

As for speed, buffering strategy (size and how often fsync is done) will often
dominate other stuff. I remember doing a benchmark between C++ and Python for
loading and saving data. Python won hands down. The reason was because it had
better buffer size defaults. There is also trickery and configuration involved
in your page cache params are setup and the schedule of how dirty pages are
flushed (if you don't fsync).

> If you’re interested in how fast using read(2) is, it’s essentially the same
> speed: 5437341 ns if I’m reading directly into the std::vector.

Yap, mmap is not always better, it is more complicated and depending on access
pattern not always worth it.

~~~
kabdib
Also, the quality of the library providing stdio or streams can vary quite a
bit. I know a few platforms where you're much better off _never_ calling this
stuff (unless it's for quick bringup or other throwaway code) because the
folks writing that layer obviously hated their day jobs and wanted to write an
OS instead . . . so they did.

------
xemdetia
This is a confusing article. Does the author know about byte alignments? Or
that read/write binary values like floats is just a quick way to run into
platform/portability issues if relevant? Or investigating how any RDBMS
serializes and reads data as fast as possible? Or really any research related
to the field? What about comparing spinning disk via ssd? Or even how libpng
works? Or the complexity of ASN.1 notation (used for TLS certificates and many
other things)?

Protobuf solves a completely different problem than 'load a file from disk,'
which at least FlatBuffers does.

It seems like the only complaint the author has is that it doesn't work
particularly at the level of abstraction they prefer. The FlatBuffers timing
results also seem dubious because of the lack of profiling to say what is
killing the time. Are the comparisions ruining pipelining? Why not write a
transpiler to glue the protobuf output to something you actually want?

It's hard to get past the first half with any confidence in methodology of
testing, especially when the author leads with 'Loading things from disk is a
surprisingly unsolved problem in C++.'

