
FlatBuffers: a memory efficient serialization library - rw
http://google-opensource.blogspot.com/2014/06/flatbuffers-memory-efficient.html
======
onedognight
Interesting that Cap'n Proto[1] is not even mentioned given that this library
has similar if not the same goals and lineage.

[1] [http://kentonv.github.io/capnproto/](http://kentonv.github.io/capnproto/)

~~~
cakoose
Deeper in the site they have a benchmarks page[1] that has a short "why not
Cap'n Proto":

"Cap'n'Proto promises to reduce Protocol Buffers much like FlatBuffers does,
though with a more complicated binary encoding and less flexibility (no
optional fields to allow deprecating fields or serializing with missing fields
for which defaults exist). It currently also isn't fully cross-platform
portable (lack of VS support)."

[1]
[http://google.github.io/flatbuffers/md__benchmarks.html](http://google.github.io/flatbuffers/md__benchmarks.html)

~~~
kentonv
Hmm, my personal (biased) opinion is that using variable offsets and vtables
(as FlatBuffer does) is a lot more complicated than Cap'n Proto's fixed
offsets.

"Optional" fields (ones that don't take space on the wire at all) are not
clearly much of an advantage. The usual use of optional fields in Protobufs
was to emulate unions, which Protobufs itself never supported directly, but
Cap'n Proto does. Also, when wire size really matters, Cap'n Proto offers a
very fast compression pass which deflates zeros, and unset fields are always
zero, largely mitigating this issue.

The lack of VS support in Cap'n Proto is of course a very fair criticism, and
a huge sore point in Cap'n Proto adoption. MSVC is finally catching up in its
C++11 support so I hope this can be solved soon.

(Disclosure: I'm the author of Cap'n Proto, and also the former maintainer of
Protocol Buffers.)

~~~
mortarion
I think you have maybe misunderstood the purpose of why flatbuffers exists;
message packing for real time high performance games. It's not a replacement
for Protocol Buffers or Cap'n Proto.

You've heard of Valve Software right? Their game engine, being a distant
relative of Quake, used the old quake message packing format up until 1 or 2
years ago. Since then they have retrofitted all their popular games to use
protocol buffers instead since it allows them to more easily extend their net
code without breaking backwards compatibility (up to a point of course). In
the context of games, optional fields are very important as are default
values.

Also in this context; the author of flatbuffers is Wouter van Oortmerssen,
author of the Cube and Cube2 game engines:
[http://sauerbraten.org](http://sauerbraten.org)

From the white paper:

> In particular, FlatBuffers focus is on mobile hardware (where memory size
> and memory bandwidth is even more constrained than on desktop hardware), and
> applications that have the highest performance needs: games.

~~~
kentonv
I think there's some confusion here. To be clear, Cap'n Proto allows you to
add new fields to a message without breaking backwards compatibility, and
Cap'n Proto implements default values.

The question is how a field is represented on the wire if it's defined in the
schema but you don't assign it a value before serializing. With FlatBuffers,
it won't take any space on the wire (although it still takes space in the
vtable, but that can be amortized with many instances of the same structure).
With Cap'n Proto, all fields are located at fixed offsets from the start of
the struct, therefore they take space even if they aren't set. You can
truncate unused fields off the end, but not from the middle.

Also, note that Cap'n Proto uses pointers for variable-width fields. The
pointer can be null if the field is not set. So the maximum wasted space for
an unset field is 8 bytes, all of which are zero and therefore compress
nicely.

Cap'n Proto's approach is exactly the approach used by in-memory data
structures in C, C++, and all other high-performance programming languages. If
your C struct contains a field you don't use, it still takes space in memory.
I would think this would be preferable for games.

~~~
judk
IME, optional fields are used for...optional fields. Like when you have a
50-field data object but only want to transmit 5 of those fields.

~~~
ahminus
I think if you have a 50 field data object, and you have cases where you
transmit only 5 of those fields, the problem is your data modelling, not the
serialization library.

~~~
ssylvan
I think you haven't worked on games. 99% of the fields in game objects are the
defaults, with some minor customization (e.g. position).

~~~
ahminus
Games are all I've worked on for over a decade. See my other comments.

------
kentonv
Huh. So Google is releasing a competitor to Cap'n Proto. As the former
maintainer of Protobufs (at Google) and author of Cap'n Proto (after Google),
I'm pretty surprised that I hadn't heard about this. I also don't recognize
any of the names, so this is not from the people who were working on Protobufs
at the time I left.

I'm the main competitor, so take me with a grain of salt here.

The docs don't look very detailed, but taking a quick look through...

> "On purpose, the format leaves a lot of details about where exactly things
> live in memory undefined, e.g. fields in a table can have any order... To be
> able to access fields regardless of these uncertainties, we go through a
> vtable of offsets. Vtables are shared between any objects that happen to
> have the same vtable values."

Hrm, that sounds like a lot of complexity and a big performance hit. In order
to read a field from a struct, you have to do a vtable lookup to find the
offset? Maybe you can still get that done in an inline accessor, but it will
make the optimizer sad.

How is de-duping of vtables accomplished? It doesn't look like the API exposes
any details about vtables, meaning you'd have to do it magically behind the
scenes, which seems expensive?

It looks like the authors may have been trying to maintain the concept of
"optional fields" from Protobufs, where if you don't set an optional field, it
takes zero bytes on the wire. But the main use of optional fields in practice
is to implement unions -- i.e. a list of fields where only one should be
filled in at a time.

Cap'n Proto's solution to this was just to build unions into the language,
something Protobufs should have done a long time ago.

> "FlatBuffers relies on new field declarations being added at the end, and
> earlier declarations to not be removed, but be marked deprecated when
> needed. We think this is an improvement over the manual number assignment
> that happens in Protocol Buffers."

That's not an improvement at all. Declarations should be ordered semantically,
so that related things go together. More importantly, even if you say that
they can't be, developers will still do it, and will accidentally introduce
incompatibilities. To make matters worse, it tends to be hard to detect these
issues in testing, because your tests are probably build from the latest code.
Simply adding field numbers as in Protobufs and Cap'n Proto has proven an
effective solution to this in practice.

Very surprised to see Google get this wrong, since they have more experience
with this than anyone.

> Benchmarks

I couldn't find the benchmark code in the github repo, so it's hard to say
what they may be measuring here. FlatBuffers presumably shares Cap'n Proto's
property that encodes and decodes take zero time, which makes it hard to
create a meaningful benchmark in the first place. Indeed the numbers they put
up kind of look like they're timing a noop loop. I can't see how the
"traverse" time could be so much better than with protobufs, which makes me
think the code might be written in such a way that the compiler was able to
optimize away all the traverse time for FlatBuffers because it thought the
results were unused anyway -- a mistake that's easy to make.

But I'd love to see the code to verify. Hopefully I just missed it when
digging through the repo.

~~~
kentonv
OK, AFAICT there is _no bounds checking_. When you want to read a message, you
give FlatBuffers a bare pointer to the start of the message -- no size. So you
can't use this to read data you don't trust I guess.

Which is an OK trade-off for certain situations (like reading your game data
from disk). But... not for any kind of secure network protocol.

Maybe I'm missing something, though. I've only been looking at this for a few
minutes.

~~~
Aardappel
Most readers of binary file formats can be made to read memory outside the
buffer by corrupting the data, and FlatBuffers is no different.

That said, an option to bounds-check every offset would be possible, at a
certain cost. Might be a nice optional feature to have.

~~~
kentonv
> Most readers of binary file formats can be made to read memory outside the
> buffer by corrupting the data

I'm pretty sure that would be considered a serious security bug for any format
likely to be displayed in a web browser. For instance, an image format with
such a bug would allow you to implement a heartbleed-like attack on a user's
browser by displaying a malicious image and then reading back the pixel
values. That would be very, very bad.

But I can believe that your statement applies to formats used by games for
their own assets, where those assets come directly from the game developer.

------
ipkn
If you don't want the code generation step and C++ is the only language to
use, Dumpable[1] can be useful.

[1][https://github.com/ipkn/dumpable](https://github.com/ipkn/dumpable)

------
emmelaich
Just because I'm a fan of his, I'd like to point out that the author, Wouter
van Oortmerssen aka Aardappel (strlen.com) is creator of the Cube 3d engine
and lots of other interesting stuff.

Oh I'm also a fan of Kenton's in case it matters :-)

~~~
digitalboss
And lobster [http://strlen.com/lobster](http://strlen.com/lobster) (game
programming language open source). He's a personal friend as well, one of the
wisest dorks I know :)

"Wouter van Oortmerssen is a humorous guy preferring strange names for his
programming languages and often also programming examples."

------
clhodapp
Their doc site doesn't scroll in Chrome for Android, which seems a bit
embarrassing if you're Google.

~~~
jbroman
This has been fixed. Thanks for reporting.

------
mantraxB
The problem with FlatBuffers and Cap'n Proto is that while in-memory rep
matching serialization formats are much faster (infinitely faster as Cap'n
Proto says tongue in cheek) for languages with unsafe direct memory access
(like C/C++), they're actually slower and more cumbersome to use in any
language without unsafe direct memory access (like Java, Rust, Python, Go,
JavaScript/Node, Swift and... well, most languages).

The other problem is that Google tends to release random stuff as open source
and then modify it (breaking BC) or abandon it without regard for the people
who are using it.

So caveat emptor, I guess.

~~~
kentonv
Java has ByteBuffer, Javascript has TypedArrays, Python has the "struct"
module, and other languages have other fine ways of reading raw data in this
kind of format. Some languages may lack the ability to inline calls well
enough for true zero-copy to be efficient, but the worst case is you fall back
to parsing the message upfront like you would with any other encoding. That
upfront parse is still going to be faster, because it's just easier to read
fixed-width values than variable-width.

~~~
mantraxB
I don't get it, you can't read anything out of a ByteBuffer, except a bunch of
bytes.

You can't map 1:1 any more complicated in-memory structure to that buffer, or
am I wrong?

~~~
kentonv
ByteBuffer has methods for reading primitive values of all sizes -- ints,
floats, etc. -- from arbitrary offsets within the buffer. So you just need a
wrapper object with nice generated methods for each field which in turn call
the ByteBuffer getters.

None that the C++ code works this way too. We don't cleverly generate a struct
that just happens to have the right layout; we generate a class that wraps a
pointer and provides inline accessors that read/write values from the correct
offset relative to that pointer. After inlining, it's just as fast.

