
FlexBuffers - excerionsforte
https://google.github.io/flatbuffers/flexbuffers.html
======
uluyol
So here's how I think this fits into all the other different types of data
serialization:

Schema-ful, copying: Protobuf, Thrift, plenty more

Schema-ful, zero-copy: Cap'n'proto, Flatbuffers

Schema-less, copying: Json (binary and other variants included), XML

Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

~~~
tannhaeuser
While it's true that XML is a generic serialization of angle-bracket markup
that doesn't _need_ a schema for serialization (which was exactly the main
motivation for its "invention" as a SGML subset), the reason it's being used
in inter-party or long-term/loosely-coupled service payload serialization is
because it has fairly powerful and well-established schema languages (XML DTDs
and XML Schema) for validation. This is unlike JSON which thrives in ad-hoc
serialization for tightly-coupled back-ends and web front-ends (and only in
that scenario IMHO). So I don't think XML belongs into the "schema-less,
copying" category.

~~~
randombytes6869
Very true. IMO many people that hate XML config files just haven't used an IDE
that validates schema. Its super nice to have auto-complete and property
validation on config files, something not offered by JSON or YAML. A good
reason to stick with XML for complicated configs.

Its one of reasons I don't mind maven. Yeah there's 1000+ line XML config
file, but maven DTD is so tight nearly any syntax issue will be flagged.
Something easy to appreciate when you're used to giant config files that don't
get validated until runtime.

~~~
Negitivefrags
Visual studio does schema validation for JSON and gives errors inline. There
is a big list of supported schemas built in and you can define your own. Never
actually looked at the format personally.

~~~
nojvek
Json schema is what vscode uses. It’s a standard nowadays. Just as effective
as XMLSchema.

[https://jsonschema.net/](https://jsonschema.net/)

[http://www.schemastore.org/json/](http://www.schemastore.org/json/)

------
8organicbits
> if you supply a buffer that actually contains a float, or a string with
> numbers in it, it will convert it for you on the fly as well, or return 0 if
> it can't. If instead you actually want to know what is inside the buffer
> before you access it, you can call root.GetType() or root.IsInt() etc.

I've started to prefer functions that are explicit about their error cases and
have interfaces that make it obvious about what errors you'd want to handle.
Unexpectedly getting a zero when you get garbage data may cause issues.

Thinking about it, zero is often used as a success code: ERROR_SUCCESS is 0x0
on Windows. So garbage in, ERROR_SUCCESS out??

Something like:

    
    
        value, err = root.AsInt64();
    

makes it clear you have an error path. You can still ignore `err`, like when
you are moving fast and breaking things or it's somehow certain to always
succeed, but it's clear that there's an unhandled error path.

~~~
Aardappel
As you can see from the documentation, `AsInt64` is a convenience method that
either says, I know this is an int, or make it so. There are also ways to
check the type before you access, if you prefer. You can even check if its
unsigned, if you want that level of type-safety.

~~~
bowmessage
Right, but GP is suggesting that the requirement to ignore an 'err' value that
would make the API user think twice about possible failure cases that could
otherwise go ignored with the convenience API.

~~~
Aardappel
You are choosing to use a schema-less, dynamically typed representation (where
a strongly typed alternative is directly available). I'd say, the convenience
of being able to just say I am going to assume this is an int is fitting. If
you wanted to be forced to error check (which needs an if-then in most
languages), you might as well use the schema based, strongly typed version of
the system instead, which would guarantee correct types for you, and requires
no error checking.

------
sillysaurusx
[https://google.github.io/flatbuffers/flatbuffers_white_paper...](https://google.github.io/flatbuffers/flatbuffers_white_paper.html)
has the “why”, if anyone is curious.

------
Aardappel
I made this thing! AMA :)

~~~
XCSme
What do you think is the best buffer protocol to use for multiplayer games? We
used Protobuf for a fast-paced .io game, but encoding-decoding turned out to
be pretty slow and generated a lot of garbage in JS. We were in the process to
switch to FlatBuffers (before the company went bankrupt), but the syntax made
it feel harder to use compared to Protobuf, not sure about the performance
though (we expected it to be faster and less garbage created because of the
zero-copy).

So, would you recommend Protobuf, Cap'n'Proto, FlatBuffers or FlexBuffers for
multiplayer games? The usual packets are game states or user input sent at
high frequency.

~~~
Aardappel
I originally designed FlatBuffers for games (though admittedly more for things
like level data or save game data than network packets), so I'd think it is
pretty suitable. I had actually used Protobuf on a game project just before,
and its performance problems led directly to the no-unpacking no-allocation
design that FlatBuffers has.

So FlatBuffers will make an incoming packet waaay faster to work with than
Protobuf. On the downside, Protobuf tends to be a little smaller, so if
bandwidth is a greater concern than (de-)serialization speed, you might still
prefer it. Additionally, receiving data over the network raises the question
of how you handle packets that have been corrupted (or intentionally
malformatted by an attacker), and in the case of FlatBuffers you'd need to at
least run the "verifier" over the packet before accessing it if you don't want
your game servers to crash when this happens. That slows it down a little bit,
but is still fast, i.e. still doesn't allocate etc.

Cap'n Proto will perform similarly, though does have the downside that all
fields take space on the wire, regardless of whether they're set or not. So
which is better depends on the kind of data you want to send and how it is
likely to evolve.

Frankly, for the absolute highest performance (and lowest bandwidth) game
networking you still need a custom encoding.

~~~
genocidicbunny
> Frankly, for the absolute highest performance (and lowest bandwidth) game
> networking you still need a custom encoding

This is really the truth here. The efficiency of a networking protocol for a
multiplayer game is somewhat sensitive to the context; Are you trying to do a
fast-paced game with lots of client prediction? Do you need to have guaranteed
delivery? Do you expect your game to be used with highly-lossy networks, or
mostly from stable connections?

It's not uncommon to find something like flatbuffers or flexbuffers available
in a multiplayer game engine, but the high-performance systems like movement
or ai will probably utilize a custom protocol better suited to their task.

------
secondcoming
I'm looking to replace our current Boost.Serialization code with something
else. I have the following requirements:

\- serialisation cannot be intrusive (i.e. I don't want a class definition to
be generated from a schema file)

\- zero copy

\- some sort of versioning so any accidental message version mismatch between
sender and receiver doesn't cause a crash / undefined behviour.

\- many language support

Protobufs are intrusive, so that rules them out. JSON is just too verbose and
slow. MessagePack looked like the most promising path to pursue at the moment.
Flatbuffers look quite similar.

~~~
awild
My predecessor has employed smile in all our m2m communication. As far as I
have understood it, it aims to be a binary representation of json. I cannot
speak for its speed but I trust him to a certain extent.

------
ccktlmazeltov
there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant
it is outside of C++. Any idea?

~~~
mmastrac
We use FlatBuffers (the parent project) in our code @ FS... that means it has
to work in C++, Rust, Java, and TypeScript. It's mostly the schema processing
that requires a C++ compiler to make it work.

It's a pretty stellar cross-platform serialization toolkit and the zero-copy
support for reading is no joke.

~~~
maximilianburke
We've been using it across multiple languages too as the data serialization
format between server and (web) client. It's fantastic.

------
person_of_color
What does zero-copy mean in this context?

~~~
sillysaurusx
Normally, zero-copy refers to the fact that data can be decoded without
copying it in memory. I.e. a pointer cast is zero-copy because there's no need
to copy the memory elsewhere; that memory can simply be reinterpreted as the
target type.

I'm not sure if that's the same meaning in this context, however.

------
je42
I wonder how it compares to messagepack's zero copy serializer:
[https://blog.treasuredata.com/blog/2011/11/21/messagepack-
th...](https://blog.treasuredata.com/blog/2011/11/21/messagepack-the-missing-
serializer/) ?

------
m0zg
I wish Google open sourced RecordIO instead (or in addition). People reinvent
this particular bicycle, poorly, pretty much in every project where engineers
are smart enough to introduce a _structured_ application log.

~~~
vore
It looks like
[https://github.com/google/riegeli](https://github.com/google/riegeli) might
be what you're looking for? (from a search of "RecordIO")

~~~
m0zg
Far too complicated for what it's supposed to do. Google's RecordIO basically
just concatenates messages together, with a few additional provisions for
compression and handling small messages. In contrast, Riegeli depends on three
different compressors and Abseil, and its file format specification is quite
elaborate:
[https://github.com/google/riegeli/blob/master/doc/riegeli_re...](https://github.com/google/riegeli/blob/master/doc/riegeli_records_file_format.md).
This could explain its lack of market penetration (only 221 stars at the time
of this writing). The authors apparently don't care that it's not popular,
since they don't even provide a code example on how to use it. It also
apparently uses HighwayHash for integrity checking, which is overkill - such
things should use crc32c, for which most modern CPUs have a dedicated, fast
instruction.

------
86J8oyZv
So, is this basically Android's Bundle semantics made standard and cross-
platform? Are there differences implementation-wise?

------
armitron
I'd rather use MessagePack. Clean, simple, small.

~~~
iso8859-1
It has worse performance though. Not zero-copy.

~~~
ludocode
It depends what you mean by zero-copy. It can't be traversed without parsing
the structure, but strings and binary blobs can be used in-place without
copying. (This is also possible with JSON even, for example with RapidJSON's
in-situ parser.)

This has worse performance for decoding, but far better performance for
encoding. Encoding any traversible format is guaranteed to be expensive
because you have to encode it inside-out, calculating the nested sizes of
everything as you go. For messages that are only encoded once and read once,
for example network messages, a format that is traversible in-place is a poor
choice.

~~~
Dylan16807
That's too general. Sometimes there's [almost] nothing to calculate, so no
slowdown. Sometimes the traversible format can be used as your working data
structure and then encoding is a no-op.

------
vbezhenar
Java example is very weird.

