
Call for comments - RethinkDB wire protocol - coffeemug
http://www.rethinkdb.com/blog/rethinkdb-wire-protocol-call-for-comments/
======
zellyn
See the "required is forever" warning here:
<https://developers.google.com/protocol-buffers/docs/proto>

You may want to make even your conceptually required fields optional.

As an example, if you tried to proxy a newer version of the protocol through
an old proxy (assuming it's deserializing), unknown enumeration values for
required fields could cause errors.

~~~
coffeemug
Thanks!!

------
MichaelGG
Why bother trying to encode the query syntax into the wire protocol? Why not
just accept queries as a string, then have some simple way to pass parameters?
It seems like encoding the whole query language syntax makes the clients far
more complex.

~~~
alexpopescu
Transmitting a string over the wire would pose a wide range of problems:

1\. it's not efficient

2\. we'd need to design a complete new protocol (keep in mind that you'd need
to include query parameters) which for advanced query language like RQL is
challenging

3\. with a new protocol we'd need also versioning (for ensuring backward
compatibility)

And the list is probably much longer.

alex @ rethinkdb

~~~
MichaelGG
Thanks for entertaining me :). I'm only talking about the actual encoding of
the query itself, not parameters, return types, etc.

Looking at the github page, it seems all the names of things are still being
transmitted as strings (of course), so "r.add(1,2)" is unlikely to be smaller
decomposed into discrete items (each with at least a 1-byte pb tag overhead).

I'd think it'd be passing a main query string, with placeholders for
parameters. "r.add(@a,@b)" or something, then have a parameter block where you
can efficiently encode non-string parameters.

Basically, keep protocol buffers and the rest, just don't encode the syntax
tree - let the server parse it.

~~~
LeafStorm
So, basically using SQL prepared statements. Which are fine for simple queries
where the behavior is always exactly the same, but become very complicated
when you outgrow the "one hour blog" demo phase. It's much easier to build
queries out of data structures in most languages, and if you're already
building a data structure, there's no reason to convert it to a human-editable
text form just so it can be parsed again.

~~~
MichaelGG
Oh, I see. So in the ReQL docs all those examples aren't really a "language"?
There's no underlying "RQL", it's just using a language with some nifty client
libraries to build up an internal state that gets serialized. That's a neat
approach.

~~~
coffeemug
Right. Technically we could write a "language" by just writing a string parser
that converts strings to protocol buffers (some day we might do that). But
embedding into host languages is so nice, there was no need to do it so far.

------
coffeemug
Hey all, slava @ rethink here. As the post mentions, the new protobuf spec is
here:
[https://github.com/rethinkdb/rethinkdb/wiki/protobuf_rfc_raw...](https://github.com/rethinkdb/rethinkdb/wiki/protobuf_rfc_raw_spec)

Would really appreciate anyone's questions or comments!

------
tedchs
Maybe consider re-using an existing wire protocol, such as that for Postgres
or MongoDB (whatever is closest conceptually to your database)? Or use HTTP?
That way you don't need to re-invent a client for every language.

~~~
coffeemug
We did (even had a few implementations). Nothing could come close to enabling
an awesome user experience as a custom one, so we decided it's worth the
effort.

