I have a lot of respect for Kenton and I'm sure this is high quality work.
The idea of a fixed-length encoding of a protobuf-like structure is a good idea, and something I have seen explored elsewhere. To me, though, it's unfortunate to create an entirely new schema language for it, instead of just making it an alternate encoding of the existing protobuf data model, which it closely resembles. To have two schema languages that are nearly, but not quite, isomorphic means that you get no interoperability benefits with any existing protobuf-based code or data.
The way I see it, the most important feature of protobufs is not the on-the-wire encoding, but rather the schema. If you start from a schema (and its associated data model), then you can create as many different encodings as you want, and convert between them losslessly at any time, while taking advantage of their different performance characteristics.
For example, Dremel (as described in this paper: http://research.google.com/pubs/pub36632.html) is an extremely fast SQL-based query engine at Google, which powers the Google cloud product BigQuery (which I work on). Because Dremel uses Protocol Buffers as its data model, any set of Protocol Buffers can be dumped into Dremel for fast querying, without any kind of data massaging or converting step. I think this is a really powerful idea.
I can see reasons why Kenton might have chosen not to use .proto schemas directly, but I hope that they can eventually converge, because having a shared schema language is a powerful idea.
This is a good point, and something I considered. There are a few reasons I decided against using the .proto format:
* In .proto format, it's generally assumed that _removing_ a field, or having a gap in field numbers, is OK. In Cap'n Proto, it's not OK, because offsets of subsequent fields are affected.
* The language frankly has a lot of little ugly quirks that I wanted to get away from.
* I want control over my format so that I can add features that protobufs may not be interested in, like the ability to define constants.
* After proto2 I'm kind of tired of working around legacy. :)
* I'm not working for Google anymore, so I don't have any particular reason to cater to what would make the most sense for Google. Why not use Thrift or POJOs or anything else as the base language? :)
I think most of the benefits you describe could be achieved by writing a .proto<->.capnp translator, which should not be very hard given that, as you say, the languages are pretty much isomorphic.
The idea of a fixed-length encoding of a protobuf-like structure is a good idea, and something I have seen explored elsewhere. To me, though, it's unfortunate to create an entirely new schema language for it, instead of just making it an alternate encoding of the existing protobuf data model, which it closely resembles. To have two schema languages that are nearly, but not quite, isomorphic means that you get no interoperability benefits with any existing protobuf-based code or data.
The way I see it, the most important feature of protobufs is not the on-the-wire encoding, but rather the schema. If you start from a schema (and its associated data model), then you can create as many different encodings as you want, and convert between them losslessly at any time, while taking advantage of their different performance characteristics.
For example, Dremel (as described in this paper: http://research.google.com/pubs/pub36632.html) is an extremely fast SQL-based query engine at Google, which powers the Google cloud product BigQuery (which I work on). Because Dremel uses Protocol Buffers as its data model, any set of Protocol Buffers can be dumped into Dremel for fast querying, without any kind of data massaging or converting step. I think this is a really powerful idea.
I can see reasons why Kenton might have chosen not to use .proto schemas directly, but I hope that they can eventually converge, because having a shared schema language is a powerful idea.