
Cap’n Proto - bleakgadfly
https://capnproto.org/index.html
======
kentonv
Hi all, Cap'n Proto author here. Thanks for the post.

Just wanted to note that although Cap'n Proto hasn't had a blog post or
official release in a while, development is active as part of the Sandstorm
project ([https://sandstorm.io](https://sandstorm.io)). Cap'n Proto --
including the RPC system -- is used extensively in Sandstorm. Sandboxed
Sandstorm apps in fact do all their communications with the outside world
through a single Cap'n Proto socket (but compatibility layers on top of this
allow apps to expose an HTTP server).

Unfortunately I've fallen behind on doing official releases, in part because
an official release means I need to test against Windows, Mac, and other
"supported platforms", whereas Sandstorm only cares about Linux. Windows is
especially problematic since MSVC's C++11 support is spotty (or was last I
tried), so there's usually a lot of work to do to get it working.

As a result Sandstorm has been building against Cap'n Proto's master branch so
that we can make changes as needed for Sandstorm.

I'm hoping to get some time in the next few months to go back and do a new
release.

~~~
lux
I recently ran into flatbuffers
[https://google.github.io/flatbuffers](https://google.github.io/flatbuffers)),
which claims the same no-serialization step. Curious how the two compare?

~~~
bchallenor
I recently evaluated the two and went with FlatBuffers, largely because its
Java support appeared to be more mature.

The FlatBuffers encoding is based on vtables and is relatively straightforward
(the runtime library is tiny). This also means it's inefficient for small
messages, but in my testing its vtable deduplication worked great for my use
case (~100k messages of the same type per memory-mapped file), in that the
vtable overhead tends quickly to zero.

Cap'n Proto has a more complex encoding that is probably more efficient in
terms of wire size, and particularly for small/standalone messages, but the
runtime is larger as a result.

------
Cyph0n
For some reason, the banner (infinitely faster?), name, and introductory FAQ-
style responses made me think the whole thing is a joke - similar to Vanilla
JS [1].

Anyways, it seems like a cool project, so I'll be sure to follow its
development closely.

[1]: [http://vanilla-js.com/](http://vanilla-js.com/)

~~~
pjscott
The best jokes in software usually have running code. The best of the best are
practical.

~~~
StavrosK
Like Flask.

~~~
Cyph0n
It was a joke aimed at Bottle, right? And now it's probably the second most
popular Python web framework. Armin knows his shit.

~~~
StavrosK
Yeah, it was an April fool's joke satirizing web microframeworks. He does, but
I don't know if that was more because Flask is a better framework or because
of better marketing. I don't think Bottle is really any worse than Flask.

------
niftich
I've always liked Cap'n Proto because it was (quite literally) the ideas
behind Protobuf taken to an extreme, or, depending on your point-of-view,
reduced to its most basic components: data structures already have to sit in
memory looking a certain way, why can't we just squirt that on the wire
instead of some fancy bespoke type-length-value struct?

Of course, the hardest part is convincing everyone that it's not _your
bespoke_ type-length-value struct, but that you have _good reasons_ for what
you're doing. I think the humorous, not-so-self-serious presentation has
worked in its favor (but that's just a subjective opinion and I can't back it
up with data).

~~~
jackmott
>data structures already have to sit in memory looking a certain way, why
can't we just squirt that on the wire instead of some fancy bespoke type-
length-value struct?

In C/C++ ya can! When making games in college that is exactly what we did.
Take the struct, dump it into the socket. I was rather shocked when trying to
recreate the same system in C#. "I can't? I _CAN 'T_?"

~~~
QuercusMax
That's an absolutely terrible idea if you want any kind of forward / backward
compatibility. Just about any system is better than dumping structs over the
network, once you're dealing with a system that can evolve.

~~~
heywire
You might be surprised just how many successful applications simply dump a
packed struct to disk as a serialization format (or even transmit it over the
wire). If the operating environment (OS/hardware) is known to be of a certain
type, endianness is not a concern.

~~~
QuercusMax
It's not just endianness; it's about maintainability. If you ever think you
might change the struct, EVER, you need to worry about these things.

------
throwaway13337
To get an overview of the area of binary interchange formats that are language
agnostic, the author of Cap'n Proto does a good job in this:

[https://capnproto.org/news/2014-06-17-capnproto-
flatbuffers-...](https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-
sbe.html)

"Protocol Buffers" has been the go-to for a long time but there are more
options now.

For uses where serialization/deserialization CPU time is a concern, it seems
to really a question of Cap'n Proto versus flatbuffers (
[https://google.github.io/flatbuffers/](https://google.github.io/flatbuffers/)
).

~~~
kyrra
I'm wondering how it compares to proto3, which I understand is a fairly large
change to the original protobuf.

~~~
kentonv
Proto3 is not a large change. In fact it shares most of its code with proto2,
as I understand it. The underlying encoding is the same.

Proto3 removes some features from proto2 which were deemed overcomplicated
relative to their value (unknown field retention, non-zero default values,
extensions, required fields) and adds some features that people have wanted
for a long time (maps). But all of these features are things that are "on top"
of the core, not really fundamental changes.

I think the only change which affects my comparison post (linked by GP) is
removal of unknown field retention. This is actually noted in the comparison
grid. I'm honestly very surprised that they chose to remove this feature since
it is critical to many parts of Google's infrastructure.

~~~
dekhn
proto2 has maps, but like proto3 they aren't maps, they're an randomly ordered
sequence of key value pairs _ugh_.

~~~
kentonv
Presumably the lookup table is built at parse time?

(Proto2 definitely didn't have any built-in notion of maps when I was working
on it. I thought maps were added as a proto3 feature...)

~~~
dekhn
[https://developers.google.com/protocol-
buffers/docs/proto#ma...](https://developers.google.com/protocol-
buffers/docs/proto#maps)

I don't think any lookup table is provided (the wire order of entries is
undefined). They are not lookup maps, they are syntactic sugar for repeated
key/value pairs.

~~~
kentonv
The "lookup table" is constructed at parse time. Yes, the items are just
key/values on the wire, but when parsed (in C++, at least) they are placed in
a google::protobuf::Map, which is hashtable-based. I guess it could differ
across languages -- some might not have any specific support for maps yet.

------
Perceptes
Big fan of what Sandstorm is doing, both with Sandstorm itself and this
component. I _really_ want to use this instead of gRPC, as it seems
technically superior, but language bindings and adoption across language
ecosystems are likely to be a big downside given that (as Kenton mentions in a
comment elsewhere here) Sandstorm isn't really interested in Cap'n Proto being
widely adopted. All my new stuff is built in Rust, so the Sandstorm team's
interest in and use of Rust are a good fit for me. But when it comes to
interoperability with other languages, this may end up being a concern
compared to gRPC. In any case, I hope to see the Rust implementation
eventually replace the C++ one as the official reference implementation.

~~~
kentonv
FWIW, language interoperability is of interest to Sandstorm since apps can be
written in many languages. But we're only 7 people with a lot on our plate, so
unfortunately we can't currently be the ones to go around implementing Cap'n
Proto in every language. That will change as Sandstorm grows.

~~~
Perceptes
Great news, and thanks for the response! I don't have an immediate need for a
system like this, so perhaps by the time I do there will be broader adoption.

------
venning
Correct me if I'm wrong, but some of this sounds like _blitting_ , except
optimized for the in-memory structure and not the on-disk structure.

In 2008, Joel Spolsky wrote about 1990s-era Excel file formats and how they
used this technique to deal with how slow computers were then [1]. Same
technique, new problem set.

[1]
[http://www.joelonsoftware.com/items/2008/02/19.html](http://www.joelonsoftware.com/items/2008/02/19.html)

------
setori88
Fractalide
([http://githib.com/fractalide/fractalide](http://githib.com/fractalide/fractalide))
is an implementation of dataflow programming (specifically flow based
programming). Component build hierarchies are coordinated via the Nix package
manager. Capnproto contracts are weaved into each component just before build
time. These contracts are the only way compenents talk to each other. Thanks
Sandstorm.io for this great software.

~~~
edraferi
FTFY:
[https://github.com/fractalide/fractalide](https://github.com/fractalide/fractalide)

------
megak1d
I've always liked the look of this, saw it a while back but we are still using
protobuf in our .NET environment simply due to the "free" schema generation
using AOP/attributes [ProtoContract]/[ProtoMember] in Marc Gravell's excellent
protobuf-net ([https://github.com/mgravell/protobuf-
net](https://github.com/mgravell/protobuf-net)) project - I assume this would
also be possible for cap'n proto.

------
zaptheimpaler
Could this be used as an alternative to Apache Arrow[1]?

[1] [https://arrow.apache.org/](https://arrow.apache.org/)

------
mrfusion
I hate to ask but can anyone explain this like I'm from 2000?

I guess it's a way to send data to your front end java script but not use json
and this compresses it so it's faster? How much better than using json is it?

~~~
striking
It lets you put data on the wire, in a structured format, right out of memory.
Asking the question "how much faster is it" isn't even valid here, because it
skips the usual serialization process.

Cap'n Proto generates you some code that contains some data structures. You
put data into these structures, and they will automatically be in the right
shape to put directly on the wire. And then that data can be pulled right off
the wire and right into memory and be fully ready to access, with no
intermediate step.

It's, in a sense, infinitely faster than JSON serialization or
deserialization. Because it doesn't even perform any serialization. It's just
data.

There are some other tricks at play here, but I won't go into them. This is
plenty cool.

~~~
Matthias247
It's infinitely faster if you have control over the data layout in your
application - which most likely means your are developing your application in
C/C++, or maybe Rust. In the case of JS you don't have, so accessing and
writing the data is slow (since you would need [de]deserialization there to
write it in a byte array). In fact any serializations in JS are slower than
JSON, since this is natively implemented in the JS VMs while others are not.

~~~
kentonv
No no, this is a common misunderstanding about Cap'n Proto. It does _not_ take
your regular in-memory data structures from your regular programming language
(even C++) and put them on the wire. What it does is defines its own specific
data layout which happens to be appropriate both for in-memory random-access
use and for transmission.

Cap'n Proto generates classes which wrap a byte buffer and give you accessor
methods that read the fields straight out of the buffer.

That actually works equally well in C++ and Javascript.

~~~
Matthias247
Ok, but these accessor methods will still have a very different performance.
In C++ copying a UTF8 string from one byte array into an std::string is super
fast. Whereas in JS it's really slow, since you need to read from an
ArrayBuffer, convert code points to UTF16 in JS and then store these in a
string (which is not efficient, since the strings are immutable). In node you
could at least speed that up through some native extension, but even then it
would most likely be slower than JSON. In the browser it would be in any case.
But that's really a speciality of JS.

In general I then think the difference (for non-C++) between your method and
others (protobuf, thrift, ...) is that yours would require the cost of a field
serialization in the moment the field is accessed. In others all fields are
deserialized at once. But in the end it should have the same cost if I need
all fields, e.g. in order to convert the data into a plain
Java/Javascript/C#/... object, or am I missing something there? For C++ is
absolutely believe that you can have a byte-array backed proxy-object with
accessor methods that have the same properties as accessing native C++
structures.

~~~
kentonv
Even if you access every field, Cap'n Proto's approach should still be faster
in theory because:

\- Making one pass instead of two is better for the cache. When dealing with
messages larger than the CPU cache, memory bandwidth can easily be the
program's main bottleneck, at which point using one pass instead of two can
actually double your performance.

\- Along similar lines, when you parse a protobuf upfront, you have to parse
it into some intermediate structure. That intermediate structure takes memory,
which adds cache pressure. Cap'n Proto has no intermediate structure.

\- Protobuf and many formats like it are branch-heavy. For example, protobuf
likes to encode integers as "varints" (variable-width integers), which require
a branch on every byte to check if it's the last byte. Also, protobuf is a
tag-value stream, which means the parser has to be a switch-in-a-loop, which
is a notoriously CPU-unfriendly pattern. Cap'n Proto uses fixed widths and
fixed offsets, which means there are very few branches. As a result, an
_upfront_ Cap'n Proto parser would be expected to outperform a Protobuf
parser. The fact that parsing happens lazily at time of use is a bonus.

All that said, it's true that if you are reading every field of your
structure, then Cap'n Proto serialization is more of an incremental
improvement, not a paradigm shift.

------
wtbob
> The Cap’n Proto encoding is appropriate both as a data interchange format
> and an in-memory representation, so once your structure is built, you can
> simply write the bytes straight out to disk!

Eh, I'd rather pay the cost of serialisation once and deserialisation once,
and then access my data for as close to free as possible, rather than relying
on a compiler to actually inline calls properly.

> Integers use little-endian byte order because most CPUs are little-endian,
> and even big-endian CPUs usually have instructions for reading little-endian
> data.

 _sob_ There are a lot of things Intel has to account for, and frankly little-
endian byte order isn't the worst of them, but it's pretty rotten. Writing
'EFCDAB8967452301' for 0x0123456789ABCDEF is perverse in the extreme. Why?
_Why?_

As pragmatic design choices go, Cap'n Proto's is a good one (although it
violates the standard network byte order). Intel effectively won the CPU war,
and we'll never be free of the little-endian plague.

It's all so depressing.

~~~
kentonv
Big-endian vs. little-endian is an ancient flamewar that isn't going to go
away any time soon, but sure, let's argue.

Once you've spent as much time twiddling bits as I have (as the author of
proto2 and Cap'n Proto), you start to realize that little endian is much
easier to work with than big-endian.

For example:

\- To reinterpret a 64-bit number as 32-bit in BE, you have to add 4 bytes to
your pointer. In LE, the pointer doesn't change.

\- Just about any arithmetic operation on integers (e.g. adding) starts from
the least-significant bits and moves up. It's nice if that can mean iterating
_forward from the start_ instead of _backwards from the end_ , e.g. when
implementing a "bignum" library.

\- Which of the following is simpler?

    
    
        // Extract nth bit from byte array, assuming LE order.
        (bytes[n/8] >> (n%8)) & 1
    
        // Extract nth bit from byte array, assuming BE order.
        (bytes[n/8] >> (7 - n%8)) & 1
    

There's really no good argument for big-endian encoding except that it's the
ordering that we humans use in writing.

I think the correct answer won here.

~~~
wtbob
> To reinterpret a 64-bit number as 32-bit in BE, you have to add 4 bytes to
> your pointer. In LE, the pointer doesn't change.

But one shouldn't do that very often: those are two different types. The
slight cost of adding a pointer is negligible.

> Just about any arithmetic operation on integers (e.g. adding) starts from
> the least-significant bits and moves up. It's nice if that can mean
> iterating forward from the start instead of backwards from the end, e.g.
> when implementing a "bignum" library.

\-- is a thing, just as ++ is.

> There's really no good argument for big-endian encoding except that it's the
> ordering that we humans use in writing.

That's like saying, 'there's really no good argument for pumping nitrogen-
oxygen mixes into space stations except that it's the mixture we humans use to
breathe.'

It's simplicity itself for a computer to do big-endian arithmetic; it's
horrible pain for a human being who has to read a little-endian listing. A
computer can be made to do the right thing. Who is the master: the computer or
the man?

~~~
kentonv
That line of argument suggests you'd be happier with a human-readable format
like JSON. Which is another eternal flamewar that we aren't likely to resolve
here. Needless to say I like binary formats. :)

~~~
wtbob
> That line of argument suggests you'd be happier with a human-readable format
> like JSON.

Oh, that's rude. There's a huge difference between flipping a few bytes and
committing a public indecency against God and man like JSON.

------
sandGorgon
How do you pronounce the name? If libreoffice is bad.. This name is absolutely
impossible.

Is it captain? Is it cap+n+proto?

A lot of collaboration is verbal - people sit around and talk about stuff. I
don't know if it is a fun take on an American word... But it is impossible to
use in the rest of the world.

I really wish you would call it something else... Unless it is personal for
you :(

~~~
kentonv
CAP-ən PRO-to

But "Captain Proto" is acceptable if you have trouble with the contraction.

Or you can also think of it as "Cap-and-Proto". Which is an intentional pun
("capabilities and protocols", or something).

Googling either of these will get you to the right place, so I think it gets
the job done.

~~~
nemo1618
>Or you can also think of it as "Cap-and-Proto". Which is an intentional pun
("capabilities and protocols", or something).

I never realized that! I like the name much more now.

------
joshuawarner32
Here's the discussion from a while ago:
[https://news.ycombinator.com/item?id=5482081](https://news.ycombinator.com/item?id=5482081)

------
chaotic-good
I think that compression is a must for the serialization library. Protobuf
uses almost twice less memory than Cap'n Proto. Using an external compression
is not an option in some cases. E.g. consider building tcp-server that
communicates with thousands of clients simultaneously. Each client connection
will have its own LZ4 context that should be heap allocated. I believe it's
about 16KB per connection + buffers. This results in large memory consumption
and a lot of random memory access and TLB misses.

~~~
kentonv
Cap'n Proto offers "packed" encoding which applies light compression (removing
zero-valued padding bytes), brings it in-line with Protobuf, and ought to be
much faster than the things Protobuf does for "compression" (varint is a very
slow encoding!).

~~~
chaotic-good
I'm glad that I was wrong about it.

------
flatline
Interfaces! Inheritance! Looks promising. Protocol buffers are nice for their
compact encoding and multi-language generator support but as a schema language
they are really cumbersome. Composition is pretty much all you get, there are
no longer required fields, you can't even use enums as a key type in a map.
I'm sure their use cases are not necessarily the same as mine but sometimes I
miss just using plain old XML.

~~~
kentonv
To be clear, Cap'n Proto's serialization layer, from a schema perspective, is
almost exactly the same as Protobuf (though with a very different underlying
encoding).

The interfaces and inheritance relate to the RPC system. The interfaces are
for remote objects.

See: [https://capnproto.org/rpc.html](https://capnproto.org/rpc.html)

------
morecoffee
> capability-based RPC system.

This sounds like a cool idea, but so far I haven't seen any good explanation
of how it works, and why it will save me from rolling my own ACL system. For
bragging about it in the very first sentence, there is surprisingly little
detail about how it works.

~~~
kentonv
It's a complicated topic -- it requires thinking about things in a different
way, and tends not to make a lot of sense until at some point it "clicks" and
you realize all sorts of patterns you were already using are actually special
cases of capabilities.

Here is some reading:

[https://capnproto.org/rpc.html#security](https://capnproto.org/rpc.html#security)

[https://sandstorm.io/how-it-works#capabilities](https://sandstorm.io/how-it-
works#capabilities)

[http://zesty.ca/capmyths/usenix.pdf](http://zesty.ca/capmyths/usenix.pdf)

------
mixmastamyk
Apache thrift doesn't seem to be mentioned, how does it compare?

~~~
kentonv
Thrift is very much equivalent to Protobuf for the purpose of everything
discussed on the web site. As the site mentions, I prefer to pick on Protobuf
because I am also the author of Protobuf v2. :)

------
Twonneilb22ll
I'm happy to hear from you all in glad to see you're doing all you can for
updates Programs codes appreciate the hard work

------
Twonneilb22ll
I'm happy to hear from you all in glad to see you're doing all you can for
updates Programs codes appreciate the hard work

------
Twonneilb22ll
I'm happy to hear from you all in glad to see you're doing all you can for
updates

------
Paul_S
Would probably be a good idea to have a no-exceptions version.

~~~
kentonv
You can compile Cap'n Proto with -fno-exceptions, and it does a bunch of
things differently to make that work. Basically, invalid-input exceptions
instead replace the invalid data with a reasonable default value and set a
flag on the side that you can query to see if there was any invalid input.
Assertion failure exceptions (where there is no way to recover) largely turn
into fatal errors.

------
imaginenore
Is it faster than MsgPack?

~~~
kentonv
Depends on the use case! In fact, the answer to "is X format faster than Y
format" always depends on the use case. It's always easy to construct cases
where one or the other looks better. People of course want to know "on
average", but in reality there's no such thing as an "average" use case.
You'll ultimately have to test the case you have in mind to find out.

With that said, here are some considerations:

\- msgpack is usually used as a binary encoding of JSON, with no schemas. That
means that textual field names are included in the encoded message. Formats
like Protobuf and Cap'n Proto that have schemas known in advance can avoid
this bloat, making them faster and smaller.

\- msgpack is _not_ a zero-copy encoding. It's necessary to parse the whole
message upfront before you can use it, like with protobuf. Cap'n Proto is
zero-copy, the advantages of which are described extensively on the page. For
example, if you have a multi-gigabyte file containing a massive Cap'n Proto
message, and you just want to read one field from one place in that message,
you can do that by memory-mapping the file. No need to read it all in. That's
not possible with Protobuf or Msgpack.

I think it's best to focus on these kind of paradigm-shifts when trying to
reason about performance. You can always micro-optimize the encoding path
later on, but you can't suddenly switch to zero-copy later if your data format
wasn't designed for it.

------
matmann2001
.

------
TazeTSchnitzel
Blender's file format does something similar, it essentially saves a core dump
to disk.

