
Cap'n Proto 0.6 released – 2.5 years of improvements - 0x1997
https://capnproto.org/news/2017-05-01-capnproto-0.6-msvc-json-http-more.html
======
DonbunEf7
As usual, nobody has said "capability" yet, which is unfortunate, because one
of Capn's biggest strengths is that it embodies the object-capability model
and is cap-safe as a result.

Edit: Why does this matter? Well, first, it matters because so little software
is capability-safe. Capn's RPC subsystem is based directly upon E's CapTP
protocol. (E is the classic capability-safe language.) As a result, the
security guarantees afforded by cap-safe construction are _extended across the
wire_ to cover the entire distributed system. This security guarantee holds
even if not every component of individual nodes is cap-safe, and that's how
Sandstorm works.

Continuing on, there's also historical stuff going on. HN loves JSON; JSON is
based on E's DataL mini-language for data serialization. HN loves ECMAScript;
ES's technical committee is steered by ex-E language designers who have been
porting features from E into ES.

~~~
ludwigvan
Is this the E language you are referring to? Had never heard of it before.

[https://en.wikipedia.org/wiki/E_(programming_language)](https://en.wikipedia.org/wiki/E_\(programming_language\))

~~~
kentonv
Yes. It's not widely known because it was somewhat of a research language,
never really production ready. But, it presented clean solutions to a huge
number of hard problems, and as a result has been pretty influential behind
the scenes. E.g. Javascript promises are directly inspired by E.

------
drej
This might be a contrarian view, maybe I'm misunderstanding it. To me, much of
message passing is not performance critical, so it would be well served by
JSON/YAML/XML for easy implementation/testing/debugging. If one needs
performance, he can just send bytes over, which can be (de)serialised in a
(couple) dozen SLOC. Sure, when you're talking about very complex structures,
RPC, dynamic schemas etc., then you might opt for something like this, but
let's be honest - that's quite a minority of current users, isn't it?

I never minded these frameworks, but then I wanted to write a few parsers for
some file formats and they used Thrift/Flatbuffers to encode a few ints, which
seemed like a major overkill. There was no RPC, no packet loss, no nothing.

~~~
kentonv
Your view isn't contrarian at all -- in fact it's widely-held by smart people.
But I'll present the counter-argument.

True, in many use cases, performance is not critical. However, when you're
doing big data processing or serving at scale, serialization performance does
in fact start to matter a lot. If Google Search passed all backend messages as
JSON rather than Protobuf, it would probably require 10x the hardware (we may
be talking about _millions_ of machines here) and wouldn't be able to respond
as quickly. Similarly, Cloudflare couldn't handle its logging load without
Cap'n Proto.

Many times, companies who started small and thought performance didn't matter
later found themselves needing to do massive, expensive rewrites in order to
keep up with scale (e.g., famously, Twitter).

Hand-written binary serializers may be easy to write _initially_ , but they
are very hard to maintain. Over time, you'll need to add and remove fields,
while maintaining compatibility with old binaries already running in the wild,
or compatibility across languages. It's very easy to screw this up with a
hand-written encoder, but very easy to get right with Protobuf or Cap'n Proto.

Finally, even if performance isn't an issue, you may find that type safety is
useful. If you are writing code in a type-safe language, correctly dealing
with JSON actually tends to be a huge pain involving lots of branching. By
defining a schema upfront and generating code, you can get a much nicer
interface. And if you're doing that anyway, then you might as well take
advantage of the more-efficient encoding while you're at it -- it's easy to
dump human-readable text for debugging when needed. (As of this release, Cap'n
Proto ships with a library for converting to/from JSON, and there are lots of
third-party libraries to do the same with Protobuf. Both libraries also have
one-liner "dump a debug string" functions.)

~~~
kevin_thibedeau
You don't need a schema for type safety. Just an encoding that preserves type
information like all the schemaless binary serializations (BSON, CBOR, etc.)
do.

~~~
kentonv
You're talking about a different kind of "type safety".

I mean that when I write code to consume a message, I would like to detect (at
compile time) if I typo a field name, and I also don't want to worry about
what happens if a (possibly-malicious) sender sending me a different type than
I expected.

~~~
EdSharkey
I like to use the term "binding" to differentiate between the compile-time
type checks and static typing of messages in a programming language vs. the
runtime type equivalence checks performed between sender and receiver of a
message.

~~~
bborud
Please don't use the term "binding". It has many uses and it mostly tends to
confuse people.

~~~
EdSharkey
What kinds of uses of the term "binding" causes confusion in the context of
message data types?

I think binding is precisely what is going on when a compiler generates code
for marshaling and unmarshaling messages based on a schema.

~~~
bborud
Binding, as a term, is used for anything from assigning values to variables to
what is essentially dependency injection of implementations of more or less
abstract interfaces.

I'm merely pointing out that the term is vague and can cause confusion. That
you make a strong association with one use of the term doesn't really govern
what others associate with it.

------
pagnol
Some time ago I wanted to use Cap'n Proto in the browser but then I found that
the only existing implementation written in JavaScript hadn't been updated in
two years and the author himself recommended against its use somewhere in a
thread on Stackoverflow. I would love to use Cap'n Proto but for me a robust
JS implementation is a sine qua non. Does anyone here happen to know if
there's been any progress in this regard or have I missed something?

~~~
kybernetikos
I've got about 70% of a capnproto implementation written in pure javascript
that works in the browser.

Part of the problem is that there is a bit of an ideological difference, where
I prefer dynamic code to adding build steps but all the existing tools and
code assume you want to generate source. I also found it quite irritating to
bootstrap too, because the tooling itself uses capnproto. It sounds neat, but
then it means that you have to be able to read capnproto in order to be able
to read it.

The documentation was also incredibly patchy at the time which made it quite
hard to develop for, although I'm told that the kentonv & gang are very
helpful.

In the end, I was just starting to struggle with capnproto generics when my
requirement went away as the server I was connecting to added a msgpack
option.

~~~
kentonv
FWIW, the Python implementation of Cap'n Proto loads schemas dynamically --
there is no code generator.

However, this is done through a C extension that calls into the C++
implementation. For browser-side Javascript, that's obviously problematic.
(Emscripten isn't really the answer -- would be way too large a download.)

~~~
TazeTSchnitzel
> Emscripten isn't really the answer -- would be way too large a download.

Could the emscripten result be slimmed-down?

(I fear the answer is probably “in theory, yes, in practice, it's >100KB”)

------
hellofunk
There is some strange geeky enjoyment from browsing all the serialization
libraries out there. For my taste, I have centered on Cereal. When workind
with C++ end to end, I have found it to be the easiest and fastest way to
throw data around.

~~~
Heliosmaster
Serialization can easily be a bottleneck, especially for write-heavy systems
(like we do, with Event Sourcing). So i think it's quite natural that people
try and squeeze the last few milliseconds out of it. It can easily make a huge
difference when serializing/deserializing billions of entities

~~~
fh973
Serialization is order of microseconds, not milliseconds.

~~~
kbenson
That depends entirely on how much (in both quantity and size) you are
serializing, doesn't it?

------
forrestthewoods
Woohoo! Lack of first class Windows support always held me back. Looking
forward to playing with this and seeing hopefully more regular future updates.

------
ipsum2
Great stuff! Since this release comes after the release of GRPC and (slightly
less related) Graph API and ships with a web server, how does it compare?

~~~
kentonv
Cap'n Proto RPC was originally released before gRPC. That said, gRPC is
heavily based on Google's internal RPC system which has been around for a very
long time, albeit not publicly.

If you read the Cap'n Proto RPC docs, everywhere where it mentions
"traditional RPC", I specifically had Google's internal RPC in mind (having
previously been the maintainer of Protobufs at Google). So, you can more-or-
less substitute gRPC in there for a direct comparison.
[https://capnproto.org/rpc.html](https://capnproto.org/rpc.html)

There are two key differences:

1\. Cap'n Proto treats references to RPC endpoints as a first-class type. So,
you can introduce a new endpoint dynamically, and you can send someone a
message containing a reference to that endpoint. Only the recipient of the
message will be able to access the new endpoint, and when that recipient drops
their reference or disconnects, you'll get notified so that you can clean it
up. This is incredibly useful for modeling stateful interactions, where a
client opens an object, performs a series of operations on it, then finally
commits it. Put another way, this allows object-oriented programming over RPC.
Also note that you can easily pass off object references from machine to
machine -- currently this will set up transparent proxying, but in the future
we plan to optimize it so that machines automatically form direct connections
as needed, which will be really powerful for distributed computing scenarios.

2\. Relatedly, Cap'n Proto supports "promise pipelining", which allows you to
use the result of one RPC as an input to the next without waiting for a round-
trip to the client. This makes it possible to use object-oriented interaction
patterns with deep call sequences without introducing excessive round-trip
latency. This is described in detail at the RPC link above.

~~~
Matthias247
For me another key difference (and really a killer feature if you need it) is
that grpc supports streaming usecases natively besides RPC. In server->client,
client->server and bidirectional ways.

Server->Client streaming is e.g. a powerful way to get realtime updates about
some state on the server to the client without needing to poll (not
performant) or needing to define callback services (ugly to maintain, service
lifecycle questions, and if you need an extra connection for from the service
to the callback service (client) there are also challenges around routing).

Native support for streaming also removes the need to model explicit flow
control (backpressure) behavior in the user defined APIs (like functions for
requesting some items in addition to the callback functions for delivering
items).

Another difference is that grpc utilizes HTTP/2 as underlying transport
protocol. However I'm thinking that's mainly an implementation detail. If
Google had designed grpc slightly different (not relying on barely implemented
features like HTTP trailers) it could have been a bigger differentiator, e.g.
by allowing browsers to directly make grpc calls without proxies.

~~~
kentonv
gRPC-style streaming can be implemented in terms of Cap'n Proto object
references: When starting a streaming call, the client creates a callback
object to be called by the server every time a new message is available, and
then sends a reference to that object as part of the request. Or for
client->server streaming, the server returns such an object reference.

The drawbacks of callbacks that you mention don't apply in this scenario: The
same network connection is utilized for the callback, solving the routing
question. The callback object receives a notification when the caller is done
with it (or disconnects), so it can clean up, solving the lifecycle question.
Flow control / backpressure can be achieved by pausing calls to the callback
if too many previous calls have not returned (I actually plan to bake this
pattern into the library in the future to make it dead simple).

So, it seems to me streaming in gRPC is actually a narrow subset of what Cap'n
Proto can express.

For web stack compatibility purposes, it's straightforward to implement
Cap'n-Proto-over-WebSocket. I'm not sure that HTTP/2 buys much on top of that.

~~~
Matthias247
Thanks for the explanation. If it works like you describe it seems to be a
reasonable way to obtain streaming behavior. Will take a look at it if I find
some time.

Regarding backpressure and HTTP/2: You get some different kind of backpressure
behavior with your approach (application level flow control) and the grpc
approach (transport level flow control). Let's say you have 2 functions which
are called in parallel. For one the arguments are very big (let's say 100kB),
for the other one they are small (some bytes). With HTTP/2 and transport level
flow control the small function could get the bandwidth as the big one, which
means more requests/s for the small function. With pure application level flow
control the small function needs to wait until the big one is fully sent
before it can be put on the wire.

Which means if I have 2 tasks that execute concurrently and look like

    
    
        A: while (true) { await callBigFunction(); }
        B: while (true) { await callSmallFunction(); }
    

then with grpc I get more calls for B and without Cap'N'Proto I get the same
for both (correct me if I'm wrong).

I don't think it's a huge disadvantage in practice, because if you have large
data chunks you should probably design your API different - and if you want
realtime behavior then probably both protocols are not optimal. But it's still
something to keep in mind.

~~~
kentonv
You can achieve what you describe under my scheme by counting the total size
of the parameters of all calls in flight rather than just counting the calls.
This is what sophisticated systems are doing in practice and is what I plan to
do when incorporating the pattern into the library.

~~~
Matthias247
The described approach sounds very reasonable!

Besides building more support in the library it would maybe also be very
worthwhile to have these patterns described together with examples on the
website. E.g. how to achieve callback interfaces which don't user another
connection, server->client streaming, etc.

For me as a potential user information about these kind of features would make
Cap'n'Proto immediately look more interesting. When I look at the website I
only see Promise Pipelining described as a main feature - which is novel for
sure, but not something that attracts my interest if I look for the other
features we discussed about.

------
makmanalp
What always trips me up about capnproto is that it's billed as a serialization
library, but what it is is an in-memory storage layout, and "serialization" is
mostly just dumping memory into a file, right? (which is cool)

What confuses me is, then what are the costs of migrating to this system? Am I
essentially dumping my programming language's object model for my capnproto
implementation's? When can this be annoying? Or does it vary from
implementation to implementation?

In a similar tangent - how similar is this to apache arrow, not because of the
columnar analytics part, but could I expect to just dump a bunch of data in
shared memory and read it from another process to eliminate IPC
serialization/copy costs?

~~~
kentonv
Generally I'd recommend using Cap'n Proto in much the same way as you'd use
Protobuf. It's not intended that your in-memory state be in Cap'n Proto
objects, only the messages you intend to transmit, or data stored on disk.

~~~
na_ka_na
Why is not intended that in-memory state be in Cap'n Proto / Protobuf objects?
What are the down-sides?

~~~
rkv
After integrating protobufs in my application for messaging I decided to use a
separate schema for storing the current state of the program. Ie. When state
changes, the protobuf is updated and written to disk. When the program
restarts, the state file is loaded into memory. I have not run into any
problems doing this.

Edit: Your question is addressed here:
[https://news.ycombinator.com/item?id=14249367](https://news.ycombinator.com/item?id=14249367)

~~~
na_ka_na
Thanks, but I don't follow how that comment addresses my question. Is it that
cost of constructing Cap'n Proto / Protobuf is quite a bit higher than
constructing objects defined natively?

~~~
kentonv
> Is it that cost of constructing Cap'n Proto / Protobuf is quite a bit higher
> than constructing objects defined natively?

I discussed in more detail in reply to your first post, but just to be really
clear on this:

No. In fact, for deeply-nested object trees, constructing a Cap'n Proto object
can often be cheaper than a typical native object since it does less memory
allocation. However, there are some limitations -- see my other reply.

(Constructing Protobuf objects, meanwhile, will usually be pretty much
identical to POCS, since that's essentially what Protobuf objects are.)

There is a common myth that Cap'n Proto "just moves the serialization work to
object-build time", but ultimately does the same amount of work. This is not
true: Although you could describe Cap'n Proto as "doing the serialization at
object build time", the work involved is not significantly different from
building a regular in-memory object.

------
grandinj
Does it do protocol negotiation? i.e. can a client ask the server what
interfaces it implements?

~~~
kentonv
There are a few answers to that:

1\. If you have a remote object reference, you can (explicitly) cast it to any
interface type, and then attempt to call it. If it doesn't implement that
interface (or that method), an "unimplemented" exception will be thrown back.
It's relatively common to do feature detection this way.

2\. It's easy for the application to define a Cap'n Proto interface to support
fancier negotiation. For example, you could have all your RPC interfaces
extend a common base interface which has a method getSchema() which returns
the full interface schema, or a list of interface IDs, or whatever it is you
want.

3\. We actually plan to bake in schema queries in a future version, such that
all objects will support some sort of getSchema() call implicitly. This would
especially be useful for clients in dynamic languages that could potentially
connect to a server without having a schema at all, and load everything
dynamically.

------
mhogomchungu
I think this[1] API among others should be renamed and "future" be used
instead of "promise" to better reflect C++ naming conventions.

[1] [https://github.com/sandstorm-
io/capnproto/blob/247e7f568b166...](https://github.com/sandstorm-
io/capnproto/blob/247e7f568b1664cbcca1455243da06a7ab78c6a3/c%2B%2B/src/kj/async-
io.h#L54)

~~~
kentonv
Cap'n Proto Promises are highly analogous to promises in Javascript. Both are
derived directly from the E language, which has been around for decades. It
makes sense to stick with the terminology used in similar designs, so that
people don't have to re-learn the same concept when switching languages.

C++'s standard library for some reason decided -- relatively recently -- to
introduce the term "promise" to mean something different (what most people
call a "resolver" or a "fulfiller" for a promise), which is unfortunate. It's
C++ that is being inconsistent here.

There are also subtle historical differences between the meaning of "promise"
and "future". Historically, futures have usually existed in multi-threaded
designs rather than event-loop/callback designs; you wait() on a future,
blocking the calling thread, whereas with promises you call promise.then() to
register a callback to call when a promise completes. The C++ committee again
seems to have gotten this confused with their definition of "future", which
looks more like a traditional promise.

~~~
moosingin3space
How do Cap'n Proto Promises and Rust futures relate?

~~~
dwrensha
They have a lot in common!

For a while, capnp-rpc-rust used `gj::Promise`, which is based directly on the
C++ Cap'n Proto implementation of promises (i.e. `kj::Promise`). Back in
January, capnp-rpc-rust was updated to use `futures::Future` instead, and it
was a fairly straightforward transition, as described in this blog post:
[https://dwrensha.github.io/capnproto-rust/2017/01/04/rpc-
fut...](https://dwrensha.github.io/capnproto-rust/2017/01/04/rpc-futures.html)

The trickiest part of the transition was dealing with scheduling. The
implementation of `kj::Promise` has a built-in scheduling queue that
guarantees a certain form of deterministic FIFO semantics, and those semantics
are heavily depended upon in the Cap'n Proto RPC implementation. Rust's
`future::Future` is less batteries-included, requiring capnp-rpc-rust to
explicitly create queues where deterministic scheduling is needed.

Confusing the terminology perhaps even more, in capnproto-rust there is a type
`capnp::capability::Promise` that implements `futures::Future`.

------
educar
Windows support is great but why is this a showstopper? This is a common
problem in many libraries and the common solution is to mark those platforms
as unsupported until sometime steps up. Also platform support is a dynamic
list - platforms are kept alive by presence of contributors/maintainers.

~~~
kentonv
I think it would set a pretty bad precedent to have Windows support in one
release and then drop it in the next release, then bring it back, etc. Windows
users would likely be left very confused. Saying "it's up to contributors to
step up" is nice in theory but in practice I don't think a well-maintained
library can operate that way. Few people are eager to volunteer to run tests
over and over fixing tiny issues for a coordinated release...

That said, it probably would have been better to drop Windows temporarily than
to go two years without a release. But it always _seemed_ like I'd find time
in the next month.

Also note that Windows wasn't the only thing. I have a huge test matrix that I
run for every release and, again, to avoid confusion, I want the whole thing
to pass for any release. Things like building with -fno-exceptions or 32-bit
builds or Android or ancient GCC versions tend to break frequently as the code
evolves, but we really ought to have all these working for a release.

But maybe I'm just too OCD about this... :)

We now have AppVeyor (and Travis-CI) set up to build a chunk of the test
matrix on every commit, which should help a lot going forward.

~~~
speps
> But maybe I'm just too OCD about this... :)

It's not OCD, it's good practice and I wish more open-source projects would
follow that! Good job!

------
justforFranz
Would it kill developers of open source to provide a simple, two-sentence
explanation of what their app does?

~~~
paulddraper
How's this?

> Cap’n Proto is an insanely fast data interchange format and capability-based
> RPC system. Think JSON, except binary. Or think Protocol Buffers, except
> faster.

[https://capnproto.org/](https://capnproto.org/)

~~~
kentonv
I guess I should make the top banner be a link to the home page, so that
people on mobile (who can't see the sidebar very easily) can just mash the
screen.

EDIT: done

------
TeeWEE
Ver nice and technically better than gRPC. However i better bet my company
success on a standard that the big giant Google is betting on, and other
companies are embracing. Also client side libraries for all langauges is
important. Probably gRPC is probably better here too?

~~~
ocdtrekkie
FWIW, Cloudflare is betting on Cap'n Proto, and Cloudflare is pretty big too.

~~~
kentonv
Though to be fair, Cloudflare is still 2-3 orders of magnitude smaller than
Google by most measures. :)

------
halestock
A bit off topic, but that title banner is a massive waste of space. It takes
up more than half the space on my screen (15" laptop).

~~~
taftster
I know, totally agree. It's actually one of those things that have made me shy
away from recommending its use. I continually end up back at Protocol Buffers,
simply because the Cap'n Proto website doesn't seem professional to me.
Instead, it feels like the back of a breakfast cereal box.

~~~
kentonv
Working as intended. :)

~~~
taftster
Ha ha, indeed! :)

------
DonHopkins
The name "Cap'n" was forever tainted for me, from my traumatic experience with
"Cap'n Software Forth".

[http://www.art.net/~hopkins/Don/lang/forth.html](http://www.art.net/~hopkins/Don/lang/forth.html)

"The first Forth system I used was Cap'n Software Forth, on the Apple ][, by
John Draper. The first time I met John Draper was when Mike Grant brought him
over to my house, because Mike's mother was fed up with Draper, and didn't
want him staying over any longer. So Mike brought him over to stay at my
house, instead. He had been attending some science fiction convention, was
about to go to the Galopagos Islands, always insisted on doing back exercises
with everyone, got very rude in an elevator when someone lit up a cigarette,
and bragged he could smoke Mike's brother Greg under the table. In case you're
ever at a party, and you have some pot that he wants to smoke and you just
can't get rid of him, try filling up a bowl with some tobacco and offering it
to him. It's a good idea to keep some "emergency tobacco" on your person at
all times whenever attending raves in the bay area. My mom got fed up too, and
ended up driving him all the way to the airport to get rid of him. On the way,
he offered to sell us his extra can of peanuts, but my mom suggested that he
might get hungry later, and that he had better hold onto them. What tact!"

