Hi all, Cap'n Proto author here. Thanks for the post.
Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project (https://sandstorm.io). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n Proto socket (but compatibility layers on top of this allow apps to expose an HTTP server).
Unfortunately I've fallen behind on doing official releases, in part because an official release means I need to test against Windows, Mac, and other "supported platforms", whereas Sandstorm only cares about Linux. Windows is especially problematic since MSVC's C++11 support is spotty (or was last I tried), so there's usually a lot of work to do to get it working.
As a result Sandstorm has been building against Cap'n Proto's master branch so that we can make changes as needed for Sandstorm.
I'm hoping to get some time in the next few months to go back and do a new release.
As of MSVC 2015 Update 3, the support for C++11 is nearly complete. The missing pieces are that Expression SFINAE is partially supported (what's there is good enough for the STL and Boost, with certain workarounds that you must be aware of), and C99 preprocessor support is full of bugs. Aside from that, all features are present and generally usable. (Some are buggier than others; e.g. we're overhauling multithreading library support for the next major version of the libraries.)
There's still a bunch of minor differences/annoyances with the MS toolchain that require me to go in and make a bunch of small changes everywhere to my project, which compiles fine on clang/g++ (and I wouldn't say I'm doing anything _crazy_ like the aforementioned "Expression SFINAE").
Right now, my project's rather small 100 kLOC codebase compiles (with very minimal #ifdef hackery) on Android, iOS, Mac, and Linux; but Windows is still very much a WIP, even some of my third-party libraries don't compile on VC. I'm actually considering trying the MINGW toolchain at this point, and I'd be curious to hear anyone's thoughts?
I'm sure part of it is that the compiler itself probably developed with the tight feedback loop from developing and maintaining large codebases in house at Microsoft (like Windows and Office). It's pretty hard when part of the internal pressure to support large codebases like that conflicts with the need to conform to outside third-party standards. I've heard great things about the compiler people at Microsoft, and I'm sure they have a technically strong team, but they are most likely caught in the middle of this organizational deadlock.
I'm sure it doesn't help that I'm not mentioning any specifics, I'm going to be revisiting the Windows desktop port of my mobile app soon (which I gave up and haven't touched for a couple months), and everything will be fresh in my mind again. I do vaguely remember something about having to explicitly add more #include statements to pull in header files that were already getting pulled in by my other compilers.
I used to develop the Windows port of WordGrinder (my console word processor) using Mingw for Linux, testing with Wine and packaging using nsys for Linux. This actually worked pretty well and allowed a complete Windows development flow from Linux using the same rolling automation for both platforms.
The big problem was Wine, which can behave quite differently to real Windows in some places. It's certainly not bug-for-bug compatible. So I kept finding myself having to boot into Windows anyway for debugging, so eventually I just installed an msys development system on Windows with gvim and used that.
There was also something else wrong with mingw for Linux, but for the life of me I can't remember what. There was something about mingw being different to mingw32? Missing headers? It's been too long.
You might mean the differences between MinGW and MinGW-w64. The
former is the version hosted at
http://mingw.org that used to be called mingw32 (not to be confused with mingw-w64-i686). The latter is what a lot of Linux distributions
package today and what you install with MSYS2 [1] on Windows. It was initially
forked from MinGW to integrate support for the 64-bit Windows API, hence the name,
but it has gained other features since and is more actively maintained.
I've run into a difference between the two recently. It turns out that
MinGW implemented POSIX glob() post-fork, which hasn't been ported to
MinGW-w64.
[1] As I side note, I found that MSYS2 makes a huge difference in how you develop software
for Windows with a POSIXy toolchain. Anyone who uses MinGW on Windows should try it.
Kenton, thank you for your work on this! I currently use ZeroRPC (which uses protobuf and msgpack) and I was blown away by Cap'n Proto. Really excited to try it out soon! Some questions:
- Do you guys have an RPC library written in anything other than C++? If not, could you point me to protocol specs so I can start writing my own?
- Since it uses a streaming model to support random access, what encryption method do you think would work best with Cap ' n Proto that would keep it speedy and still retain all functionality?
> - Do you guys have an RPC library written in anything other than C++? If not, could you point me to protocol specs so I can start writing my own?
People have written implementations in Rust, Go, and Erlang, and wrappers around the C++ library in Javascript and Python: https://capnproto.org/otherlang.html
Scroll down that page for some info on how to start writing an implementation in another language.
> - Since it uses a streaming model to support random access, what encryption method do you think would work best with Cap ' n Proto that would keep it speedy and still retain all functionality?
Hmm, I'm not clear on what you mean by "streaming model" -- I think of "streaming" as the opposite of random access.
Regarding encryption, this is a very big question and there are a lot of different needs and use cases to consider. Mostly I don't think that use of Cap'n Proto affects encryption decisions much, but if you want to make sure you don't lose random access, you should of course use a cipher that supports random access, like chacha20 or AES-CTR.
> Hmm, I'm not clear on what you mean by "streaming model" -- I think of "streaming" as the opposite of random access.
Right, sorry, I re-read my comment and confused myself too. Seems like it's bad for me to go on HN without a fresh cup of coffee (it's 10am now here in the Philippines).
Thanks for your swift response, I'll experiment with AES-CTR (since I'm more familiar with it than chacha20). And thanks for pointing out that there are wrappers for Python/Go already, the programming language I use daily and was thinking of building libs for! Again, great work, and I'll stay posted.
Also as a note, both ciphers suggested are stream ciphers, so you will need some way of authenticating. Depending on your needs, you may consider using something like libsodium that takes care of all of that for you:
Recently I've started compiling more of my code for Windows using Mingw (or clang). Performance is very good and you can build in Linux for Windows inside a Docker container. You can even build Windows installers using NSIS. I've also tried to do this for OSX but with less success.
Cap'n Proto supports MinGW as a target platform. (It has also supported Cygwin since very early on.) However, Cap'n Proto is a library, so it needs to support whatever compiler the users of that library are using, and for the vast majority of Windows developers that's MSVC.
It sounds really well designed. Great job Cap'n Proto, I've been following your work for some time!
I've also built another message format recently (yes I know, they are never ending). It can't do everything Cap'n Proto can do, although it shares a lot of the same values. One thing I chose to do is to have order preservation on types, which can be very useful. It does mean that the wire format is largely BE though. Anyway, that's an aside.
I'm curious what you think of Amazon ION by the way?
I haven't looked at Ion before, but it appears to be similar to BSON or Msgpack in that it's a binary format that encodes field names as textual identifiers, to be "self-describing" and avoid the need for an external schema. I generally like schemas, because I like static typing. Of course, static types vs. dynamic types are another ancient flamewar and I'm unlikely to cover any new ground by stating arguments here. :)
Schemas are IMO the only way to go when you are transferring between heterogenous languages, even when all languages involved are untyped.
Consider javascript talking to common lisp. Of course JSON has a canonical mapping to javascript, but it does not for common lisp. Should a JS array be a lisp list or vector? Should lisp's NIL be false or null? Should a JS object decode to an alist, plist, or hash-table? &ct.
Interesting; I've always written &ct, which wikipedia informs me is considered "archaic" Thanks for that note, it will save me one character of typing 3 years from now when I've finally broken the muscle memory of the old way :)
Wikipedia probably mentioned it, but & is a combination of "e" and "t", or "et" - french (or perhaps latin?) for "and". Thus &c is short for "etc", or "et cetera". I wasn't aware of the form "&ct", I presume it stands for "ET CeTera", but I'm not entirely clear on why that extra "T" would ever make sense. I doubt anyone ever used "etct"?
I think there is a lot in common between Common Lisp, Python and JavaScript in general.
For many years I was in the schemaless camp before JS came along. Then for a number of years I was in the self-describing camp because I was thinking that if we don't accept JavaScript and JSON are pretty fundamental on the web we're fools and everyone seemed to be passing around JSON. So in that period was thinking that MsgPack was pretty damn good.
Recently I've switched back to the schemaless view, but with strict order preservation and richly typed fields. Very "tuple" based... so works well with Lisp and JavaScript but also C++. Highly inspired by Linda.
I don't do what Cap'n Proto does and lay out the fields and all that good stuff so that you can kind of memory map it onto structs.
That is nice and I understand the motivation for sure, and I have worked on systems that do that in the past with very good results, but currently my thinking is that compactness without additional compression is a good balance.
Also, since the protocol is order preserving (where it matters) you can do radix sort operations or hash maps on the server side extremely quickly. That was the ultimate motivating factor.
Inner tuples or BLOB's are length prefixed of course so you can skip them, but basic types and strings are not. Strings are zero terminated while Ints/Floats/Doubles are BE and complement encoded to preserve sorts, and also Integers are packed to minimum size.
Memory mapping is possible on this system too, but it's very "functional"... there's no attempt at pointer preservation. I don't go that far and Cap'n P seems to be preserving some of the semantics of ProtoBuf at least in that regard, which I'm sure is a good thing for many scenarios.
You could still do that with what I'm doing but it would be at the application level. Same with self-description actually, you could easily build something that looks like JSON if you wanted... either:
[["foo",1.0],["boo","cat"]...] etc
or
[[1.0,"cat"],["foo",boo"]
the choice is really up to the application programmer.
That might be a bit "loose" for a lot of people to stomach, but it works very well for what I'm doing, has a lot of flexibility and packs really well
Like I was suggesting, protocols are something of an art and I don't think we're at a final solution yet, which is why many people are constantly inventing new ones :-)
Also I might add that I very quickly became quite disenchanted with the lack of types in JSON; and I'm sure many of us have come to that same conclusion.
Really hope that it can be fixed at some point, but not holding my breath on that one.
I think it will take a major effort to reform that format although I am hopeful now that we at least have UInt8Array and friends that are starting to expose a broader set of machine friendly types.
I've actually seen people use types with JSON. JSON objects are clearly more than sufficient to represent the data you need, and you can enforce the schema in a library. It's just a much less efficient form of something like protobufs.
I think it depends on what data you need based on the problem space, but yes it does suffice for many use cases.
The thing is that it is least common denominator, and when you are dealing with high perf, cross language systems, it really isn't a good wire format or storage format.
It takes ages to parse, it's lossy, lacks commonly used types (or you have to annotate it with non standard attributes)... or worse guess the intention, and it's pretty verbose.
But again, that said it is a widely used standard and one that we have to live with. So there is that.
I like schemas for validation and strong types. I also like self-describing systems when all goes to hell and I'm debugging either my own protocols or someone else's.
I have this crazy idea that there's a middle ground: self-describing and schemas?
We can kind of glue this together (there's lots of json schemas floating around out there now, it seems), but it would be awfully interesting to see these well-supported as a pair (with explicitly language agnostic schema definitions -- which seems to be a sticking point for most of the json schema strapons).
Doesn't really have an implementation-agnostic schema spec, as far as I'm aware.
(I use CBOR a lot -- I'm otherwise quite happy with it!)
EDIT: I guess there's a "CDDL" listed on the tools page, but... It's still single implementation (ruby) and I don't see a clear link to a grammar for it.
Hahaha very true let's not go there. Anyway yes I'm a big fan of richly typed schema-less formats as well, even better when they can be read easily by both JS stuff and C++.
Except in this case you have flatbuffers which is a perfectly reasonable replacement and works across a wide array of platforms.
We rejected Cap'n Proto for the same reasons. I don't see a need to use bleeding edge C++ features when the same(or better) results can be achieved with code generation.
Kudos on the effort but if you're looking for wide adoption you've missed the mark.
Well... We're not, actually. We're looking to support the needs of Sandstorm.
It's nice if my code is useful to others, and if people want to contribute better Windows support I'll be totally happy to review and merge those changes! But wide adoption of Cap'n Proto (outside of the Sandstorm context) is not part of our business model, unfortunately.
(Note: The poster you were replying to isn't associated with Cap'n Proto nor Sandstorm.)
Slightly off-topic but it's good to see after all these years that companies like Microsoft (and Apple?) have to make a serious attempt to embrace the open-source community or miss out on a lot of cool new software and developers. Times have changed somewhat; the mantra used to be "doesn't work on Linux", while nowadays we've got a lot more heterogeneous environment.
I could imagine implementing something like this as an alternate wire format and client library for protocol buffers. Can you outline the main reason you chose not to go this route? In particular, what aspects of the proto descriptor language don't fit Cap'n Proto? Or what aspects did you feel needed to be changed for some other reason?
To be clear, by "alternate wire format and client library for protobuf" you mean "reuse the protobuf IDL but change everything else", right?
There are a few major reasons I didn't go that route:
* The .proto language has a lot of weird quirks that I don't like. Some of the quirks are specific to the protobuf encoding (e.g. int32 vs. sint32 vs. fixed32 being different types), while other quirks have no particular rationale behind them. I didn't want that baggage.
* The .proto language does not treat interfaces (aka services) as a first-class type. That is, you cannot define a field whose type is an RPC interface type -- a reference to a remote object. The ability to do this is a critical part of Cap'n Proto's interface design.
* It's highly unlikely that the protobuf team would be interested in accepting changes to the language which were not actually supported by protobuf. This means that if I shared the language, I would have my hands tied when it comes to new features -- or I'd also have to implement Protobuf equivalents to make them happy.
For cases that don't use features specific to one format or the other, it should be easy to write a tool that automatically converts between schema formats, FWIW.
I'd like to see Ruby bindings (not just serialization). We are using Go, Node.js and Ruby in production, and we have been looking to move from plain HTTP to gRPC. If Cap'n Proto is better, we might use it, but not without Ruby bindings.
I don't think anyone is currently working on Ruby bindings, but if you're interested, feel free to take it on!
(Note that Sandstorm is focused on making Cap'n Proto work well for Sandstorm. We welcome contributions, but we generally don't have resources available ourselves to work on third-party feature requests unrelated to Sandstorm. That is, unless you want to pay us a bunch of money, in which case, feel free to contact me. ;) )
I recently evaluated the two and went with FlatBuffers, largely because its Java support appeared to be more mature.
The FlatBuffers encoding is based on vtables and is relatively straightforward (the runtime library is tiny). This also means it's inefficient for small messages, but in my testing its vtable deduplication worked great for my use case (~100k messages of the same type per memory-mapped file), in that the vtable overhead tends quickly to zero.
Cap'n Proto has a more complex encoding that is probably more efficient in terms of wire size, and particularly for small/standalone messages, but the runtime is larger as a result.
Does flatbuffers have an RPC mechanism? Or is it purely a serialisation tool?
Cap'n Proto is both. I.e. you have the serialisation, which can be used on its own, but you can also define interfaces with methods that can pass those serialised structs as parameters, and which return asynchronous promises.
The "lists" are actually arrays, so access by index is constant-time. You can build a hashtable on top of that. I'd like to add built-in support for maps at some point, but it's one of a very large number of wishlist items...
The simplest implementation would be a sorted list with binary search for access. Segments create problems though.
I thought of the hashtable, but that means I have to keep the hash-table in RAM. By contrast, the entire capnproto structure itself could be memory mapped, thus not having any RAM constraints.
I'm thinking larger datasets here, large enough where overhead of e.g. JSON becomes a problem with RAM (on mobile devices), but not yet large enough to have to use SQLite.
I might be overthinking it, and could move to SQLite straight away. Just trying to keep my stuff as simple as possible. Conceptually simple, that is. Capnproto is conceptually simple: a tree dumped to disk, allowing on-demand memory mapped access to requested parts.
The capnp structure itself could be a hashtable, and doesn't need to be (entirely) loaded into RAM.
I'm saying you create a capnp list, and then you store elements into the list at position according to their hash -- i.e. how you'd build a hashtable, but the capnp list itself is the backing array.
You would have to do this at write time, of course, and make sure the hashing is consistent between runs.
From reading the examples, it looks like one can only serialise to a file descriptor. What if i wan to serialise to a byte array? Am I perhaps missing something obvious?
You can certainly serialize to a byte array. The example uses a file descriptor but in fact the code defines an abstract stream type which you can implement any way you want, and you can also obtain pointers to the message's underlying storage in order to extract the bytes directly (avoiding a copy).
Thrift to me is a Protobuf clone. Very similar design. Early on it was not very well optimized compared to protobuf, but I imagine they've fixed that by now. The big advantage they had was that they included an RPC system in their first release -- although it was a FIFO RPC system which struck me as an odd design choice (I think they may have fixed this more recently?). But now GRPC exists, so I don't think there's much reason to choose Thrift over Protobuf/GRPC.
In general, because I felt there was too much resistance to me pursuing ideas that I wanted to pursue. Google has become increasingly top-down, whereas it was fairly bottom-up when I started in 2005. That's not necessarily a bad thing, but I felt that I personally would be happier running a startup where I could call the shots.
FWIW, if you just want to write code, be comfortable, and make a crapton of money, and you don't care if you're implementing someone else's ideas, I highly recommend working for Google. That's not meant to be sarcastic or disparaging -- I totally respect that approach and there are days when I wish that were me. But if you have ideas of your own and you won't be happy unless you see them implemented... it probably won't happen at Google.
(To be fair, some people at Google would surely argue that my problem is that my ideas are crazy and bad, and I don't have any firm evidence -- yet -- that they are wrong.)
I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company.
It may not be that your ideas are crazy and bad, but rather that your sane good ideas simply aren't appropriate in the context of advertising.
Enough with the Stockholm syndrome. There are pleanty of perfectly great ideas that Google would never pay their employees to work on, however explicable or inexplicable the reason might be.
>A blow for mobile advertising: The next version of Safari will let users block ads on iPhones and iPads [1] : [...] An Apple realist might argue that its great rival Google makes more than 90 percent of its revenue from online advertising — a growing share of that on mobile, and a large share of that on iPhone. Indeed, Google alone makes about half of all global mobile advertising revenue. So anything that cuts back on mobile advertising revenue is primarily hurting its rival. (Google has been less friendly to adblockers than its “open” positioning would suggest.)
I didn't downvote you (in fact, I can't, since you were replying to me), but:
Your post was snide. Snide remarks are commonly downvoted on HN, regardless of their accuracy.
Regarding your follow-up, you're taking a very simplistic view of Google that doesn't really capture reality. Google obviously builds lots of technology that doesn't directly affect advertising revenue. Most decision-makers at Google are not asking "how does this affect advertising revenue" for every decision. You'd know that if you ever worked there.
> I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company.
Several reasons, but most critically:
The phrase "advertising company" generally refers to a firm in the business of creating advertising. Google's main business is an online media company selling advertising space (both in its own advertising-supported services and alongside online media provided by others.)
Is what the article says not true that "Google makes more than 90 percent of its revenue from online advertising" and "Google alone makes about half of all global mobile advertising revenue"?
That sounds like a pretty major advertising company to me.
Can you name a bigger advertising company than Google?
If they're actually a technology company driven by the demands of their users and striving for technical excellence, then do you expect Google to support built-in ad-blocking in Chrome and Android like Apple already supports it in Safari and iOS any time soon?
Personally, I find it very difficult to motivate myself to apply for Google. Because of your reasons, and also becoming the 50,000th engineer. Will take almost ~2 yrs to work on anything interesting (Let's face it, most engineers aren't the world famous engineer in the Google Brain team).
... some people at Google would surely argue that my problem is that my ideas are crazy and bad ...
To what extent are such disagreements purely technical (e.g. byte-oriented wire encodings vs. roughly-like-memory) and to what extent are they "political".
By "political" I am talking about your claims about decentralisation and diversity, as if you are selling Sandstorm as the IBM-PC for the cloud millennium: a neutral platform any old software developer. That is very different from Google's way of doing doing business.
I didn't actually propose massive decentralization while at Google, nor did I propose zero-copy encoding. So, no, I don't think I was proposing anything that threatened Google's business model.
For some reason, the banner (infinitely faster?), name, and introductory FAQ-style responses made me think the whole thing is a joke - similar to Vanilla JS [1].
Anyways, it seems like a cool project, so I'll be sure to follow its development closely.
Yeah, it was an April fool's joke satirizing web microframeworks. He does, but I don't know if that was more because Flask is a better framework or because of better marketing. I don't think Bottle is really any worse than Flask.
If Cap'n Proto is a joke, JSON beat it to the punchline.
What's an efficient binary representation? C Structs. What's an efficient text representation? Javascript objects. But casting arbitrary data to a struct is a horrible idea. And eval'ing anonymous javascript is a horrible idea. Back to the drawing board.
Then N years later someone has the brilliant idea of just... not parsing these formats that idiotic way. And wrote a code generator because the smart way is tedious.
In my view, JSON beat XML because it matched the way we structure data in code: records and lists. XML, meanwhile, wants us to represent data structures as text with tags, which isn't how we represent data internally at all, requiring laborious translation steps.
The fact that it was "built-in" to Javascript, of course, helped, by allowing it to compete on even footing (XML is also built in via XMLHttpRequest).
I've always liked Cap'n Proto because it was (quite literally) the ideas behind Protobuf taken to an extreme, or, depending on your point-of-view, reduced to its most basic components: data structures already have to sit in memory looking a certain way, why can't we just squirt that on the wire instead of some fancy bespoke type-length-value struct?
Of course, the hardest part is convincing everyone that it's not your bespoke type-length-value struct, but that you have good reasons for what you're doing. I think the humorous, not-so-self-serious presentation has worked in its favor (but that's just a subjective opinion and I can't back it up with data).
I thought that the main reason we didn't do this was because it's was hard - platform inconsistencies like 32/64 bit, endianness, not to mention differences between how languages store things, etc.
The thing that irks me about these methods is that if you're using a capnproto Int rather than a regular Int, doesn't that mean that you're basically forgoing a lot of functionality that was built around and works with the regular old data types?
For example, we also do that with numpy data types in python, but there the performance benefit is super clear - numerical operations dominate. I guess it really depends on your use case. If most of your time is spend on serde, then perhaps it's worth it.
In terms of data layout, 32-bit vs. 64-bit architecture only really affects pointer size. But Cap'n Proto does not encode native pointers (that obviously wouldn't work), so this turns out not to matter.
> endianness,
It turns out almost everything is little-endian now. Also, big-endian architectures almost always have efficient instructions for loading little-endian data. So Cap'n Proto just has to make sure to use those instructions in the getters/setters for integer fields.
> not to mention differences between how languages store things, etc.
Cap'n Proto actually doesn't attempt to match how any language stores things. Instead, it defines its own layout that is appropriate for modern CPUs. It ends up being very similar to the way many languages store things (especially C), but isn't intended to exactly match.
The C++ implementation of Cap'n Proto generates inline getter/setter methods that do pointer arithmetic that is equivalent to what the compiler would generate when accessing a struct.
For Java, Cap'n Proto data is stored in a ByteBuffer, which effectively allows something like pointer arithmetic. Again, getters/setters are generated which use the right offsets.
Most other languages end up looking like either C++ or Java.
> In terms of data layout, 32-bit vs. 64-bit architecture only really affects pointer size. But Cap'n Proto does not encode native pointers (that obviously wouldn't work), so this turns out not to matter.
Unless you're programming in C or C++ (or using a library from them), where size of int, long, etc. may change depending on architecture and compiler.
>data structures already have to sit in memory looking a certain way, why can't we just squirt that on the wire instead of some fancy bespoke type-length-value struct?
In C/C++ ya can!
When making games in college that is exactly what we did. Take the struct, dump it into the socket. I was rather shocked when trying to recreate the same system in C#. "I can't? I CAN'T?"
That's an absolutely terrible idea if you want any kind of forward / backward compatibility. Just about any system is better than dumping structs over the network, once you're dealing with a system that can evolve.
Yeah, I once had to write an Android / Desktop C++ implementation of such an API that was squirting iOS datastructures over socket. The API was built by a programmer who didn't know what endianess (or size of the structs) was.
You might be surprised just how many successful applications simply dump a packed struct to disk as a serialization format (or even transmit it over the wire). If the operating environment (OS/hardware) is known to be of a certain type, endianness is not a concern.
"Protocol Buffers" has been the go-to for a long time but there are more options now.
For uses where serialization/deserialization CPU time is a concern, it seems to really a question of Cap'n Proto versus flatbuffers ( https://google.github.io/flatbuffers/ ).
Proto3 is not a large change. In fact it shares most of its code with proto2, as I understand it. The underlying encoding is the same.
Proto3 removes some features from proto2 which were deemed overcomplicated relative to their value (unknown field retention, non-zero default values, extensions, required fields) and adds some features that people have wanted for a long time (maps). But all of these features are things that are "on top" of the core, not really fundamental changes.
I think the only change which affects my comparison post (linked by GP) is removal of unknown field retention. This is actually noted in the comparison grid. I'm honestly very surprised that they chose to remove this feature since it is critical to many parts of Google's infrastructure.
I don't think any lookup table is provided (the wire order of entries is undefined). They are not lookup maps, they are syntactic sugar for repeated key/value pairs.
The "lookup table" is constructed at parse time. Yes, the items are just key/values on the wire, but when parsed (in C++, at least) they are placed in a google::protobuf::Map, which is hashtable-based. I guess it could differ across languages -- some might not have any specific support for maps yet.
For our use (serializing streaming market data from direct NASDAQ and BATS feeds) we found both Capn'Proto and Flatbuffers to perform similarly.
Ultimately we went with flatbuffers because we found the API much cleaner across languages (C++/Go), but it's ultimately going to depend on your use case which one you use. Performance is pretty much identical. We populate our own custom structs from the capn'proto/flatbuffers structs anyways so we're never really doing zero-copy. That said, since both of these formats transmit numerical data as little-endian memory-representations of ints/float/longs/doubles their performance is fantastic.
When we had to choose a few months back, we picked Cap'n Proto over flatbuffers mainly over the fact that we liked the API better. I couldn't retrieve my notes from back then so I sadly can't point out. In any case, we never looked back.
Big fan of what Sandstorm is doing, both with Sandstorm itself and this component. I really want to use this instead of gRPC, as it seems technically superior, but language bindings and adoption across language ecosystems are likely to be a big downside given that (as Kenton mentions in a comment elsewhere here) Sandstorm isn't really interested in Cap'n Proto being widely adopted. All my new stuff is built in Rust, so the Sandstorm team's interest in and use of Rust are a good fit for me. But when it comes to interoperability with other languages, this may end up being a concern compared to gRPC. In any case, I hope to see the Rust implementation eventually replace the C++ one as the official reference implementation.
FWIW, language interoperability is of interest to Sandstorm since apps can be written in many languages. But we're only 7 people with a lot on our plate, so unfortunately we can't currently be the ones to go around implementing Cap'n Proto in every language. That will change as Sandstorm grows.
Great news, and thanks for the response! I don't have an immediate need for a system like this, so perhaps by the time I do there will be broader adoption.
Correct me if I'm wrong, but some of this sounds like blitting, except optimized for the in-memory structure and not the on-disk structure.
In 2008, Joel Spolsky wrote about 1990s-era Excel file formats and how they used this technique to deal with how slow computers were then [1]. Same technique, new problem set.
Fractalide (http://githib.com/fractalide/fractalide) is an implementation of dataflow programming (specifically flow based programming). Component build hierarchies are coordinated via the Nix package manager. Capnproto contracts are weaved into each component just before build time. These contracts are the only way compenents talk to each other. Thanks Sandstorm.io for this great software.
I've always liked the look of this, saw it a while back but we are still using protobuf in our .NET environment simply due to the "free" schema generation using AOP/attributes [ProtoContract]/[ProtoMember] in Marc Gravell's excellent protobuf-net (https://github.com/mgravell/protobuf-net) project - I assume this would also be possible for cap'n proto.
I hate to ask but can anyone explain this like I'm from 2000?
I guess it's a way to send data to your front end java script but not use json and this compresses it so it's faster? How much better than using json is it?
It lets you put data on the wire, in a structured format, right out of memory. Asking the question "how much faster is it" isn't even valid here, because it skips the usual serialization process.
Cap'n Proto generates you some code that contains some data structures. You put data into these structures, and they will automatically be in the right shape to put directly on the wire. And then that data can be pulled right off the wire and right into memory and be fully ready to access, with no intermediate step.
It's, in a sense, infinitely faster than JSON serialization or deserialization. Because it doesn't even perform any serialization. It's just data.
There are some other tricks at play here, but I won't go into them. This is plenty cool.
It's infinitely faster if you have control over the data layout in your application - which most likely means your are developing your application in C/C++, or maybe Rust. In the case of JS you don't have, so accessing and writing the data is slow (since you would need [de]deserialization there to write it in a byte array). In fact any serializations in JS are slower than JSON, since this is natively implemented in the JS VMs while others are not.
No no, this is a common misunderstanding about Cap'n Proto. It does not take your regular in-memory data structures from your regular programming language (even C++) and put them on the wire. What it does is defines its own specific data layout which happens to be appropriate both for in-memory random-access use and for transmission.
Cap'n Proto generates classes which wrap a byte buffer and give you accessor methods that read the fields straight out of the buffer.
That actually works equally well in C++ and Javascript.
Ok, but these accessor methods will still have a very different performance. In C++ copying a UTF8 string from one byte array into an std::string is super fast. Whereas in JS it's really slow, since you need to read from an ArrayBuffer, convert code points to UTF16 in JS and then store these in a string (which is not efficient, since the strings are immutable). In node you could at least speed that up through some native extension, but even then it would most likely be slower than JSON. In the browser it would be in any case. But that's really a speciality of JS.
In general I then think the difference (for non-C++) between your method and others (protobuf, thrift, ...) is that yours would require the cost of a field serialization in the moment the field is accessed. In others all fields are deserialized at once. But in the end it should have the same cost if I need all fields, e.g. in order to convert the data into a plain Java/Javascript/C#/... object, or am I missing something there? For C++ is absolutely believe that you can have a byte-array backed proxy-object with accessor methods that have the same properties as accessing native C++ structures.
Even if you access every field, Cap'n Proto's approach should still be faster in theory because:
- Making one pass instead of two is better for the cache. When dealing with messages larger than the CPU cache, memory bandwidth can easily be the program's main bottleneck, at which point using one pass instead of two can actually double your performance.
- Along similar lines, when you parse a protobuf upfront, you have to parse it into some intermediate structure. That intermediate structure takes memory, which adds cache pressure. Cap'n Proto has no intermediate structure.
- Protobuf and many formats like it are branch-heavy. For example, protobuf likes to encode integers as "varints" (variable-width integers), which require a branch on every byte to check if it's the last byte. Also, protobuf is a tag-value stream, which means the parser has to be a switch-in-a-loop, which is a notoriously CPU-unfriendly pattern. Cap'n Proto uses fixed widths and fixed offsets, which means there are very few branches. As a result, an upfront Cap'n Proto parser would be expected to outperform a Protobuf parser. The fact that parsing happens lazily at time of use is a bonus.
All that said, it's true that if you are reading every field of your structure, then Cap'n Proto serialization is more of an incremental improvement, not a paradigm shift.
Well, to be fair, serialization is still performed - but interchange format is cleverly picked out to be as similar as possible to common memory representation.
It's a lot higher-performance, because you virtually don't have to parse. Remember when MySQL started supporting the memcached protocol because it'd reached the point where form simple pkey lookups it was spending more time parsing an SQL query than actually executing it? It's like doing that for your program.
But for me at least, the real advantage over JSON isn't the performance but the schema compatibility. You have a spec for your data and generate code from that, which means the spec is guaranteed to be correct, and there's clear documentation about what changes to the spec are or aren't forward or backward compatibile. (You get the same thing from the original Protocol Buffers though).
> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk!
Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly.
> Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data.
sob There are a lot of things Intel has to account for, and frankly little-endian byte order isn't the worst of them, but it's pretty rotten. Writing 'EFCDAB8967452301' for 0x0123456789ABCDEF is perverse in the extreme. Why? Why?
As pragmatic design choices go, Cap'n Proto's is a good one (although it violates the standard network byte order). Intel effectively won the CPU war, and we'll never be free of the little-endian plague.
Big-endian vs. little-endian is an ancient flamewar that isn't going to go away any time soon, but sure, let's argue.
Once you've spent as much time twiddling bits as I have (as the author of proto2 and Cap'n Proto), you start to realize that little endian is much easier to work with than big-endian.
For example:
- To reinterpret a 64-bit number as 32-bit in BE, you have to add 4 bytes to your pointer. In LE, the pointer doesn't change.
- Just about any arithmetic operation on integers (e.g. adding) starts from the least-significant bits and moves up. It's nice if that can mean iterating forward from the start instead of backwards from the end, e.g. when implementing a "bignum" library.
- Which of the following is simpler?
// Extract nth bit from byte array, assuming LE order.
(bytes[n/8] >> (n%8)) & 1
// Extract nth bit from byte array, assuming BE order.
(bytes[n/8] >> (7 - n%8)) & 1
There's really no good argument for big-endian encoding except that it's the ordering that we humans use in writing.
There's really no good argument for big-endian encoding except that it's the ordering that we humans use in writing. And not even always that. We call our numbering system "Arabic", but for the Arabs, it little-endian.
For some reason humans seem to want high powers on the left, even if it makes no sense in a left-to-right language.
Take polynomials, they are typically written big-endian
ax^2 + bx + c
But infinite series have to be little-endian.
c_0 + c_1*x + c_2*x^2 ....
If you think for a moment about how you would write multiplication, you will see the latter form is much easier to reason about and program with.
I think figure 1 illustrates a common misunderstanding. People who object to little-endian are often imagining it in their heads as order (3): they imagine that the bits of each byte are ordered most-significant to least-significant (big-endian), but then for some reason the bytes are ordered the opposite, least-significant to most-significant (little-endian). That indeed would make no sense.
But because most architectures don't provide any way to address individual bits, only bytes, it's entirely up to the observer to decide in which order they want to imagine the bits. When using little-endian, you imagine that the bits are in little-endian order, to be consistent with the bytes, and then everything is nice and consistent.
> When using little-endian, you imagine that the bits are in little-endian order, to be consistent with the bytes, and then everything is nice and consistent.
But isn't that kind of at odds with how shifting works? (i.e. that a left shift moves towards the "bigger" bits and a right shift moves toward the "smaller" ones.) Perhaps for a Hebrew or Arabic speaker this all works out nicely, but for those of us accustomed to progressing from left to right it seems a bit backwards...
> To reinterpret a 64-bit number as 32-bit in BE, you have to add 4 bytes to your pointer. In LE, the pointer doesn't change.
But one shouldn't do that very often: those are two different types. The slight cost of adding a pointer is negligible.
> Just about any arithmetic operation on integers (e.g. adding) starts from the least-significant bits and moves up. It's nice if that can mean iterating forward from the start instead of backwards from the end, e.g. when implementing a "bignum" library.
-- is a thing, just as ++ is.
> There's really no good argument for big-endian encoding except that it's the ordering that we humans use in writing.
That's like saying, 'there's really no good argument for pumping nitrogen-oxygen mixes into space stations except that it's the mixture we humans use to breathe.'
It's simplicity itself for a computer to do big-endian arithmetic; it's horrible pain for a human being who has to read a little-endian listing. A computer can be made to do the right thing. Who is the master: the computer or the man?
That line of argument suggests you'd be happier with a human-readable format like JSON. Which is another eternal flamewar that we aren't likely to resolve here. Needless to say I like binary formats. :)
Floating point isn't legible either, nor are are bitfields, opcodes, instruction formats, etc. That's why we use computers to do the 'right thing' and format the data if we need to read it.
I don't have a preference for either one, but using little-endian when most/every processor you will be targeting supports it makes more sense than using big-endian + extra work on x86 just so you can read it with less effort in a memory dump.
I've just began learning about endianness and I'm sorry if this comes off as pedantic, but doesn't the last example refer to bit numbering rather than (byte) endianness?
Endianness applies to both! Or, it should, if you're being consistent. It makes no sense to say that the first byte is the most-significant byte, but that bit zero is the least significant bit of the byte.
Because all modern computer architectures assign addresses to bytes, not bits, it's up to us to decide which way to number the bits. But we should always number the bits the same way we number the bytes.
> Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly.
The thing is, as close to free as possible is surprisingly expensive. Protobuf's varint encoding is extremely branchy, and hurts performance in a datacenter environment (where bandwidth is free, and CPU is expensive).
> As pragmatic design choices go, Cap'n Proto's is a good one (although it violates the standard network byte order). Intel effectively won the CPU war, and we'll never be free of the little-endian plague.
Did they though? Arguably there are far more ARM CPUs (like the one in your pocket) than there are server CPUs. Since cellphones and other low power devices are almost all big endian, it seems like network byte order would have been better to use. High powered servers can pay the cost of coding them, but battery powered devices cannot afford to do so.
ARM (since v3) is bi-endian for data accesses and defaults to little. You can confirm this by searching the ARM Information Center [1] for 'support for mixed-endian data', but I can't get a working URL for that exact page.
> > Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data.
> sob There are a lot of things Intel has to account for, and frankly little-endian byte order isn't the worst of them, but it's pretty rotten. Writing 'EFCDAB8967452301' for 0x0123456789ABCDEF is perverse in the extreme. Why? Why?
Little endian means that a CPU designer can make buses shorter, which makes the CPU more efficient and smaller. There are also several benefits from the programming side. So it is actually better than big endian in /some/ cases, at the cost of being less intuitive to humans.
So while Intel did choose little endian, they had very good reason to (and it's probably why everything except SystemZ and POWER use it).
Hah. According to Wikipedia: "The Datapoint 2200 used simple bit-serial logic with little-endian to facilitate carry propagation. When Intel developed the 8008 microprocessor for Datapoint, they used little-endian for compatibility. However, as Intel was unable to deliver the 8008 in time, Datapoint used a medium scale integration equivalent, but the little-endianness was retained in most Intel designs." Which I guess is a reason to use little-endian, though not one that's really relevant to anyone that's ever actually used an Intel CPU. (ARM, incidentally, was little-endian by default because the 6502 was.)
How do you pronounce the name? If libreoffice is bad.. This name is absolutely impossible.
Is it captain? Is it cap+n+proto?
A lot of collaboration is verbal - people sit around and talk about stuff. I don't know if it is a fun take on an American word... But it is impossible to use in the rest of the world.
I really wish you would call it something else... Unless it is personal for you :(
Seems like a good opportunity to point out that the pun is that the RPC model is based on capabilities. Capabilities and Protobuf -> Cap. 'n' Proto. -> Cap'n Proto
Gee, all this time I thought he was hitching his wagon to the right honorable, dapper, charismatic, articulately spoken, upstanding guru of phone phreaking culture.
I think that compression is a must for the serialization library. Protobuf uses almost twice less memory than Cap'n Proto. Using an external compression is not an option in some cases. E.g. consider building tcp-server that communicates with thousands of clients simultaneously. Each client connection will have its own LZ4 context that should be heap allocated. I believe it's about 16KB per connection + buffers. This results in large memory consumption and a lot of random memory access and TLB misses.
Cap'n Proto offers "packed" encoding which applies light compression (removing zero-valued padding bytes), brings it in-line with Protobuf, and ought to be much faster than the things Protobuf does for "compression" (varint is a very slow encoding!).
Interfaces! Inheritance! Looks promising. Protocol buffers are nice for their compact encoding and multi-language generator support but as a schema language they are really cumbersome. Composition is pretty much all you get, there are no longer required fields, you can't even use enums as a key type in a map. I'm sure their use cases are not necessarily the same as mine but sometimes I miss just using plain old XML.
To be clear, Cap'n Proto's serialization layer, from a schema perspective, is almost exactly the same as Protobuf (though with a very different underlying encoding).
The interfaces and inheritance relate to the RPC system. The interfaces are for remote objects.
This sounds like a cool idea, but so far I haven't seen any good explanation of how it works, and why it will save me from rolling my own ACL system. For bragging about it in the very first sentence, there is surprisingly little detail about how it works.
It's a complicated topic -- it requires thinking about things in a different way, and tends not to make a lot of sense until at some point it "clicks" and you realize all sorts of patterns you were already using are actually special cases of capabilities.
Thrift is very much equivalent to Protobuf for the purpose of everything discussed on the web site. As the site mentions, I prefer to pick on Protobuf because I am also the author of Protobuf v2. :)
You can compile Cap'n Proto with -fno-exceptions, and it does a bunch of things differently to make that work. Basically, invalid-input exceptions instead replace the invalid data with a reasonable default value and set a flag on the side that you can query to see if there was any invalid input. Assertion failure exceptions (where there is no way to recover) largely turn into fatal errors.
Depends on the use case! In fact, the answer to "is X format faster than Y format" always depends on the use case. It's always easy to construct cases where one or the other looks better. People of course want to know "on average", but in reality there's no such thing as an "average" use case. You'll ultimately have to test the case you have in mind to find out.
With that said, here are some considerations:
- msgpack is usually used as a binary encoding of JSON, with no schemas. That means that textual field names are included in the encoded message. Formats like Protobuf and Cap'n Proto that have schemas known in advance can avoid this bloat, making them faster and smaller.
- msgpack is not a zero-copy encoding. It's necessary to parse the whole message upfront before you can use it, like with protobuf. Cap'n Proto is zero-copy, the advantages of which are described extensively on the page. For example, if you have a multi-gigabyte file containing a massive Cap'n Proto message, and you just want to read one field from one place in that message, you can do that by memory-mapping the file. No need to read it all in. That's not possible with Protobuf or Msgpack.
I think it's best to focus on these kind of paradigm-shifts when trying to reason about performance. You can always micro-optimize the encoding path later on, but you can't suddenly switch to zero-copy later if your data format wasn't designed for it.
Is X faster than Y will be tough to determine without a really detailed treatment of the use case. For example in C++ you want to account for the unavoidable construction of your class type, its inevitable destruction, how long it takes to put it on or take it off the wire, how often you might expect to miss the cache when referencing it, whether the type of movable or copyable and how expensive that is, whether or not you can reset it for reuse without destroying it, and much more. If you were to compare to protobufs I doubt that you'd find serialization and deserialization to be the predominant costs. I don't know what the cost breakdown looks like for capnproto, but maybe kentonv has standing benchmarks.
Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project (https://sandstorm.io). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n Proto socket (but compatibility layers on top of this allow apps to expose an HTTP server).
Unfortunately I've fallen behind on doing official releases, in part because an official release means I need to test against Windows, Mac, and other "supported platforms", whereas Sandstorm only cares about Linux. Windows is especially problematic since MSVC's C++11 support is spotty (or was last I tried), so there's usually a lot of work to do to get it working.
As a result Sandstorm has been building against Cap'n Proto's master branch so that we can make changes as needed for Sandstorm.
I'm hoping to get some time in the next few months to go back and do a new release.