Protobuf is the worst serialization format and schema IDL, except for all the others.
The truth is, a lot of these critiques are valid. You could redesign protobuf from the ground up and improve it greatly. However, I think this article is bad. Firstly, it repeatedly implies that protobuf was designed by amateurs because of its design pitfalls, ignoring the fact that it's also clear that protobuf grew organically into what it is today rather than deliberately. Secondly, I do not feel it is giving enough credit to the idea that protobuf simply was not designed to be elegant in the first place. It's not a piece of modern art. It is not a Ferrari. It is a Ford F150.
What protobuf does have right is the stuff that's boring; it has a ton of language bindings that are mostly pretty efficient and complete. I'm not going to say it's all perfect, because it's not. But protobuf has an ecosystem that, say, Cap'n Proto does not. This matters more than the fact that protobuf is inelegant. Protobuf may be inelegant, but it does in fact get the job done for many of the use cases.
In my opinion, protobuf is actually good enough. It's not that ugly. Most programming languages you would use it from are more inelegant and incongruent than it is. I don't feel like modelling data or APIs in it is overly limited by this. Etc, etc. That said, if you want to make the most of protobuf, you'll want to make use of more than just protoc, and that's the benefit of having a rich ecosystem: there are plenty of good tools and libraries built around it. I'll admit that outside of Google, it's a bit harder to get going, but my experiences on side projects have made me feel that protobuf outside of Google is still a great idea. It is, in my opinion, significantly less of a pain in the ass than using OpenAPI based solutions or GraphQL, and much more flexible than language-specific tools like tRPC.
For a selection of what the ecosystem has to offer, I think this list is a good start.
I think this kind of gets to the point of what protobuf is, and what a lot of tech is like at Google in general. I spent a lot of my career believing elegance and expressiveness was so important to strive for, even if we as engineers often fall short.
But tech at Google tends to be pragmatic in the specific way that protobuf is. It's not perfect, it doesn't fit neatly into a grand ideology, but it is (at least within Google itself) simple enough, easy enough to understand, portable and fit for purpose. In a similar way to bazel, it's full of components worthy of criticism, but those fixes get made when they become the ecosystem's most pressing issues, and not before.
> Protobuf is the worst serialization format and schema IDL, except for all the others.
I don't even feel this way. I can complain for days about most technical things, including things Google has made (e.g. Golang), but not protobufs. They're just good. They don't totally replace JSON of course, but for the intended use case of internal distributed systems, they're solid.
JSON is good for dumps of your data, for humans reading it, and for interoperability without the schema - On your backends, you should always serialized to proto because it's darn more efficient.
If it's a startup or a personal project with no special performance requirements, chance of me using protobufs for anything is very slim. Human readability aside, just bothering to set up the serdes and other boilerplate is too low on my list of priorities. Makes more sense at medium scale.
Slapping it all into gRPC is less boilerplate than whatever else you'd be doing (unless you're doing GraphQL or something). I'd always default to doing that (or thrift, honestly, but same difference) unless there were a particular reason to use something like a manual API.
gRPC requires HTTP/2, which is a nontrivial added requirement. Prevents it from running on Heroku and ironically Google AppEngine, and wherever it does work, load balancers will probably need special setup for it. Even Kubernetes. You can translate it to/from JSON, but that defeats the purpose.
> gRPC requires HTTP/2, which is a nontrivial added requirement. Prevents it from running on Heroku and ironically Google AppEngine
Then use thrift, which does plain http. But most of the design decisions in the article are the same there as well. Turns out there's really only one way to do this.
Oh, gRPC over JSON still gives you the advantage of a well-defined API. But I use OpenAPI aka Swagger for that because it's a much thinner layer, especially if for whatever reason I need to handle HTTP GET (callbacks/webhooks) in addition to my typical HTTP POST JSON.
It's also possible to use JSON directly with gRPC/DRPC/Connect, so you can take advantage of a Protobuf IDL and codegen with a bit less of the logistical complexity.
Note that a solution does not (and should not) solve every problem. That does not make it bad. Very few (read: NO) solutions can solve every problem.
I've known this forever, but when my company started doing outreach to customers, the point was really driven home.
Find a problem. Solve it. That is it. If the solution is popular enough it will pick up momentum. There will always be naysayers. It might not be the best solution, or even the right one, but it IS a solution, which is better than what others are doing.
Think something is bad? Fix it. Submit a PR if it is open source. Don't complain for the sake of complaining unless you are willing to show how it should be done.
Note that I'm not one of those open source jerk holes, however I do recognize that unless I can show I can do better, my opinions are meaningless.
There are some legit criticisms here (remember, protobuf is a more than 20 years old format with extremely high expectation of backward compatibility so there are lots of unfixable issues), but in general this article reveals a fundamental misunderstanding of protobuf's goal. It is not a tool for elegant description of schema/data, but evolving distributed systems composed of thousands of services and storage. We're not talking about just code, but about petabytes of data potentially communicated through incomprehensible level of complex topology.
Yeah, I stopped reading at "by far, the biggest problem with protobuffers is their terrible type-system" for similar reasons. What did the author want, ASN.1? Cause that gives you a full type system, except that's not what I want.
Edit: kentonv's rebuttal says this too. "This article appears to be written by a programming language design theorist who, unfortunately, does not understand (or, perhaps, does not value) practical software engineering." And when my angry hat is on, this is also how I would generalize programming language design theorists.
Funny you should mention ASN.1. Protobuffers is Really just a toy ASN.1 written by some guys who just solved the problem at hand instead of looking further afar.
It's great for what it is. Many of us have written something similar.
It's not a pro level system designed by a serious telecom team.
I worked on LTE stuff during a research/engineering position. It was just 1 year and not my life's work, but I had to deeply understand ASN.1 for it, and I'd never heard of Google's Protobuf. I didn't like ASN.1 even for its intended use case. That big type system was far more complex than necessary for the protocols I was dealing with (surrounding the EPC mainly).
ASN.1 is more optimized in other ways that make sense, sacrificing some flexibility in favor of performance because the ends are expected to be more coordinated, but that's separate.
The type system woes are quite legit. I wonder how many of these were improved in FlatBuffers. Which has a ton of other good things going for it, chiefly ideally less serialization demands.
Once you start doing rpc, the demand goes so up. Cap'n'Proto's ability to return & process future values is such an exciting twist, that melds with where we are with local procedure calls: they return promise objects that we can pass all around while work goes on.
Kenton popped up a couple months ago to mention maybe finding time for mutli-party support, where I can for ex request say the address of a building & send the future result to another 3rd party system. Now this is less just a way to serialize some stuff, & more a way to imagine connecting interesting systems.
I've used Cap'n'Proto for a work project, and aside from not-always-amazing language compatibility it really is awesome. It's what I wish Protobufs was. The performance improvements don't matter much to me, the biggest thing for me is the type system fixes most of the mistakes that are mentioned in this article.
It’s an untruth. There is a validation step of course, which means that there is a decoding step. There is no encoding step because the wire format is the same as the in-memory representation.
No, there isn't. Not any more than there is a decode step anyway.
The truth is that encoding, decoding, and validation all occur when you call the accessor methods for specific fields. If you never actually call the getter for some field, it won't be decoded nor validated at all.
If you are going to be reading the entire message tree, then zero-copy is only an incremental improvement on something like Protobuf -- the use of fixed-width values may make decoding faster (or slower, in some cases). The real magic is when you want to read just one field of a 10GB file. Just mmap() the whole thing and treat it like a byte array, and it'll be efficient. Cap'n Proto will not attempt to scan the whole file, only the parts that you explicitly query.
Missing the point. The whole point of protobufs is to be a stable wire format that's forward compatible. They can't have sum types because you can't add one of those in a forward compatible way; oneof works the way it does because that's the only way it can. (I do think repeated is overengineered and should just be a collection field, which would solve the problem of not composing with oneof, but that's honestly just a minor wart).
Also if your language's bindings don't preserve the relationship between x and has_x and don't play nice with recursion schemes, use a better binding generator! You don't have to use the default one.
Protocol buffers were written by Jeff Dean and Sanjay Ghemawat. Whether you see issues with the implementation or not and whether they are fit for your use case or not are valid discussions, but if your argument starts with "omg these idiots couldn't design a simple product" then I'm already reading the rest of it with a huge grain of salt.
I think the emotional response is probably due to the fact that Protocol buffers is often forced upon teams, due to the desire for a common standard and the marketing power that Google have. But the author raises many valid issues and you would be wrong to dismiss them completely. The first version of Protocol buffers, IIRC, didn't even support static offsets and extracting a field without deserialising the entire message. So it was arguably both inelegant and inefficient. I was never impressed myself personally and have seen quite a few better closed-source wire formats inside various corporates.
Yes you can have many valid issues with Protocol buffers. "Written by amateurs" is not one of them, and that is the specific one I was dismissing.
Reading through the rest of the post though, it is pretty clear that it is a case of trying to use what is a standard for data transfer over the network to describe your entire application's object model. It isn't the right tool for the job, and doesn't have to be.
If it is being forced on you then, well, that's a complaint for your management, not the technology itself.
It's also a really bold claim that he doesn't back up, so I don't trust the rest. It's not like he dug around and found out that the main protobuf author was actually some hack who forced something bad on the rest of the company then bailed. That would be Google Plus.
This was hard for read because of the glaring omissions of existing engineering practices. It feels author only cares about programming purity and writing programs which transform metadata.
Making all fields in a message required makes messages into product types but loses compatibility with older or newer versions of protocol. Auto-creating objects on read dramatically increases chances that a field removal in future version of protocol will be handled properly. Auto-creating on write simplifies writing complex setters, etc...
You could argue that those problems could be solved better (and I might agree, protobufs are one of my least favorite serialization protocols) but not even acknowledging reasons and pretending the designers didn't know better makes writer seem either ignorant or arguing in bad faith.
protobufs are a data exchange format. The schema needs to map clearly to the wire format because it is a notation for the wire format, not your object model du jour.
If the protobuf schema code generator had to translate these suggestions into efficient wire representations and efficient language representations, it would be more complex than half the compilers of the languages it targets.
I do mourn the days when a project could bang out a great purpose-built binary serialization format and actually use it. But half the people I hire today, and everyone in the team down the hallway that needs to use our API, can no longer do that. I'm lucky if they know how two's complement works
> protobufs are a data exchange format. The schema needs to map clearly to the wire format because it is a notation for the wire format, not your object model du jour.
Yes, exactly this. I don't understand the vitriol here. Protobufs work fine for a wide variety of purposes and where the criticisms matter people use a different tool.
> I do mourn the days when a project could bang out a great purpose-built binary serialization format and actually use it. But half the people I hire today, and everyone in the team down the hallway that needs to use our API, can no longer do that. I'm lucky if they know how two's complement works
Would be great to work at a place where everyone knew how the machine worked, but the vast majority of developers entering the workplace since about 2000 have learned Java and web sh*t exclusively.
The problem is that protobufs aren’t just an interchange format, they’re also a system for generating the code used to interact with said interchange. Said code has a habit of leaking into the types used by your code base. It’s too easy to just pass protoc-generated objects around and use them all over your code base, hence the majority of the criticisms.
Protobuf seems to encourage this… instead of a hard boundary where your serialization logic ends and your business logic begins, every project I’ve worked on that uses protobuf tends to blur the lines all over the place, even going as far as to make protoc-generated types the core data model used by the whole code base.
You can take the wire format and ignore the code generator, if you like. I often do, especially for languages other than C++. For C++ I quite like the generated code.
I mean can you blame them? There's just so much java and web shit to learn. When are they going to pick up bit-banging when every week there's a new build toolchain or minifier or CSS standard or whatever other pile of abstraction upon abstraction of the day.
We use C and all our nodes are x86... so we just dump packed structs on the wire and the schema is a header file. I suspect this sort of simple approach works in most cases...
And really that's how the network stack on Linux and Co. is implemented (modulo taking care of endianness)
Till one day someone wakes up and wants a UI in C#. Which is also not a big deal, but you have to either hand roll the serialization code or build a header file parser and generate it. And then someone needs to add a field - so we just tack it onto the end of the struct. This works fine as long as you have the length represented out of band somehow. If not then you get to make struct FooBar2 with your extra field. Then a year later someone has the great idea that it would be great to send text, and now you're making a variable length packet or just always sending around structs with a bunch of empty space and a length field (which is also not too bad). But wait, now the UI team totally needs that data in JavaScript for their new Web UI, so you're back to either generating or hand jamming dozens of structs. All the while tamping out bugs in various places where someone is running old code against new data or old data against new code - all of which have various expectations. Or that microcontroller that is blowing up because it just doesn't like that unaligned integer that is packed into the struct.
Not that I'm bitter about that life - it just involved a lot more troubleshooting protocols over the years than I'd have liked. Anyway, those are problems that Protocol Buffers helps solve. But as long as you're using just C and not changing much - packed structs are quite lovely.
Sure, a small header that contains type and length is trivial and pretty much implied in my previous comment.
I don't think people should think too much about "what if we change language", etc. because (1) that's unlikely to happen, (2) you have years ahead of you, (3) it may be simpler to convert structs (not the most complicated thing in the world amd supported in most languages in one form or another) into whatever else when/if you actually need it than to overengineer now 'just in case'.
The main downside I see is performing schema changes. A very common pattern I use with protobufs in distributed systems is:
1.) add a field to the schema
2.) update and deploy the grpc servers to start populating that new field. The old grpc clients will continue to work ignoring the new field
3.) update and deploy the grpc clients to start using the new field
With packed structs you don’t get the reverse compatibility, so schemas and endpoints have to be immutable which is not the end of the world but it can make changes more difficult.
I notice this too. What happened to CS degrees? All I meet are React-infused-piles-of-tools people that don't seem to be able to code anything without a mile long pile of tools.
There are ~30K CS graduates from US universities per year. A big chunk of them are international students who have to (or want to) go back home. The remaining ones are being fought over to fill an estimated 500K-1.5M open software engineering jobs. So chances are your company is likely not hiring the cream of the crop.
The recent round of layoffs all put together made up a tiny percent of just the net new headcount added to the industry in just the last ~2 years. And all these companies have already started ramping hiring back up. There is no broad change in tech hiring.
No your just using old numbers. headcount has also shrunk.
You can't use numbers and raw logic because it has various levels of inaccuracies and things you haven't accounted for especially when the numbers and data come from times prior to the layoffs.
Reports from recruiters and people who are unemployed are telling me things that are significantly different. There is a fundamental change in the job market.
I've worked with many serialization formats (XDR, SOAP XML w/ XML schema, CORBA IDL and IIOP, JSON with and without schema, pickle, and many more). Protocol buffers remind me of XDR (https://en.wikipedia.org/wiki/External_Data_Representation). Which was a great technology in the day; NFS and NIS were implemented using it.
I do agree with some of the schema modelling criticisms in this article, but the ultimate thing to understand is: protocol buffers were invented to allow google to upgrade servers of different services (ads and search) asychronously and still be able to pass messages between them that can be (at least partly) decoded. They were then adopted for wide-range data modelling and gained a number of needed features, but also evolved fairly poorly.
IIRC you still can't make a 4GB protocol buffer because the Java implementation required signed integers (experts correct me if I'm wrong) and wouldn't change. This is a problem when working with large serialized DL models.
I was talking to Rob Pike over coffee one morning and he said everythign could be built with just nested sequences key/value pairs and no schema (IE, placing all the burden for decoding a message semantically on the client) and I don't think he's completely wrong.
I find rolling my own serialization protocol easy enough, and I want some friction from using it: i've been burned when it is too easy and people start to use IPC where a function call would just as well and be more efficient. Premature pessimization is a large problem these days.
Personally I avoid Map fields in protos. I don't find a huge amount of value for a Map<int, Foo> over a repeated Foo with an int key. That tends to be more flexible over time (since you can use different key fields) and it sidesteps all of the issues about composability of them.
I think required fields are fine, provided that you understand that "required" means "required forever". If you're already using protos this isn't exactly a brand new concept. When you use a field number that field number is assigned forever, you can't reuse it once that field is deprecated. It requires a bit more thoughtful design, and obviously not all fields should be required, but it has value in some places.
"They’re clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has."
Quite a statement from someone that spent less than a year at Google.
Personally I really appreciate protobuf; it has saved me and teams I've worked with a ton of time and effort.
I worked a lot with protocol buffers and for the problem it claims to solve it solves that problem well. I can't really grasp this criticism. That extreme type hell that Java coders like is not really a thing outside the Java world.
And databases have had it since forever, you throw in some SQL and you get back some data over defined protocols like TDS. You can even invoke some small programs via stored procs.
The truth is, a lot of these critiques are valid. You could redesign protobuf from the ground up and improve it greatly. However, I think this article is bad. Firstly, it repeatedly implies that protobuf was designed by amateurs because of its design pitfalls, ignoring the fact that it's also clear that protobuf grew organically into what it is today rather than deliberately. Secondly, I do not feel it is giving enough credit to the idea that protobuf simply was not designed to be elegant in the first place. It's not a piece of modern art. It is not a Ferrari. It is a Ford F150.
What protobuf does have right is the stuff that's boring; it has a ton of language bindings that are mostly pretty efficient and complete. I'm not going to say it's all perfect, because it's not. But protobuf has an ecosystem that, say, Cap'n Proto does not. This matters more than the fact that protobuf is inelegant. Protobuf may be inelegant, but it does in fact get the job done for many of the use cases.
In my opinion, protobuf is actually good enough. It's not that ugly. Most programming languages you would use it from are more inelegant and incongruent than it is. I don't feel like modelling data or APIs in it is overly limited by this. Etc, etc. That said, if you want to make the most of protobuf, you'll want to make use of more than just protoc, and that's the benefit of having a rich ecosystem: there are plenty of good tools and libraries built around it. I'll admit that outside of Google, it's a bit harder to get going, but my experiences on side projects have made me feel that protobuf outside of Google is still a great idea. It is, in my opinion, significantly less of a pain in the ass than using OpenAPI based solutions or GraphQL, and much more flexible than language-specific tools like tRPC.
For a selection of what the ecosystem has to offer, I think this list is a good start.
https://github.com/grpc-ecosystem/awesome-grpc#protocol-buff...