
Ask HN: What do you use for microservice RPC? - atombender
First, some background:<p>In our distributed microservice-based stack, we currently use JSON-over-RESTful-HTTP exclusively for the internal APIs exposed by our microservices. Apps, including browser front ends, talk to each other using HTTP.<p>In particular, we&#x27;ve come to realize is that it&#x27;s really a poor man&#x27;s RPC, and that REST has no practical utility in internal APIs, which benefit from a service&#x2F;verb perspective, but not from a resource-oriented one.<p>Some current issues we want to solve:<p>- Performance. HTTP&#x2F;1.1 is not suited for tiny RPC payloads where you want seamless pipelining. HTTP&#x2F;2 is starting to look good as a much more performant transport, however.<p>- Wire formats. JSON is increasingly an expensive format to shuffle data around in.<p>- &quot;Service discovery&quot;. A client wants simply to communicate with a bunch of services (preferably using the same connection) — it shouldn&#x27;t concern itself with where those services run or how.<p>- Fault tolerance. Clients should be resilient against failing services and networking blips.<p>- Latency. Right now every HTTP request goes through HAProxy. Adds complexity, adds latency. We&#x27;d like microservices to talk to directly.<p>We could solve these issues for ourselves in a proprietary way. But we&#x27;re not in the business of reinventing wheels, nor are we in the business of creating libraries. Developing the required glue from scratch in all the languages we use (Go, Node.js and Ruby currently) would consume resources we&#x27;d like to devote to app code.<p>What do devs on HN use?
======
rdli
Here's what I've seen:

1\. HTTP/JSON, with resiliency semantics layered on via SmartStack, Hystrix,
or DIY library.

2\. GRPC/protobuf. Haven't seen anyone other than Google run this in
production, and Google doesn't really use gRPC (they use Stubby).

3\. Finagle/Thrift. Twitter and some other folks use this.

4\. Async message bus. Not as popular because it's not point-to-point, but it
works. It's technically not RPC so it requires a different architectural
approach that lots of people aren't willing to adopt.

5\. Super fancy custom RPC protocol. Uber does this, for example. As does
Google. You need lots of engineering to do this well, though.

<plug> We've seen these design patterns over and over again, which is why
we're building Quark (github.com/datawire/quark) which is a language for you
to build your own custom RPC protocol (case #5 or to a lesser extent #1).
We're also building Datawire Connect on top of Quark which gives you resilient
RPC semantics in a way that's compatible with your existing tech stack.
</plug>

~~~
lobster_johnson
Quark looks quite interesting, if odd — it's an entire language, with classes
and generics and everything? Just for implementing protocol glue?

I think it's something I'd like to revisit once it's matured, as your set of
languages is quite small at the moment, and I think it's overkill for us at
the moment. In particular, I'm much more concerned about connection and
discovery semantics than about things like wire protocol and how to express
the "schema".

~~~
rdli
If you're more worried about connection and discovery semantics, you'd want to
use Datawire Connect which is built on top of Quark. Quark is being used by
companies that for one reason or another need/want to build their own
protocol. This is admittedly overkill for most people.

Datawire Connect integrates discovery as well as connection semantics (next
week we'll have a release that supports full Hystrix semantics, e.g., circuit
breakers and timeouts and load balancing).

~~~
lobster_johnson
Looks interesting. As with Quark, I think we'd hve to wait until you have
reached a higher level of maturity and a broader selection of languages. We
need at least Go, Ruby and Node.js to get going, which is what our current
apps our built with. We're also evaluating Elixir and Swift as possible future
languages.

~~~
rdli
Great feedback! We will have Ruby in a few weeks (code is sitting on a GitHub
branch), and we have Node already. Go is on the roadmap, but we also have a
proxy that lets you interface with languages we don't support natively.

I'd be super interested in what tech you find that works well with your
language set of Go, Ruby, and NodeJS. GRPC won't. Thrift might (don't know
offhand). NATS might, but there's lots of coding there you need as you mention
and it's a SPOF.

~~~
lobster_johnson
gRPC does have native bindings for all of Go, Ruby and Node.js. Not sure why
you say it won't work well?

NATS being a SPOF isn't necessarily a problem for us if it's stable, which it
seems to be. We are already completely dependent on HAProxy as a SPOF, anyway.
Moreover, most client bindings seem to support providing a host list they use
for failover.

I will take a closer look at Quark and Datawire Connect when I have a spare
moment. Do you have it running in production? Happy to hear that Go is on the
roadmap.

~~~
rdli
gRPC bindings for languages that aren't popular at Google (e.g., Ruby, Node)
aren't particularly mature. Each of the different language bindings is
independently maintained, so there is no guarantee of feature parity across
the languages.

We do not have Quark/DC running in production yet. Most of the adoption has
been in the realm of testing scenarios (it's super handy to be able to
generate client stubs).

------
lobster_johnson
Here's some tech we are thinking about:

* gRPC. Good attempt at solving cross-language RPC. Downsides: Alpha quality; Protobuf requires build steps on the client side (codegen, schema versioning); Doesn't yet solve discovery or fault tolerance; 3.0 beta is required if you want to ship free-form blobs of data around.

* Protobuf: Similar to gRPC.

* NATS. Fast in-memory messaging broker which seems suitable for RPC. In particular, this solves the discoverability issue, since services are simply topics. Downsides: Something of a single point of failure. Also, we still need to add an HTTP compatibility layer for browser clients, and we'd still need to write bindings for each language we use so that services can declare endpoints, and clients can talk to them, without using the NATS library directly. Fault tolerance also needs to built into this glue. Not saying it wouldn't be a nice open source project, though.

* Thrift. Similar to gRPC/Protobuf. Requires schemas. We've talked to people who use this in production, and they're not too happy. Seems rather biased towards the Java universe.

~~~
rdli
protobuf and Thrift are serialization formats and an alternative to JSON, not
RPC.

~~~
lobster_johnson
I know, I should have said "Protobuf + custom connection handling".

Thrift does come with RPC client/server code. You can choose between different
transports. Example [1].

[1]
[https://github.com/apache/thrift/blob/master/lib/nodejs/exam...](https://github.com/apache/thrift/blob/master/lib/nodejs/examples/client.js)

