
Show HN: Shisetsu – An RPC-Like Protocol on Top of Redis - kixpanganiban
https://github.com/KixPanganiban/shisetsu
======
dozzie
So, basically, you allow network programmer to not understand network
programming, which is considered a major (main?) benefit, but at the expense
of operators needing _one more service_ to maintain?

How many programmers need to write a client and how many users would need to
run an additional service? Redis clearly doesn't pay off here.

~~~
kixpanganiban
> So, basically, you allow network programmer to not understand network
> programming, which is considered a major (main?) benefit, but at the expense
> of operators needing one more service to maintain?

That's true. My ultimate goal is to have a way to reconcile parts of a stack
written in different languages to interop, without the burden of having to
learn network programming (ie strict RPC). While it's true that Redis is yet
another service, it is precisely that which I hope would appeal to some people
if they use Redis anyway.

> How many programmers need to write a client

Well, when implementations in other languages come out (PHP and Node are in
the works), programmers don't even have to 'write a client' at all. The point
of the project is to be able to swap out existing functions with a Shisetsu-
wrapped remote executables for easy integration.

> how many users would need to run an additional service? Redis clearly
> doesn't pay off here.

See my first answer. If they don't want to run an additional service, they can
use existing RPC protocols instead (ZeroRPC, Thrift, ZeroC Ice) at the cost of
additional development to integrate those.

~~~
dozzie
> While it's true that Redis is yet another service, it is precisely that
> which I hope would appeal to some people if they use Redis anyway.

It doesn't make sense. Instead of a system with one service one now has a
system with two services, each of which can fail in its own unique way. More
moving parts for absolutely no reason at all. This way you could add there
ZeroMQ or AMQP, just to appeal to people who have these two already deployed.

>> How many programmers need to write a client

> when implementations in other languages come out [...], programmers don't
> even have to 'write a client' at all.

So the main benefit of "not needing to use network sockets to write a client"
drowns down the toilet. Have you ever needed to implement your own XML-RPC
client? I only needed that once, and only because I insisted on using a
runtime without a satisfactory one. This is what _libraries_ are for, not
sprinkling a database over the smallest things.

> The point of the project is to be able to swap out existing functions with a
> Shisetsu-wrapped remote executables for easy integration.

This is the whole point of _RPC_ , not putting _Redis_ there.

> If [users] don't want to run an additional service, they can use existing
> RPC protocols instead (ZeroRPC, Thrift, ZeroC Ice) at the cost of additional
> development to integrate those.

So you say Redis acts as a magic faerie dust that eases integration process?
How?

~~~
kixpanganiban
Thanks for your insights. Clearly, I should have put more thought about the
whole project before showing it to the world. You're absolutely right on most
points, this introduces more moving parts, another unnecessary service, and
defeats the purpose of an RPC. In fact, I have had to implement my own XML-RPC
client (and an RPC-over-HTTP client too) on a certain occasion, and I tried to
remember my gripes on that occasion and avoid them with this approach.

Regarding your last line:

> So you say Redis acts as a magic faerie dust that eases integration process?
> How?

I guess what I said was too vague. It should have been "it makes an RPC-like
system easier to integrate when you _already have Redis_ ".

Shisetsu is not a magic bullet. It won't make things magically easier for
everyone. All it is really is a niche project that, given you already use
Python and Redis, is something for you to consider. If not, then you can move
along, there's absolutely nothing to see here. :)

~~~
dozzie
>> So you say Redis acts as a magic faerie dust that eases integration
process? How?

> I guess what I said was too vague. It should have been "it makes an RPC-like
> system easier to integrate when you _already have Redis_ ".

OK, let's work with this: in what way is Redis helping here over XML-RPC or
JSON-RPC? What kind of problems does it solve that are pain in the ass with
the other RPC protocols? (And now don't take my word that it doesn't, try to
justify your choice, and next time start your README with that.)

I'm not against custom protocols or adding daemons to a system (I've
recently(ish) written a set of RPC daemons with their own protocols; the
daemons are replacing Celery task queue, which itself is a widely tested
software), they just have a decent reason to be introduced. Additional
complexity just needs to pay for itself somehow.

~~~
kixpanganiban
Thanks man, I really appreciate you giving me points and guiding me here.

> OK, let's work with this: in what way is Redis helping here over XML-RPC or
> JSON-RPC? What kind of problems does it solve that are pain in the ass with
> the other RPC protocols?

Redis vs XML-RPC/JSON-RPC is a mismatch. It's more of a msgpack-RPC with Redis
as broker vs XML-RPC or JSON-RPC:

1\. Most brokerless RPC protocols make it difficult to inspect the message on
the fly. With Redis, you can open up redis-cli and subscribe to the
appropriate channels to extract the raw messages. For most users this might
not mean anything, but for others this could mean easier debugging, measuring
performance, and so on.

2\. msgpack's serialized format is small. This makes it faster to send it
across a network especially when the message is considerably large (vs, say,
JSON or XML). That, and msgpack itself performs serialization/deserialization
quickly on its own. see:
[http://webcache.googleusercontent.com/search?q=cache:hYaRwEU...](http://webcache.googleusercontent.com/search?q=cache:hYaRwEUpbdwJ:maxondev.com/serialization-
performance-comparison-c-net-formats-frameworks-xmldatacontractserializer-
xmlserializer-binaryformatter-json-newtonsoft-servicestack-
text/+&cd=2&hl=en&ct=clnk&gl=ph) (I'm using a Google Cached link because main
link seems to be down)

> I'm not against custom protocols or adding daemons to a system (I've
> recently(ish) written a set of RPC daemons with their own protocols; the
> daemons are replacing Celery task queue, which itself is a widely tested
> software), they just have a decent reason to be introduced. Additional
> complexity just needs to pay for itself somehow

I totally agree. However, the idea of introducing Redis into the equation to
use as the broker didn't seem like additional complexity to me because a lot
of other network messaging protocols rely on a broker anyway. For example, the
widely popular Web App Messaging Protocol (WAMP) ([http://wamp-
proto.org/why/](http://wamp-proto.org/why/)) relies on brokers to decouple
callee/caller on an RPC operation. ZeroRPC, another popular RPC choice, uses
ZMQ.

For Shisetsu, what I see is the main advantage of Redis is that I can delegate
multiplexing work to it instead of having to code myself, making the protocol
and its implementation easier to manage, just by relying on Redis' PubSub. The
client and server don't even have to know each other's address and port, they
just have to know which Redis instance to connect to and assume that there
exists one of the other to work with. Redis also provides the channel buffer
and the queue instead of me having to write those as part of the client/server
code, which means anyone else looking to implement the protocol in another
language wouldn't have to deal with those as well.

~~~
dozzie
> Redis vs XML-RPC/JSON-RPC is a mismatch. It's more of a msgpack-RPC with
> Redis as broker vs XML-RPC or JSON-RPC:

I used it as a sort of mental shortcut. MessagePack is very similar in data
model to JSON or what XML-RPC allows, so I treat them as essentially the same.
I was comparing the architectures where the client connects directly to RPC
server, as opposed to connecting to a third party (broker, Redis in this
case), which is supposed to relay the communication.

> 1\. Most brokerless RPC protocols make it difficult to inspect the message
> on the fly. With Redis, you can open up redis-cli and subscribe to the
> appropriate channels to extract the raw messages. For most users this might
> not mean anything, but for others this could mean easier debugging,
> measuring performance, and so on.

This sort of things is usually implemented using logging, but I agree that
while _usually_ it's not an important feature, there are cases when it is
extremely useful.

> 2\. msgpack's serialized format is small. [...]

Though this doesn't concern Redis, and could be a wire protocol over raw TCP.

> [...] the idea of introducing Redis into the equation to use as the broker
> didn't seem like additional complexity to me because a lot of other network
> messaging protocols rely on a broker anyway.

But we're not talking about messaging protocols, we're talking about RPC
protocols/systems. You introduce two services (Redis + RPC server) for a task
that usually was carried out by one. This is why I claim it's greater
complexity than usual.

> ZeroRPC, another popular RPC choice, uses ZMQ.

I'm not sure it is that popular. Thrift, Protocol Buffers, and gRPC seem much
more widely used, and XML-RPC is a little longer on the market, so it probably
got a fair share of RPC use.

And ZeroRPC doesn't seem use broker, just a ZeroMQ protocol. ZMQ is in this
setting more like a transport protocol than a separate service to dispatch
requests and responses.

> The client and server don't even have to know each other's address and port,
> they just have to know which Redis instance to connect to and assume that
> there exists one of the other to work with.

So instead of knowing an address and port of an RPC server, client needs to
know address and port of Redis. This doesn't seem to be much of a gain. At
best, Redis acts as something analogous to port mapper in this setting (though
it relays all the communication instead of telling the client where to connect
to).

> Redis also provides the channel buffer and the queue instead of me having to
> write those as part of the client/server code, which means anyone else
> looking to implement the protocol in another language wouldn't have to deal
> with those as well.

Job queueing is somewhat outside of the RPC protocol, though I agree it can be
useful. The RPC system I mentioned I wrote does have a queue and we use it.

Though I don't quite understand what you mean under "channel buffer" term.

