
Vice: Go channels across many machines - ingve
https://medium.com/@matryer/introducing-vice-go-channels-across-many-machines-bcac1147d7e2
======
lobster_johnson
The idea of using channels for network RPC has come up before, and it's still
a bad idea.

As jerf points out in the /r/golang thread [1] and this [2] older thread, Go
channel semantics don't match network semantics. In particular, Go channels
have exactly-once delivery, which is impossible to do over the network.

Go channels are designed for communicating between goroutines; they work more
like a signaling mechanism than a data pipe mechanism. They have too many
design trade-offs to be practical for anything else. They're terrible as
general-purpose queues and pub/sub-style topologies, for example. It's
tempting to use channels are "iterators" because of range{}, but that's also a
terrible idea. Buffered channels should be avoided unless you know _exactly_
what you're doing. And since unbuffered channels block on send, you have to be
super careful about goroutine interdependencies; the only way to avoid causing
receivers to block senders is to wrap the send call itself in a goroutine, but
if you do have slowness, you can accidentally fork millions of goroutines this
way. Channel ownership is tricky to get right. And so on. Channels seem
trivial on the surface, but they aren't. They look like Unix pipes, but they
aren't. (That only one party to the communication can close a channelI
consider to be a wart. Trying to close a closed channel panics.)

The reason we see people abusing Go channels like this is simple -- channels
are generic and support type-safe type matching (with select{}). They sound
like a perfect match for certain things. But they really aren't. Channels are
really a great argument for why Go could benefit from generics.

[1]
[https://www.reddit.com/r/golang/comments/6q4p1j/comment/dkv6...](https://www.reddit.com/r/golang/comments/6q4p1j/comment/dkv6u8v)

[2]
[https://www.reddit.com/r/golang/comments/2gecvq/netchan_go_c...](https://www.reddit.com/r/golang/comments/2gecvq/netchan_go_channels_over_a_network/ckix68f/)

~~~
MoOmer
On the other hand, I use channels successfully for queuing and processing data
processing pipelines in many data ingestion applications. There are patterns
that work well, and appropriate use of the context package, error receiver
channels, and sanity work very well for some problems.

------
PaulRobinson
We have an abstraction library at work we plan to release in coming weeks
(although we've been saying that for a year now) called "drumbeat".

We have libraries in Ruby, Go and Java, and the concept is that developers
should not have to worry about transports or how they work: you make it as
simple as possible for most developers in the org to work with distributed
systems, and they'll start working with distributed systems, and the arcane
understanding of exactly what the trade-offs are between different vendors and
messaging technologies can be centralised to a few devs with ops support
instead of it being bikeshedded to death.

This is nice in that it becomes scalable and transparent with little
refactoring in Go, but having multi-language support is useful, otherwise we'd
probably drop drumbeat and use this instead.

It does suggest we should think about how best to do this in Go again though -
I like the pattern a lot.

~~~
lobster_johnson
How does this compare to, say, gRPC?

One of the ideas in the gRPC space is that the protocol, at the client and
server level, is always plain HTTP/2 and DNS. To do discovery (that is, find
the IP of a peer), load balancing, routing, retrying, throttling, circuit-
breaking etc. you inject a proxy such as Istio [1] in the middle that modifies
the connection with all the necessary intelligence.

This way, the app stays super simple -- it needs no configuration. If it wants
to talk a service called "foo", it just naively dials
[http://foo/](http://foo/) and does gRPC. If you write lots of apps in
different languages, they only need the gRPC client/server glue, nothing else.

[1] [https://istio.io/](https://istio.io/)

------
yiyus
It looks quite nice.

An alternative solution, which requires a modified compiler but avoids the
need for a context and errors channel, are the channels used by Clive:

[https://lsub.org/export/golsub.html](https://lsub.org/export/golsub.html)

I'd really like these channels (or some modified version of them) made it into
Go2.

------
chenzhekl
There was once a network channel in the standard library. But it was later
deprecated. Does anyone know the reason?

~~~
kyrra
Rob didn't like the design.

[https://softwareengineering.stackexchange.com/questions/1540...](https://softwareengineering.stackexchange.com/questions/154004/why-
did-golang-discontinue-the-netchan-package)

~~~
dullgiulio
It's not just Rob. Channels don't return errors and network operations can
always produce errors.

------
i_have_to_speak
nRPC [1] provides a higher-level abstraction but only over NATS. Similar to
gRPC.

[1] [http://github.com/rapidloop/nrpc](http://github.com/rapidloop/nrpc)

------
oelmekki
Not sure I would use it, it seems a bit too much of magic to me (coming from
ruby, I really love how everything is explicit in go), but this is certainly a
clean and well executed idea. Good job, guys!

------
packetized
get go channels to work well on one machine first.

also see: heka, deprecation of

~~~
owaislone
They are meant to provide easier concurrency mechanism. A way to coordinate
concurrent routines. They are not meant to build a data pipeline for
processing. I agree that Heka's use of channel was not optimal.

------
chewxy
netchan is back! Let's see how this one fares

------
ramshanker
I see a breewing home super computer cluster here.

