

Go package: fanout – make writing parallel code even easier - sunfmin
https://github.com/sunfmin/fanout
 hide goroutines and channels from developer.
======
TheDong
Oh, hey, it's a generic package that means you lose type safety and saves you
from writing roughly 10 lines of code.

Buffered channels + goroutines + WaitGroup already allow you to implement this
trivially, and because channels are builtin generic, you can do it without the
nasty type casts. Really, []interface{} is a terrible type to work with.

~~~
coldtea
> _Oh, hey, it 's a generic package that means you lose type safety and saves
> you from writing roughly 10 lines of code._

Or actually, it's not a generic (as in generics) package, which is why you
lose type safety.

~~~
chimeracoder
> Or actually, it's not a generic (as in generics) package, which is why you
> lose type safety.

Can we please skip just one opportunity to begin this endless flamewar on a HN
post about Go? Just one? Please?

This topic has been beaten to death and beyond, and is not relevant to TFA at
all, as the word "generic" can be used in many contexts, and it's more than
clear which context OP meant.

~~~
coldtea
> _and is not relevant to TFA at all_

How is it not relevant to TFA? This is a speficic example of the problem with
not having generics!

------
jws
Just yesterday I was staring at some of my go code thinking that channels are
the "goto" of concurrency. You can make just about anything with them, but to
understand code you have to read it all, hold it in your head, and reason
about all possible outcomes. In the '60s that's how flow of control was done.
As the '70s went by "structured programming" came in and exotic things like
_while_ loops, _switch_ statements, and functions that you could only enter at
the top (so limiting!) became the norm.

This post proposes a level of abstraction to take a common 10 line idiom and
abstract it to a word. I'd much rather read code with the abstraction. (In
this case it is clean to read, but there are many complicated patterns in
common use involving auxiliary chans for cancellation and timeouts.) Sadly,
this is where it collides with the go language designers. Go is anti-
abstraction by design. If you don't like that then you descend into
_interface{}_ hell and manual runtime type checking, or change languages, or
just repeat yourself a lot and pray you get the fiddly bits right each time.

~~~
zzzcpan
Oh, they use abstractions a lot, just not the flow control ones, mainly
objects. Like sync.WaitGroup mentioned here is an object and is a completely
brain dead choice for this kind of thing. Some form of parallel map() instead
would be much easier to understand and to reason about, for example:
[http://play.golang.org/p/4I-uBJ1Tce](http://play.golang.org/p/4I-uBJ1Tce)

Anyways, I mostly agree, just not generally over abstractions, but more in the
context of proper flow control abstractions.

------
goykasi
I wrote a similar Go package for running work loads in parallel, but I used
beanstalkd for job/result transport. This allows me a bit more freedom to
spread the workers/requesters across my network. It's a bit rough around the
edges and could use some refactoring, but it works well for my uses.

[https://github.com/eramus/worker](https://github.com/eramus/worker)

~~~
sunfmin
Bookmarked and Stared!

------
zzzcpan
Seems to be way too trivial for a library:
[http://play.golang.org/p/U0URukpeO3](http://play.golang.org/p/U0URukpeO3)

Although I like the idea of having some kind of API to spread the work across
multiple machines, since channels are slow anyway.

------
drtse4
>How does it make the program run faster?

Sunfmin, wrong answer, you'd have a 20x improvement with 20 workers only with
20 cores (ignoring the limited overhead), regardless of the number of workers
your queue of jobs will be consumed in
number_of_jobs*single_job_duration/GOMAXPROCS.

~~~
sunfmin
Thanks, but there will be I/O waiting stuff right? For that whois command
example, I find the CPU usage is zero if I am not running it parallel.

~~~
drtse4
Yup, if your jobs are i/o heavy the cpu time is wasted waiting if you execute
them in a strictly sequential way. With your worker pool the cpu intensive
part of the jobs is instead being executed concurrently GOMAXPROCS jobs at a
time.

------
bradhe
Why does this need to be a dependency that you bring on? This is a trivial
pattern.

------
AYBABTME
Write yourself the 20 lines needed before pulling an extra 3rd party deps just
for that.

The cost of having another 3rd party deps is greater than the seconds you save
from using this.

------
StavrosK
I'm not sure what the usefulness of this is (apart from saving you from
writing one or two functions yourself), isn't it just a parallel queue?

~~~
sunfmin
I guess it is a parallel queue, It saves the trouble to worry about goroutines
and channels and synchronizations, encapsulated the logic that talked in
[http://blog.golang.org/pipelines](http://blog.golang.org/pipelines) last
part. To make use of that pattern easier.

I have quite a few use cases myself:

1\. Say there is a article on the web, that contains dozens of images embed in
the html. Which I want to crawl the article, and Also the images, and put the
image into S3 for faster serve. I can use the fanout concurrency pattern to
put upload all the images simultaneously to S3 to make it faster.

2\. Say I have 20 mysql node sharded data, I want to fetch top 50 latest
created data from all of them, and sort it to and show it to user after.

~~~
StavrosK
Oh, yes, there are lots of use cases for parallel queues, for sure.

