
Announcing GenStage - jbernardo95
http://elixir-lang.org/blog/2016/07/14/announcing-genstage/
======
hosh
Yep, I wished I had had access to this back when I was working on something in
the Rails side. Ingesting large amounts of data from Shopify across multiple
oauth access points while being limited to the PostgreSQL backend and Shopify
rate limit restrictions was a pain to setup with Redis and Sidekiq. I ended up
forking Sidekiq to accomplish that -- essentially using Sidekiq as a
concurrency framework to build in a different set of behavior. Reading through
the GenStage document, I could have implemented something better.

The way I implemented it (with Ruby) left gaps of idle time, and huge amounts
of data gets staged through the queues. I couldn't figure out how to make it
better. Since I only had a week to come up with something while the
infrastructure was melting down, at the time, it was acceptable. Looking at
this though, the demand-based backpressure would work very well.

There are a couple problems I think I can use this in my current work -- this
is great work!

~~~
danvayn
Install the Bullet gem and see if your queries look sharp!

~~~
hosh
Already did. That wasn't the issue. I get that you're trying to help, but you
don't really understand the problem and why GenStage would help it. The issue
isn't N+1 queries but in concurrency and reasoning with concurrency, something
that Rails, and many Rails developers seem to avoid.

See also:
[https://news.ycombinator.com/item?id=12074425](https://news.ycombinator.com/item?id=12074425)

------
lucidstack
This looks incredible, congratulations to the Elixir team!

Perhaps more exciting than the first part of the post is the second bit about
the future. It's fantastic to see such a clear path forward for concurrency in
Elixir. Definitely looking forward to GenStage.Flow

------
lpgauth
Curious, how is the back pressure actually implemented? ACK message or ETS
public table?

~~~
strmpnk
It's demand based. The consumer will request a fixed size of input and then
the producer will send up to but no more than that until more demand is made.

~~~
lpgauth
Sure it's demand based, but how is it implemented? How is the demand
communicated between the processes?

~~~
josevalim
I am on my phone (sorry) but the documentation for the GenStage module has a
section on the message protocol between consumers and producers. GenStage is
simply one implementation of this message protocol.

------
thibaut_barrere
I was already diving into Elixir for ETL-based work before discovering
GenStage, but now even more. Thanks José et al. for the hard work on this!

------
rpazyaquian
I'm not quite at the point where I can appreciate how useful this is, because
I don't really understand concurrency itself. What is a good resource from
which to learn about concurrency, and especially Elixir/Erlang's approach to
it?

~~~
vemv
Before diving into Erlang/Elixir, I'd make sure to learn the basics of
concurrency/parallelism with more basic languages such as Java, Python or
Ruby. i.e. start with the easy stuff, then with that strong foundation choose
as you prefer!

~~~
dragonwriter
Honestly, learning actor-model concurrency with Erlang (and probably Elixir)
is a lot easier than dealing with concurrency in the "basic" languages you
list. Concurrency in Java, Python, or Ruby isn't "the easy stuff" compared to
concurrency in Erlang/Elixir.

~~~
vemv
Conceptually, threads are probably easier than actors for a beginner. I'm not
saying that it's easier to implement correct code with threads (in fact it's
more complex), but most devs go from OO to FP, not the other way around.

These days there is such an explostion of languages, frameworks, techniques
etc it must be scary for a beginner. Given that, i'd advise to go by the safe
(average) path.

~~~
rubiquity
But if you learn about concurrency via threads then you have to learn about
mutexes... if you learn about mutexes then you have to learn about condition
variables... and if you learn about condition variables then you have t...

Well you see where this is going. I think it's important to learn about
Threads, Mutexes, Condition Variables, and why sharing isn't caring, but I
don't think those lessons necessarily need to come before learning about the
Actor model.

~~~
nugator
On the other hand, if you want to sell the Actor model it might be good to
first go through the complex techniques needed for trying to get concurrency
right with threads.

------
losvedir
Congrats to the Elixir team.

As I understand it GenServer is kind of a wrapper over erlang's underlying OTP
genserver abstraction. (Is that correct?)

What's GenStage's relationship to erlang? Does erlang have an equivalent?

~~~
toolz
All OTP abstractions are based on process creation/supervision/messaging
primitives. So while erlang doesn't have this abstraction, they are all based
on the same primitives that both erlang and elixir use.

------
poorman
Super excited about GenStage.Flow. Might have to start another Elixir project
just to try it out.

------
Dangeranger
Could this be used to allow Elixir to load balance a third party server?

For example: You have an Elixir load balancer that manages requests for three
other servers and distributes load based on the 'demand' that the consumer
communicates back to the balancer?

~~~
felixgallo
yep. The neat thing about this is that it's composable primitives, so you
could imagine having an arbitrarily shaped tree of demand-creating producer-
consumers and consumers. And with distribution, it'd be possible to scale this
out pretty far.

~~~
Dangeranger
That's great.

Are there any plans to use these primitives to communicate with servers not
written in Elixir? For example can you foresee Elixir being used to manage and
coordinate heterogeneous architectures?

~~~
felixgallo
I'm on the erlang rather than the elixir side of the BEAM, but it's pretty
easy and normal in the BEAM to communicate with heterogenous processes, either
by implementing a dirty or regular NIF, opening a port, building a C Node or
using JInterface with java, using CORBA haha, or just opening a socket.

------
robbles
If the back-pressure is abstracted away by this interface, how do you monitor
it?

~~~
felixgallo
You'd still have the regular BEAM reduction-counting backpressure in place,
and it would be trivial to have your flow components report their work rates
and/or demand requests to whatever monitoring service (statsd, etc.) that you
might care to set up. Or to respond to state query messages like
sys:get_state/1.

------
windor
It's definitely worth trying! I have used akka-stream once, and always expect
something like that in erlang world. Finally, here it is. Thanks for the
Elixir team.

