
Python Generators in Depth - webexcess
http://excess.org/article/2013/02/itergen2/
======
haberman
Another very nice and mind-bending presentation on coroutines, that eventually
takes things so far that he creates a mini "OS" with coroutines as tasks,
complete with system calls and a scheduler: <http://dabeaz.com/coroutines/>

~~~
manaskarekar
The talk by the same David Beazley on Co-routines: [http://blip.tv/pycon-us-
videos-2009-2010-2011/a-curious-cour...](http://blip.tv/pycon-us-
videos-2009-2010-2011/a-curious-course-on-coroutines-and-concurrency-
part-001-2006228)

The talk from David Mertz on Co-routines:
<https://www.youtube.com/watch?v=b7R3-_ViNxk>

~~~
mapleoin
Wow that David Beazley presentation is annoying. He spends the first 10
minutes in the introduction, complaining about how he won't have enough time
to go through his presentation.

~~~
d23
I didn't watch the video, but going through the slides I was absolutely
fascinated. I've been craving something new and advanced like this out of
python for a while.

------
11001
Thank you, great write up! I'd rather see more posts like this on the front
page than reviews of the latest tablets.

------
Xcelerate
Somewhat off-topic, but how are generators different than returning an object
with a closed over environment? For instance, his first example in
CoffeeScript would be:

    
    
        runningAvg = ->
                count = 0
                total = 0
    
                send: (value) ->
                        total += value
                        count += 1
                        total/count
                    
        r = runningAvg()
        console.log r.send 10
        console.log r.send 5
        console.log r.send 0
        console.log r.send 0

~~~
enjo
In that case? Not very much different at all, but that's missing the point.
Think of it this way (I apologize, this is going to get obtuse):

Imagine you're a barista. You clock in and start doing barista things. You
have three responsibilities. You have to take orders from customers and you
have to pour coffee. When there are not customers you have to clean the shop.

The rules:

\- You must take at least two orders if there is a line, but no more before
you start pouring coffee. You should never have more than two outstanding
orders at a time.

\- If you have no customers in line and no coffee to pour, you must clean the
shop.

Lets do it with generators (again pseudo code):

    
    
      function day_of_work() {
        generator take_orders() {
            orders = array()
    
            while(during_shift()) {
               while(customers_in_line()) {
                 if(length(orders) >= 2) {
                    yield orders
                 } else {
                   orders.push(take_customer_order())
                 }
               }
    
               yield orders
            }
    
        generator clean_shop() {
            unpack_coffee()
            yield
            
            clean_espresso_machine()
            yield
    
            take_out_trash()
            yield
        }
     
        while(orders = take_orders.next()) {
            if(length(orders) > 0) {
              pour_coffee(orders.pop())
            } else {
              clean_shop.next()
            }
        }
    

There are lots of cool things going on here.

First notice that these generators not only have variable state, but they also
have _control flow_ state and THAT is why they're different. Notice that
take_orders will yield control if more than two orders exist in the queue. It
will also yield control if no more customers are in line. So our main program
loop is really simple. It gets the current set of orders from "take_orders".
Once the order queue is full it starts pouring coffee.

Notice what happens when no orders are present. Clean shop is executed once
per iteration. It first unpacks the coffee, and then yields control. Now we go
back and check for more orders, if we find them we fill them until we don't
have any more again. Then it's back to cleaning shop, but we pick back up
RIGHT WHERE WE LEFT OFF! So we just start cleaning the espresso machine...
this goes on and on until the work day ends at which point take_orders cleanly
exists and orders is null. Then we pack up and go home.

Now you might go: Well I can implement that in Javascript! You'd be right!
Closures + callbacks make it totally possible. Lets take a stab at it:

    
    
      orders = [];
      current_task = -1;
    
      while(during_work_day()) {
           take_orders(
                orders,
                function(order) { pour_coffee(order); },
                function() { current_task++; clean_shop(current_task); }
           );
      }
    
      function take_orders(orders, orders_full_callback, no_orders_callback) {
          if(customers_in_line()) {
              if(orders.length > 2) {
                  orders_full_callback(orders.pop());
              } else {
                  orders.push(take_customer_order());
              }
          } else {
              no_orders_callback();
          }
      }
    
      function clean_shop(current_task) {
          switch(current_task) {
            case 0:
                unpack_coffee()
                break
            case 1:
                clean_espresso_machine()
                break
            case 2:
                take_out_trash()
                break
            default:
                do_nothing()
          }
      }
    

So that will work. Every iteration we look for customers, if the queue is full
we callback to start pouring coffee. We clean the shop if nothing is going on.

Notice that our callbacks are closures. However, the callbacks do not have
_execution state_. This means we have to manage state externally in the form
of _current_task_.

Now for a trivial example like this, that might not be a big deal. However,
imagine that we had a whole list of things to do when there weren't customers.
Lets make it even harder, and have those tasks depend on state.

For instance what if we actually had a bunch of separate tasks we had to do:

\- unload the truck

\- draw something cool on the chalkboard

\- clean the shop

What if the order those should be done in depended on state?

Before 1:00 PM the priority is:

#1 draw something cool

#2 unload the truck

#3 clean the shop

After 1:00PM the priority changes:

#1 unload the truck

#2 draw something cool

#3 clean the shop

After 3:00PM the priority changes again!

#1 clean the shop

#2 unload the truck

#3 draw something cool

Now think about how hard this is going to be in Javascript. You're going to
have to maintain state for three separate jobs (where am I in that job), but
also deal with making sure you're working on them in the right order.

What makes generators so damn handy is that the state is internalized. I only
need to change the order I invoke the generators. As long as I keep calling
next() until there aren't any items left...they'll just keep churning through
whatever they're doing in the first place. They become more expressive as the
complexity of the state they are managing increases. Things that are hard to
do in languages that don't support them, are often pretty easy with
generators. Do some work, yield, do some more work, yield... becomes a very
simple pattern to reason about while not having to manage all of the state you
have to do in other languages.

I hope that helps clear it up a bit:)

~~~
bjudson
Great example. Thank you.

------
nickporter
Unrelated, but I find that the technique used to blur the logo is really cool!
He uses two images, one normal and the other blurred. Both images are using
fixed positioning. The normal image is set as the `body` background, and the
other is set as the background of the content panels (`div.bgover`). So, when
you scroll over the panels, the blurred logo is layered over the normal logo,
giving it that cool blurred transparency effect.

~~~
webexcess
I stole that from here
<http://meyerweb.com/eric/css/edge/complexspiral/glassy.html>

~~~
nickporter
Very cool, thanks.

------
pekk
These are great. If only one could write async network code with these that
wasn't vastly more complicated than the equivalent gevent code.

~~~
masklinn
Tulip coroutines[0] look rather good. As with C#'s async/await you have to
propagate the "asyncness" to the event loop (using `yield from` with a future
or another coroutine), which is not necessary in gevent, but that should be
the extent of it.

[0] [http://www.python.org/dev/peps/pep-3156/#coroutines-and-
the-...](http://www.python.org/dev/peps/pep-3156/#coroutines-and-the-
scheduler)

~~~
pekk
The implementation described in that PEP is massive. Seems pretty hard to
explain...

------
Marazan
That link to Generator Tricks for System Programmers (
<http://www.dabeaz.com/generators/index.html> ) has been enlightening. For the
first time I've totally and fully groked why generators are good at a
_practical_ rather than just theoretical level.

------
wookietrader
If they could only be pickled...

~~~
webexcess
That's a feature!

Seriously though, not being able to define a serialization method can be a
limitation. However, it is always possible to turn a generator into an
iterator class with whatever methods you like, but the code may end up looking
totally different.

~~~
wookietrader
To me it only looks like a limitation.

So yeah, you can write a custom class. But you will need to add a custom "save
state" and "load state" in your class' __iter__ method. So all the benefits go
away.

