
A Conversation with Guido about Callbacks - vgnet
http://oubiwann.blogspot.com/2012/03/conversation-with-guido-about-callbacks.html
======
ef4
Callback-heavy programming is what you get when your language doesn't support
better ways of controlling context. It's analogous to manual memory
management: it's work that's better to outsource to a compiler or runtime,
except in specialized circumstances.

For Python, you can get the benefits of asynchronous IO without callback hell
if you use the Stackless fork.

A similar idea for Node is node-fibers.

Both let you build exception-safe coroutines.

~~~
voidr
> Callback-heavy programming is what you get when your language doesn't
> support better ways of controlling context.

UI programming is callback heavy, but I doubt that any language will support
"better" ways for that.

So basically UI programmers are used to callbacks, others not that much, and
now that event-loop servers are becoming popular, we see confusion from people
not familiar with events.

Server-side people just need to figure out how to use callbacks properly and
everything is gonna be fine.

All people need to understand is to nest callbacks properly, nested callbacks
are the things that confuse most people.

> For Python, you can get the benefits of asynchronous IO without callback
> hell if you use the Stackless fork.

You can't just abstract away every programing feature that seems hard and
expect everything to be fine, it just doesn't work that way, Java developers
are still suffering because Java tries real hard to abstract memory
management.

~~~
gruseom
_So basically UI programmers are used to callbacks, others not that much, and
now that event-loop servers are becoming popular, we see confusion from people
not familiar with events._

I don't think that's right. There is a huge difference. UI callbacks
("onClick" and the like) execute synchronously. But the callbacks in event
servers execute at completely different times. Indeed, they're typically
nested inside code that has long since terminated. This is what causes the
learning curve, not grokking events or functions-as-values, which any half-
decent programmer can easily do in the unlikely event that they didn't a long
time ago. Indeed, the way we write asynchronous callbacks is cognitively
dissonant with everything we've been trained to understand about scoping. The
idea of a nested scope that doesn't mean "contained within" or "covered by",
but rather "at some unknown time later", is inconsistent enough with how
programming languages normally work that it can easily trip you up even after
you get good at it.

Asynchronous programming is a very different thing than e.g. browser UIs in
Javascript. I've programmed those for years. It still took me a long time to
get my head around async servers.

------
sausagefeet
Seems like one has to be very out of touch with modern approaches to
concurrency to like callbacks. Sounds flamey, but I'm just not sure how one
can look at Go, or Erlang, or even Ocaml/Lwt with syntax extensions, and think
"yes, callbacks are superior to this".

~~~
yorhel
Rather flamey indeed, but let me take the bait. I've been using callback-style
programming in many applications, ranging from IRC bots to application servers
and whatnot. I've also been playing around with Go for several months and
tried very hard to implement something that is inherently asynchronous. I gave
up after about two months, I just couldn't get used to the concurrency model.
I always end up with either data races, deadlocks and have otherwise no good
overview of the program logic. This is exactly what the article describes, and
seems to apply to me as well.

Incidentally, while figuring out how to implement something inherently
asynchronous in Go I looked at some IRC client implementations, and noticed
how all of those used callback-style programming one way or another. That
doesn't inspire much confidence. :(

All in all, I am not sure yet whether this is purely due to my inexperience
with concurrent programming or whether I am just inherently incompatible with
it. I'd love to be able to play with alternatives to callbacks in the future,
but for now it's just way over my head for the things I want to do.

~~~
sausagefeet
It appears, much like FORTH, callback coding rots the brain. I am surprised
you had such issues with deadlocks in Go, it takes some effort to write code
that deadlocks in Go (I don't mean that a deadlock cannot be shown with a
trivial amount of Go code, but that the style of programming tends to lend
itself to not having deadlocks).

> and noticed how all of those used callback-style programming one way or
> another.

What do you mean precisely here, because I think you are conflating two
concepts. There is a distinction between using a callback as a unit of
concurrency (do this async task and call THIS function when you're done), and
genuinely being an event handler (call THIS function when you get receive a
particular IRC event). I would expect to see the latter in Go for IRC code,
not the former.

But there is another aspect of this too, beyond code looking prettier.
Go/Erlang/Haskell scale to multiple cores as the runtime evolves to support
this. Callback codes doesn't, it can't. This doesn't seem to be because
callback developers don't see the value in multiple cores, Node has built-in
support for spawning a VM per core. This is quite restrictive. Callback
developers have to either limit themselves to solving problems that don't need
to communicate with each other or use some other means to communicate between
VMs. That doesn't sound like progress to me. But _shrug_ , plenty of people
don't neat things in Node so maybe it doesn't matter.

~~~
yorhel
It's hard to pinpoint exactly why I was having so many problems with
deadlocks. Most of the time they could easily be solved by having channels
with an unbounded buffer, thus guaranteeing asynchronous behaviour. But since
the Go authors intentionally left that out, I must be doing something wrong at
a more fundamental level.

You're probably right about the IRC clients using the latter style of
callbacks. But then it turns pretty much into the following style:

    
    
      irc_connect();
      register_event_x(callback_x);
      register_event_y(callback_y);
      while(read())
        dispatch_event();
    

Which is essentially an event loop with callback-style dispatching. The only
major difference here and with regular callback programming is that the
callbacks are allowed to block. But in my mind that only complicates things as
I can't tell anymore whether I can receive a particular event at some time
because I have no idea whether there are any blocking handlers going on. Of
course, you can dispatch everything into a separate goroutine, but then
suddenly I can't easily argue anymore about the order of message arrival and
what effect that has on shared state. Of course, this example may be a bit
vague and abstract, but these resemble the kinds of issues I have constantly
run in to.

On the scaling on multiple cores: Sure it's a nice advantage when you're
already used to concurrent programming, but for most things I've written so
far it doesn't really matter. In the few cases that I had a component that
required heavy disk I/O or computational resources, it was only a small and
easily-identifyable component that could be implemented in a separate OS
thread. I'm not familiar with Node, but both glib2 and libev allow spawning a
separate event loop in each thread and provide mechanisms to communicate
between each other (idle functions in glib, ev_async in libev). Those are for
C, however, and I have to admit that callback-style programming isn't too
convenient there due to the lack of closures and automatic garbage collection.

~~~
sausagefeet
I would expect the code to look more like an interface and handing your
interface object off to the goroutine in charge of the IRC connection. Or the
other way and have a goroutine in charge of shuffling the bytes back and forth
and when it parses a package it has a channel to pump the information down and
then you receive it and work on it. Either way, this seems much more straight
forward than callbacks.

> But in my mind that only complicates things as I can't tell anymore whether
> I can receive a particular event at some time because I have no idea whether
> there are any blocking handlers going on

Don't really get what you mean here. The select operator seems like it would
solve the problem your implying.

> then suddenly I can't easily argue anymore about the order of message
> arrival and what effect that has on shared state

You can barely, sometimes not even, reason about the order of message arrival
and it's effect on state in callbacks, though. At every point in a callback
you have the entire state of the program to deal with, at least in Erlang or
Go you only have the context of your local process/goroutine to deal with. I
have seen plenty of callback code explode because, what seemed like, an
obvious sequence of events got reorderd or sent before the previous work was
expected to be done, or any variation, and the code didn't handle it properly.
You might be thinking "well, duh, that is easy to fix and a silly mistake to
make", but it isn't. In a shared memory, especially where mutability is the
default mode of action, an event-driven framework can be represented as a
function that takes a 2 dimensional matrix: 1 dimension is every variable in
my program, the other is every event that can happen in my program. At every
point in the program any variation of this matrix must be a valid state. In a
language like Go, or Erlang, I can cut this matrix down significantly so that
I only have to worry about the matrix involving some subset of events and some
subset of variables. In short, that's a massive win IMO.

------
aufreak3
Given how much callback based stuff exists in the Javascript world -- GUIs,
Node.js and some recent W3C specs like the FileSystem API -- I thought it
ought to be possible to write a simple sequencing function that can take care
of this in the JS world. The key to a useful sequencing function is that it
should let you customize the sequencing of the actions and recovery techniques
pretty easily.

Here is my attempt at it - <https://gist.github.com/2192413>

Thoughts?

------
bluesnowmonkey
I'm waiting for the day when everybody realizes that callbacks and preemptive
threading both suck, relative to automatic CPS transformations.

<http://matt.might.net/articles/cps-conversion/>

------
jQueryIsAwesome
Slightly on-topic, a post of mine about how to have zero nested callbacks in
Javascript: [http://javascriptisawesome.blogspot.com/2012/03/zero-
nested-...](http://javascriptisawesome.blogspot.com/2012/03/zero-nested-
callbacks.html)

