Hacker News new | past | comments | ask | show | jobs | submit login
Callbacks are Pretty Okay (andrewkelley.me)
155 points by AndyKelley on Aug 17, 2013 | hide | past | favorite | 159 comments



I am wary of designs that make code significantly longer, as both these examples do. The improvement in local readability is often superficial and the cost in overall readability (not to mention bugs, which increase with code size) is often nontrivial.

In the examples, each line of code with a named function appears more readable, but this is only because its meaning has become less dense—and as there is also now more code to read, the two factors arguably cancel each other out. Meanwhile the new code has more non-locality: the eye of the reader must jump between the various function invocations and declarations. That's a lot of jumping, and it will get much worse as complexity grows.

Introducing a name has costs as well as benefits. For example, it adds indirection: the name now stands between the reader and the thing that is named. To overcome this, a name should say something about the problem the code is solving. If all it says is something about the code itself, that's not good—it adds weight and obscures the problem (because you're now taking more code to solve it). It feels like you're simplifying code by factoring it in this way, but you may only be spreading it out more, making pieces which are logically bound together further apart from each other.

A sign that this is happening is when you can't think of a good name for the code you are factoring, or when the name refers only to the code itself. "wtfFriendAction" is a comical example, but I think "gotFriends" is more instructive: it tells you nothing more than "I am the callback for getFriends", and thus adds no information to the original code, only verbosity.


I don't think naming your functions has to make the code "significantly longer". Perhaps 2 lines per callback.

And yes it can be hard to name functions, and I think the OP has not created very good example, but a good name that accurately describes what the function does means I don't have to read it if I don't want to.

A better example would have been some node.js code that has 5 levels of nesting, each with a some significant code within.

You've just got to name them.


The examples are 40% and 20% longer. That is significant.

The trouble with small examples is that their problems don't seem like a big deal because their entire codebase isn't a big deal. You have to apply them many times over to get any sense of their compounding effect, which is what really matters. But by definition, that's out of scope of a discussion whose examples are small. The only way out of this paradox is to pay close attention to apparently small effects.


One of the things that really annoys me about modern JavaScript programmers is that they seem to have forgotten the basics.

Functions are good. Short methods are good. Reusability is good. Properly naming your functions is good. Separation of concerns is good.

Complaining about a couple of extra carriage returns and a few extra characters to properly name the method is insanity.


How many functions are good? If I add a dozen, is that better? Short methods are good—have any evidence for that? I don't think you do, because such evidence as there is points in the opposite direction (look it up in Code Complete). It's proper to name functions properly? I agree! What's "properly"?

Hammering these dogmas harder doesn't make them any truer, it just creates a feeling that you know what you're doing in a "If the English language was good enough for Jesus Christ then it's good enough for the state of Texas" kind of way. We have almost no empirically verified knowledge about software development. One of the few things we do know (sort of—the studies are old and shaky) is that code size is a predictor of error rates. Think that's relevant?


Short methods/functions are good in that they do very little and thus are easier to read and grasp than a function spanning 50-100 or more lines. Short methods/functions are also easier to name properly.

As Clojure, F#, Haskell and friends show, it's better to have 100 functions working on one datastructure, then having ten functions for ten datastructures.


gruseom's comment raised the bar for evidence in this discussion, and gave some examples of evidence. "As Clojure, F#, Haskell and friends show" - can you point to a study?

Relevant link: http://www.slideshare.net/gvwilson/bits-of-evidence-2338367


No study no. But since those languages are built around that line of thinking, and becouse those languages are great, and becouse that line following your quote is in alot of the books about learning said languages, I belive the statement to be correct.


That line is by Alan Perlis [1]. It's great, but not relevant to the question above, which is: all other things being equal, what's the best way to do functional decomposition? If you're moving between 1 and 10 data structures, then "all other things" are so far from being equal as to make for a completely different matter.

[1] http://www.cs.yale.edu/quotes.html


The important difference in those FP languages is a) they encourage pure programs with little side effects. This is much more friendly towards small functions. b) You have the freedom to create functions only where you need to. In the OP's example, he is forced to create a new named callback after each IO call.


The reason those languages are more friendly towards small functions has nothing to do with pure functions/programs. You can use the same concepts in C/C++ or Java/C# just as easily, I've done it, it's not hard.

What makes those languages more friendly towards smaller functions has to do with how easy it is to create a new function in those languages. Creating a function in C/C++ for instance, is a pain becouse of the boilerplate involved (header files...).

In the OP's example, he's imposing named functions on himself when he doesn't actually need to. For some reason he also declares functions within the same scope that he calls it, which just makes it cluttered in my oppinion. He would probably be alot more happy with promises.


> How many functions are good?

As many as needed.

> If I add a dozen, is that better?

If your programs path needs to do 12 things, then yes. Each function should do one thing. If in documenting your function, you use the word and, that's a safe bet that you are actually writing two functions as one.

> Short methods are good—have any evidence for that? I don't think you do, because such evidence as there is points in the opposite direction (look it up in Code Complete)

Unless there is another reference I missed, Code Completes reference referred to no more than a screen's height, which at that time was ~ 25 lines. Assuming not every line is a line of code performing some action, I'd still consider that a small function.

> It's proper to name functions properly? I agree! What's "properly"?

That's hard to answer, which is why so much has been written on it. Couple that with community idiosyncrasies (like Obj-C naming vs Java), you have a lot of "proper" ways to define functions. Within the realm of the community standards, I subscribe to the thought that the method names should tend to be verb oriented. Talk about actions, or perform an action. After all, functions do something.

> We have almost no empirically verified knowledge about software development.

Well, there is quite a lot. Much of it's older (in the history of software).

> One of the few things we do know (sort of—the studies are old and shaky) is that code size is a predictor of error rates. Think that's relevant?

Be careful about this. It's not as open and shut as you'd like to think. Indeed, it's about as verifiable and confirmed as cyclomatic complexity is. Without going into too much detail and sticking strictly the context here: "code size" is an awful metric.

By that notion:

    int a = b + c; 
Is dramatically better than:

    int hours = timesPerformed + hoursPerPeformance;
Layer that into the concept of lines of code: is that lines of functional code? Lines of any code that isn't a comment? Any lines that aren't a comment? Lines of code that are only functional? All lines, including comments? All lines, including comments, except for standard headers?

All of those are valid deductions. After all, while it might seem obvious to remove comments, they could cause confusion, right?

And this ignores even the basic question: is a line merely a line on the screen? Or is it a languages statement? A single operation? Would a complex for clause with multiple assignments and statements count as one line?

All this is to say that code size is not the holy grail of error rates.

My own though is that taking CC and LoC and putting them together is best. A function with low complexity and with fewer lines of code is better than a longer function. Many smaller, lower complexity functions are also better than a single highly-complex function that is longer than any single function, but smaller than all the others when combined.


Lets see, in the 2nd edition of code complete, section 7.4 "How Long Can a Routine Be?" has a bunch of studies listed. Lets take a look:

1. A study by Basili and Perricone found that routine size was inversely correlated with errors; as the size of routines increased (up to 200 LOC), the number of errors per LOC decreased (1984).

2. Another study found that routine size was not correlated with errors, even though structural complexity and amount of data were correlated with errors (Shen et al. 1985)

3. A 1986 study found that small routines (32 LOC or fewer) were not correlated with lower cost or fault rate (Card, Church, and Agresti 1986; Card and Glass 1990). The evidence suggested that larger routines (65 LOC or more) were cheaper to develop per LOC.

4. An empirical study of 450 routines found that small routines (those with fewer than 143 source statements, including comments) had 23% more errors per LOC than larger routines (Selby and Basili 1991).

5. A study of upper-level computer-science students found that students' comprehension of a program that was super-modularized into routines about 10 lines long was no better than their comprehension of a program that had no routines at all (Conte, Dunsmore, and Shen 1986). When the program was broken into routines of moderate length (about 25 lines), however, students scored 65% better on a test of comprehension.

6. A recent [sic!] study found that code needed to be changed least when routines averaged 100 to 150 LOC (Lind and Vairavan 1989).

7. In a study of the code for IBM's OS/360 operating system and other systems, the most error-prone routines were those that were larger than 500 LOC. Beyond 500 lines, the error rate tended to be proportional to the size of the routine (Jones 1986a).

8. An empirical study of a 148 KLOC program found that routines with fewer than 143 source statements were 2.4 times less expensive to fix than larger routines (Selby and Basili 1991).

The 25 lines is just one of the studies, related to students of CS in particular. In general, it seems like the data indicates that somewhere between 100 and 200 is optimal for minimum errors per line, lowest cost to modify, less need to modify, and overall undesirability. So the "short, 5-10 line functions" seems to be not only entirely unsupported by empirical data, but actually harmful in general.


Thanks for that. Though, LOC seems to fall into the "What is a line of code trap."

Study 2 also supports complexity with errors, not size (small or large).

I hadn't realized how old these studies were, however. For some reason, I thought they were from the 90's. I'd be wary about relying on, what is, frankly outdated assumptions.


I'd be wary about relying on, what is, frankly outdated assumptions.

You'd be wary, and so you'd replace these studies with what exactly? Which of their assumptions are outdated?

Let's not fall into the trap of thinking that programmers before us didn't understand as much. As far as I can tell, the issues were much the same. A greater problem with older studies is that their sample sizes were mostly small and their experiments never replicated.


> Which specifically of their assumptions are outdated?

Methodologies. Languages. Practices. Tools.

You'd have to assume that nothing has changed in the last 20-30+ years in the world of programming to combat errors and defects, which is decidedly not the case.

This isn't at all to say that programmers before us didn't understand as much. Just they had different limitations.

I'll admit, I've been doing far more reading on the subject because of this thread. I still stand by the concept that a function should do one thing, and only one thing. The size of the function isn't important, but the nature of a function doing one thing generally leads to smaller functions.

I also think duplication should be removed. This also tends to remove excessive code.

On that note, and interesting discussion can be found here:

http://c2.com/cgi/wiki?LongFunctionsDiscussion


Sorry to reply twice, but I remembered something about this:

a function should do one thing, and only one thing.

I'm sure we all agree that functions should be well-defined. However, this is curiously separate from the question of how long they should be, because there are often different possible decompositions of functions into legitimate "one things" at distinct granularities.

For example, I have a rather long function in my code that builds a graph. That's clearly "one thing": you pass it what it needs to build a graph, and it returns the graph. But I could also extract smaller functions from it that also do "one thing"—to add a vertex, say, or establish an edge. In both decompositions—buildGraph by itself vs. buildGraph, addVertex and establishEdge as an ensemble—all the functions do "one thing". It's just that in the first example there is one coarse-grained "one thing", while in the second there are three finer-grained "one thing"s.

So cohesion (a function should do one thing only) doesn't tell us much about whether to prefer longer or shorter functions. All other things being equal, the coarse decomposition is arguably better than the finer-grained one, because it's simpler in two ways: it has fewer parts (1 vs. 3 in the above example), and the total code is smaller (it saves 2 function declarations).


> because there are often different possible decompositions of functions into legitimate "one things" at distinct granularities.

Actually, it's fairly well established. DRY: Don't repeat yourself. Yes, if you need to assign 50 values, and each is unique, than breaking them up might be difficult. However, these aren't the rules. These are the exceptions. Heck, even your graph example could be an exception.

However, in my experience, most code does contain hits on how things can be broken up into smaller, more manageable pieces.

> the coarse decomposition is arguably better than the finer-grained one

=) The finer-grained one is arguably better than the coarse decomposition because it removes repeated lines of code and makes it easier to test the smaller methods. That the total number of lines of code is increase is unimportant, because you rarely look at all the total code in a module. Rather, you are focused on individual components. If the individual component is smaller, it makes it easier to understand.

I appreciate your point of view. And I agree, it's not easy to make things smaller without losing something. However, that doesn't mean we shouldn't try.


Methodologies. Languages. Practices. Tools.

I did say "specifically" :)


That's really just being lazy on your part though. I'm not going to sit here and list everything that's come onto the scene since 1990. Every new language. Every new methodology. Every new practice, pattern, and tool. And not just new things, but also improvements over way things are done.

Unless you are going to sit here and suggest that ARC or GC don't help decrease memory errors, or that languages like Python or Java haven't helped move things along. Heck, even C has been steadily improved over the years. Compilers are getting smarter. Tools like valgrind. Agile methodologies, and formalized code testing. Static analysis and even improve code reviews. Heck, even simple things like IDEs and editors.

So much has changed, so much as evolved. Does that mean everything is wrong? No. But relying on studies that can't be replicated and don't account for common programming practices and environments today is dangerous.


You claimed that the studies kazagistar listed rely on outdated assumptions. So, specifically which studies are invalidated by specifically which assumptions? It ought to be easy to name one, since there are quite a few studies and you say there are myriads of outdated assumptions to choose from.

Your answer is that "so much has changed", you're "not going to sit here and list everything", and my question is "really just being lazy"? That's a little hard to take seriously.


They really didn't.

I'm going to be frank, in the 80s, programmers sucked compared to today's programmers. If you took an 80s programmer and time shifted him to today, he'd look like he had no experience and really struggle until he'd spent a lot of time learning new stuff.

You've grown up with software engineering in the last 10 years. Have you honestly not seen the sea change in that time?

Be it programming languages that have shifted dramatically from the original OO in Java/C# 1.0, javascript inspiring massive changes in those languages with anonymous types, closures, etc., with unit testing, TDD. There's so much that's changed, so, so much I can't even think of half of it.

I said it below, who today passes ByRef? Or uses a global variable? 10 years ago that was common.

Just 10 years ago! The field is moving so amazingly rapidly!

And one of the real things I've seen, something you can't describe unless you've got experience, which I've no doubt you have, is that methods now actually do what they say they do. They didn't used to. They used to do other things, and touch global vars, and get passed in variables to muck around with.

And on top of that, people now seem to actually understand OO. Not just blindly recanting what polymorphic means, they can actually split up and create objects sensibly. They really couldn't do it very well 10 years ago. No-one really understood it, it sounded like a good idea but only a tiny percentage actually understood what it really meant.

And yet you think studies done back then, using techniques that would get people fired today, are worth referencing?

LOOK at the dates in the parent post. 1986? No Java, no Javascript, no Python, no Ruby, no C#. But COBOL had just released an update!

You'd better watch out for those GOTOs!

I honestly am perplexed that you can even start to think those studies are still relevant today.


The grandparent wasn't claiming that the 80s were as good as us, he's claiming that just the passage of time is a bad reason to forget a lesson.

You mention a long list of rules and practices, but I'm not sure they've had as unadulterated a benefit as you think. Replacing GOTOs with objects has more often than not resulted in a more structured form of spaghetti (https://en.wikipedia.org/wiki/Yo-yo_problem, for example) that's no better. There were already people programming with closures and anonymous types, there's just more of them now. That's probably better, but not night-and-day. Lots of people still write spaghetti with closures and callbacks.

Arguing about whether the past is better or the present is better is a waste of time. We're just trading generalities at this point about a topic that permits no easy generalization. One way I've been trying to avoid this trap is with the mantra (generality :), "languages don't suck, codebases suck." It forces me to remember that it's possible to write good code in a bad language, and (often) to write bad code in a good language. Intangible abilities, domain properties, accidental design decisions and cultural factors have a huge effect that it's easy to ignore and underestimate by clinging to reassuring rules about the parts that seem more tangible. But you won't grow until you try to stare into the abyss.

(Edit: By 'you' in that last sentence I didn't mean you specifically, mattmanser.)


> he's claiming that just the passage of time is a bad reason to forget a lesson.

No, I wasn't claiming that at all. It wasn't the passage of time, but what occurred during that passage of time that mattered.

mattmanser goes into more detail.


By grandparent I meant gruseom.


Without going into too much detail

Please do go into detail. What is some of this "quite a lot" of knowledge that sheds light on the questions at hand? It would be great to have new references.

"code size" is an awful metric

This has come up on HN before and the evidence does not support what you're saying. The gold standard in recent work [1] found that more complicated metrics have no value beyond simple program length: "After controlling for size none of the metrics we studied were associated with fault-proneness anymore."

it's about as verifiable and confirmed as cyclomatic complexity is

Do you mean they're both awful? Actually, if what you say is literally true, it follows by Occam that cyclomatic measurements are worthless, since they are far more complicated and no more valuable than the most trivial metric there is.

The studies may be weak and often old, but at least there is a body of evidence around this, and the body of evidence points reasonably consistently to the conclusion that code size is (a) the best measurement of complexity that we have, and (b) the best predictor of error rates. (The only other one I know of that has repeatedly been verified is code inspections, and that's a process not a metric.) Contradictory evidence is welcome.

It's true that there's a debate around how best to measure code size. But that's a separate question. That also has received some pretty high-quality discussion on HN in the past--see the link below. It changed my thinking, for one; I highly recommend the comments by anon_d in that thread.

[1] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.20.2...,

via http://blog.vivekhaldar.com/post/10669678292/size-is-the-bes...,

discussed at https://news.ycombinator.com/item?id=3037293.


Number of tokens is a better metric than number of characters.


Yeah I'm sick of that obvious strawman being trotted out by the function-bloater side everytime we try to talk about how code size is important. I'm sure there's other examples where small code size isn't necessarily better -- but nobody bothers to look for them!

In practice I find I'm a terrible judge at where the function boundaries should be. Several times I've tried to reorganize a project where functions had gotten gradually too big, ended up making them too small and numerous and having a pain debugging, then finally settling on some intermediate position. All the overly-certain talk of "as many functions as you need" misses these nuances.

(I wrote recently about global vs local readability: http://akkartik.name/blog/readable-bad)


> Yeah I'm sick of that obvious strawman being trotted out

So, you think it that code should be measure by the size of the code?

a = b + c; is better than the other?

Care to explain why?


Either he's wrong about the straw man, or he's asserting that the number of characters used is an important factor in code quality. I'm searching for clarification of which he's asserting.


Sorry, I think my response was sacrificing clarity for rantiness.

I meant that everytime somebody refers to the value of controlling code size (like gruseom did in this thread), we have to deal with the inevitable strawman of:

    int a = b + c;
vs

    int hours = timesPerformed + hoursPerPeformance;
But gruseom's comment has nothing whatsoever to do with short vs long variable names. tlrobinson interpreted right that codebase size should be measured in number of tokens. Measured in tokens, both examples above have identical size.

(PG has also said this many many times with reference to arc.)


I never suggested in any way that number of characters was a good metric. Indeed, I suggested quite the opposite.


I think tlrobinson's point is that number-of-tokens renders a trivial objection to measuring code size such as "what if we rewrite everything to have one-letter variables" irrelevant, since that doesn't change the number of tokens.


Yes, that was my point.


That's a book that's citing sources from over 20 years ago, before unit testing, improved tools, compiler warnings and code analysis tools were even available. Massive changes have happened to the way we code since then. Hell OO had only just started entering the conciousness back then. We're not even talking about things like DI, just simple concepts like OO.

Code Complete still has many lessons to teach, but it's certainly not gospel in this day and age (unfortunately my copy is buried in some boxes from a move and I can't find it at the mo to re-read that chapter!).

I personally have seen massive changes to the way we code, people just seem to understand how to split out code into more logical chunks than they used to. Even as recently as 8 years ago, when I start professionally, I remember people used to constantly ask how to pass variables by reference, global variables, all sorts of nastiness that meant that methods weren't really being used properly. These days I honestly haven't seen a by ref pass or a global variable for years (apart from in an old, truly terrible VB.Net program I've just been given to maintain & update, ugh).

Programming really has changed massively. And there certainly are fairly famous proponents of short functions out there, Kent Beck, Robert Martin. It's advice that's changed over the last 20 years, but I don't know whether there's studies to back it up, all I can give you is that the general consensus seems to have changed.

Back to the point, I certainly don't hate long methods, what I hate is methods that do many different things. That's just bad code in my book, especially if they start bleeding lines into each other and get jumbled up as tends to happen as your code ages.

I admit it's very easy to go too far down the rabbit hole, with a massive tree of confusingly nested methods and it's a balancing game.

But when you see a long function you can usually tell straight away whether it's justified or not. Is it doing a complicated thing that justifies a hundred or two hundred lines or is it just doing loads of little simple things.

And often if it's doing simple things, invariably other functions in the same code file will also be doing those simple things with tiny variations and it can be refactored to reduce the TLOC and improve maintainability when you need to update that functionality. And that's usually a sign of a programmer how hasn't learnt to do DRY yet and is relatively inexperienced (in the .Net world anyway, I'm not having a dig at all, I appreciate js still massively changes every year).

A function that does one thing and one thing only is:

- generally short

- testable

- name it intelligently and then others don't even have to read the code to know what it does

- easier to spot bugs

- functionality doesn't get mixed up

- reusable code becomes a lot more obvious

Some of these hold especially true with javascript given its lack of block scope.

Long functions are generally a strong code smell, in my experience anyway. They generally contain bad code, regardless of whether they'd have been broken up or not. Functions that start declaring random anonymous objects, which I see a lot in javascript, are especially dubious. It's an artefact of javascript's design that because it's so difficult to declare reusable objects and namespaces and manage code between files a lot of javascript programmers just end up writing these huge blobs of code.

As to where I see this happening in javascript, as an example I was reading Meteor.js's code the other day and was actually surprised at just how bad the code is. Every function seems to do 5 different things, it declares object right in the middle of functions rather than extracting it out, loads of frankly crazy coding practices.

EDIT: In fact, now that I think about it, I remember once using that bit of CC to argue that long functions were better discussing it with a friend. Funny how you can do a complete 180 over time.


>A function that does one thing and one thing only is: >- generally short >- testable > ...

This is true only if the function decomposition is intelligently done. The problem is that with callbacks, the function decomposition is forced on you based on what needs to be done asynchronously (i.e., mostly I/O). As a result, it breaks apart the flow of control into little tiny pieces that may be only loosely coupled with the most logical way to decompose functionality.

So your statement may be true in general, but it may not be applicable when it comes to how to best handle callback functions.


The problem with callbacks is they are not about reusability — they force you to organise your code around I/O and not around your app logic... hence the names like wtfFriendAction.


There is a difference between callbacks and lambdas. Nothing forces you to provide a lambda as a callback. If that lambda will only ever be used once, then it's ok as long as it's short and easy to understand what it does.

If that function is required multiple places, you make a proper function out of it, and pass that function instead of a lambda.

Callbacks allow for abstractions like map, filter, reduce... Used properly, they can actually increase reusability instead of prohibiting it.


It's like you didn't read my comment — how functions like wtfFriendAction help reusability?


In this small example it doesn't, getFriendsById however, does. I would imagine that wtfFriendAction could be used multiple places though.

wtfFriendAction is additionally a very, very poor name. It should be called "addFriendToFriendList" or something. Another thing is that the author himself states that the function doesn't actually makes sence, hence the ugly name.


> Another thing is that the author himself states that the function doesn't actually makes sence, hence the ugly name.

That's the point — using callbacks you end up with functions which do not make sense...

Edit: spelling


IMO, the biggest problem is the extra indirection, not the raw lines of code.

As for names, the problem is that he is forced to make a new callback whenever he does an IO operation and then needs to make up a name for it on the spot. Ideally you should never be forced to split code into functions unless you actually want to do so and when you can choose where to create functions its much easier to create good abstractions and good names.


Right, but now your functions are very tightly coupled to each other. The indentation might be nicer, but functionally it's just misleading.

The cool thing about promises is that your methods are pretty portable. For example:

    function getUserFriends(userName, next) {
        db.users.findOne({name:userName}, foundOne);

        function foundOne(err, user) {
            if (err != null) return next(err);
            db.friends.find({userId:user.id}, foundFriends);
        }

        function foundFriends(err, friends) {
            if (err != null) return next(err);
            return next(null, friends);
        }
    }
Can be rewritten (using q[0]) as:

    var Q = require('q');

    function getUserFriends(userName, next) {
      var deferred = Q.defer();
      db.users.findOne({name:userName}, deferred.makeNodeResolver());
      return deferred.promise;
    }

    function foundOne(user) {
      var deferred = Q.defer();
      db.friends.find({userId:user.id}, deferred.makeNodeResolver());
      return deferred.promise;
    }

    function foundFriends(friends) { ... }

    getUserFriends.then(foundOne).then(foundFriends);
You'll see here that getUserFriends, foundOne, and foundFriends (which I left out) are now all portable and can be used elsewhere in the code for other purposes (because they are no longer tightly coupled).

[0]: https://github.com/kriskowal/q


With the downside that I have absolutely no clue what this code does or how.


Once you get used to promises (it took me a couple of days) it's not confusing at all. It takes a bit of effort though. I remember when I first started programming none of the programs made any sense to me; as you said, I had absolutely no clue what the code did and how. But then I kept practicing and soon I understood how things work.


Yeah, I guess it's not fair for me to compare something I'm familiar with to something I'm not. I'm sure promises are easy to understand, but that API there does not look great.


Q has some utility functions around wrapping node style callbacks that make the code a bit shorter and arguably more easy to understand.

    var Q = require('q');

    function getUserFriends(userName) {
      return Q.nfcall(db.users.findOne, {name:userName});
    }

    function foundOne(user) {
      return Q.nfcall(db.friends.find, {userId:user.id});
    }

    function foundFriends(friends) { ... }

    getUserFriends.then(foundOne).then(foundFriends);


That's much better, thanks for sharing


The real problem with callbacks is error handling. Using anonymous functions doesn't address this problem. The .then notation has the same problem.

The await solution (or any equivalent) has the benefit to present the call as a synchronous call which is trivial to read and understand. It also has the benefit to support all existing and proven error handling methods. try catch, etc.

This is the best solution to asynchronous calls that I have seen so far.

I'm currently using c++ asio, and it's a nightmare.


Promises are actually pretty good at error handling, the first error encountered is returned.


Well, Q allows you to attach an error handler with catch:

    bla().
        then(something).
        then(someOther).
        catch(ohNoes);
First, this gives you really bad stack traces, essentially you have no chance to figure out what happened before your error handler got called. You can get slightly less bizarre stack traces with `Q.longStackSupport = true;`, but that has a serious performance impact (creating a new throwable per invocation).

Second, if you try to get better code structure by breaking up your program into named functions ('something', 'someOther' above), you loose locality, i.e. your error handler does not share a scope with its causing code. You can fix that by passing the entire promise to a processing function, but this again obscures things.

Compared to a program that just does:

    x := bla();
    var y;
    try {
      y = something(x);
    } catch(...) { ... }
    return someOther(y);
The promise code is still a lot more complicated. I agree with Miguel de Icaza, continuation passing style/callbacks do destroy 'natural' program structure, and make it much harder to use the regular programming language's idioms (loops, returns, ...).

I'd be hugely in favour of go's approach with very cheap goroutines, so you don't actually have to use CPS just because your runtime isn't clever enough to use more than one thread (ahem node). OTOH go's lack of exceptions kills the whole argument by bringing back the error handling of the 80s :-(


How can you make this argument and not even mention the word 'closure'? That's like arguing there's no need for object orientation, and demonstrating it by showing how you can model any class as just a set of global functions, by only showing classes which have no instance state.

Replacing anonymous functions with named functions works fine unless the anonymous functions are closures. If they are, you're going to have to add parameters to them to pass state in, and you're in real trouble if you want to have them modify closed state, because in JavaScript you've got no pass-by-reference. But closures that modify closed variables are hard to reason about in the usual nested callback syntax model.

What the async 'syntactic sugar' approaches do is replace nested closure scopes with simple control-structure scoping that's easy to reason about - even if the 'callbacks' are modifying state which then gets made available to subsequent callbacks.


It looks like you opened the article, hit "ctrl+f closure", saw 0, and went on a rant. In the examples where the turned an anonymous function into a named function, the named functions still close over symbols from their calling context, so... it's hard to see what you're on about.


His examples only close over parameter variables and lifted functions, not local variables that actually, you know, vary.

The fact is, in JavaScript, lifted functions can close over local variables but the fact that they do is not obvious - in fact it's so counterintuitive that you're generally best to avoid doing so.

In the first example he gives, two functions that, in the original, are nested, in the refactoring are placed in the same scope, rather than nested within one another. His amended solution prevents the second function from closing over the scope of the first, which is possible in the original, or an async form. If you wanted to share state from the first function to the second, you'd have to realize you could bring the second function into the scope of the first, and then move it, and then you're back where you started with nested callbacks. But amending this code is made harder than it should be by the decision to factor the two closures out to the widest possible scope, rather than the narrowest.

And the fact that ctrl-F closure yields no results just confirms the feeling that the examples gave me that the author is playing around with closures without knowing what he's actually doing.


I think callbacks are still pretty broken even after applying the "two rules". Yes, one avoids nesting, but you still break your procedure/function/method into chunks which make no sense by themselves.

I don't see how that can lead to good code. Better than a million nested callbacks? Sure. Readable? Nope. How do you even name all those callbacks?


"Callbacks are Pretty Okay" translation: I'm a programmer in my twenties, work mostly in Javascript, have no much experience with the historic literature and core ideas of Compute Science, don't know any better, and I like node, so there.



So much hubris in so few words.


This recent 'callbacks as goto' discussion is so utterly mundane that I've all but failed to convince my wrists to so much as twitch sufficiently to click on any of the links, just the title is enough to drive me to drink.

Callbacks are in no way "this generation's goto", they do not in any way inhibit the ability to analyse or prove correct a program, and all alternatives to callbacks amount to incredibly fancy syntax sugar for hiding what is really happening behind the scenes anyway.

A callback is a function call, or put another way, a specification of an exact set of arguments grouped in a formal specification that is sent to a deterministic little machine (the function) to execute immediately. On completion the machine will provably produce the result it promised to return in its owner's manual (i.e. the declared return code).

None of this is "goto" in any way. In goto land, without knowing the exact set of inputs a program receives and whole-system simulation, it is utterly impossible to make assumptions about even the most minimal pieces of state in the program at any given moment.

Contrast to a function call. A function call is a fixed guarantee about the state of the system at the point a machine begins executing. Furthermore a function call has a vastly restricted set of valid behaviours: at some point it must terminate, and prior to termination, update the system state (stack) to include its return value. And upon termination, result in the calling machine to resume operation.

All this syntax sugar shit is whack. Great, you typed 24 characters instead of 3 lines to declare a lambda. Hyper productive, go you. How does this progress toward the point where I can say "computer, make me a coffee!"?

If you're genuinely interested in what this generation's Goto might look like, take 30 minutes to watch http://vimeo.com/71278954 .. our UIs are utterly trapped in pre-1970s thinking, our communication networks are so utterly idiotic that we STILL have to write custom code to link disparate chunks of data and logic together, we're STILL writing goddamned function calls in a freaking text editor (something that was solved LONG ago). All the things like this.

I can't ask my computer to make me a cup of coffee, and it responds. I can't describe in restricted English a simple query problem and have the millions of idle machines around the world coordinate meaningfully to answer my question (and our best chances of this.. freebase.com.. got bought up by Google and left to rot as some vestigial appendage of their antispam dept no doubt).

Computing is in a sad state today, but not because of fucking callbacks. It's in a sad state today because we're still thinking about problems on this level at all.

Node.js wasn't an innovation. It was shit that was solved 60 years ago. I wish more people would understand this. Nothing improves if we all start typing 'await' instead of defining callbacks.

Innovation in our industry has been glacial since at least the early 90s.

Edit: and as a final note, I wholeheartedly welcome you to nitpick the hell out my rant, wax verbose on the definitions of provability, the concept of a stack, the use of English versus American language, why Vim is awesome, and in the process once more demonstrate the amoeba-like mentality 90% of our industry is trapped in.

Go and build a programming language your Mom can use. Preferably by talking to her computer. Please, just don't waste it on any more of this insanity.


>Callbacks are in no way "this generation's goto"

They are very much analogous to GOTO -- or even COMEFROM.

They have a similar effect to the control flow, and a similar adverse effect on the understanding of the program.

And Dijkstra's core argument against GOTO applies 100%:

"""My second remark is that our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed. For that reason we should do (as wise programmers aware of our limitations) our utmost to shorten the conceptual gap between the static program and the dynamic process, to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible."""

>all alternatives to callbacks amount to incredibly fancy syntax sugar for hiding what is really happening behind the scenes anyway.

If, for, while, foreach --heck even map, reduce et co, etc are also syntax sugar for GOTO. Your point? Or adding the weasel word "fancy" somehow makes this particular syntax sugar bad?

Not to mention that there's nothing "incredibly fancy" about await, async, promises et co.

And if I wanted to know "what's really happening behind the scenes" I wouldn't programmer in Javascript in the first place.


You're not the first person I read mentioning "if" as a syntax sugar for "goto". It is not. In fact, "goto" needs "if" to replicate looping behavior.

Unless you imply "goto" can branch conditionally (e.g. je/jne/jg etc. from ASM) but that's not "goto". "Goto" means unconditional jump.


I thought of that Dijsktra quote too. It sheds light on what's wrong with the OP's proposal: in return for less nesting, it worsens the conceptual gap between the program (in text) and the process (in time). A poor trade.

I've found that nesting to indicate asynchronous control flow can be quite a good device for keeping the program text closely related to its dynamic execution. (It's true that such code is hard to read in JavaScript, but I don't think that's because of nesting per se.) It allows you to lay out synchronous logic along one dimension (vertically) and asynchronous along another (horizontally). I hypothesize that there's a quite good notation waiting to be brought out there; unfortunately, such experimentation tends to be done only by language designers in the domain of language design, when it really ought to be done in the context of working systems—and that's too hard to be worth the trouble unless you're programming in something like a Lisp.


>I thought of that Dijsktra quote too. It sheds light on what's wrong with the OP's proposal: in return for less nesting, it worsens the conceptual gap between the program (in text) and the process (in time). A poor trade.

That's like saying that "by using FOR instead of GOTO we worsen the conceptual gap between the program (in text) and the process (in time)".

Only we don't worsen it -- we just abstract it to a level in which we can reason about it better.

Higher level is not worse -- except if you consider "more distance from what is actually happening at the CPU" as worse. Which is not: historically the higher the distance, the more programmers got done.

We don't need to keep track of the actual low level process in time -- which with callbacks we're forced to. We just need to know how the steps of the process relate. Which await and co offer in a far better form than callbacks.


We're not talking about distance from the CPU. At least I'm not, and I'd be shocked if Dijkstra were. No, the issue is that a program has at least two different logical structures: its lexical organization in text, and its dynamic organization in time. You want these two to be as closely related as possible, because humans rely mostly on the program text to understand it.

"Time" here doesn't mean CPU time, it means logical time--the temporal (as distinct from textual) order of operations in the program. To put it another way, the "time" that matters for program clarity is not when stuff happens at lower levels (e.g. when does the OS evict my thread) but what happens when in the source code itself. This is not so far from your phrase, "how the steps in the process relate", so I don't see the disagreement.

I certainly don't agree, though, that the OP's design recommendations lead to higher-level or easier-to-reason-about code. Do you really think they do?


> ... the issue is that a program has at least two different logical structures: its lexical organization in text, and its dynamic organization in time.

That's a concise way to put it and I'll certainly remember and reuse it! Did you come up with it or did you come across it somewhere?


It's just a paraphrase of the Dijsktra quote. The same idea is implicit in the distinction between lexical and dynamic scope, which confused me for years before I got it... probably because "dynamic scope" is something of an oxymoron.


>they do not in any way inhibit the ability to analyse or prove correct a program, and all alternatives to callbacks amount to incredibly fancy syntax sugar for hiding what is really happening behind the scenes anyway.

1) EVERYTHING is 'syntax sugar'.

2) There's nothing intrinsically wrong with gotos, they just aren't very well suited for human brains. Computers can execute goto statements very efficiently.

Callback-hell smells very much like gotos to me. It's very easy to do the wrong thing and easy to create very hard to read, hard to understand, and hard to maintain code.


> 1) EVERYTHING is 'syntax sugar'.

The OPs counter argument argument is against treating some syntactic sugar as special and novel when it is in fact not. Even voice commands are syntactic sugar, but they're way better for the lay user.

Although I dunno what he's talking about with "I can't talk to my computer." My phone is learning a lot of tricks, really fast.

> Callback-hell smells very much like gotos to me. It's very easy to do the wrong thing and easy to create very hard to read, hard to understand, and hard to maintain code.

This is mostly because people try to treat higher order functional code as in fact just a fancy syntax. Writing higher-order functional code (code that creates, consumes, and modifies functions) requires a set of alternative disciplines that most people in our industry not only don't learn, but actively despise and in many cases mock.

Even otherwise brilliant, smarter-than-me people I look up to do this and then declare the entire functional concept bankrupt. It drives me nuts. Once upon a time, when people saw OO code then turned their nose and said, "You shouldn't need to define hierarchies and ontologies just to reason about your code! How stupid is that!" But then proven models came out and the industry half-assed adopting it (remember Taligent? No? Look it up).

So many developers these days are whining and grousing about how static type systems inhibit their freedom and higher order functions are whacky propeller-head notions that only nerds take seriously. And yet they wonder why the bulk of the industry moves at a glacial pace. I'm happy that Erlang's nearly-30-year-old proposition and some of the implications Needham's duality are finally reaching mainstream computing.


> This is mostly because people try to treat higher order functional code as in fact just a fancy syntax. Writing higher-order functional code (code that creates, consumes, and modifies functions) requires a set of alternative disciplines that most people in our industry not only don't learn, but actively despise and in many cases mock.

Well it's a hell of a lot harder to write a good verb than it is to write a good noun. Similarly, writing good adverbs is nearly impossible (how many books on Lisp macros do you know of?). If I had to teach my Mom to code, I wouldn't teach her how to zip, fold, map & reduce lists-of-lists on the fly, I'd teach her the FullName noun.

Callbacks are just gotos that return. They can also pass along non-global context, like error information. It's a subtle distinction, and to be up in arms about it is indeed strange.

The industry moves slowly because they can afford to. A few million dollars can feed a hundred developers. The codebases get so large, the teams so big, that lowest-common-denominator kind of code will always prevail. Remember what I was going to teach my mom? Not lisp macros, no. Simple nouns, simple mechanisms.


Prepare for a solid fisking.

> Well it's a hell of a lot harder to write a good verb than it is to write a good noun.

See, that's your indoctrination talking. Really both are about equally hard. The actual definition of zip is pretty simple; assuming you have trained yourself to think about it the right way. This is no different from OO. The idea that imperative programming is "natural" is sort of a myth.

> (how many books on Lisp macros do you know of?)

Quite a few, actually! But I'm not sure why this matters;. Lips macros have 0 to do with not only this conversation, but this entire family of abstractions. Macros bear no resemblance to what we're talking about.

> If I had to teach my Mom to code, I wouldn't teach her how to zip, fold, map & reduce lists-of-lists on the fly, I'd teach her the FullName noun.

Why? People think verb-first all the time, describe things in verb-first ways, and act in verb first ways. They do it all the time, an it's not unnatural.

> Callbacks are just gotos that return.

Not really.

> They can also pass along non-global context, like error information.

If they are implemented with continuations, they do a lot more. But see also coroutines.

> The industry moves slowly because they can afford to.

I submit that the resurgence of the small software shop and the incredible successes that small software startups have been seeing is a counter-argument to this. As backwards as the average Node.js shop is, they're still light-years ahead of the befuddled, ossified monstrosities that they compete with.

> A few million dollars can feed a hundred developers.

You should be ashamed of this remark.

> The codebases get so large, the teams so big, that lowest-common-denominator kind of code will always prevail.

Bridges are not constructed this way.

> Remember what I was going to teach my mom? Not lisp macros, no. Simple nouns, simple mechanisms.

Stop patronizing people. You're pretty smug for someone who doesn't know lisp. I thought being smug was my job as a lisp hacker!


> The idea that imperative programming is "natural" is sort of a myth.

Yes yes yes. I definitely agree. It's "sort of a myth," but it also sort of true. Zip is indeed quite simple, but it wouldn't make any sense at all unless you knew what a list was. Case in point, zipping two lazy (infinite) lists like they're eager won't work at all; each version of a list would have its own zip. The verb will be more or less derived from the noun. A verbless noun makes sense, but a nounless verb? I think there is some dependency.

Ask some programmers if they learned function calls before they learned variable assignment. I'm obviously betting they didn't, but I'd be curious if I were wrong.

I really don't know what's "natural". I'd like to know, but I don't. I do play guitar. (poorly.) Playing a good chord is a lot harder (for a beginner) than playing a good note; I've seen plenty struggle, including myself. Now though, both are about equally hard. I have no preference. The actual structure of a -7 chord is pretty simple once you start to think in the right way. For some reason, though, beginners seem to like playing major chords and single notes. Similar story: when I was a child, I learned how to write the letters before I learned how to write the words. When I was a slightly older child, I learned the chess pieces before I learned the chess openings. It all goes hand in hand, but something's got to come first. I figure it's probably got something to do with how the brain acquires new patterns, but I know even less about neurology than I do programming.

I intended to relate adverbs to lisp macros, but I don't have a rock solid thesis on the matter. Macros can arbitrarily change the nature of functions just like adverbs change the meaning of nouns. In either case, overuse leads to crappy writing. I'd argue they're more awkward to write, but not because of any indoctrination. Just a personal thought.

I'd even bet cash that even a room of grad students could brainstorm/generate new nouns much faster than they can generate adverbs. This might not prove anything.

> You should be ashamed of this remark. > Bridges are not constructed this way. > Stop patronizing people.

I got confused by all this. I didn't intend to patronize anyone. Sorry.


> The verb will be more or less derived from the noun. A verbless noun makes sense, but a nounless verb? I think there is some dependency.

Linguistically inaccurate. But it also seems irrelvant. "Zipping things" is a great example of a placeholder noun, anything that can be zippable, right? People do this all the time. "Driving" implies that you have a thing to drive, but the act of driving is clear and distinct in people's heads despite the fact that it can represent a lot of different actions.

> Ask some programmers if they learned function calls before they learned variable assignment. I'm obviously betting they didn't, but I'd be curious if I were wrong.

The answer to this question is irrelevant, but also hard to understand. Variables and functions are deeply intertwined ideas because most function calls take variables.

SICP taught functions first, and it was widely acclaimed.

> Macros can arbitrarily change the nature of functions just like adverbs change the meaning of nouns.

I do not see an interpretation of macros that is concordant with this metaphor. Macros let you define new parts of speech entirely, hand-tooled to let you perfectly express what you want the way you find most natural.

> I'd even bet cash that even a room of grad students could brainstorm/generate new nouns much faster than they can generate adverbs. This might not prove anything.

I do not think this is relevant. But if you'd like to see an example of how complicated this is, look at any note from one partner to another. Mine go like this: "Dave, Please pick these items on your way home:" and then a list. Which is a function (in IO, so monadic since it has side effects)> But that is a verb THEN a list of nouns.


> There's nothing intrinsically wrong with gotos... Computers can execute goto statements very efficiently.

That's because the problem is analyzing the goto statements, not executing them. And computers suck at it, less than humans, but still suck.


You snipped out the part where he said exactly that.


>Go and build a programming language your Mom can use. Preferably by talking to her computer.

People who think this is doable with current technology usually don't have much clue in how to build a programming language.


https://ifttt.com/

"Computer, create a rule for my car."

"OK. Which car? Do you mean Red Toyota Purchased 1997?"

"Yes."

"OK."

"Computer, when car is at location House, then boil kettle."

"Do you mean Kettle in Kitchen of 48 My Road?"

"Yes."

"OK. Is that all?"

"No. Computer, this rule is only valid on weekdays."

"OK. So when car Red Toyota Purchased 1997 is at location 48 My Road on Monday, Tuesday, Wednesday, Thursday, Friday, boil Kettle in Kitchen of 48 My Road?"

"Yes, but only if my iPhone was not inside the car."

"Do you mean Jemima's Apple iPhone 4Gs?"

"Yes."

"OK."

...


This just demonstrates that the problem, for laypeople, isn't so much the programming as the debugging. What does "your mom" do when she realizes the kettle is on for nearly 18 hours every day because that's how long her car is there?

Being able to communicate effectively, in a bidirectional manner, with a computer is a skill. It's a skill people will probably always have to learn, one way or another. It's more likely that more people will learn these skills than that we'll devise a way for computers to be the smart ones in the conversation any time soon.


The distinction between "is at" and "arrives at" is going to result in a high power or house gas bill. :)


I'm a PL researcher who has also worked a bit on dialogue systems. Getting those rules encoded for even a small domain would be a huge task, primarily because machine learning hasn't helped us in this area yet.

We will get there, but maybe in 5 or 10 years.


Exactly. I think OP's argument was that we should devote more resources to these sorts of problems instead of bickering about low-level stuff so much (if they aren't directly helping that goal).

I don't see how that view implies ignorance of current technological limitations.


The OP wrote:

>Computing is in a sad state today, but not because of fucking callbacks. It's in a sad state today because we're still thinking about problems on this level at all.

He seems to imply that it's sad today is today and not 20 years from now. So what?

People have dreamed about replacing "programming" with "dialogue systems" since at least the 60s; they are like flying cars, always 10 or 20 years away. We might be closer now, but it's not like we were not dreaming about this since before I was born.

In the meantime, we have code to write, maybe it's worth doing something about this callback thingy problem just in case the singularity is delayed a bit.


There are not that many researchers (or other people) working on 'less code'. And the response from the community is always; don't try it, it's been done and it won't work (NoFlo's Kickstarter responses here on HN for instance).

Instead I see these new languages/frameworks which have quite steep learning curves and replace the original implementation with the same amount of code or even more.

As long as 99% of 'real coders' keep saying that (more) code is the way to go, we're not on the right track imho. I have no clue what exactly would happen if you throw Google's core AI team, IBM Watson's core AI team and a few forward thinking programming language researchers (Kay, Edwards, Katayama, ...) in a room for a few years, but I assume we would have something working.

Even if nothing working would come out, we need the research to be done. With the current state we are stuck in this loop of rewriting things into other things, maybe marginally better to the author of the framework/lib/lang and a handful of fans resulting into millions of lines of not very reusable code. Only 'reusable' if you add more glue code than the original code in the first place adding only to the problem.


Nothing was new with NoFlo, it was just trying the same thing that failed in the past without any new invention or innovation. The outcome will be the same.

Trust me, the PhBs would love to get rid of coders, we are hard to hire and retain. This was at the very heart of the "5th gen computing movement" in the 80s and it's massive failure in large part led to the following second "AI winter."

> I have no clue what exactly would happen if you throw Google's core AI team, IBM Watson's core AI team and a few forward thinking programming language researchers (Kay, Edwards, Katayama, ...) in a room for a few years, but I assume we would have something working.

What do you think these teams work on? Toys? My colleague in MSRA is on the bleeding edge of using DNNs in speech recognition, we discuss this all the time (if you want to put me in the forward thinking PL bucket) over lunch...almost daily in fact. There are many more steps between here and there, as with most research.

So you are unhappy with the current state of PL research, I am too, but going back and trying the same big leap that the Japanese tried to do in the 80s is not the answer. There are many cool PL fields we can develop before we get to fully intelligent dialogue systems and singularity. But if you think otherwise, go straight to the "deep learning" field and ignore the PL stuff, if your hunch is right, we won't be relevant anyways. But bring a jacket just in case it gets cold again.


I agree with you on NoFlo and criticism like yours is good if it's well founded. I just see a bit too much unfounded shouting that without tons of code we cannot and can never write anything but trivial software. The NoFlo example was more about the rage I feel coming from coders when you touch their precious code. Just screaming; "it's impossible" doesn't cut it and thus i'm happy with NoFlo trying even if doomed to fail, it might bring some people to even consider different options.

> What do you think these teams work on? Toys? My colleague in MSRA is on the bleeding edge of using DNNs in speech recognition, we discuss this all the time (if you want to put me in the forward thinking PL bucket) over lunch...almost daily in fact. There are many more steps between here and there, as with most research.

No definitely not toys :) But I am/was not aware of them doing software development (language) research. Happy to know people are discussing it during lunch; wish I had lunch companions like that. Also I should've added MS; great PL research there (rise4fun.com).

I wasn't suggesting a big leap; I'm suggesting considerably more research should be put into it. Software, it's code, it's development and bugs are huge issues of our time and I would think it important to put quite a bit more effort in it.

That said; any papers/authors of more cutting edge work in this field?


My colleague and I talk about this at lunch with an eye on doing a research project given promising ideas, but I think I (the pl guy) am much more optimistic than he (the ML/SR guy) is. I should mention he used to work on dialogue systems full time, and has a better grasp on the area than I do. I've basically decided to take the tool approach first: let's just get a Siri-like domain done for our IDE, we aren't writing code but at least we can access secondary dev functions via a secondary interface (voice, conversation). The main problem getting started is that tools for encoding domains for dialogue systems are very primitive (note that even Apple's Siri isn't open to new domains).

The last person to take a serious shot at this problem was Hugo Liu at MIT. Alexander Repinning has been looking at conversational programming as a way to improve visual programming experiences; this doesn't include natural language conversation, but the mechanisms are similar.


I would think that this is why PL research if very relevant here; until we have an 'intelligence' advanced enough to distill from our chaotic talking about an intended piece of software, I see a PL augmented with different AI techniques to explain, in a formal structure (with the AI allowing for a much larger amount of fuzziness than we have now in coding; aka having the AI fix the syntax/semantic errors based on what it can infer about your intent after which, preferably on a higher level of the running software, you can indicate if this was correct or not) how a program should behave.

With some instant feedback a bit like [1] this at least feels feasible.

[1] http://blog.stephenwolfram.com/2010/11/programming-with-natu...


> We might be closer now, but it's not like we were not dreaming about this since before I was born.

It's not about dreaming. It's about action and attitude. Continuing down the current path of iterating on the existing SW/HW paradigm is necessary, but in that 20 years, it's not going to lead to strong AI. Our narrow minded focus on Von Neumann architecture permeates academia. When I was in college I had a strong background in biology. Even though my CS professor literally wrote a book on AI, he seemed to have a disdain for any biologically inspired techniques.

Recently, I've seen a spark of hope with projects like Stanford's Brains in Silicon and IBM's TrueNorth. If I was back in school, this is where I'd want to be.

http://www.technologyreview.com/news/517876/ibm-scientists-s...

http://www.stanford.edu/group/brainsinsilicon/


Sounds familiar!

http://en.wikipedia.org/wiki/Fifth_generation_computer

1982 was 30 years ago, BTW.


Thanks for the link. After 60 years of promising to make computers think even the fastest machines on the planet with access to Google's entire database still have trouble recognizing a cat from a dog, so I have to agree with the article that yes it was "Ahead of its time...In the early 21st century, many flavors of parallel computing began to proliferate, including multi-core architectures at the low-end and massively parallel processing at the high end."

As 5th generation shows small pockets of researchers haven't forgotten evolution has given each of us a model to follow for making intelligent machines. I hope they continue down this road because faster calculators aren't going to get us there in my lifetime. You feel differently?

As the article mentions "CPU performance quickly pushed through the "obvious" barriers that experts perceived in the 1980s, and the value of parallel computing quickly dropped", where as 30 years later single threaded CPU performance has gone from doubling every 2 years to 5-10% per year since ~2007. Combine that with the needs of big data, and the time is right to reconsider some of these "failures".

Parallel programming is the biggest challenge facing programmers this decade, which is why we get posts like this on callback hell. Isn't it possible that part of the problem lies in decisions made 50 years ago?

I'm not saying we need to start from scratch, but with these new problems we're facing, maybe it's time reconsider some of our assumptions.

"It is the mark of an educated mind to be able to entertain an idea without accepting it." -Aristotle


The problem is that we've been down this route before and it failed spectacularly (we've gone through two AI winters already!). It does not mean that such research is doomed to fail, but we have to proceed a bit more cautiously, and in any case, we can't neglect improving what works today.

The schemes that have been successful since then, like MapReduce or GPU computing, have been very pragmatic. It wasn't until very recently that a connection was made between deep learning (Hinton's DNNs) and parallel computing.


Yes, I did say continuing down the current path was necessary. From the languages to the libraries, today's tools allow us to work miracles. As Go has shown with CSP, sometimes these initial failures are just ahead of their time. Neuromorphic computing and neuroscience have come a long ways since the last AI winter.

The hierarchical model in Hinton's DNNs is a promising approach. My biggest concern is that all of the examples I've seen are built with perceptrons, whose simplicity makes them easy to model but share almost nothing in common with their biological counterparts.


Funny a cryptographer was just explaining why you should never question your field's fundamental assumptions.

http://www.extremetech.com/extreme/164050-discovery-may-make...


That's more of an orthogonal point than a counter point though. Just because we haven't accomplished something yet doesn't mean we shouldn't invest any more effort into it.

I was responding to your insinuation that OP must not be knowledgeable about language implementation to hold the views he/she does. In this context, technological limitations are irrelevant because that was not the point; the point was an opinion about the relative effort/attention these problems receive.

I'm admittedly not really invested in this area myself so I don't really care, but it's disingenuous to try and discredit OP's views like that. At least this response is more of a direct counter-opinion.


Indeed!

Intelligent machine learning is 5 to 10 years off, just as it has been since the early 1980s.


Well put. I wasn't even born then, but things have changed so little that the first thing thing that comes to mind is that I really need to see Tron someday. I doubt computers will ever learn to talk humna, rather, as we all see more and more every day, humans are going to have to learn to talk computer.


I was born in 1973 and by the 1980s a lot of people were convinced intelligent machines and nearly general purpose AI was just around the corner. There was tons of university and military funding going into projects to achieve this.

Of course, now the idea of that sort of blue-sky research is almost universally dead and the most advanced machine learning will likely be developed at Google as a side effect of much effort put into placing more relevant ads in front of us, which is kind of hilarious.


We will get there. It is just taking a lot longer than initially anticipated. I for one am not looking forward to trans-humanity, but I see it as inevitable.


If most humans eventually learn to talk computer, what's the difference between "human" and "computer" language?


The primary challenge is the stated example is speech and NLP. It's not building the decision trees. We can do that today with off the shelf technology. Hell, it's the kind of thing people love to hack together with their Arduinos.

Microsoft even has a pretty slick thing called on{X} that does some amazing tricks with rules reacting to network events and geolocation tools.

Although I confess that commercializing a system that lets you asynchronously start potentially unmonitored exothermic reactions in your home? Probably hard. :D


There are these wonderful inventions called automatic electric kettles, you should look into them. They seem to be commonplace in Europe, not so much in the US. But seriously, all it takes is a bimetallic strip to turn the kettle off when it reaches the desired temperature.

Eliza ran in 16KB of RAM. The logic flow is easy, all you need is a sufficiently large dictionary.


People who think interactive decision tree building is hard generally haven't done even basic research in the most elementary AI techniques.

It is not hard.

The real barrier to something like that is in finding the right way to infer context for speech recognition and natural language parsing. You could easily bolt a system like that onto IFTTT and Twilio and get good results right out of the gate.

Free idea for you, there. Go make a startup.


> I can't ask my computer to make me a cup of coffee, and it responds.

Question: what exactly would you like your computer--that beige box under your desk--to do, if you ask (or rather, command) it do do that? This sounds more like an embodied intelligence/robotics problem than a problem with constraint-solving goal-directed AI, per se.

Unless you want it to use Mechanical Turk to fire off a task to have a human make you a coffee. I could see that working, but it's probably not what you'd expect to happen.


If you truly believe that we're all missing the point, how do you propose it be fixed? How far down the stack do we need to start disassembling parts and rebuilding? And who, if anyone, is doing that now?


> [Callbacks] do not in any way inhibit the ability to analyse or prove correct a program

It's obvious to me that they do: humans are not that good at keeping track of these fragmented chains of flow control. They're just more balls in the air for the juggler.

In building systems, we are using many layers of abstractions all the time (or "syntax sugar" as you say).


The answer to that is obvious. At least a handful of people are expert jugglers - in fact, these people are so good at it, they've never actually been good at handling one thing at a time - they constantly juggle everything, and have been doing so for decades. They've just been dying for the world to make juggling mandatory, because now they can truly shine.

All the other 99% of us need to do is take juggling classes for the next several years and reorient all of our work around juggling. Just quit doing things the way you've done (successfully) for the past several years. Problem solved.


    Computing is in a sad state today, but not because of
    fucking callbacks. It's in a sad state today because 
    we're still thinking about problems on this level at all.
The unfortunate part about thinking about problems is it that problems are relative - i.e. they are problems only relative to a context. Nobody even in the tech world has a clue how solving a simple problem just to satisfy some intellectual curiosity can affect life on the planet. What meaning does Wiles' proof of Fermat's last theorem have on the part of the world that's sleeping hungry every night?

The stuff we're talking about here has irked some pretty decent brains enough for them to go out and make ... another textual programming language.

Folks like Musk are an inspiration indeed for boldly thinking at the high level they do and seeing it through. However, that's exactly what's easy for press to write up on. Stuff like the moon speech is what's easy to communicate. It is also simple to attribute some great deed to one of these great "visions". But behind each such vision lies the incremental work of millions - each of it, I say, worth every penny - unsung heroes, all of them.

On a related point, not all the time where we've heard a call to greater action do we see the ability in those folks to imagine how that might come about.

edit: typos.


> The unfortunate part about thinking about problems is it that problems are relative - i.e. they are problems only relative to a context. Nobody even in the tech world has a clue how solving a simple problem just to satisfy some intellectual curiosity can affect life on the planet. What meaning does Wiles' proof of Fermat's last theorem have on the part of the world that's sleeping hungry every night?

Like the city you live in right now?

> Folks like Musk are an inspiration indeed for boldly thinking at the high level they do and seeing it through.

"Folks like Musk" are smart enough to know that the progress of humanity, the great and meaningful leaps of technology to strive for, are accomplished by people solving those hard problems under a common banner.

A consequence you've evidently yet to learn, or are afraid to realize.


> "Folks like Musk" are smart enough to know that the progress of humanity, the great and meaningful leaps of technology to strive for, are accomplished by people solving those hard problems under a common banner.

That's an interesting aspect of their work you point out. It made me think about describing my "banner" and add it to my HN profile.

It would be cool if we can all identify such "banners" under which we work in our HN profiles. Note that the banner doesn't have to be a technological breakthrough. In my case, for example, I wish for cultural impact.


> It would be cool if we can all identify such "banners"

These days these "banners" are LLCs or C-corps. :)


Though true, the "banners" aren't always obvious. Some folks may have a complementary mission. The golang authors are perhaps not directly working to "organize the world's information", for instance and their banner would read "making it easy to build simple, reliable, and efficient software".

It is still, a nice exercise to do it though ... and I might turn mine into an LLC anyway :)


> Furthermore a function call has a vastly restricted set of valid behaviours: at some point it must terminate, and prior to termination, update the system state (stack) to include its return value.

Hooray! We've solved the halting problem. Without formal analysis, it's not possible to show (in general) that a function will terminate.

More seriously, I've seen function behavior that's just as bad as other types of flow control (including gotos).


Most structured programming ignores this problem as unless you assume some best effort from the programmer, all programming is futile.

You don't exercise your Big-O thinking here, you exercise your Big-Theta.


You're looking in the wrong places. Nobody thinks we're close to building a computer you mom can talk to, but we are getting closer to understanding what computer languages are and their properties. On that end research is blistering and exciting.

But the industry has no need for that stuff.


My Mom prefers to do her development in C, because she needs to make sure her search algorithms are very high performance. So it seems she already has a pretty good language for her use cases?


>incredibly fancy syntax sugar In JavaScript yes but the way python handles async stuff is not 'fancy syntax sugar' it is actually very clever - I'm not saying Javascript should do things the way python does, they are fundamentally different and I actually quite like callbacks in javascript but I think you should be clear that your comments are made in relation to Javascript (assuming they were).


I agree with you.

It occurs to me that if you're using a whole bunch of anonymous functions with nested lexical closures, then the shared state between them starts looking a lot like global variables.

And globals are evil, but relative to the other nightmares that Gotos could produce, they're pretty mild.


That's not callback's fault, but closure's. Callback's need not have shared state.


So true: `Node.js wasn't an innovation. It was shit that was solved 60 years ago.`

We're just reinventing the same thing because some sh*ty stuff got momentum. Take websockets, how is now a webserver better then IRCD hacked with an implmentation of fcgi?

PS: vim is awesome :)



I don't like callbacks but I don't have anything to add to the discussion except this irony:

Do you want callbacks? why instead of writing var a = 1 + 2; don't like to write:

var a; sum(1,2, function(result) { a = result; });


Thank you for providing some sanity to this callbacks inquisition. I am getting so tired of all the "callbacks are evil, js is evil" crusade. Sorry your pet language isn't supported by every browser ever.


> a freaking text editor (something that was solved LONG ago)

It wasn't solved, it was replaced by a different, and worse, set of problems. Other people having minds organized differently from yours isn't evidence of a usability problem.

The problem with programming isn't the syntax. The problem is teaching a Martian to smoke a cigarette: Giving instructions to something which shares no cultural background with you and has no common sense.

(Of course, the classic 'teach a Martian to smoke a cigarette' problem is rigged, because the students are never told what instructions the Martian can follow; therefore, anything they say will be misinterpreted in various creative ways depending on how funny the teacher is. On the other hand, playing the game fairly by giving the students a list of things the Martian knows how to do would reduce the thought experiment to a tedious engineering problem.)

> Go and build a programming language your Mom can use.

This sexism is worse than anything else in your barely-thought-through rant of a post. The blatant, unexamined sexism in this statement keeps fully half of the potential programmers from picking up a damn keyboard and trying.


> This sexism is worse than anything else in your barely-thought-through rant of a post. The blatant, unexamined sexism in this statement keeps fully half of the potential programmers from picking up a damn keyboard and trying.

I'd throw in 'ageism' in there as well, being someone who is now 'of a certain age'.


I think you both are overreacting. "Something your mom could use" could easily be translated into "something a typical person could use." I don't think any girls or retired folk were dissuaded from programming by this comment.


Dissuaded by that single comment, probably not, but a constant flood of low-level sexual bias does send a message to girls that they don't belong. If you follow some of the gender-flipping reactions to pop culture, it begins to strike you just how pervasive these messages are.


The alternative "build a programming language that Dad can use" would have been much worse. I hope you see that. Unless gender bias is removed completely from our language (build a PL that your parent can use), the OP actually selected the lesser of two evils.


Wasn't aware that it was a strict dichotomy. There are plenty of ways to get the point across without gendering it. And I'm not trying to bash the OP at all. It's just that I do think it's important that we all hold each other accountable. But thanks for engaging, a lot of people don't even recognize there being a problem in the first place.


"Something your dad / grandfather / uncle" can use is just as apt. The point isn't "durr hurr you need a Y chromosome to do turing complete mathematics" its "99.99% of the population can't program. Write a language they can use"


That would offend even more as it strongly implies that "mom could never program of course, but maybe we could build a language where dad could?" If we want to be all PC about it, we should use gender neutral terms; like "something your parent figure could write programs with."

Crazy.


This is so absurd. There is absolutely nothing sexist about this statement unless you WANT it to be sexist. If a woman said this, would it be sexist? No. But a man saying it would make it so. What if a female programmer said "something your Dad can use"? Would she be sexist and ageist also?

Probably not. I'm a left-of-center guy, but I'm so sick of this politically correct bullshit coming out of left field. The complete inability to understand that not everyone is thinking of the broad, social contexts of oppression everytime they utter a statement.

You KNOW WHAT HE MEANT when he said that statement, but in the typical, annoying trait unique to yuppie white assholes, you seek to distance yourself from a heritage of being an oppressor by constantly pointing out racism/sexism/ageism. It's a game, and it doesn't matter whether what you are pointing at is REAL, AND AFFECTS SOMEONE, it just matters that you score your points to prove to the professional victim class that you're not one of the bad guys, even though you look like one.

Please, feel free to protest the JIF peanut butter slogan: "Choosy moms choose JIF". But no, there isn't an organized professional victims organization around single dads, so nobody will be protesting that, because why would you? There are no points to score.

Newsflash: Your parents and grandparents were probably racist bigots like every other cracker in this country. Your game does nothing to help. A woman who is being denied a promotion at work because of her gender will get zero help from your bullshit game. She will instead be hurt by it, because of the "crying wolf" that idiots like you do for your game that causes eye-rolling in 95% of the population.


but in the typical, annoying trait unique to yuppie white assholes, you seek to distance yourself from a heritage of being an oppressor by constantly pointing out racism/sexism/ageism

It seems to be a common tactic to accuse someone of white-guilt to quiet them down.


Fuck you and anyone who uses the concept of social justice as a hammer to try and silence people they disagree with. That is another serious barrier to actually solving any of these problems.


Social justice as a hammer is exactly the idiocy I was pointing at. Nitpicking a comment taken out of context as an affront to an oppressed group is exactly that.


I have a project in Go which is pretty heavily functional. Over time, I've begun to notice the general goodness of his 2 rules. If you didn't RTFA, they were

1. Avoid nontrivial anonymous functions.

2. Function declarations after the code that actually does things.

I wonder if his advice is good for all functional-style code in non-functional languages.


>In this refactored code, we've only reduced the maximum nesting by 1 - from 8 to 7. But consider how much easier it is to follow.

Are you fucking kidding me?


Nesting is not the problem, you see what you're doing (when you perform an async io request). And you still have the "if (e) return cb(e)" line... And on top of that, it does not look better.


I guess it comes down to preferences in readability.

Personally I agree that not putting too much logic in anonymous functions makes it more readable. I write my own code that way, but some might disagree.

I don't care about whether functions are above or below where they are used, as long as it is consistent - but event then it is a minor thing. And since it isn't enforced by the language (e.g. like in Clojure where you have to def the var/function above the reference) it is going to vary across code bases anyway.

I don't really think either solves callbacks, nor that callbacks is a problem to be solved. Most comes down to pros and cons seen through the filter of your personal preferences.

Sure you can use a language that compiles to javascript. Does the pros outweigh the cons for you? Great, use it/too bad move right along.

Not everything is right or wrong, good or bad - in fact most things aren't.


Thing is that it actually matters in JS because declaring a function this way triggers the hoisting mechanism, so named functions are in scope within the entire function in which it is declared, regardless of block nesting.

If one isn't careful, and because JS doesn't have the visual delineator of blocks, instead all these rules you have to memorize, this could easily could lead to bugs, especially in large code-bases.

I think it's good to rely on closures to keep from polluting your scopes and to not use uneccesary function names, which is going to lead to one giant imperative statement (the one with the sequence of commands for the computer to perform) rather than a lot of little functions that you can pass around. Secrets of the JS ninja goes really in depth about this.


There is no such thing as an actual hoisting mechanism. Yes, declared functions are available anywhere in the surrounding scope, just like in any other language. It would be ridiculous if for example you couldn't call a method in Java inside another method unless it was syntactically placed before the target.

Global scope pollution is completely irrelevant here - declarations are available in the surrounding scope, not global. If the surrounding scope is global (which it never is in node.js), then that's your problem.


I don't think that's true about the hoisting mechanism. The javascript interpreter looks ahead to find all the var and functions and hoists them to the top of the function. I don't how know Java deals with forward-references, it was already considered passe before I even got into this stuff.

Also, where did I write anything about global scope? I said it was good to rely on closures to keep from polluting your scopes, and that there was no need for with unnecessary function names. You're literally belittling me out on things I didn't even write. Am I upvote-target practice here or something?


In the ML family of languages (e.g. OCaml, F#) declarations are only visible to code that occurs syntactically later. This has obvious disadvantages, but also the benefit of making dependencies obvious.


Yes I fell into the "all languages" trap. Maybe "most mainstream languages" is more in line with what I meant. :P


    test()

    def test():
        print "Hello."


An operation queue where dependencies can be set is also very relevant. For example, if you have to do 2 different json requests, followed by a render - but they are mutually exclusive, you can do a 2->1 dependency. If they do depend on each other's result, you can do a 1->1->1

I like this async syntax but I'm more fond of native JS generator trampoline. I think the JSOperation with dependencies could even use generators in its implementation.

All these solutions coming about after the C# async article are very enlightening to this problem of the synchronic vs diachronic vs asynchronic.

The JSOperations lib is in alpha but can be found here https://github.com/iandrewfuchs/JSOperations/blob/master/REA...


Thanks for this OP. I use these 2 rules in my every day JS because where I work, readability wins every time. The anon function / indentation code format is something I've never really tried to "pick up" when I migrated over to javascript. Its ugly and takes longer for someone outside of my head to figure out what is going on in my code. Verbose function names help the other 4 guys on my team know exactly which function is handling the different scenarios of an async call. Its much more important that the developers are able to work efficiently than it is to save a few characters per function or a few lines per callback.

With the state of JS compilers/minifiers and gzip, I really feel that that particular argument can't even be made in most JS situations.


I like Andrew's rules here, and reading over one of my recent node.js projects, I can see how they would have helped.

My code becomes cleaner and more maintainable when I am forced to name each chunk of 5-10 lines by putting them into separate functions/methods with defined inputs and outputs. This compartmentalization is naturally encouraged by java, C#, ruby, and python, but Javascript actually seems to discourage it.

The ready availability of anonymous callbacks in JS tends to allow for "function creep", where you blink twice and suddenly you have 50 tangled lines of code without obvious control flow or separation of concerns; sticking to named functions seems like it could definitely help.


I agree with the OP. Its all about how you structure and name your code to avoid deeply nested callbacks. With the use of Prototype constructs and decent directory structures you can keep code clean and concise and completely avoid callback hell.

Take a look at the following users repos to see exactly how clean javascript code can be.

https://github.com/learnboost https://github.com/component https://github.com/visionmedia


This is essentially how I have been writing JavaScript and Node.js code for some time. I actually flatten the code even more, like in these examples[0].

For some people, callbacks take more cognition than other solutions, but for someone who has been using callbacks through and through, this is a natural progression and makes sense. It is more straightforward and you don't have to understand further abstractions like in a compile to JS language or with promises.

[0]: https://gist.github.com/chapel/6255720


I think I've done pretty much exactly the same thing:

https://gist.github.com/radiosilence/2fd82aaf3b1721448ae1

Except your function names make more sense.


I never thought of using nested functions like that. It does make the code much more readable, and using 100% JS is a big win. Not as pretty as yield/await, but much more debuggable and compatible.


I think this is a step backward toward procedural programming (needing to name each function, ignoring the hoisting mechanism that happens when you declare a function that way) and then took another step back by putting it all in one huge imperative statement at the top). Probably going to lead to bugs if one isn't careful with the function scope. The best part about JS is that you can be use a functional, declarative style, which is awesome.


The main problem I see with callbacks is that people forget that deeply nested structures are too cumbersome to hold in one's brains. The OP rules to break down things in smaller, named function is a solution to a more general problem: don't nest things if you can avoid to.

Callbacks aren't the problem. People using them are the problem. Keep your logic as flat as possible and callback-hell will dissolves before your eyes.

Programming dogma is so annoying.


"Coffee-Script actually prohibits this kind of code organization, because all functions are necessarily assignments"

How about this then? I haven't tested. Perhaps functions need to be in a different order, but we only have one level of nesting:

https://gist.github.com/radiosilence/2fd82aaf3b1721448ae1


You don't need any nesting. All those methods could be declared at the same level, though that entails passing more params to them since you can't rely on closures. That's probably how you'd divide your synchronous code into methods anyway, so you might as well do the same with your async code.


This dude fails the understanding-LiveScript test. Next time you bash something, read up about it, okay? 'backcalls' are purely syntactical -- they're callbacks, but with a special syntax so you don't have to indent.


That's what I thought, but someone claimed otherwise: https://news.ycombinator.com/item?id=6227958

Instead of investigating and finding out for sure, I communicated my incomplete knowledge without context. You're right - I should have investigated and then communicated the correct knowledge. Any LiveScript bashing was unintentional.


This seems misled response to misled article.


callbacks are also known as continuations which are known as generalized goto.


No, they are not.




Applications are open for YC Winter 2021

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: