Hacker News new | past | comments | ask | show | jobs | submit login
What Color Is Your Function? (stuffwithstuff.com)
410 points by jashkenas on Feb 2, 2015 | hide | past | favorite | 143 comments

We can massively generalize this by calling "blue" "pure" and "red" "impure". The end result is essentially Haskell (but you can take it much further, too!).


There's something horrifyingly restrictive about having merely (blue -> pure, red -> impure), though. All "blue" functions behave roughly the same (e.g., very, very nicely and compositionally) but "red" functions come in many classes: async, stateful, exception-throwing, non-deterministic, CPS'd, IO-effectful, combinations of the prior.

What we want is a nice way of representing different kinds of red functions and how they all interact.

What we'd also like is a nice way of composing different kinds of red functions so that the bundled piece has a sensible kind of redness too and we can keep composing.

And this is exactly monads and monad transformers.

There are other ways to achieve this end as well all under the general name "Effect Typing". Very cool stuff.

But what I'd like to emphasize is that Java/C#/Go have not solved this larger problem. They each introduce a fixed number of "rednesses" and very specific ways that different kinds of red can be connected together. Monads are a generalizable solution.

The situation is exactly the same as HOFs themselves. We've had subroutines and functions for a long time, but first-order and higher-order functions are a marked increase in power since you can now refer to these "function things" directly.

Monads take the notion of "CPS Transform" and allow you to refer to it directly, to mark sections of code which use it and compose them intelligently. They allow you to invent your own kinds of redness on the fly and ensure that your custom colorations compose just as nicely as the built-in ones.

If this article is even slightly interesting then you owe it to yourself to learn much more about effect typing. It'll change your world.

(That and sum types, because nobody is served by a bunch of integers named Token.LEFT_BRACKET.)

Effect Typing and Monads are yet another colour for the lipstick you put on the pig. You still have non-composability (i.e. the async monad spreads through your code), you still have the complexity, you still don't have proper tracebacks without special runtime support, and you still have the inefficiency that Bob didn't actually mention (unwinding all those stacks all the time).

Languages with threads (Java, C#) don't quite solve the problem as their threads have other disadvantages, e.g. creating too many threads causing contention, though it's much less of a problem than people make it to be.

Go actually completely removes the problem - you don't get a nice tool to pack the problem into boxes more easily, you don't get a different fancy colour for your lipstick, you just don't have the problem at all - you wouldn't know it existed was it not for those other languages.

There might be a deeper lesson here about good, pragmatic engineering vs awesome computer language geekery.

Go's "good, pragmatic engineering" solution to the problem - lightweight threads, channels, and the communicating sequential processes model - is built on the back of decades of academic research, including programming languages research.

And now for some computer language geekery:

Go solves one "code color" problem (concurrency/asynchrony), by hard-coding the solution into the language. Go programs are written in a particular monad (the "Go" monad, if you will), and so have one color. Indeed, monads originally arose in the study of the semantics of imperative languages; to be imperative is to be inside a particular monad. This is exactly the "async monad spreading through your code" problem taken to its logical conclusion: just put everything in the same monad!

This hard-coding solution has very concrete advantages. For one, you don't need to think about separate colors and how to transition between them, combine them, define them, etc. (IOW, monads are complex; by picking one, you reduce cognitive load.) For another, it's easier to optimize your implementation and tooling for that monad (lightweight threads, useful tracebacks).

But PL academics aren't satisfied with this solution, because it only solves one color problem, and it solves it non-composably: you can't put the solution in a library, you have to invent a whole new language for it. That's the holy grail (some) PL research is chasing. It's a question of concrete benefits in the here-and-now versus abstract benefits in the hopeful-future. But it pays to remember that today's concrete benefits are yesterday's green-field research.

Indeed, Go only solves one problem by baking it into the language/runtime, it doesn't have a generic mechanism to solve similar problems.

But my point is that it solves the problem much better than monads do - in particular if you don't just look at the syntax/user interface level, but at the non-functional aspects, like debuggability, performance, etc.

I remember a time when Aspect Oriented Programming was all the rage, with point cuts etc to capture cross-cutting aspects of your program. AOP is certainly less generic than monads, but even with rather specialized tools, it turns out the two "killer apps" of AOP were much better served by runtime support in the VM (logging/debugging) and explicit code (transactions).

I'm skeptical about the holy grail that you describe. I see how it's attractive, but in my work experience, the pragmatics of solving a particular issue often turn out to be harder (sometimes much) than the theoretical aspects.

There's certainly influence from more theoretical research in all we do in CS, but for example with Go, it's only based on Hoare's CSP model in the most abstract sense, and its implementation of lightweight threads, its stack model etc are definitely in the domain of clever engineering, not breakthrough research.

"AOP is certainly less generic than monads, but even with rather specialized tools, it turns out the two "killer apps" of AOP were much better served by runtime support in the VM (logging/debugging) and explicit code (transactions)."

Indeed... beware the tech promoted with the same example over and over again. That's a huge red flag.

> This hard-coding solution has very concrete advantages.

True, but you missed the major disadvantage of doing everything in a monad: there may be horrible side-effect demons lurking in every line of code!

In my experience, the most egregious offenders are shared mutable state (AKA multithreading) and exceptions. Introducing multithreading or exceptions into an application which previously shunned them feels like switching to a whole different language; one where all our intuitions and guarantees can be undermined at any time.

This assumes that the primary problem is merely asynchronous code. If that's the main problem you're facing then there's no reason in the world to believe that having more distinction at no gain is the right tradeoff.

I'm not claiming that you must program in a language which distinguishes lipstick colors. I'm merely claiming that you can and that it's a valuable tool.

To respond to your concrete criticisms:

The non-composability you refer to is actually what I call composability itself. You want the isolation of effectful worlds so that you have different things to choose to compose in the first place. You can do this "horizontally" with binding or "vertically" with free products or lifting (or any number of other techniques). Without this distinction things "compose automatically", but only because you've decided to mix all of your colors and go with brown.

Proper tracebacks and stack unwinding can be a problem, but fundamentally these are problems of how you compile/interpret languages like this instead of semantically whether these languages allow you to express what you would like to at high fidelity. Both of those problems can be addressed, though. The degree to which they're painful in practice is also a question.

The comment about threads is somewhat off topic. There's nothing in what I was saying which precludes the introduction of nice green threads and competitive scheduling. You can implement Go in Haskell quite neatly, for instance.

But yes, I agree. If your problem is pure multithreading then languages like Go and Erlang make it go away completely. They do so by collapsing layers of semantics down to a single one (completely antithetical to the red/blue distinction here) but choosing one such that multithreading is nice.

This can be an excellent spot in solution space for many problems. I would never suggest otherwise.

But as long as we're ostensibly on the topic of "blue/red"-ness of your language then it's high time to talk about effect typing.

> Effect Typing and Monads are yet another colour for the lipstick you put on the pig.

When I started to read the essay, I thought that it would be an anti-typing screed (it's nothing to do with the author, whom I don't know!), with blue functions being untyped (or, pace Harper, unityped) and red functions being typed. An untyped function can call a typed function by ignoring the types; a typed function cannot call an untyped one, because that might violate its contract. If you buy this analogy, then your first sentence might equally well have said "typing is yet another colour for the lipstick you put on the pig"—but lightweight HM inference shows that is not so. Maybe we're just awaiting the revolutions in effect typing that languages like ML and derivatives, and Haskell, brought to typing itself?

Actually, you can call both ways - from statically typed to dynamically typed and back. This is the premise of the work done in Racket on contracts, and more generally of the "gradual typing" movement. Harper's "unityped" term suggests exactly how this can work. An untyped function takes arguments that are all of one type, and returns one of the same type (that type being the "dynamic" type that includes every possible value, and thus is useless for static checking).

To call an untyped function from typed code, simply cast all your arguments to this type (trivial, since it contains anything), and check its return value has the type you expect (throwing an error if it's of the wrong type). To call a statically typed function from untyped code, simply check the types of all the arguments you wish to pass to it (throwing an error if they don't match its signature), and cast the returned value to the dynamic type (again, trivial).

I'm glossing over a very important difficulty, which is that you can't dynamically check something of a function type -- if I'm handed a function, how do I know at runtime whether it's of type `int -> int`? I can't! So the solution is to "wrap" this function with a contract that delays checking until the function is actually called, at which point you can check the argument is an `int` and the result is an `int`.

> To call an untyped function from typed code, simply cast all your arguments to this type (trivial, since it contains anything), and check its return value has the type you expect (throwing an error if it's of the wrong type).

To me, as soon as a function includes dynamic checks, it ceases to be statically typed; so I would say that a function that operates this way becomes only dynamically typed.

Do you consider any function which may raise an error to be "dynamically typed", then? (For example, any function which performs an array index; or any function which performs I/O; or, to be absurd, any function which allocates memory (since memory is a finite resource that may run out).) If so, I understand the distinction you're making, but it's a very strong position; most people do not mean that when they say "statically typed". If not, I don't understand the distinction you're making.

Admittedly, gradual typing does not mean that calling into untyped code from typed code is "safe" from runtime type errors (although it is safe in the technical preservation-and-progress sense). But it gives you a partial guarantee: if a type error occurs, you know that the fault does not lie in the typed code. This aids debugging, and helps you port over existing codebases gradually. So it's still quite useful.

> Do you consider any function which may raise an error to be "dynamically typed", then? (For example, any function which performs an array index; or any function which performs I/O; or, to be absurd, any function which allocates memory (since memory is a finite resource that may run out).)

That's a good question! Yes, in an idealised sense, I do; but, in practice, I think that I would make an exception for functions where the error comes from the host system, not from the program. (For example, if I were too strict about it, then I would have to disallow even the safest of Haskell functions because it might be executed on a computer whose hard drive is failing.) Thus, I would call Haskell's `head` function dynamically typed.

> Admittedly, gradual typing does not mean that calling into untyped code from typed code is "safe" from runtime type errors (although it is safe in the technical preservation-and-progress sense).

I think that I don't understand the definition. Isn't encountering a runtime error precisely getting 'stuck' (in the "opposite-of-progress" sense)?

> I think that I don't understand the definition. Isn't encountering a runtime error precisely getting 'stuck' (in the "opposite-of-progress" sense)?

Sorry, perhaps I should have said "exception" rather than "error". Raising an exception is absolutely "safe"/progress-obeying, as long as your language specifies the behavior of exceptions. That the exception happens to be called a "TypeError" doesn't change anything.

Getting "stuck" means being in a state such that there's no defined next state - a state which the language spec doesn't cover. So most "dynamically-typed" languages are in fact safe in progress-and-preservation terms, because they have well-defined behavior in all cases. C, otoh, is "statically-typed" but unsafe, because you can write C code with undefined behavior.

(That dynamically typed languages can be (statically) typesafe should be unsurprising, since they're just statically-unityped languages. Nothing says a unityped language can't be typesafe, it's just a very uninteresting property!)

Total agreement generally, but I wanted to mention that the Any -> X downcasting is trouble sometimes. It is difficult to do this in a type-safe way (e.g. without accidentally introducing holes) and if you have typecasing universally then you lose parametricity, one of the most powerful tools in any polymorphic type theory.

(Late to the party; sorry)

> you can't dynamically check something of a function type

Another approach, besides contracts, (one that my lab is working on) relies on whole program type-checking and path-sensitive program analysis.

Go seems to have solved the IO problem, but what about the general asynchronous issue? How do you handle calling a function several thousand times on dozens of threads and combine the results?

How does Go solve "the IO problem"? (Probably what I want to ask is: which IO problem does Go solve? It can't be the one about purity and side-effects, and if it doesn't solve that...)

How do you do IO without stalling your whole program while it accesses the data, specifically the fact that the asynchronous call is terminated in your main loop, not where you are accessing the IO itself.

Is there any reason you couldn't just spin up so goroutines to do this?

Telling a hundred threads to do work is easy, figuring out how to sync all of the answers is the hard part.

I am not saying you can't pull it off, but the proposed solution doesn't simplify the complex part of those operations.

> We can massively generalize this by calling "blue" "pure" and "red" "impure". The end result is essentially Haskell (but you can take it much further, too!).

Yes, I wish I'd taken the time to work more Haskell into the post but this thing has been marinating on my hard drive for months and I wanted to just get it done before the Superbowl ended and the wife and kids got home.

> (That and sum types, because nobody is served by a bunch of integers named Token.LEFT_BRACKET.)

What said it was an integer? ;)

> What said it was an integer? ;)

Ha, well, totally fair. But, as long as this watercooler language is essentially "Javascript++", I'm not going to let it get away ensuring sum types are bolted on :)

Unfortunately, monads are a bad way of encapsulating and controlling effects. Effects are the most important -- and most bug prone -- of any interesting software aside from compilers, and monads take this central element and make it un-composable. This is why I don't like the current advanced type systems: they help most where they're least needed (the relatively bug-free portions of the code, namely data transformations), and don't help -- interfere, even -- where they're most needed. Effect systems -- if we can get them right and easy to use -- will be a great help. In the meantime, some of the most production-ready effect systems (though they're not general, but have to be built ad-hoc for a specific use) can be found in Java 8's pluggable type systems provided by the Checker framework[1]. It's got a rudimentary lock-use effect system and a GUI effect system (plus some other cool type systems, like physical units, immutability, security and even a linear type system for aliasing).

[1]: http://types.cs.washington.edu/checker-framework/current/che...

> Effects are the most important -- and most bug prone -- of any interesting software aside from compilers, and monads take this central element and make it un-composable.

Can you elaborate on this? Because, AFAICT, the big selling point for monads for dealing with effects is precisely composability.

I think the point is that there is no good effect-union type; this has to be pieced together with monad transformers where you might have that types "Atransformer (Btransformer C)" is not equal to "Btransformer (Atransformer C)".

This is, for example, why Haskell has one monolithic "IO" monad instead of one for hitting the filesystem, one for HTTP requests, one for IORefs, etc. Haskell does not, for example, represent the difference between program nondeterminism (reading a file and doing stuff differently based on its contents) and permanent side-effects on the file system (writing a file); and I'd say the reason that it doesn't do that is because NondeterministicT (FilesystemAlteredT Identity) x would be different from FilesystemAlteredT (NondeterministicT Identity) x.

In some ways the granularity is nice because those two types are not isomorphic; but you might want to "erase" the granularity, which is not possible in Haskell. In terms of the "unioning" that you want, it's something like a dual of the unions that appear in typed exceptions: once you trace through some code you find out that it might fail via `FileNotFound` or `HTTP404`; this means that the relevant effects are `FileSystem :+: HTTP` in some sense.

I know there are a lot of solutions to this, but I've personally found the `mtl` mechanism in Haskell to be completely sufficient. I describe whatever effect languages I want my application bits to be able to operate over as separate type classes and then build one or more interpreters of these classes to run.

Essentially it's mtl + Oleg's "Finally Tagless" mechanism [0] [1].

The problem with this mechanism is not felt by the application writer but instead the library writer. It forces you to instantiate (n x m) combinations of effects for your (concrete instantiators) x (effectful languages). For the core classes mtl bears this well, but it might explain why this technique is not as well represented in written libraries.

That said, as an application writer the number of concrete instantiators is often between 1 and 3 so the pain is very reasonable.

(Edit: I should add that there is another downside in the mtl method in that authors of these finally-tagless code fragments cannot choose priority order of effects---the interpreter is free to decide this on their own. There's nothing wrong with this as long as you, as interpreter writer, consider the burden of correctness. You should be anyway, though, so... It ends up being a minor pain point, imo.)

    [0] http://hackage.haskell.org/package/mtl
    [1] http://okmij.org/ftp/tagless-final/

It took me a while to start to grok the "Finally Tagless" thing; thanks for that!

I'm kind of interested in knowing why you don't think that there isn't an isomorphism along the ordering of the monad transformers. Wouldn't it just be a question of rebuilding combinators so that they're targetting the right depth? Or am I missing something

I'll demonstrate the lack of an isomorphism via a counterexample.

Consider MaybeT and WriterT (and the basic Identity monad for the bottom of our stack)

    newtype MaybeT    m a = MaybeT   { runMaybeT   :: m (Maybe a) }
    newtype WriterT w m a = WriterT  { runWriterT  :: m (a, w)    }
    newtype Identity    a = Identity { runIdentity :: a           }
I'm going to claim that MaybeT (WriterT w Identity) is not isomorphic to WriterT w (MaybeT Identity). If we expand the first

    MaybeT (WriterT w Identity) a
    WriterT w Identity (Maybe a)
    Identity (Maybe a, w)
    (Maybe a, w)
And if we expand the second

    WriterT w (MaybeT Identity) a
    MaybeT Identity (a, w)
    Identity (Maybe (a, w))
    Maybe (a, w)
So we can see that the Maybe is wrapped around something different in each stack. So now to the claim that there's no isomorphism should be obvious---any witnesses to the isomorphism f and g would have to have that for all w, f (g (Nothing, w)) = (Nothing, w), but since g :: (Maybe a, w) -> Maybe (a, w) cannot fabricate an `a`, it must be such that g (Nothing, w) = Nothing which means that f :: Maybe (a, w) -> (Maybe a, w) cannot determine the right `w` to return.

Monads themselves are composable via bind, but the values encapsulated by the monad are not. Consider there is no real converse to the return operator: once something has been wrapped by a monad, then it is forever stuck in that monadic world. That can be rather annoying, because in languages such as OCaml (not sure about Haskell &c), monad-like interfaces are a convenient abstraction, yet using them leads to highly idiosyncratic code. Rather similar to the sync/async, red/blue distinction in the original article.

Sure the values are. Any function `a -> b -> c` can be lifted into a function `m a -> m b -> m c` for any Monad. That's (part of) the exact thing that bind buys you.

"Escaping" from a monad is an artifact of a bad intuition. There's no reason for a monad (m a) to "contain" values a and thus no reason to believe that they can be removed.

Even Maybe is a sufficient model of this. I can make a whole big compositional pipeline with all kinds of compositions at the value level, but if my inputs are Nothing then there won't be anything to "pull out" at the end.

Ultimately, "monadic worlds" is a good intuition. Just realize that your monadic world may be a strange one when you pull it back into "your" world---the interpretation may not be obvious, it may demand something of you, and it may not look very much like the "values" you felt like you were playing around with.

Let's take a simple, quotidian example: logging and errors. I write functions that produce logs and may generate errors. If I were using monads, I'd have a log monad and an error monad. For logs, my functions would produce a value and a list of log messages. The monad would combine the list of messages. For errors, my functions return an OK and a value, or an error, and the monad short-circuits errors. Now I write functions that might produce both logs and errors. How do the two monads compose?

Of course there are two ways. The problem is that there is not enough information in the mere product of the two algebras to indicate what proper behavior is.

If you don't mind deferring the definition of "correct" to the interpreter writer then that intentional choice is captured by

    (MonadWriter Log m, MonadExcept m) => m a
If you want more controls then we need to start specifying laws. For instance, the following law distinguishes between the two instances

    throwError e >> m == throwError e
and this now clearly indicates that EitherT e (WriterT Log) is wrong while WriterT Log (Either e) is fine.

So how do we specify what additional laws are needed to cut down the free product of the effectful languages? Dunno, that's a hard one. You could clearly do it using dependent types, but that's still today very expensive.

But given that we literally lack proper information to make a choice it's very important that all of those choices are allowable. It's my position that there is no meaningful default. It's also my position that the `mtl` style free products are perfectly acceptable.

I would even go so far as to suggest that there are applications where we want the other ordering of these two.

Me too!

But in general, effects are not "the values encapsulated by the monad", so the fact that the values are "stuck" within the monad doesn't have any bearing on whether using monads to manage effects makes effects non-composable.

To be fair, I like pluggable type systems a lot. I don't think that privileged type systems are the end-all-be-all.

I disagree a lot that the current effect systems don't prevent errors, however. You're right that linear types are difficult to encode in Haskell at least, but you can get away with a lot of bang for relatively little buck even without them.

The right choice (or choices) in design space here is a big challenge. It'll be exciting to see new research as it develops. I'm especially excited to see what comes of Pfenning's Pi calculus linear types [0].

[0] To be clear, I don't know if they're "his" excepting that he appears to be doing a lot of modern research there and that's where I learned about them.

I'm really out on most of the "async" stuff, after having used it. (Mostly in Node and Tornado)

Remember in the early 90s when Windows and Mac OS were "cooperatively" multitasked? Which is to say, you had to explicitly yield to allow other applications to run (or risk locking up the entire system). And then it was replaced with pre-emptive multitasking, which allowed the scheduler to figure out what process deserved CPU time, while allowing the programmer not to have to think about it. You could call a blocking IO function, and the OS would just go do something else while you waited.

All this "async" stuff seems like a return of cooperative multitasking, only worse. Not only do I have to explicitly yield, but now it's to some event loop that can't even properly use multiple cores, or keep a coherent stack trace. It's a nightmare to debug. It's theoretically fast... except if one request forgets to yield, it can clog up the entire thing. I guess you use multiple processes for that and a dispatcher, but at that point you've basically reinvented preemptive multitasking... badly.

Threads aren't perfect, but excluding STM and actor models they definitely suck the least.

100% agree.

Think of your code as an operating system. And heck if you write anything complicated it will start to look like that.

As you said, if you write code that needs to co-operatively yield, needs to use defereds, promises, futures, you are back to Windows 3.1 Which is nice, when it came out.[+]

The next level in evolution is pre-emptive multi-tasking. A scheduler interrupts your task and make sure to run other task. You don't have to worry about fairness (or another task being unfair). This is your classic threads running in the same shared memory -- C++,Java,C#. In OS equivalent this is Windows 95. It was a glorious OS for its time. It really was, I am not kidding.

Next level in evolution is pre-emptive multi-tasking with isolated memory. Aha, now one broken/evil/greedy task can't easily foul up other tasks. This was huge. Think jumping from Window 95 to NT or to Unix/Linux. Awesome things like multi-user system, reliable systems with multiple processes starting/stopping that could run for more than 3 days without rebooting. This is what makes the world run today in general computing. In programming languages paradigm there is ... (sorry, if this is dissapointing) only Erlang. You can add its friends (Elixir, LFE and few others running on same VM). Yap, the only one I know that has built-in heap isolation and light tasks (a couple of K's of memory). The non-programming language specific paradigm is of course OS processes, with some messaging. Now you can call them micro-services living in containers to be cool and that's fine. But is really nice. You've caught up with OS technology finally.

[+] : One kind of in-between is semi-pre-emptive multi-tasking and that is when pre-emption happens implicitly during certain system calls (usually IO calls like socket.recv() or say disk.write() or time.sleep(). I think Go works like this and Python's eventlet and gevent work that ways.

How does one request "forget to yield"?

I'm mostly thinking Tornado/Python, where the async stuff happened via generators (IE, the "yield" keyword). But that meant there were large chunks of the python standard library that were basically off limits because they blocked and couldn't be used with a generator, so if you used those functions the main event loop would be stuck waiting.

For node, we happen to have a server that calls into a geometric modeler (for collaborative 3d modeling). Since it's doing a lot of math, you could totally conceive that while an expensive modeling operation is running and chewing through CPU cycles, all the other sessions on the system are just waiting. That's kind of a specific use case admittedly, but with threads it wouldn't even be an issue, but with async it's a problem. I get there's ways around it (offload the work to a worker process asynchronously, for instance, which is what we're doing), but it's annoying that it's a thing I have to think about when the functionality is built into the OS.

Some of these cooperatively multithreaded implementations have "green" varieties of all your standard functions; these are greenlet-aware (that is, aware of the cooperative threading & I/O loop that's happening) functions that do things like, for example, sleep. So, you might have a my_green_library.sleep and a calls_the_os.sleep; the latter of which will yield the hardware thread directly to the OS, and block that thread completely until its done. Whereas the former will perform a sort of userland context switch, and note something to the I/O loop, and then sleep until the next event.

Worse, this problem makes composition hard: you need to know the entire implementation of any function you call, in order to be aware of whether or not it will cause the calling thread to block.

So I've been writing javascript full time for a couple years at this point, client, server, and open source, and what I have adopted is coercing everything into promises, which I suppose would be the author's way of saying making everything red.

If you have something that is not async mixed in with something that's async, you can still add it to the promise chain and it will resolve right away. If you have a library that uses callbacks or some other thing, you can just wrap it such that it now uses promises. And then of course you can always look for alternate libraries that use promises from the start and skip step as well.

I've found that using promises for everything works super well. There is no confusion or doubt at all. Everything has the potential to branch into async at any time with no consequences and without complicating the flow. And an additional benefit is that rather than checking for errors after any operation you do, you choose where to check for errors. When a promise rejects, it skips everything else in the chain until it gets to a catch. So rather than running 4 async operations and doing an "if error do this" after each operation kind of deal, you can catch the error in once place and handle it once. Promises surpress the error in the promise chain until you choose to handle it, which is dangerous if you don't understand how promises work, but really useful once you do.

There are really solid promise-based libraries for all common operations in node right now. When.js for general promises, composition, and coercion, rest.js for network requests, bookshelf and knex for database connection and orm stuff, etc. If you are a js developer, give them a shot!

Don't get me wrong, I'm not trying to claim that this is better than any other language-level construct by any means, but if you are working in javascript, where you have to deal with javascript's limitations as a language, from experience I can say that working in an all-promises environment makes things quite pleasant.

This is great for one's own projects, but if creating something for more than one's immediate project (i.e. libraries), it forces everyone else to adopt the same style.

Maybe those other projects are also using other libraries that don't use promises, so now there is a problem. Do you wrap the other library in promises too, if that is even a viable option for you?

Colorness is a problem for the whole ecosystem too.

Promise is the best thing an async library method could return because it's always trivial to convert into anything you want (callback, stream, async/await, yield whatever you want) because it's the only thing that's standardized. However, nobody gets the callback contract (https://gist.github.com/CrabDude/10907185) right. Even node core gets the callback contract wrong in different ways in its different APIs.

In practice however most library callback apis resemble the callback contract enough so that one-line promise-wrapping of the entire library is possible.

Maybe I didn't get this across in my original comment, but my primary job is open source software. So I spend ~40h/week working on public projects that are not all my own. I spend tons of time contributing to other peoples' projects, and I authored and maintain a significant number of libraries that are used by other developers.

What I was saying is that it does work. I can use dependencies with no problems no matter what they expose by coercing to promises internally, and have had no complaints about my libraries' APIs use of promises.

I agree that colorness is a problem, but there are also fantastic libraries that handle easier conversion between most of the common styles. For example, `bluebird` promises allow you to swap between error first callbacks in both directions with `promisify` and `Promise.nodeify`.

I agree. I think that there are two things that are missed out from the this article.

1. Programmers are lazy and will waste cycles like crazy around IO if given a chance. Core library async get's them thinking in red mode but it also get's them actually thinking in ways that allow the overall app to usefully get on with other things. Go sounds like it might be a nice exception to this but I'm not familiar enough to comment.

2. He kinda skips over the fact that threads are really really hard to work with and test (I immediately think about wasting time with race conditions that are just about impossible to reproduce or fix). Node.JS style continuation hell has the advantage of being a model that can actually be understood and made to work reliably in a way that threading, in theory can, but in practice cannot.

I had no clue this was about async functions. I assumed it was safe/unsafe functions until I got to the part about it not being. I think that is a much stronger issue than sync/async.

These kind of distinctions happen all the time. The classic http://www.joelonsoftware.com/articles/Wrong.html - one colour for html-encoded strings, one colour for plain strings. Functions that might fail with an error or functions that never fail. Logging versus non-logging. Database-accessing or not. Callable from user scripts or not.

Keeping track of colour - in a way where correctness is guaranteed by the computer - is such a useful general feature that any serious modern programming language should support it. We should have easy, simple syntax for specifying colour, and should make polychromatic functions almost as easy as monochromatic ones.

When you do that - when the red of "async" is just one colour among a whole rainbow that your program is already tracking - then the cost of making an explicit distinction between async and not is minimal, and the benefits - being able to track, control, and see instantly whether a function is async or not - are more than worth it.

I'm not sure if you're leading this or not, but to be clear... this is exactly a type system.

(Edit: read your blog; it's clear you're leading it. Poe's Law is in effect sometimes here, sorry 'bout that!)

It's a good type system. Many programming languages don't support this kind of type information, notably C.

I'm always curious how far a sufficiently motivated masochist could go with C... I guess it's just C++ ;)

You can write very safe code by using structs (even 1-element structs) for everything. But there's no generics system and very little in the way of safe tools for making one; writing a safe "generics runtime" and macros to use it puts you halfway to writing a new language.

>structs (even 1-element structs) for everything.

Are there any large C projects written in this style?

I don't recall the LOC, but I used this style at a recent position. Basically, the idea is not to use bare primitive types anywhere - instead wrapping them in something semantic. Probably the two biggest wins from this approach were distinguishing types of indexes and (in the particular project) distinguishing price from quantity.

It does add a bit of boilerplate. For the indexes in particular, foo_lookup(foo, idx) is substantially safer than foo[idx.value] - but then you've gotta write the lookup functions (which may just be a macro but is still a bit of cruft). Locally, if using var.value everywhere in math is getting ugly, you can of course pull things out into temporary primitive variables - enforcing things at the boundaries between functions provides almost all of the benefit.

All I'm aware of is a friend's private project of around 20KLOC.

I would love to see even small examples of this.

I intend to write a blog post on this at some point... but I recently colored functions in C by passing an empty struct as the first argument, with a standard name and annotated with __attribute__((unused)). The propagation was manual, but trivially easy, and it proved a major help in refactoring when I had to move some functionality to a differently colored context. I was even able to get some (extremely limited) polymorphism using unions to express "any of these colors". It could even be marginally nicer if C allowed passing one union as another if the first was a subset of the second - which I think would be correct - but really parametric polymorphism would be the bigger win...

The Linux kernel is a good example of just how far you can push C (plus GCC and Sparse extensions), including type system enhancements (__user and __kernel), pseudo-OO (kobject, many structs of function pointers), and other safety features.

I was convinced it was going to be about pure vs impure (monadic) code, but then he just picked out one particular monad instance (coroutines/asynchronous operations).

Any time you have side effects, you potentially have "red" code, in that it can add arbitrary restrictions on how you must use the functions.

I guess more generally, it's about effects. The same rant could apply to any particular effect, because they share the property of being infectious: async, unsafe (in C#, not Rust), throws (checked exceptions), IO, etc.

My question is: In a language that has a first-class effect system, does the red/blue problem disappear? Does being able to generalize over effect allow you to avoid cutting the world in half, and allow you to compose effects easier?

Absolutely, yes.

For instance, here is an abstraction of code which reads and writes

    class Monad m => MonadTeletype m where
      writeLn :: String -> m ()
      readLn  :: m String
Here is one which receives the current time

    class Monad m => MonadNow m where
      now :: m UTCTime
And here is code which transparently combines them

    echoTime :: (MonadNow m, MonadTeletype m) => m ()
    echoTime = do
      line <- readLn 
      t0   <- now
      writeLine (show t0 ++ ": " ++ line)
You then, when actually executing echoTime, have to create a monadic implementation which you prove to instantiate both MonadTeletype and MonadNow. For instance, we can always show that `echoTime` can be satisfied by the Haskell "sin bin" type, IO:

    instance MonadTeletype IO where
      writeLn = putStrLn
      readLn  = getLine

    instance MonadTeletype IO where
      now = getCurrentTime
That said, it's easy to write monadic languages like MonadTeletype and MonadNow which aren't trivially satisfied by IO. This occurs when you've imputed new meaning and language into your monad, which is really cool. IO is the "sum of all evils", but it's not terrifically expressive.

Except that's not an effect systems because you only have one real effect - the IO type. The challenge of effect systems is to describe effects and how they interact. For example, you'll have an effect that says "a lock is obtained" and one that says "a lock is released", and if you call them both, in the right order, in the same function, then that function has an effect of "mutating something under lock".

Sure it is! The effect `(MonadTeletype m, MonadNow m) => m a` is just as real as IO. There's nothing special about IO except that the Haskell RTS knows how to interpret it.

I could, for instance, build a pure interpreter of those effects into, say, a stream transformer or compile it into a different language (though we'd have to do some tricks to expose Haskell's name binding).

If you want your lock obtained/lock released bit then you want indexed monads. It's easy enough to do, but the safety/complexity tradeoff in the Haskell community has landed on the other side of that... probably for more historical reasons than actual technical ones.

> There's nothing special about IO except that the Haskell RTS knows how to interpret it.

But that is what makes it an effect. Sure, you can model effects in a "hosted" language this way, but that's not what we mean by effects.

I disagree. In force, the Haskell RTS is just a "model" of the IO effects as well. In particular, there are at least two such models, the GHCi interpreter and the GHC compiler!

As another example, it'd be completely possible to write your own IO and interpret it in another language. You can read about this on Edward Kmett's blog where he talks about implementing IO for Ermine [0]


Of course, there's something important that makes us want to differentiate "real world" side effects from internal "model effects". Ultimately, from a correctness and reasoning POV, there ought to be no difference. From a practical point of view, some models are more interesting or important than others. But as a compiler writer you're put right back into the same hot seat.

[0] Unfortunately, his blog appears to be OOC right now, so here's a weird chinese mirror!

At first, it reminded me about const-correctness in c++.

A lot of commentors are mentioning that this is just a specific case of effect typing. Haskell and monads have been brought up as an example of effects typing, but I'd like to present another example that more closely resemble familiar static type systems.

Nim[1], at least at one point (I'm looking at the current manual and can't find it documented), had support for tagging functions with a pragma and the compiler would enforce that functions without the pragma can't call function with the pragma outside of a special block. The compiler interpreted certain pragmas like "impure" and "exception" in a special way, outputting a warning when certain language features were used inside functions marked with the pragma. The language manual shows that the compiler still at least supports these special pragmas. It's possible that it never supported custom pragmas and I'm just misremembering.

Interestingly, the author dismisses promises as not a major improvement and calls async/await and generators at least a half-way solution. It turns out that the are just a simple syntactic transform that isn't powerful enough to express everything that coroutines can. Promises, on the other hand, can. Promises are actually a monad: `.then(...)` is the bind function (`>>=`). This is essentially how the IO monad in Haskell works.

[1]: https://nim-lang.org/

Also, what about Koka ?

http://koka.codeplex.com/ and http://research.microsoft.com/en-us/projects/koka/

From the latter:

> The Koka project tries to see if effect inference can be used on larger scale programming. The language is strict (as in ML), but seperates pure values from side effecting computations (as in Haskell). Through the effect types, there is also a strong connection to its denotational semantics, where the effect type of each function has a direct translation to the type signature of the denotational function.

I just wanted to say that I've finally taken a look at Koka after hearing you mention in on HN many times. It's very nice! I appreciate how it provides most of the bang of effect types much more conveniently than one might expect with the explicit monadic structuring going on in Haskell.

Have you looked at Frank [0] by any chance? It has a very similar row-effect type (I guess these were both first explored in Eff?) but in a CBPV language which makes it clear how pure values are separated from effectful computation.

[0] http://homepages.inf.ed.ac.uk/slindley/papers/frankly-draft-...

I'm not versed in this area, but I listen to Daan talk about Koka a lot. You might want to ask him directly about Frank.

> It turns out that the are just a simple syntactic transform that isn't powerful enough to express everything that coroutines can.

Sure, but at least with async/await you can reuse control flow structures from the original language, like for loops and try-catch. With promises you need to reimplement all of those as a library.

You can recurse and branch just fine, you just lose for/while loops.

Recursing can build up very long promise chains if you aren't careful and also just isn't a natural way for many programmers to express many kinds of code.

Even simple branching gets nasty in many cases if, for example, one branch does something async while the other does not.

You also lose try/catch.

Yeah, agreed. It's functional, but technically weak.

This might be a place to introduce functional pipelines like Haskell's Pipes, Machines, Conduits. They solve the effectfull iteration and streaming problem, but the mechanism is heavily transformer based.

The problem is that while you can introduce FP abstractions to structure your programs, they are not the same abstractions that you use in the synchronous (imperative) code. You still end up in a situation where your programs is written in two colors that don't play nice with one another (for example, a "foreach" combinator doesn't work on both sync callbacks and async callbacks)

Custom pragmas in Nim are supported although I'm not sure they do what you're after. You may also be talking about Nim's effect tracking.

Nim also supports async await.

(I can't give you any links as I am on mobile right now)

For those that haven't noticed this article is by the chap who wrote the absolutely wonderful http://gameprogrammingpatterns.com/

It seems like a lot of people are interested in fixing this, and would be keen to see a solution. I believe StratifiedJS is precisely that solution (for JS at least), and it has existed in working form for years: http://stratifiedjs.org/ (it's not just an experiment - it's remarkably stable).

StratidiedJS completely eliminates the sync/async distinction at the syntax level, with the compiler/runtime managing continuations automatically. A `map` function in SJS works just like you would want, regardless of whether the mapping function is synchronous or not.

In addition, it provides _structured_ forms of concurrency at the language level, as well as cancellation (which most async APIs don't support).

Disclosure: I work on StratifiedJS, and just wish more people knew about it.

Looks like exactly what this article is talking about. I have no idea why noone else has commented on it....

How does stratifiedjs work under the hood? Does it switch out the stack?

It compiles down to JavaScript that you can then run on Node or the browser.

I once wrote a big long rant about the mess that JS and Node have made trying to cope with async code and got tons of comments proposing X or Y library that would "fix" the issue. Not a single person mentioned StratifiedJS. I wonder if there was some history to it that prevented it from getting momentum.[0]

[0] http://notes.ericjiang.com/posts/791

I think there are a couple of problems:

1 - SJS effectively 'solves' the concurrency problem, but it is not a problem that is on the top of most people's mind when they write an application. To a first approximation, the concurrency problem in JS looks "solved" to people already (promises, generators, etc), and it is only when you get down to it and look at it in detail you see that SJS is actually a substantially more complete solution to the problem.

2 - Many people see it as a 'cute' solution that doesn't scale to big applications. To counter that point we've developed a complete SJS client/server framework - https://conductance.io - and are writing big complicated apps on it (such as http://magic-angle.com/ ). It's still rough around the edges, but we're pretty confident that the upcoming release (scheduled for end March) will show just how powerful the SJS paradigm is. There is a presentation on it here: http://www.infoq.com/presentations/real-time-app-stratified-...

Disclaimer: I work on SJS!

I don't have anything of substance to add, but author, if you're reading this, I enjoyed your writing style a lot.

Thank you! I know the reader's time is precious so I try to cram as much entertainment and information in there as I can.

Spidermouth the Night Clown will stay with me for years.

At some point, the phrase "Night Clown" popped into my head and it's so deliciously evocative I've had it stuck rattling around in there since then.

Positively "Stross-ian" :-)

(see the exploits of a certain fictional Bob Howard, former programmer)

He also wrote a book. You might find it interesting.


Tornado made async code slightly less painful by using yield and coroutines, but you still have to run blocking methods on thread pools using futures. They abstracted it really nicely and I can now write clean code if I need an occasional blocking library in my tornado code.

But after writing tons of Go over the past 2-3 years, going back to async code, even with the tornado sugar, just feels like driving a manual car after getting used to automatic. It's just redundant. I've seen better, I've written way cleaner code and got better concurrency. Promises, futures, yielded generators - they are all syntactic hacks. The only language I've used that really addresses this properly is Go (disclaimer: I haven't written any Erlang).

Really, any language that has threading gives you the easy concurrency that you want. Now, if you limit your choices to "concurrency, but not with OS threads", then your pool is a lot smaller.

I think Clojure (core.async), Haskell (GHC), and Rust would also give you what you're looking for.

downvoted for this? really?

No actor-model based language has this problem, so perhaps all it comes down to is baking in the right(or even any decent) concurrency support from the start, at the language level.

Of course they have. Well, everything is nice and dandy until your actors never block. As soon as you start blocking, your async model has the same problems as threads have. And you cannot really write real-life systems without blocking actors.

Ask Erlang programmers whether they have dealt with this kind of stuff.

The main process or any parent process can be made to not block by spawning sub-processes that handle an entire group of child processes , so I don't see how that is a problem.

[1] is a nice read regarding continuations (and the Cont monad), though a bit more advanced.

[1] http://blog.sigfpe.com/2008/12/mother-of-all-monads.html

His solutions is threads? Really? Has he read no history? There are problems with threads. That's why async I/O is hot right now. Threads is a limited resource. Threads are expensive to create and dispose. Context switches are espensive. Threads must be synchronized. Threads can have race conditions.

Good rant, but I didn't expect him to serve such a shallow conclusion after a solid and insightful introduction.

Since he's a Go fan, he might prefer lightweight threads running in an event loop rather than real threads with their context-switches. Moreover his concern is syntactic, not semantic: so maybe he'd like something which "looks thread-like" but "complies-to-CPS" too.

Some Microsoft engineers are working on a nice solution to the thread-race-condition problem with a somewhat different approach: pretend your threaded environment is a DVCS. When you want to spawn a new lightweight-thread, think of it as a git clone. It makes its own changes to its own state, then you can eventually pull its changes into the present state -- so you get deterministic threading if you've got a deterministic merge algorithm.


If you are interested in concurrent revisions, you might also be interested in Glitch:


In contrast to Burckhardt et. al's work, Glitch re-executes computations to reach a fixed point (logging side effects so they can be rolled back along the way). This paper compares the various approaches in solving this problem (including concurrent revisions, but also a few others):


This has been a fascinating read, but I worry that there's too much pressure in the paper to view all of these different approaches as somehow "solving the same problem". I'm not sure they do.

There is this "whoops, my state updated out from under me" concurrency problem. To steal a metaphor from physics, the problem is that we expect the spacetime to be "locally flat" but to curve at long scales -- similarly we expect the state to somehow be locally private but globally we discover it's shared. The multiple timelines (of operations in various threads) contain updates to the shared state which are noncommutative; when we synchronize we try to throw up these big walls, global across all timelines, across which operations cannot pass.

Glitch's approach is to break these threads into commutative-and-noncommutative parts (fixed-points and events). So the focus is not actually on the fixed-points; they can be parallelized without fear because the operations on the state commute. The focus is instead on the events. And there it's not clear that the events solve the concurrency problem at all. (Please don't take that as a criticism; I don't think you were trying to solve this problem. Your approach reminds me a little of Sussman's propagators, and definitely it has some nice implications for live-coding.)

Concurrent revisions are a more direct response. The basic insight is that no matter what, "there is one authoritative timeline, let's call it the consumer-timeline, which is how all of these noncommutative events from multiple timelines will actually be ordered." Given that insight, and the need for the state to look locally-private, these explicit joins are an obvious solution. The joins look a little jarring because in the spacetime analogy we're talking about a 'piecewise-smooth' function, so there's a sort of derivative-discontinuity happening here.

What would be really interesting is if the shared commutative operations of Glitch could change "piecewise smooth" to "smooth", but I don't think these ideas have that power.

> That's why async I/O is hot right now

Well, async I/O is good. However, the reason why it's super hot right now on the internetz is because people don't understand that race conditions and synchronization is something you _cannot avoid_. Your program either needs coordination or not. If it does, then there will be some kind of synchronization used. Dependencies and synchronization is not something that only occurs when accessing shared collections. In any condition, where the semantics of the algorithm requires ordering and/or transactionality and that algorithm is somehow distributed(between threads, processes, machines, you name it) and concurrent, then you will end up using lock/barrier/etc. semantics.

No language itself will solve this issue as it stems from the problem you're trying to solve, not from the language constructs.

Also, on a side note: I love what Clojure does with transactions and STM, but it essentially converts a problem into something that is easier to handle in 99% of the cases, and hard to handle in the rest. (Transaction rollbacks, etc.)

Anyway: every single programmer needs to be comfortable with thread-level concurrency, as you'll face the same problems in on higher abstraction levels where it'll be more difficult to recognize them. He's not saying threads are the solution: he's saying that these 'modern' approaches don't make your life as easy as you would think. Abstracted coordination makes simple programs simpler, but if you don't understand the underlying principles then suddenly your ocean of callbacks will fail somewhere, and you won't understand why.

He is talking of threads as an abstraction. Not operating system threads, more like green threads. Different languages call it different things, so I'm fine with the word thread. He even explains that he doesn't mean operating system threads, so your response doesn't make complete sense to me.

I've always wondered why Joyent built Node.js when they already used Erlang. I've tried to research it but there seems to be no articles about it. It's something I would really like to know because it seems like most people's uses for Node.js could easily be done in Erlang, and they were already using Erlang so did it fall short in other important ways? Or Stackless Python if you don't like goofy languages.

Both existed when Node.js first was released. Nowadays Go can fill in many people's use cases too. And I would also argue that green threads are higher level than CPS. So there has to be a reason to use Node.js other than its async stuff.

Well, we weren't using Erlang much (and had written very little, if any). And while it's unclear if it's related to Erlang or not, over time our Erlang-based systems have been ripped out of our stack because of their failure to meet our production needs. In particular: Erlang-based systems are fine when they're fine, but undebuggable when they're not. While JavaScript might not be known for its observability, we have been able to advance the state of the art there considerably[1]; at this point there is no question that node.js is much more observable and debuggable in production than Erlang.

[1] https://www.joyent.com/developers/node/debug

His solution isn't raw OS-level threads. Thread semantics are the solution. It could be greenthreads that all run within the same OS-level thread. Basically, when a function A calls function B, if function B blocks it blocks function A as well. The entire stack blocks on whatever it going on the top frame.

I've wondered that myself. node-fibers basically does this, but for some reason it's pretty hated in the JS world.

I really don't understand this point of the article: I hated the fact that in Java any call might block. To me, this isn't some better abstraction, this is like a language being impure by default: any function can make changes to global state. So actually I preferred seeing the color of the functions - it made the program much more understandable and "safer".

The problem is not so much threads as the current implementations of threads. So instead of inventing your own crappy thread-like continuation passing style to circumvent threads, we could be spending time fixing them. Linux used to have huge scalability problems with threads. Not so much any more, but the situation could still be improved. I find the culture of circumventing instead of improving problematic and endemic.

Threadpools. Pin threads to processors. Hardware synchronization extensions. Avoid shared state between threads and use language-level primitives to enforce appropriate locking when needed (this isn't really much harder than language-level primitives to make CPS easy). IOCP.

Threadpools and innovative synchronization approaches (RCU) are exactly how the Linux kernel handles IO and interrupts internally. They can be unperformant if used poorly, but they don't have to be.

> Has he read no history?

Considering the author is Bob Nystrom, one of the main engineers of Dart, I'd wager the answer is yes.

> one of the main engineers of Dart, I'd wager the answer is yes.

I should point out that I'm on Dart, but I'm pretty low on the totem pole. I just happen to write more publicly than many of my teammates. :)

>Threads is a limited resource. Threads are expensive to create and dispose. Context switches are espensive. ...

Languages like Go with green threads don't have those issues. They execute about the same way languages that expose async IO do.

>Threads is a limited resource. Threads are expensive to create and dispose. Context switches are espensive.

There is absolutely no reason they have to be. If you have an async callback API, you can plaster over it with green threads and they are very cheap, and don't have to involve any new context switches.

>Threads must be synchronized. Threads can have race conditions.

You don't have to allow your threads to run concurrently just because you added threads.

I don't really consider this solved in languages or runtimes that lack green threads. If you want to make 300,000 threads in Lua or Go, go right ahead, but if you port that application to Java you're going to have a bad time.

An orthogonal useful thing that is sometimes not solved in languages with green threads is the ability to copy continuations. If you have call/cc or coroutine.clone, you can e.g. use rollback netcode in your fighting game and store state in execution states, but if you cannot copy execution states, you will have to choose one or the other.

>If you want to make 300,000 threads in Lua or Go, go right ahead, but if you port that application to Java you're going to have a bad time.

In Java itself perhaps, but with Java you can use lightweight threads via Quasar. http://blog.paralleluniverse.co/2013/05/02/quasar-pulsar/

"a single machine can handle millions of them"

A related way of composing code is railway oriented programming already commented on HN (1)

Underneath there are monads, sure, but the author has deliberately chosen a more mechanical metaphor which may help those for whom monads sound too abstract.

The post focuses more on how to handle errors that asynchrony, but it shows well the key steps to lift a blue function into red ones and to compose these constructions.

(1) https://news.ycombinator.com/item?id=7887134

I don't get it ... But hey, it took about six month for me to figure out how to write asynchronous JavaScript ... The key is to not use anonymous functions, it will flatten out the "Christmas tree" of callbacks. And it makes it possible to read what the code does, or at least what the programmer wants the code to do. It's much better then "then", then what, but, then, why complicate things when it's actually possible to be verbose.

It helps syntactically but does nothing about the main problem: composability. Each of your functions has to know which function to call next, instead of just having one function that invokes n other functions in order. That's what next() is meant for.

For composability in JS I use small utility modules like "inOrder":

  function beforeWifeComesHome() {
    function takeTheTrashOut() {
      inOrder([openDoor, goOut, emptyTrash, goIn, closeDoor]);
But it would be easier to let the compiler take care of the async calls and insert callbacks automatically so that you can think synchronously while the program works asynchronously. And if you want to run stuff in parallel you should use child processes.

Funny to consider this alongside Guido's refusal to add full anonymous functions to Python. His argument seems to be "If it's too long for a single-line lambda, then it's long enough to deserve a name"

Guido has a different, legitimate reason to not add multi-line lambdas to Python: they are super nasty to integrate with Python's grammar.

Python has a strict grammar where statements (which use indentation) contain expressions, but never vice versa. Allowing statement-body lambdas would give you an expression form that contains significant indentation that could be embedded in the middle of some larger expression, like:

    ))) # Ugh, where do these go?!
Handling that in a way that's readable and easy to parse is hard, and, I think, there's no solution that would fit naturally with Python's look and feel.

CoffeeScript does handle it, but I think it's one of the hairier corners of its grammar and the Python community isn't quite so gung ho about accommodating grammatical weirdness like that.

Python also has `yield` generator syntax which can approximate `async` syntax rather well (everything is still colored but less painful). There seems to be no concensus whether generators or promises ("futures") are better but it seems to be agreed that if you want nice syntax you should be happy with generators.

The worst problem is stdlib APIs weren't designed for any async style. => gevent's monkey-patching all of stdlib is considered among the _less_ nasty solutions :-(

On a deeper level, I don't think the Python community even agrees async is a very important problem. Unlike the self-selection Node enjoyed, many people work on software where mostly sync code is good enough.

Nim also handles it and it works pretty well I think.

https://glyph.twistedmatrix.com/2014/02/unyielding.html is a good read - there are reasons why explicit sync-async "coloring" (i.e. await/yield) is better than green threads/coroutines which author admires.

That's a really long post, and I dimly recall reading it a while back, but after you scrape off it saying the same thing over and over again, I think it reduces down to an uncompelling argument.

It's basically, "I want context switches syntactically explicit in my code. If they aren't, reasoning about it is exponentially harder."

And I think that's pretty clearly a strawman. Everything the author claims about threaded code is true of any re-entrant code, multi-threaded or not. If your function inadvertently calls a function which calls the original function recursively, you have the exact same problem.

But, guess what, that just doesn't happen that often. Most code isn't re-entrant. Most state isn't shared.

For code that is concurrent and does interact in interesting ways, you are going to have to reason about it carefully. Smearing "yield from" all over your code doesn't solve.

In practice, you'll end up with so many "yield from" lines in your code that you're right back to "well, I guess I could context switch just about anywhere", which is the problem you were trying to avoid in the first place.

I don't think you read the article very carefully. Specifically, what I am claiming about (shared-state) threaded code which is not true of "any re-entrant code" is the fact that you cannot tell whether threaded code is re-entrant or not without a comprehensive, combinatorial whole-program analysis. It's not feasible to know what you might be re-entrant with, because you might be re-entrant with anything. With preemptive threads you really just can't do it at all, but with green threads you have to follow every call stack all the way to the bottom, because nothing else about it the bottom of the stack tells you whether you're going to context switch or not. When you get back a Deferred (or a Promise or a Future or a thunk or whatever), then the stack-inspection can be shallow; O(1) on the depth of your stack instead of O((stack_depth) * (function_length)).

Sadly, you can build a Java API to introduce callback hell if you want to.


It's nice you don't have to, though.

Funny. I thought he was talking about Java 8 for a while. We (eventually, when the rest of the stack catches up) get functors / lambdas, but:

* red = instance methods.

* blue = static methods, which cannot call an instance method.

That's not true. You can call an instance method from anywhere given an object. There's just no implicit 'this' in a static method.

Er, what's that? I seem to have lost a receiver for your message...

Once you go functional, instance methods start to look like a special klugery for currying the first argument to a function. Go and Nim (for instance) hint at this more than a little, as well, with their OOP syntax.

Bundling two kinds of methods within a "class" starts to feel weird when you start using individual functions.

What about isolates in Dart? I mean isolates are isolated processes, which also can be a thread and they also can communicate with each other.


1. Isolates can only communicate with each other using asynchronous method calls. So even though you can move some work to another isolate, you can't block waiting for it to complete, so your function still has to be red.

2. Isolates are very limited in what you can send between them, which makes then not very useful in practice for much of anything.

But waiting is non-blocking right? It means that we can just make one isolate's role to wait for all other isolates to complete.

For me the bad part of "red" functions is that testing is harder for it or I just suck at it.

You can always make a function that sync all async operations: sleep until a global variable is changed by the callback.

A pain, but still not that bad.

Yes, I've used a similar technique myself; however I consider it only a last resort. But doesn't that require your language to have threads? Is it possible to do this in eg. Javascript?

There is no sleep call that i know of in javascript, but you are right it depends on the language and its implementation of sleep.

Edit: although you could emulate a sleep call by setting a (pediodic) timeout

For node.js, there is a module: https://www.npmjs.com/package/deasync

you can't if all code is running in the same thread, as sleeping would prevent any code to run at all.

      CMP $C050
It's not a function, it's a procedure. It doesn't need to return anything, it produces an effect. Interrupts will break it.

   await until an as yet indeterminable time in the future

Relevant paper: http://www.info.ucl.ac.be/~pvr/VanRoyChapter.pdf Especially p34+ regarding concurrency paradigms.

Interesting allegory. Pretty much the exact same thing could be said about pure/impure functions in a language like Haskell -- where I thought this was going (until I realized it was about JS)

js-csp enables go-like concurrency in javascript: https://github.com/ubolonton/js-csp

Or you could be like scheme and make all functions red.

What about FRP?

FRP has nothing to do with this. Concurrency is only an implementation detail of FRP.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact