
The Problem of Async Programming, and a Crazy Idea for Solving It - lorean_victor
https://medium.com/connect-platform/the-problem-of-async-programming-and-a-crazy-idea-for-solving-it-cf368a9ea949
======
ricardobeat
Callback hell was never about promises, but callbacks (!) which would require
a third-party library or some gymnastics to implement control flow. Promises
and async/await both solve it reasonably well. The problem presented can be
rewritten as:

    
    
        Promise.all([
            doA().then(a => doC(a).then(c => [a, c])),
            doB().then(b => doD(b), b_; return b; })
        ]).then(([a, c], b) => doE(c, a, b))
    

No need for `new Promise(resolve => ...)`. Or with async/await:

    
    
        const [a, b] = Promise.all([doA(), doB()])
        doD(b)
        const c = await doC(a)
        doE(c, a, b)
    

Is it worth introducing the complexity of visual programming to solve this
'problem'? I have a much harder time understanding the graph than the code.

~~~
lorean_victor
yeah I think the article actually starts with that Promise-based code snippet.
Even the sequential graph is based on the promise-based code.

~~~
ricardobeat
Look closer - the original is way more convoluted - there is no reason to use
`new Promise` constructors when you're dealing with other promise-returning
functions.

~~~
lorean_victor
yes you are right the constructors aren't indeed needed.

as for the complexity of the graph, basically the way I see it is that even
this promised based method is a way to solve the problem of putting the graph
into the proper promise-based format (for example, chain doA and doC and
return an array of their result) vs simply drawing the intended graph. I mean
it does take sometime to get used to it specially if you are really used to
just working with code (which I myself originally was), but since the graph is
the more intuitive way to represent it compared to other methods, after going
through the learning curve it seems to be less complex and easier to manage.

that said its not like I have conducted proper academic research on the matter
and it might as well be a pretty subjective thing.

------
karmakaze
Doesn't seem new at all, just like functional programming with lazy values.
The new part isn't the async it's the visual programming of which there are
others. Does incorporation async make visual programming different?

~~~
lorean_victor
Yeah I don't think so that on its own the visual programming makes any
difference (or even adds any value). The main point here is that the "visual
programming" is coincidentally a solution for async programming.

~~~
karmakaze
The actual problem with async functions is mixing them with non-async ones.
Having all futures/promises makes it easy. Now if the visual style could
represent current mixed js code, that could be awesome for display and editing
I'd have to try before deciding.

~~~
lorean_victor
Yeah we reached that point pretty early on while developing the PaaS with the
tool, so had to incorporate support for mixed synchronous JS blocks. you can
check out how to do that here: [https://medium.com/connect-platform/inline-
coding-in-connect...](https://medium.com/connect-platform/inline-coding-in-
connect-platform-534fce3c8cdf)

------
jmull
> To see that for yourself, try to re-write our async code snippet again using
> only async/await statements and see how does that pan out.

With async/await, wouldn't the example just be:

    
    
        let aOut = doA();
        let bOut = doB();
        let cOut = doC(aOut);
        doD(bOut);
        doE(cOut, aOut, bOut);
    

That is, it looks the same as the sync version.

The difference is in the "do" functions themselves: they would be async
(meaning they return a promise, and they accept promises as parameters).

~~~
lorean_victor
so in your code sample, would 'aOut' be a promise or a value? in case it is a
promise, then basically doC() would needed to be re-written to handle when it
gets a promised value instead of an actual value, and if not, then doA() must
be invoked synchronously, i.e. the `let bOut = doB()` statement must be
awaiting `let aOut = doA()` statement which is not what the desired async
execution flow is.

~~~
jmull
Yes, aOut would be a promise. It would look like this:

    
    
        async function doA() {
            ...
            return "the-result-of-a";
        }
    

(Which returns a promise. Assuming execution reaches that last line of my
sample code, the promise will resolve to "the-result-of-a" when you "await"
it.)

> ...then basically doC() would needed to be re-written to handle when it gets
> a promised value instead of an actual value

Well, wait a minute... it wasn't written in the first place! Let's write it
though. It would look like this:

    
    
       async function doC( param ) {
           ...
           // we use "await param" where we want to
           // access the value of param. e.g.:
           const paramValue = await param;
    
           ...
        }
    

So not bad.

~~~
lorean_victor
yeah really not bad at all, didn't think of that solution originally. edited
the article accordingly.

------
flukus
I don't think the leap to visual programming is necessary considering we have
tools like make that work essentially identically and remain text based. All
this is really doing is moving the dependency declaration to lines instead of
words and this would get very messy very quickly.

An example in make (if I'm reading in the right direction) format which I'd
argue is more readable and could scale much better would be:

    
    
      doA: doC doE
      doB: doD doE
      doC: doE
      doD:
      doE

~~~
lorean_victor
Shouldn't it be like

doE: doC, doB, doA doD: doB

etc?

anyways the problem with this approach is that doD() is assumed to be a
general function (or statement or async command) that is not by definition
bound to doB(), but rather in this specific case we need it to be executed
after doB() is done.

~~~
flukus
> but rather in this specific case we need it to be executed after doB() is
> done.

Why is that? If it needs any result from doB() then it is a dependencies that
must be specified. If it needs to be done only after doB is successful then it
is a dependency, just an implicit one we rely on the procedural execution to
handle. In either case some sort of surrogate dependency (or better syntax)
would have to be introduced.

But the point is that either would necessitate or be aided by graphical
programming.

~~~
lorean_victor
ok imagine doB() is a function that loads a record from the database and doD()
is for example a password hash matcher (really imperfect example, just trying
to get the point across). You might want to use both doB() and doD() in other
contexts as well, so basically it is not the case that you would want to
execute doD() always after executing doB(), neither is it the case that
executing doD() always requires executing doB() beforehand. It just happens
that in a specific case you would want to execute doB() and then pass its
result to doD().

------
Ooolin
Is the author aware of Dataflow programming ?

These do give the described benefits, however, the tradeoffs are massive.

~~~
lorean_victor
to a pretty limited degree, could you elaborate more?

~~~
Ooolin
\- Syntax is slower to write. Copying and pasting is a non trivial operation.
In order to benefit for the clean representation of the process, you need to
spend a good amount of time ordering the boxes around. The representation can
be misleading (wires crossing, overlapping boxes)

\- Very low density of information per pixel compared to any textual language
(if no zoom feature). Any long name that has to be rendered on screen become
fugly. Since we tend to order the boxes in a process from left to right, only
the equivalent of a single line of code can be represented on the screen at
once when displaying the names of variable and functions.

\- Functions / sub diagrams generally sit in their own separate window (or
worse, file). This discourage making functions, to a degree depending on how
good/bad the UI is. Getters and setters in OO visual code are a pain.

\- Execution order is not guaranteed to be sequential, so no try/catch
structure. Error handling has to be explicit. Witch mean dragging along an
additional error variable all over the place.

\- The graph representation doesn't translate well into source control and
diff tools.

\- If you need to fire an arbitrary number of parallel stuff at runtime, you
still need a "call this asynchronously" structure.

Coming from a LabVIEW dev. The benefits described in the article are still
absolutely valid.

~~~
lorean_victor
first off, many thanks for the feedback.

> \- Syntax is slower to write. Copying and pasting is a non trivial
> operation. In order to benefit for the clean representation of the process,
> you need to spend a good amount of time ordering the boxes around. The
> representation can be misleading (wires crossing, overlapping boxes)

copying can be made much easier. on these kind of features, we are acting on
user feedback to prioritize. but since this one specifically is one that my
co-founder has been bugging me since the day he joined, I don't think it'll be
too much down the pipeline.

also, automatically re-ordering the boxes to get the flatest version of the
graph (which seems to correlate to the "cleanest code") is something on our
mind, which would alleviate that issue.

> \- Very low density of information per pixel compared to any textual
> language (if no zoom feature). Any long name that has to be rendered on
> screen become fugly. Since we tend to order the boxes in a process from left
> to right, only the equivalent of a single line of code can be represented on
> the screen at once when displaying the names of variable and functions.

that is partly by design to encourage modularizing. on that front, we have
gone for possible overshooting and then slowly tuning towards the optimal
point.

> \- Functions / sub diagrams generally sit in their own separate window (or
> worse, file). This discourage making functions, to a degree depending on how
> good/bad the UI is. Getters and setters in OO visual code are a pain.

is that because of the increased inefficiency of switching between sub-
diagrams? if that is the case, thats also a user feedback that has been on our
radar for sometime and will probably tend to it soon-ish.

> \- Execution order is not guaranteed to be sequential, so no try/catch
> structure. Error handling has to be explicit. Witch mean dragging along an
> additional error variable all over the place.

actually it does not need to be sequential for proper error handling. each
node in the graph is equipped with its own error-handling, i.e. if an error is
somehow thrown during its execution the node will report it properly, which
has in many cases led to faster detection of error-sources in the context of
the projects we have done with CONNECT up until now. still, admittedly the
error-handling system needs to become more robust, and all of this is
basically only referring to errors that you wish you would catch and fix
during development phase. for errors that should be handled on runtime, yes
they need to be somewhat dragged along, which I do not necessarily see as a
bad thing, as it forces you to explicitly choose the corner-cases that might
occur and need to be handled in runtime. this separation actually helps
completely avoid using one big 'try/catch' that would blind you to 'bugs' that
should not make it to runtime in the first place.

> \- The graph representation doesn't translate well into source control and
> diff tools.

that is true as mentioned in the article as well. we've tried to make it
really human-friendly, however we have plans for further tooling specifically
on that front.

> \- If you need to fire an arbitrary number of parallel stuff at runtime, you
> still need a "call this asynchronously" structure.

that is true unfortunately. personally I am still thinking on how to represent
such dynamic branching within the graphs without causing more confusion. since
this is also a corner case, we have kept it on the drawing board until a
proper solution is found.

------
imcotton
Yes, because XML is too old to us.

~~~
lorean_victor
and to be fair its a really redundant format. I mean I personally would prefer
the current undesirable situation rather than doing XML-based coding all day.

