

An introduction to NoFlo and flow-based programming - robhawkes
http://rawkes.com/articles/an-introduction-to-noflo-and-flow-based-programming

======
SomeCallMeTim
I think the _types_ of problems each programmer tends to confront make up a
large part of the disconnect between the ones who "drink the koolaid" of a
programming paradigm or process and the people who are unimpressed.

I typically do games. Games are useful to think, in part, in OOP terms; there
is very little "data flow" and (typically) there are a lot of objects that
need to interact in asynchronous ways. There are specific parts that could be
reduced to flow based programming (FBP), but it seems like the wrong paradigm
for writing an _entire_ game, as an example close to me. I could imagine FBP
certain AI logic state machines being useful, for instance.

But other areas of development have flow-style tools and that's clearly the
_right_ way to write code. Signal processing is an obvious example: When you
want to send an audio signal through a reverb, split it to two different types
of processors and re-merge it later with a fader tied to an external control,
laying those components out with boxes is 1000x better than trying to write
code in text to do the same thing.

So I think articles like this are useful to introduce everyone to various
other paradigms, in case they might be applicable to a problem that you're
working on. But it's never a "silver bullet" simply because each tool works
best in certain problem domains. A friend tried to convince me that I should
drink the functional programming koolaid, for instance, but on drilling deeper
I found that he had never worked on a game, and didn't really grok what
_kinds_ of code you would normally end up putting in a game.

What I'd _love_ to see is an article that compared a lot of different
processes and paradigms and talked about _where_ each is most useful.
Especially if that article surprised me. :)

Problem is that most people are familiar with their own areas and toolsets;
we'd probably need a team of people from different specialties to collaborate
on it. Maybe a wiki where people could add in their experiences trying each
approach or tool on their own problem domains? Then when someone _does_ use
functional programming to write a game, they could talk about the pros and
cons and the kind of game they wrote. And _that_ would be useful data.

~~~
agentultra
> I think the types of problems each programmer tends to confront make up a
> large part of the disconnect between the ones who "drink the koolaid" of a
> programming paradigm or process and the people who are unimpressed.

Sounds a lot like selection bias.

Even in games there is a growing movement to stop the OOP paradigm and focus
on data-oriented or "data flow".

It's hard to know whether there are universally-applicable truths in a given
paradigm... but it's rather nice to have the option. Flow-based tools are
becoming popular in procedural content generation, have been in
audio/music/dsp for some time, and are probably quite useful in other sort of
"pipeline" analogies.

But certainly past experience will play a large role in determining which
tools you think are applicable to a given problem.

~~~
SomeCallMeTim
Selection bias implies that people just like what they're familiar with; I'm
sure that does keep people from switching to new tools, but that's not what
I'm referring to.

I'm talking about people who actually try new tools and reject them as useful
to their problem domains; I try a lot of new tools, and some fit with the
problems I solve better than others.

I didn't quite remember what "data oriented meant," but a quick search brought
up this article [1] on using data-driven design in games. It brings up some
important points, but really the point of data flow design seems to be
optimization and efficiency rather than making it easier to write the code.

And I'm all for that, and I thank you for making me look that up since I will
likely use that idea, but I'll keep my text editor for most of the game
development I do going forward. Except maybe the particle systems, which as
the article below points out, are typically already data oriented, and have
had good data-flow-based tools for years. ;)

[1] [http://gamesfromwithin.com/data-oriented-
design](http://gamesfromwithin.com/data-oriented-design)

~~~
agentultra
I think you're referring to _confirmation bias_ which is related but not quite
the same. Never the less I understand what you're getting at. I took it to
mean that the kinds of problems a programmer encounters is not sufficiently
random to properly determine the usefulness of a given language or "paradigm."
Which may keep people from switching to new, better tools (or from trying them
out at all).

The data-oriented design thing is interesting and something I've heard come
and go over the years. Check out:

[http://www.dataorienteddesign.com/dodmain/](http://www.dataorienteddesign.com/dodmain/)

and

[https://www.youtube.com/watch?v=ZHqFrNyLlpA](https://www.youtube.com/watch?v=ZHqFrNyLlpA)

A video demonstration of Jai; a new compiler from Jonathan Blow designed
around this paradigm.

Cheers!

~~~
SomeCallMeTim
Ahh, I see. Yeah, I guess I meant that each paradigm _is inherently_ a
different value for different domains, and so therefore has a different value
for programmers who specialize in their domain(s). And when someone checks out
a paradigm that doesn't match their domain, they may end up complaining about
how the tool isn't good.

Love the links. Listening to Blow talk now. Will look at the longer book
later.

------
otakucode
Visual programming has interested me for a long time because it is amazing how
many times it has been tried and the result been unusable.

If I had the time and resources, I'd very much like to try developing a visual
programming system that uses a user interface change that I think would
greatly change things. Basically, using the middle mouse wheel to zoom in/out.
If you've played the Supreme Commander game series, you will have the best
idea of what I mean. As it is, you end up with absolutely everything trying to
be jammed in to one gigantic flat space. That's terrible. Better would be
zooming in on a 'box' (function, data structure, etc) and having its guts
become the new 'context' you're working in. So far as I have been able to
find, this has never been attempted.

~~~
falcolas
A ZUI would be great for working with the FP paradigm, but good UIs are hard
to make; I think it's one of the way that NoFlow is gaining the traction it
is, the UI is pretty good.

~~~
robhawkes
The GUI with NoFlo is one of the major reasons why this is different to the
previous attempts at FBP (that and it's JavaScript). What I also love is that
it's all released under MIT:
[https://github.com/noflo](https://github.com/noflo)

------
aikah
"Flow" programming is great for non programmers.I used it to develop VST
plugins on pc and when I was working in video editing(compositing).

However,I wouldn't use that to develop web apps.It's often quicker to actually
write the code,since app development is rarely just about piping stuff.

~~~
robhawkes
You'd be surprised how quickly you get used to the idea of connecting
components together with pipes, even for Web applications. The additional
benefit being that once you've written a component once you don't need to
rewrite it ever again, or even copy and paste the code.

The NoFlo site has some good examples:
[http://noflojs.org/example/](http://noflojs.org/example/)

~~~
petercooper
In particular, I think the growth of Docker/containerization and microservices
will make this an area worth keeping an eye on.

I'm not convinced it makes sense at a micro level, but at the macro level of
connecting larger services that have been coded in the traditional way..
there's something very appealing about the idea, especially in the way it
could interact with ops' concerns too (like having a visual representation of
scaling or moving services between machines).

~~~
dharma1
[https://jujucharms.com/](https://jujucharms.com/) sounds pretty much like
what you are describing

~~~
petercooper
That screenshot does look somewhat like what I pictured in my head. It's a
small world nowadays :)

------
valarauca1
One of the main issues with Flow-Based Programming is often while it optimizes
generally in the opposite direction of Functional-Programming (Currying vs.
Topological Sorting I.E.: Execute when needed vs. Execute as soon as possible)
is that you tend to lose control of your software.

While I'm not suggesting a purely imperative approach, to a certain degree
gains can be had in an imperative context over either Functional/Flow
Programming via in depth knowledge of OS/Hardware processes and are generally
NOT developer fast/friendly.

From a personal stand point Flow-Based Programming is amazing for concurrency
since it lets you see where you can use threads/processes. As which sections
of code are less _attached_ to the rest.

~~~
falcolas
> it optimizes generally in the opposite direction of Functional-Programming

Really? It strikes me as exactly the opposite - the output of each discrete
component is solely dependent on its input. There is no shared state between
components, only what is passed between them.

~~~
valarauca1
Your addressing the platform specific solution not the global problem I am.

If you build a compiler that uses Flow Based programming without dipping into
imperative blocks (very possible, and they exist). You get the problems I
describe.

~~~
falcolas
Flow based programming actually has quite a bit in common with purely
functional programming, in that the individual processes are generally
stateless; their output depends solely on their input. The fact that the
individual processes might be written in an imperative language simply doesn't
matter at this point, so long as they always produce the same output for any
given input.

This is not true for all FBP processes obviously - there exist processes which
read to and write from file handles, but much like the popular FP languages,
these "tainted" processes are contained and separated from the remaining
portions of the program via their connections to other processes.

Most of the non-functional "programming" in Flow based programs actually
occurs at the level where you are composing the connections between processes
(be that the GUI, or in a hand-written method which makes the connections).

One additional benefit of FBP which it shares with traditional FP programs is
that they are capable of exhibiting "lazy" execution properties through the
application of back pressure - by limiting how many packets of data a given
connection can contain, you create a program which consumes only enough
content from any given generator to keep the entire system occupied.

Now then, can these abstractions break down if you start digging into the
implementations of individual processes? Of course. It's not turtles all the
way down, after all. Then again, all FP languages which run on Von Neumann
based architectures suffer this problem. They all get compiled down to
imperative machine code, and are all subject to the same state interruptions
as every other program.

~~~
valarauca1
>Most of the non-functional "programming" in Flow based programs actually
occurs at the level where you are composing the connections between processes
(be that the GUI, or in a hand-written method which makes the connections).

This is false, there are entire languages that do line by line compilation via
Flow Based Programming.

>One additional benefit of FBP which it shares with traditional FP programs is
that they are capable of exhibiting "lazy" execution

This is blatantly wrong and the main thing I was trying to express.

Flow Based Programming is based off a topographical analysis of the code body.
Which is a analysis of object dependencies (where a dependency is code that
must be executed before the object (code) in question can be run).

Meaning objects are executed in the order in which they have the least number
of dependencies. This means one off sections of code, that do not necessarily
need input from external, or exist independent of otherwise coherent processes
he processed first.

For example (Example 1)

    
    
            Path p = resolve_path();
            if(something_happened())
            {
                   FP fp = open(p, "rw);
                   write(fp, "logloglog");
                   close(fp); 
            }
    

This above code is bad, because path P is always resolved, regardless of error
state.

A Currying analysis (lazy evaluation) will change this flow via lazy
evaluation (or should ideally):

(example 2)

    
    
            if(something_happened())
            {
                   Path p = resolve_path();
                   FP fp = open(p, "rw);
                   write(fp, "logloglog");
                   close(fp); 
            }
    

A topological flow based compiler will prefer path P be constant. It will
prefer example 1, to example 2 every time (if its given example 1 as base
code). And likely ensure that path P is solved for absolutely first in a
function AS resolve_path() depends on no code being executed before it.

This is the flaw in most flow based programs. Is they don't care errors like
this, and when the errors do exit, they _optimize_ for them to be more
apparent.

:.:.:

This is what I mean by a topographical analysis being the opposite of a
Currying analysis. They group code differently, in almost an opposite fashion.
Curry trys to group code by purpose. Topological by how much code has to be
executed before it.

Both have their uses.

------
zeLaur
I have to admit, haven't read all comments... but:

FBP, is nothing new ... In the enterprise world ( the world that I`m working
in ) it is used quite frequently, and there are some products that live of
this for example ( windows workflow foundation, workflowgen ) for the c#
world.

FBP is a way of doing things, let's say it is a pattern, like MVC ( less than
more, but bare with me ) they each serve a specific task. To be a tad more
clear, let's say that MVC is more of a generic on size fits all approach while
the FBP is more of a custom tailored way of doing things for (highly) specific
things. So, MVC would be great for a wiki and FBP would be great for the whole
wiki item management from treating the adding, verifing, drafts, plublish etc
etc. ( I`m sure there are a lot of better examples out there ...)

The benefit of NoFlo is that they actually made something that is usually
regarded as "enterprise-ish crap" and made it appealing. And again they did a
great job with the visual editor, but from my own experience with visual flow
design ... sooner that later you will be back to the code ( we you need to
stray a bit from the Happy Path ).

Also, I think that frameworks like NoFlow are a very good step for Node in
general showing that a lot more interesting things could be done apart for the
tasteless and colorless REST apis.

------
jdmichal
Things like this are great for data-processing pipelines with an enumerated
set of common operations, most of which input and output the same data type.
For instance, any data that can be redefined as a matrix, since a lot of
matrix operations result in another matrix (or a vector, which is just a
restricted matrix).

However, I can't help but feel that applying it any more generally is a
hammer->nail issue. Or, at least, completely invalidates the idea of
"massively reusable components". To make components more reusable, they have
to make fewer assumptions about their input. To make fewer assumptions, they
will typically need to do less. The reductio ad absurdum argument is that you
end up with a bunch of components that each correspond to what would have been
a line of code in a text editor, and all you've gained is a pretty picture.

------
falcolas
One other thing I'd like to bring up - Go is a surprisingly strong tool for
building flow based programs. It does have a few weaknesses, though, such as
the lack of a graphical front end, but this isn't a requirement to write a
flow based program.

The other (more significant) weakness is that you will have to either deal
with interface{} casting quite a bit, or write type specific versions of the
basic utilities.

On the plus side, you get virtually free concurrency and nearly purpose-built
communication channels!

~~~
bergie
Yep, there is
[https://github.com/trustmaster/goflow](https://github.com/trustmaster/goflow)
for that

------
dharma1
Thanks for the write up, has inspired me to give NoFlo a go. Reminds me of
Reaktor (an app used to design audio DSP effects/instruments)

Will be checking your blog for a more hands on series.

~~~
robhawkes
Glad to hear it! I'm sure you'll enjoy trying out NoFlo, even if you don't end
up using it for anything.

------
zobzu
one advantage of this is that you can sandbox much smaller pieces of code.

~~~
kornish
Could you clarify how visual programming helps with building smaller isolated
projects over traditional text-based means?

~~~
robhawkes
There are a number of ways, the most obvious being that you can build that
smaller, isolated project incredibly quickly using flow-based programming. The
thing is though, if you find traditional approaches quicker and easier then
they're probably the ones you should be using instead (I'd still recommend
trying FBP).

------
jamesfisher
How is flow-based programming different to FRP?

~~~
robhawkes
I don't know myself (not used FRP) but based on a quick look it seems to be
based on asynchronous dataflow programming so I imagine there are a lot of
similarities (seeing as FBP is based on dataflow too). I'd love to hear more
from someone who has experience with both.

