
Node-based a/v composition: programs as graphs and graphs as compositional tools - kousun12
https://github.com/kousun12/eternal/blob/master/background/README.md
======
jcelerier
If someone's interested in moving the node-based media software status quo,
I've been developing ossia score for the past few years :
[https://ossia.io](https://ossia.io)

A big difference with other similar environments, is that the dataflow graph
evolves dynamically based on a timeline which is useful for interactive shows,
etc. I've recently been working on doing the same for shader-based stuff :
[https://twitter.com/jcelerie/status/1211354889873510400](https://twitter.com/jcelerie/status/1211354889873510400)

~~~
anigbrowl
Intriguing. Very surprised I've never heard about this before, and
refreshingly different from most other tools in this field. I look forward to
trying it out.

------
BubRoss
He didn't mention a few other big ones with good demo versions: Nuke, Touch
Designer and Digital Fusion (much more awkward, but free).

Figuring out why programs like these work well and what gaps are between them
and general programs is interesting because many of the reasons come down to
pragmatic workflow and not technical limitations.

They all operate on a very limited set of data types (just images, images +
geometry, etc.) and don't have to deal with state, branching, or loops.
Results can be viewed easily at each stage and each operation can be isolated
and iterated with real time interactivity.

This project tries to use message passing nodes as a backbone and data flow as
the heavy lifting to get to visual C++ programs:

[https://github.com/LiveAsynchronousVisualizedArchitecture/la...](https://github.com/LiveAsynchronousVisualizedArchitecture/lava)

~~~
mroche
> Digital Fusion (much more awkward, but free)

Small nitpick, only applies to Fusion 9 and below. Fusion 16 only comes in the
Studio variant, otherwise you need to use DaVinci Resolve 16 to access the
embedded Fusion for free.

> Figuring out why programs like these work well and what gaps are between
> them and general programs is interesting because many of the reasons come
> down to pragmatic workflow and not technical limitations.

In the case of Nuke, the pragmatic workflow is a large part of it, but not the
only one. When it comes to digital compositing, it is very easy to get up and
running with layer based solutions quicker (such as Adobe products), but they
hit a wall very quickly on memory usage, process duplication, and general
difficulty of navigation and organization as the composite becomes more
complex.

Nuke is in a different class of its own, being developed originally at Digital
Domain (a VFX studio) before eventually sold to Foundry (then The Foundry).
Nuke's success was a technical one, being far more memory aware and efficient
(a scanline renderer and processor), as well as catering directly to the user
base it served. It is still the only specialist compositing tool that has
support for modern deep data, has proper handling and support for multi-
layer/multi-part EXRs, and has a clean and proper handling of channels and
layers throughout the comp. It has some modernization to undergo nowadays, but
it is still the most trusted tool in VFX.

Software in computer graphics to note as being primarily node based that
expose that directly to the user:

    
    
      - Autodesk Maya (is node-based)
      - SideFX Houdini (formerly Prism)
      - Foundry Nuke (originated from Digital Domain)
      - Foundry Katana (originated from Sony)
      - Substance Designer
      - Isotropix Clarisse BUiLDER
      - Natron
      - Blackmagic Design Fusion (an in-house tool to Eyeon Fusion -> Blackmagic Design)
    

Other tools that integrate node based workflows into their application:

    
    
      - Autodesk 3DS Max
      - Foundry Mari (originated from Weta Digital)
      - Maxon Cinema 4D
      - Blender (personally unsure of Blender and being nodal-based)
      - Blackmagic Design DaVinci Resolve
    

There are a smattering of others I haven't gotten the chance to use, from
other imaging tools to those specializing in tracking (e.g. PfTrack) and in-
house non-commercialized tools.

~~~
PunksATawnyFill
The most trusted tool since Apple shitcanned Shake, that is.

It's too bad that Shake couldn't be open-sourced. Several big post facilities
(ILM, Weta, and others) bought the source code and have probably done great
things with it.

~~~
BubRoss
No one really did much with it. Nuke is actually much better than shake by a
wide margin, using shake feels like going back to the stone ages due to
interactivity.

------
ssalazar
From a creative perspective the modular/node-based way of thinking has always
been a niche, having seen many attempts such as this to place it into a
broader model of creativity (including my own [1]). Its telling that the
enduring musical instruments of the electronic age have been relatively fixed-
function (Minimoog, Yamaha FM synths, 808/909) with constrained options for
patchability, rather than modular synths highlighted by the article. The
metaphor is further strained by modular synthesis' lack of basic affordances a
programmer would expect, such as a memory model, data types,
composability/reusability of "code", etc.

These issues translate to digital node-based systems- dealing with structured
data in e.g. Max or TouchDesigner becomes an exercise in patience, and their
mechanisms for composing and reusing code are somewhat baroque as well.

The other major creative limitation of flow-based programming (and in fact,
most computational art systems) is the absence of a timeline (as mentioned by
another commenter). This is a showstopper for many creative pros Ive talked
to. But it grazes on topics that are interesting purely from a computer
science perspective- imagine if you could freely jump to any point in a
program's execution?

To be clear, having spent my doctoral studies in this space I find these to be
interesting problems to solve rather than indictments of node-based
programming as a whole. Im interested to see how eternal and other efforts
mentioned in the comments here tackle these and other challenges.

[1]
[https://purl.stanford.edu/mf249vj6694](https://purl.stanford.edu/mf249vj6694)

~~~
kousun12
Agree on most of this; most systems I've used so far feel a bit clunky
(Houdini has by far been the most sophisticated system I've worked in), and my
sense is that if it had more mindshare / attention, the paradigm could
actually be pretty fluid. Using most node-based tools today (including
eternal), I get the admonition that it's like using text edit to write python
compared to a pro at using pycharm.

The timeline thing is really key. Houdini does it in a sort of clever way
where "time" as a variable is exported globally to any node that wants to use
it, and the timeline feature is a consequence of using the time variable in
any of your functions. i.e. it's not jumping you to a point in the program's
execution, but rather acting as a controller for all time-dependent functions.
sort of like exporting a time uniform to graphics shader

~~~
ssalazar
Agreed- a lot of the issues people bring up with node-based programming have
promising solutions, its just a matter of exploring the space and finding
successful implementations.

As far as timeline, I also think the digital audio workstation model is
interesting- modules export parameters that can be assigned to an automation
channel and dragged/tweaked visually on some hierarchy of timelines.

------
meatsock
this is a great project, i am interested to see more competitors to pd, vvvv,
and max matthews little program. a lot of great work has been made on tools
like this.

~~~
adamnemecek
I've never found pd nor max to be productive.

~~~
pixelface
why not? i use neither but they both have rich histories as audio programming
tools. autechre are/were heavy max/msp users at the forefront of IDM, max for
(ableton) live is a popular thing, and i have several friends who write pd
things that then get run on their eurorack modules.

------
bilekas
This eternal project is crazy crazy crazy good.. Achieved in the space of only
a year.. This is incredible.

Beautiful. Really.

~~~
kousun12
thanks! appreciate it. would loved to have spent some more time on it / made
it more of a primary thing for me, but i suspect it's a sort of all or none
kind of work && i hope someone can get a team together and do a version of it
for real

------
th0ma5
A git or otherwise nice version control with diffs and such for a visual
language would be killer.

------
FlowNote
For a flow-based programming DSL, check out FlowNote

[https://emblem21-opensource.github.io/flownote/docs/02-featu...](https://emblem21-opensource.github.io/flownote/docs/02-features.html)

