
Show HN: Early-stage Yahoo Pipes spiritual successor - onli
https://www.pipes.digital/
======
zeptomu
I've some experience in visual programming languages (both professional and in
my studies) and I advise _against_ the graph-form where you connect nodes and
edges.

This does not mean that text is the only way to implement programs (and even
if you connect nodes, you're still programming), but maybe a good local
optimum is in-between, e.g. interactive shells like Jupyter's Notebook and the
Mathematica interface. I know there is LabView, the Blender-Editor and AFAIK
some Unreal-Engine tool that uses this model, but bigger programs seem really
incomprehensible to me.

It's always easy to show-case 5 programs in these node-&-edge editors, but I
do not think it is the best approach for visual languages, as one should not
under-estimate the layout problems and how to represent information in boxes.

So, I am all for new ideas in visual programming, but I am not sure if the
"free canvas" approach works.

~~~
robinhowlett
I work for SnapLogic (an enterprise visual integration product) - the "free
canvas" works great for us with live preview of results/data; here's an
example linking a third-party code review tool's webhooks with JIRA:
[https://i.imgur.com/gNr1gGD.png](https://i.imgur.com/gNr1gGD.png)

All of our integrations can be converted into a API with a click of a button
too.

~~~
zeptomu
The image shows some of the problems quite well. E.g. the following things are
not clear to me:

[0] I suppose Rhombi symbolize choice. But where are the branches? What if
"Already Done" is true? Or does the else-branch enforce implicit fail-state
somehow?

[1] What's the difference between parallelograms and squares?

[2] What's the scope of the "For Each Issue"?

[3] Why is the second output of "Copy Webhook" named ("output1"), but not the
others? Are they implicit default output (maybe "output0")?

[4] What happens with longer pipelines? Do you wrap or have a scrollbar?

Furthermore the size of the modules have same size, so one gets into rendering
issues (e.g. the names of some nodes don't fit the "boxes"). Sorry, I didn't
want to sound harsh, I just think that visual programming is not automatically
simpler, just because it doesn't use text (at least _directly_ \- one has also
to name "modules"). You also don't use the "free-canvas" in its extreme form I
described (where you can rearrange everything), as you've e.g. implicit edges
between nodes, which is a good idea probably, as users get a feeling about the
complexity of pipelines by measuring their length (which would not work if
there are different lengths for edges).

~~~
robinhowlett
[0] Kind of. It's a filter, so "false" means "do not advance". We have
different Snaps for If-Else (the Snap would have 2+ outputs and you would
route to a respective "branch" of the pipeline)

[1] They denote different categories - parallelogram = READ (i.e fetch data
from somewhere), square = Transform (modify existing data)

[2] For Each Issue - for each object in the specific, targeted array of
incoming JSON object... (basically split the array chosen)

[3] Correct

[4] You can adjust the rotation of each output, so it could snake around. You
also can drag the output of one to the input of another that are not side-by-
side and it will mark them as connected, so you have layout in any way.
Finally, you could split up into separate pipelines and use another Snap to
tie them together.

I don't think you are being harsh - I'm not sure there is any way for a visual
programming language to be both powerful enough to permit massive
customizations and concise enough to be immediately readable. Some level of
interpretation has to be done.

------
beardicus
I'm surprised nobody has mentioned Node-RED yet:
[http://nodered.org/](http://nodered.org/)

Node.js-based with a Pipes-like visual wiring interface. It's quite popular
with the Raspberry Pi and Arduino crowd. Lot's of input and output plugins,
and you can drop into Node scripting when necessary. I quite like it.

~~~
perryprog
Node Red is very cool, especially for IOT type apps.

------
michaelbuckbee
This is cool, but I feel like Zapier, IFTTT and their open source equivalents
have moved far past Yahoo Pipes. How do you see this stacking up?

~~~
wocram
As far as I Kno, IFTTT only supports simple if a: then b triggers, you cannot
have multiple inputs or outputs. Do zapier or others improve on this?

~~~
bryanh
Yeah, Zapier has multi-step workflows [https://zapier.com/learn/getting-
started-guide/multi-step-za...](https://zapier.com/learn/getting-started-
guide/multi-step-zaps/).

~~~
Muanh
This doesn't include if true do A else do B logic right? I see they have
filters, maybe this can be abused somehow?

------
onli
I'd be interested to know which blocks and functionality is missing to support
the use cases you used Yahoo Pipes for! And of course general feedback is
always welcome.

------
insomniacity
Nice. It's not very clear that you need to link the pipeline to the right-hand
side. (I can't remember if you had to do that on Yahoo, it's been so long!).
It's particularly a problem on a wide screen because the eye starts on the
left, and I built my pipeline on the left and it took me a while to figure
that out.

But well done on scratching an itch that lots of people have had!

~~~
onli
> _I can 't remember if you had to do that on Yahoo, it's been so long!_

I have a screenshot on my disk:
[https://imgur.com/a/TExPX](https://imgur.com/a/TExPX) \- it had a bigger pipe
output block at the bottom. I can see how the red circle alone is not enough,
thanks!

------
unityByFreedom
Nice. Anyone else here ever use Ab Initio? It's an ETL tool for transferring
data between databases and manipulating in the process. Seems very similar.

I've always wanted to recreate something with those ideas because it was a
very efficient cross between visualizing code and writing it. This looks
pretty similar although for different data sources.

------
bergie
Interesting, have to take a proper look tomorrow. We've done similar things
with NoFlo ([https://noflojs.org](https://noflojs.org))

------
mgkimsal
would like to suggest a couple of "starter pipes" \- samples to more easily
illustrate some possibilities.

------
houshuang
This discussion is very interesting to us, as we are building a workflow
system for collaborative learning (students might do an activity individually,
the output of that activity is aggregated and transformed, students are
grouped based on an algorithm that processes data from the previous activity,
output from one activity is redistributed to another activity, etc). (quick
demo:
[https://www.youtube.com/watch?v=HQ9AyzLOn3Q](https://www.youtube.com/watch?v=HQ9AyzLOn3Q))

So this discussion about visual languages, data transformation etc, is very
relevant to us. One thing we're working on is how to make data transformation
more intuitive... Right now we are using JSONPath to enable selection of one
field to aggregate on (ie. you have a form where people input ideas, and
another activity that takes a list of ideas, so you can input a JSONPath for
the field to get aggregated). However, looking at JMESPath
([http://jmespath.org/examples.html](http://jmespath.org/examples.html)), it
looks much more powerful. Has anyone seen any examples of graphical interfaces
for going from one data representation to another, with preview, selecting
fields, aggregation etc?

------
fareesh
Unrelated to pipes specifically but along the same lines, I feel like app
developers ought to conform to a pre-defined standard like schema.org for
their respective content so that everything can be inter-operable in theory.
That way if I'm using Microsoft Todo or Google Keep or whatever, the potential
for Google Assistant or Siri or Cortana to add to whichever Todo App I'm using
is already there.

What are the drawbacks behind something like this?

~~~
comboy
No incentive. If your service is used through something else then you are
easily replaceable and users don't spend time in your app.

edit: Just to be clear, I'd love that too. Hopefully it will start happening
when everything is driven more by micropayments rather than ads. I'd imagine
you'll have "gui companies" and "service companies".

------
anonfunction
I used to use yahoo pipes for a few twitter bots I had. There was an rss to
twitter webapp that I hooked up and it was fully automated for years before it
stopped working.

I didn't see the thin red circle on the right, which I now understand to be
the output. I almost gave up before realizing my mistake.

------
Xeoncross
Using yahoo! pipes I created a lifestream application[0] some years ago that
won an honorable mention for the 10k javascript challenge. It was nice to have
a dynamic site that required no server work on my part.

Updating it for this would probably be a neat way to test out the RSS feed
usage.

[0]:[https://github.com/Xeoncross/MicroStream](https://github.com/Xeoncross/MicroStream)

------
mortadelegle
Very interesting, I do think that visual programming tends to work better when
constrained to a small niche of computing, that's the hypothesis behind
[https://github.com/AlvarBer/persimmon](https://github.com/AlvarBer/persimmon)

~~~
onli
Wish I had that when I was still doing machine learning. Looks nice!

------
Mister_Snuggles
I'd love something like this that I can run locally. From looking at the
comments, it looks like Node-RED and NoFlo are two possibilities.

I was intrigued by Yahoo Pipes a while back, but didn't want to invest much in
it in case it was shut down. Sadly, that worry was well founded.

------
jonincanada
I archived all the public yahoo pipes as Python modules -- before it was shut
down #oldhardrives

~~~
jwives
I'm not sure what the licensing on that would be; would it be possible for you
to share that archive?

------
hehheh
Neat. Any chance of adding JSON processing where you can pick a key (ala
`jq`[0]) and then combine it with RSS or other JSON or what have you?

[0] [https://stedolan.github.io/jq/](https://stedolan.github.io/jq/)

~~~
fiatjaf
I've made something[1] that used "jq for everything" in the past.

RequestHub was initially a way to connect webhooks from one service to API
calls to another services, using jq scripting for all the customization.

Later I realized it could be used for processing anything (even text!), not
only webhooks, but it was too late. Maybe someday I'll do a Yahoo Pipes
revival that will just use jq for scripting inside the boxes.

[1]: [http://archive.is/nGyH3](http://archive.is/nGyH3),
[https://github.com/fiatjaf/requesthub.xyz](https://github.com/fiatjaf/requesthub.xyz)

~~~
gingerlime
Why did you discontinue requesthub?

~~~
fiatjaf
I couldn't imagine a way to monetize it, and it was using all my free Heroku
hours.

What do you think? Do you think it had a bright future?

~~~
gingerlime
I think Zapier and the likes fill this need pretty well. I wonder if there's
still a niche for more lower-level hosted service for plumbing webhooks
though. It might be a rather small niche unless you figure out a way to offer
better value than those guys.

One thing that definitely makes it hard is the chicken&egg situation. I can't
imagine many people paying for a service (or even using it for free), unless
it has a solid reputation and is highly available. Otherwise, they'll either
host something on their own, or use one of the bigger players.

------
infectoid
Where I work we have a custom made system that was inspired by Yahoo Pipes. We
are in the slow process of rewriting it.

Also this reminds me of and IoT solution I was shown recently.

[https://nodered.org/](https://nodered.org/)

------
hehejo
What about Talend ETL?

You connect and configure various compontents (Input, Output and Manipulation)
to achieve you desired dataflow. It's compiled to Java and really Vera
versatile.

------
Toast_
Do you plan on supporting scraping content via css selectors/xpath/regex?

~~~
reubano
You can check out my library riko [1, 2]. While it doesn't have a slick GUI,
it _does_ support most of the original yahoo pipes (including xpath [3] and
regex [4]).

[1] [https://github.com/nerevu/riko](https://github.com/nerevu/riko)

[2]
[https://www.youtube.com/watch?v=bpn2G3TAAYY](https://www.youtube.com/watch?v=bpn2G3TAAYY)

[3]
[https://github.com/nerevu/riko/blob/master/riko/modules/xpat...](https://github.com/nerevu/riko/blob/master/riko/modules/xpathfetchpage.py)

[4]
[https://github.com/nerevu/riko/blob/master/riko/modules/rege...](https://github.com/nerevu/riko/blob/master/riko/modules/regex.py)

~~~
Toast_
Hey, just saw your comment. Looks like a pretty slick library; would something
like this run in a jupyter/azure notebook?

~~~
reubano
Most definitely [1].

[1] [http://nbviewer.jupyter.org/github/reubano/riko-
tutorial/blo...](http://nbviewer.jupyter.org/github/reubano/riko-
tutorial/blob/master/Tutorial.ipynb)

------
arjie
Nice. Used to love Yahoo! Pipes What's your data model look like?

~~~
onli
Every pipe is a recursive list of blocks, like a tree. Basically, a pipe has a
starting block that produces the pipe output. To do that it calls the process
function of its input block, which again run their input blocks, ..., the feed
blocks serve as tree leaves.

------
pbiggar
Cool product, and great to see this get traction! My new company is very much
inspired by Yahoo pipes and making something similar. Would love to chat about
your approach!

~~~
onli
Absolutely, I'll send you a mail

------
scottmcleod
I really loved Yahoo Pipes - Was ahead of its time if you ask me..Built some
very cool automations on top of this. Basically was IFTTT and Zapier well
before them.

------
j45
Pipes was a handy tool, hope this project keeps up!

------
dundercoder
I thought RSS had died a rather unfortunate and inconvenient death... am I
wrong and it's still been kicking all this time?

~~~
Viper007Bond
Still very much alive, just not quite as popular due to the death of Google
Reader.

------
timvdalen
Looks interesting but the editor didn't give me any feedback/output and the
login email never arrived.

~~~
onli
So far you need to log in to save the pipe and then see the output.

Shoot me a mail to admin@pipes.digital (or to the one in my HN profile) if you
want to try to sort out the non-arriving email. Or if you have a gmail
address, log in with that, portier (the login system) supports the OIDC flow
for gmail.

------
technologia
Its interesting and I do applaud your work, but I'll probably just stick to
Apache Nifi with its flexibility.

~~~
JPKab
How do you like NiFi? I work at Alteryx (we have a closed source visual data
integration tool targeted at business analysts and data scientists), but was
actually investigating NiFi quite heavily before I joined my current company.
It didn't appear to me to have much adoption, but that was over a year ago. It
certainly looked extremely compelling as a tool targeted at advanced users.

~~~
technologia
So I would agree that its not intended to be a tool that you could use like
yahoo pipes. To the extent that it requires programming, I guess I would agree
it requires an advanced user.

Putting that aside, I would say the Nifi is easy to work with, performant with
large datasets, and I personally love the notifications I get when tasks get
screwed up, etc. I would not suggest using this to expose data flow to your
user as its UI is not really geared towards folks who are programming averse
(to a certain extent).

------
nthcolumn
Pipes was awesome. Would it have killed them to just let it go free instead?
Yahoo engineering did a lot cool of stuff.

------
laretluval
Doesn't seem to work on mobile Safari.

------
yownie
couple of code examples would be nice

------
rjeli
Dead (could not connect to server)

~~~
onli
Please try again. It should be up (works for me and
[http://downforeveryoneorjustme.com/www.pipes.digital](http://downforeveryoneorjustme.com/www.pipes.digital)
supports that). Though I forgot scaling up the droplet it is running on before
posting here...

~~~
rjeli
huh, it fails immediately for me about 5/6 times on iOS Safari. weird

------
RRRA
Librarians are gonna go wild over this...

