
Yahoo Pipes - tosh
https://en.wikipedia.org/wiki/Yahoo!_Pipes
======
brettproctor
A couple of fun behind the scenes stories which I think at this point are past
the statute of limitations.

Source: I was the Service Engineer on pipes for around a year or so.

1) When you went to the pipes landing page, there were a few demo pipes to
show people what was possible. One of them combined search results from say
ebay/craiglist/amazon to show prices for things.

One day I was looking through the source of these, and noticed there were
affiliate ids in the ebay/amazon links. They all belonged to some early team
member who had long since left.

I showed a couple people on the team, we all guessed at how much they were
making from this, said good on em and went about our day. I still wonder how
much they ended up making from it.

2) It was my first on-call, and all of a sudden the west coast pipes cluster
just went bananas. After ~5 minutes, east coast started to go nuts and the
west coast subsided.

Someone, despite numerous defensive measures, had found a way to create a
pipe-bomb that would recursively call multiple versions of itself. Once the
west coast load balancer failed over, one of these requests would hit the east
coast and the 'virus' would jump over there. This flipped flopped back and
forth until I figured out how to blacklist pipe ids (and eventually got a code
fix).

~~~
psadri
Heh... just chiming in to say the affiliate person wasn’t me :-)

~~~
idlewords
I wish it was me. That's genius.

------
kentbrew
As a fresh Yahoo! Developer Network employee I was the person who blabbed
about the top-secret _callback parameter the weekend Pipes launched, which
caused the service to be taken offline for capacity upgrade.

I had an interesting series of voicemails, ranging from "dude, WTF?" to "okay,
we need to talk to you right away" to "we're pretty sure you're fired" but
ending with "hey, Tim O'Reilly picked it up and ALL IS FORGIVEN THANK YOU!!!"

[https://web.archive.org/web/20070302201929/http://developer....](https://web.archive.org/web/20070302201929/http://developer.yahoo.net/blog/archives/2007/02/yahoo_pipes_sud.html)

~~~
p0sixlang
What were the implications of knowing the _callback parameter?

~~~
gopalv
You could invoke pipes from any webpage without worrying about cross-domain
access.

If you look at some janky JS code[1] I had on my website at the time, you can
see what you can do with it.

That would basically source

://pipes.yahoo.com/pipes/pipe.run?_id=7CTRtbtL3BGX_AHbjknRlg&amp;_render=json&amp;_callback=load_daily_show

as a static asset in my page, which would load the JS and call
load_daily_show(<json>) as a result.

Now, _EVERYONE_ who hits my page is invoking the pipe as a backend API call,
with no caching and unfortunately with the entire Y/T cookies intact.

[1] -
[https://web.archive.org/web/20081007043923/http://t3.dotgnu....](https://web.archive.org/web/20081007043923/http://t3.dotgnu.info/code/j002ube/)

~~~
Guillaume86
Yeah I remember using it as a CORS proxy, it allowed me to put together a full
client side music app (tracks source was not very legal) but tags, similar
tracks/artists etc were pulled from lastfm and allowed to quickly build a
decent playlist.

------
stevenjohns
I used Yahoo! Pipes in a somewhat greyhat way. I started pulling in data from
Reddit's RSS feeds for certain subreddits and using that to create and post
content for fairly spammy Twitter, Tumblr and Facebook pages. I'd parse the
titles (for example, change things like "cake day" to "birthday") and adding
hashtags at the end.

You could ensure only top content made it by setting a minimum karma limit. It
was a way of producing content 24/7.

I only did a couple of trial pages to test out the concept -- I never made any
money from it nor had any intention to monetize -- but I'm almost completely
certain that it was a method used by dozens of blackhats to make autoblogs for
the purposes of ad revenue. You could even fake a comment history or have a
"unique" title by selecting the top comments from the Reddit posts. Once you
have one recipe down, you can just copy it for other subreddits. And there was
an endless amount of hyper-focused niche subreddits that you can instantly
plug into.

Fun times.

~~~
chatmasta
People are still doing this, only now it's reddit -> instagram.

~~~
Breza
Not to mention Reddit -> Reddit I'm amazed at how successful some crosspost
bots are.

------
peteforde
I was at the O'Reilly Emerging Tech conference session that functioned as
(what I remember to have been) the Pipes launch party. Everyone involved with
the project seemed young and full of optimism, and while they didn't know
exactly how Pipes would make money, I remember being as impressed with the UI
as I was the forward-thinking ideas that motivated the project. Keep in mind
that in 2006, IE 6 was still omnipresent so the fact that they had a usable
drag and drop interface running put it in rarified air along with other UX
trailblazers like Flickr and DabbleDB.

To this day, I am still bewildered and not just a little angry about how
Yahoo's entire M&A machine was really just a catch-and-kill trap for taking
the most promising ideas out of the ecosystem and slowly resource starving
them into irrelevance.

~~~
madrox
I remember that ETech. Back when Yahoo was doing a bunch of things that were
cool and not profit-driven, like FireEagle.

In a lot of ways that tech was all ahead of its time. It's a shame the
monetization schemes weren't similarly evolved. If it were, Yahoo may still be
more relevant today.

~~~
peteforde
I met one of the FireEagle devs there, at the same ETech event, and he was a
really great person. We proceeded to bump into each other a few more times
over the years, specifically at the Ajax Experience conferences. He really
tried to convince me that Yahoo wasn't all that bad, although he eventually
did have to move on.

As evidenced by the previous paragraph, I've forgotten his name which makes it
exceptionally difficult to reconnect.

Consider this a HN "missed connections" attempt. ;)

~~~
kentbrew
Possibly Tom Coates?

~~~
peteforde
Damn, that is a great guess that is so close it hurts: you've made me
confident that I'm mis-remembering enough to think I'm actually talking about
the FireEagle launch event. The Pipes demo was a different session, possibly a
year or two earlier.

Either way, the person I am doing a laughably poor job of describing was at
the FireEagle event but I definitely remember him being a peer/coworker of
Tom's - he wasn't the center of attention that day.

Sigh.

~~~
kentbrew
Maybe check Tom's leaving-Yahoo post? There's a paragraph listing a bunch of
the devs:

"So I want to personally thank Seth Fitzsimmons, Samantha Tripodi, Jeannie
Yang, Chris Martin, Ben Ward, Kevin Ryan, Phil Pearson, Rabble, Arnab Nandi,
Simon King, Mor Naaman, Ayman Shamma and everyone else who worked on Fire
Eagle at any point in its life. I learned an enormous amount from all of you."

[http://plasticbag.org/archives/2010/05/my_last_day_at_yahoo](http://plasticbag.org/archives/2010/05/my_last_day_at_yahoo)

------
psadri
Thanks for posting and the nice comments. Glad to have built something that
people found useful...

~~~
ohashi
It feels like Zapier and IFTTT are the successors to Pipes, but I never quite
got the same feeling from them (maybe I haven't checked Zapier in a while).
But Pipes was ahead of its time and I'm still not sure of anything that really
comes close to what I could do with pipes even back then. Thank you.

~~~
mintplant
Integromat [0] somewhat approximates Pipes. Haven't heard as much about it as
IFTTT and Zapier. I found it because they've got one of the few remaining
active API integrations with RunKeeper, which since closed the door to new
application registrations.

[0] [https://www.integromat.com/](https://www.integromat.com/)

------
sn41
Yahoo pipes was really fun and great, and I was happy that someone besides
Google was still innovating in an exciting way. I remember being very sad when
Yahoo Pipes was finally unplugged. That day I realised that the web is not
about users, not about building fun and useful things, not about building
lego-like structures, it's just filter bubbles, echo chambers and troll farms.

------
netfl0
Oh boy do I remember yahoo pipes. I used it heavily for some time. I always
suspected their servers were under massive load running all their jobs so
sometimes things were slow.

Pipes made you think the future of the web was going to be technical and
brilliant.

~~~
8bitsrule
>Pipes made you think the future of the web was going to be technical and
brilliant.

Oh well, that was at least half right.

------
dankohn1
I was a big user of Yahoo Pipes for preparing custom RSS feeds.

I have now found that [https://feedity.com/](https://feedity.com/) does a good
enough job scraping pages to produce feeds. I then run that through
[https://fivefilters.org/content-only/](https://fivefilters.org/content-only/)
to fetch the full content of the page and read it in
[https://www.inoreader.com/](https://www.inoreader.com/), my favorite Google
Reader clone.

Some people are amazed that I am always on top of new content they produce.

~~~
nreece
Hey Dan, Feedity's founder here. Thank you for using our feeds service! Just
so you know, besides HTML webpages you can also create feeds for dynamic
webpages with client-scripts, as well as social media profiles.

------
Mizza
Pipes was _awesome_. I built many personal services with it - (and even a few
low traffic production services!)

Really tragic they killed it, that's when I knew the Mayer era of Yahoo would
be nothing more than a hatchet job.

I wish they made it F/OSS as part of the shutdown. Nothing has filled the
void.

~~~
diggan
As someone who just briefly used Pipes for a couple of things and don't
remember the service fully: What unique features of Pipes do you remember that
nothing else been able to replicate?

~~~
t0mbstone
A drag-and-drop engine for building extract/transform/load machines that ran
in the cloud at no cost?

You could make HTTP REST requests to pretty much anything and parse the result
and do whatever you wanted with it.

You could use Yahoo pipes to glue APIs together.

If you were the creative sort, the possibilities were endless.

Can _you_ name another product currently on the market that can do that?

~~~
plusfour
Node-RED?

~~~
t0mbstone
Oh wow! You are right! This is very similar. Thanks

------
egfx
Ah pipes. Thanks for the memories of a product that both came and went before
it’s time. And technologically an amazing frontend web app. There is an
attempt to remake pipes [https://pipes.digital](https://pipes.digital)

~~~
egfx
I really wish the pipes.digital project was open source. It could have really
gone somewhere. Without it being open source, it seems to have gone the way of
yahoo pipes itself. : /

------
elsurudo
To scratch a similar itch today, I self-host (well, on Heroku) an instance of
Huginn ([https://github.com/huginn/huginn](https://github.com/huginn/huginn)).
It's a bit of a different model, but works nicely. It's also pretty
straightforward to extend by adding your own agents (especially if you know
ruby/rails, which it is written in).

------
wiremine
Pipes where awesome!

The product seemed to have influenced a few products (off the top of my head):

1\. IBM's Node Red [1]

2\. AWS IoT Things Graph [2]

I'm sure there are others?

[1] [https://nodered.org/](https://nodered.org/)

[2] [https://aws.amazon.com/iot-things-
graph/?nc=sn&loc=3&dn=6](https://aws.amazon.com/iot-things-
graph/?nc=sn&loc=3&dn=6)

~~~
mikeleeorg
Though not exactly the same thing, I like to think of these as spiritual
successors:

[https://ifttt.com/](https://ifttt.com/)

[https://zapier.com/](https://zapier.com/)

~~~
addicted44
I agree. When I first heard of it, IFTTT was basically Pipes done right for
me.

In many ways Pipes was far too ahead of its time. The Web simply did not have
as much computation and reason for integrating data between different web apps
at the time.

------
hagmonk
Surprised to see no mention of Apache NiFi, Quartz Composer, or Simulink:

[https://en.wikipedia.org/wiki/Apache_NiFi](https://en.wikipedia.org/wiki/Apache_NiFi)

[https://en.wikipedia.org/wiki/Quartz_Composer](https://en.wikipedia.org/wiki/Quartz_Composer)

[https://en.wikipedia.org/wiki/Simulink](https://en.wikipedia.org/wiki/Simulink)

I feel like every few years GUI based workflow tools make a revival, but then
slowly die out. In the 1990s the selling point was that your "business logic"
could be written by "the business" instead of those expensive programmers, and
we all know how that panned out.

There is obviously something here, however, especially when you consider what
people are doing with Simulink. But why do these tools remain niche? Is there
some threshold of complexity past which these tools don't scale well, but
source code does? Considering these tools are effectively representations of a
program's call graph, could there be a future where I toggle between text and
graphical versions of my program depending on what works best?

~~~
stefan_
Simulink is hardly niche, it's running right now in hundreds of millions of
car engine controllers. And it isn't a programs call graph; Simulink models
usually don't have control flow at all. It's all about data flow.

------
mikepk
Wow memories. My first startup, grazr.com created a feed processing system
that could achieve similar (imho superior) results to pipes and worked in a
different way
([https://mikepk.com/2007/03/grazrscript-v12-beta/](https://mikepk.com/2007/03/grazrscript-v12-beta/)).
The idea was to use actual code (rather than graphical drag-drop) to write
feed processing and transformation applications.

We used server side javascript (before Node, I like to think we were ahead of
our time, our own fork of Spidermonkey with network and other server code!).
We used OPML as a container format with <grazr:script> tags in them analogous
to html <script> tags. The "pipes" in our system were much more like unix
pipes, where the output of one script could be the input to another. In the
running script you could access the feed with a customized "Feed Dom" with
items and do manipulations.

With grazr's real-time freshness, feed normalization and database it worked
amazingly well. We never were able to get people to understand it (it was
super nerdy). I built some pretty amazing apps with it, unfortunately none of
it exists anymore :).

Yahoo pipes seemed like the sign of the end for Grazr though. Since we had a
widget and feed display tech. a _lot_ of people wired their pipes to grazr's
widgets for display and publishing. We saw an initial "burst" on pipes
announce and then saw the yahoo pipes usage drop pretty fast and never really
recover. As a proxy, it looked like there wasn't enough "there" there to
continue to pursue our programming environment.

I still miss it though, it was fun to work on.

------
2bitencryption
I remember not too many years ago (2011/12?) Yahoo was _the_ best source of
free APIs for weather/stock/news information.

It was well documented, fully-featured, had a great web dashboard, and most
importantly of all it was super stable.

Not sure if it shut down or if it just became irrelevant in the modern web.
But many a engineering student plugged into it to make "hello world" weather
apps.

~~~
geerlingguy
They were shut down. I remember building many a widget that would pull
information from Yahoo's simple and thorough APIs, and they were all sunset
sometime between 2010 and 2015. Their stock API was unique in that it was one
of the few with good data on almost any ticker.

Most stock tracking APIs require licenses to use, or have terrible latency.
Similar with weather APIs now.

------
eddyg
Pipes was great!

Microsoft has Flow [0] now, but not sure how it compares.

[0] [https://flow.microsoft.com/](https://flow.microsoft.com/)

------
akavel
Mandatory fanboi mention: see also [https://luna-lang.org](https://luna-
lang.org) for a product that may hopefully become a worthy successor (and is
fully open-source!).

------
fosk
And I also remember its competitor, Microsoft Popfly [1].

[1]
[https://en.wikipedia.org/wiki/Microsoft_Popfly](https://en.wikipedia.org/wiki/Microsoft_Popfly)

------
peterburkimsher
I used Pipes for RSS processing as well. I set up a rather convoluted system
that let me forward email to SMS for free.

It used a weird mix of free services, including Yahoo Pipes and IFTTT. It went
from email to RSS to Google Calendar to SMS. It was nice while it lasted, but
now Google Calendar doesn't send SMS notifications so it's no longer possible
for free.

------
imartin2k
I remember I used Pipes to create a new single RSS feed containing dozens of
separate, rarely updated RSS feeds. For some reason I preferred seeing that
one feed subscribed in my — wait for it — Google Reader, than a long list of
feeds that rarely got updated. Well. I managed to survive even after Pipes
disappeared.

~~~
Lammy
I also remember using it to create RSS feeds for sites that lacked them.

------
tosh
[https://news.ycombinator.com/item?id=9660682](https://news.ycombinator.com/item?id=9660682)
(related discussion from the 2015 end-of-life announcement)

------
prepend
I really liked pipes. I used it to combine the RSS feeds from lots of google
code and GitHub repos into a single project feed so I could see all commits
across my group in a single, chronological flow.

It let me ignore lots of Designers limitations and work around them.

Although it was possible to code it all directly in many languages it’s an
example of how making something more accessible is good, not just making it
possible. Also free cloud running.

------
donatj
Pipes was great. I miss it constantly. I used it largely to filter RSS feeds -
for instance where I was only interested in posts by certain authors or posts
from certain categories of a website.

It was very easy to build these sorts of filters with their visual ui.

------
WA9ACE
Pipes was such a good product and sorely missed. Is there anything equivalent
to it today?

~~~
geraldbauer
You can use modern scripting languages such as Python, Ruby and friends. They
will never go out of fashion if you want to put together a pipeline (also
known as workflow or script :-).

~~~
naikrovek
I really don't think Python qualifies as an _equivalent_ replacement for
Pipes.

~~~
geraldbauer
Of course not :-) Python, Ruby and friends can do so much more (and you can
run your pipes on your own computer, for example, ...).

------
jangid
Yahoo Pipes, Friendfeed were all fun. The world was more open at that point of
time.

------
newshorts
The comments here are validating. I thought I was going to be one of the few
“old guys” who actually remembered/used pipes.

Glad to see there was so much love for one of my favorite things about the
internet!

------
megahz
i liked yahoo pipes so much that after decommissioned i wrote this article (in
greek sorry) of how much i used and liked the product:
[https://medium.com/andreas-personal-
blog/%CF%80%CE%B1%CE%AF%...](https://medium.com/andreas-personal-
blog/%CF%80%CE%B1%CE%AF%CE%B6%CE%BF%CE%BD%CF%84%CE%B1%CF%82-%CE%BC%CE%B5-%CF%84%CE%BF-
yahoo-pipes-alternatives-47b0f1417f2f)

Huge pipes that might cause issueS? that could have be me :P

------
edgar971
I feel like it's similar to Airflow
[https://airflow.apache.org/](https://airflow.apache.org/)

~~~
jldugger
It is, but it provided a ton of web-centric APIs in an era where people
published useful RSS feeds. Half the value was in publishing modules that
spoke RSS in some form, and the other half was a number of online query
systems that spoke HTTP and responded with RSS.

There were many technical problems with pipes and one major business problem.
The technical problems as usual, were solvable:

\- branching. the system really didn't have a clean way to cause branching. if
you wanted to only apply a regex on certain items, that would fail hard. I
feel like some part of the anime fansub system I built must have been
stressing the system because I'd occasionally find it disabled or key
components removed from Pipes. (Deluge can consume RSS in the background, but
often the fansub RSS feeds have duplicate items for different qualities, or
are missing the enclosure field the system keys off of and other field
normalization problems)

\- types. the system really operated on RSS as a medium of data exchange. Any
operation that worked on whole RSS feeds was great. So the typical 'put a ton
of blogs into a pipe and union them all into one big feed' hello world app
worked great. As soon as you wanted to operate on a specific line of input
items and do subprocessing, you were in a sort of hell. It wasn't clear which
modules worked on which types, or why.

\- composition. a general engineering principle is to build things out of
small parts. That's the fundamental reason pipes works, but you never had the
ability to treat a pipe you built as a small part itself. You couldn't build
pipes that accepted RSS in. The subpipe command they eventually added only
allowed RSS out, without even any parameterization. Judging by the comments
from support engineers here, the reason is mostly obvious: they couldn't
prevent people from building pipes that called themselves indirectly.

But the business problem was most fundamental. The most useful APIs were often
monetized via web ads. Yea, someone was collecting amazon referral revenue,
but beyond that, no money was changing hands. That was a fundamental challenge
to further adoption. Bloggers started publishing truncated or headline blog
feeds, and what sort of evolved was a cat and mouse game with content
creators: people invent new ways to generate RSS feeds for sites that don't
have them while the sites fuzz their UI to prevent it.

------
parentheses
I used pipes to build custom RSS feeds to feed into google reader. I miss this
product and google reader so very badly.

------
sajithdilshan
It was really a good product. I've used it to normalize XML from couple of
sources to create a news aggregator

------
romdev
Yahoo Pipes -> Google Reader is now Feedrinse -> Newsblur

~~~
cbzbc
Yeah, though given the history of both of the former I'm a bit wary of using a
non-open source solution I can't host myself (Newsblur can be self hosted, but
AFAIK Feedrinse can't).

------
OrgNet
pipes looked like the future ... but it was too early, I guess

------
karlerss
Alteryx is a 7B dollar company. Their Alteryx Designer product looks like this
& does the some of the same things.

