Hacker News new | past | comments | ask | show | jobs | submit login
Yahoo Pipes (wikipedia.org)
224 points by tosh on Nov 25, 2019 | hide | past | favorite | 94 comments

A couple of fun behind the scenes stories which I think at this point are past the statute of limitations.

Source: I was the Service Engineer on pipes for around a year or so.

1) When you went to the pipes landing page, there were a few demo pipes to show people what was possible. One of them combined search results from say ebay/craiglist/amazon to show prices for things.

One day I was looking through the source of these, and noticed there were affiliate ids in the ebay/amazon links. They all belonged to some early team member who had long since left.

I showed a couple people on the team, we all guessed at how much they were making from this, said good on em and went about our day. I still wonder how much they ended up making from it.

2) It was my first on-call, and all of a sudden the west coast pipes cluster just went bananas. After ~5 minutes, east coast started to go nuts and the west coast subsided.

Someone, despite numerous defensive measures, had found a way to create a pipe-bomb that would recursively call multiple versions of itself. Once the west coast load balancer failed over, one of these requests would hit the east coast and the 'virus' would jump over there. This flipped flopped back and forth until I figured out how to blacklist pipe ids (and eventually got a code fix).

Heh... just chiming in to say the affiliate person wasn’t me :-)

I wish it was me. That's genius.

i know who it is, but i'll never tell :)

As a fresh Yahoo! Developer Network employee I was the person who blabbed about the top-secret _callback parameter the weekend Pipes launched, which caused the service to be taken offline for capacity upgrade.

I had an interesting series of voicemails, ranging from "dude, WTF?" to "okay, we need to talk to you right away" to "we're pretty sure you're fired" but ending with "hey, Tim O'Reilly picked it up and ALL IS FORGIVEN THANK YOU!!!"


And now, Pipes can be mobilized and viewed on any mobile phone using SMS and WAP! Simply get the RSS feed of the Pipe and mobilize it using the 411Sync Mobile API. Within seconds, view it on your cell phone

Oh wow. There is so much in this paragraph that was best left forgotten.

What were the implications of knowing the _callback parameter?

You could invoke pipes from any webpage without worrying about cross-domain access.

If you look at some janky JS code[1] I had on my website at the time, you can see what you can do with it.

That would basically source


as a static asset in my page, which would load the JS and call load_daily_show(<json>) as a result.

Now, EVERYONE who hits my page is invoking the pipe as a backend API call, with no caching and unfortunately with the entire Y/T cookies intact.

[1] - https://web.archive.org/web/20081007043923/http://t3.dotgnu....

Yeah I remember using it as a CORS proxy, it allowed me to put together a full client side music app (tracks source was not very legal) but tags, similar tracks/artists etc were pulled from lastfm and allowed to quickly build a decent playlist.

At that point in history APIs that returned a JSON object wrapped in a callback were few and far between (and Yahoo! had almost all of them). Pipes would give you anything you wanted in a callback, allowing for nothing-but-front-end mashups of anything that would hold still while being scraped.

Craigslist banned Pipes and un-banned it after YDN employee Jeremy Zawodny went to work there:

https://techcrunch.com/2009/12/01/craigslist-yahoo-pipes-fli... https://techcrunch.com/2009/12/16/craigslist-yahoo-pipes/

I used Yahoo! Pipes in a somewhat greyhat way. I started pulling in data from Reddit's RSS feeds for certain subreddits and using that to create and post content for fairly spammy Twitter, Tumblr and Facebook pages. I'd parse the titles (for example, change things like "cake day" to "birthday") and adding hashtags at the end.

You could ensure only top content made it by setting a minimum karma limit. It was a way of producing content 24/7.

I only did a couple of trial pages to test out the concept -- I never made any money from it nor had any intention to monetize -- but I'm almost completely certain that it was a method used by dozens of blackhats to make autoblogs for the purposes of ad revenue. You could even fake a comment history or have a "unique" title by selecting the top comments from the Reddit posts. Once you have one recipe down, you can just copy it for other subreddits. And there was an endless amount of hyper-focused niche subreddits that you can instantly plug into.

Fun times.

I used to use pipes for a network of real estate listing websites. I would combine Craigslist, Ooodle, Trulia and some others and then place Adsense ads around the feeds.

My sites were getting a lot og organic search traffic, but were ultimately dropped for not having unique content

Oh you bet Pipes was a godsend for anyone gaming Google SERPs at the time!

People are still doing this, only now it's reddit -> instagram.

Not to mention Reddit -> Reddit I'm amazed at how successful some crosspost bots are.

I was at the O'Reilly Emerging Tech conference session that functioned as (what I remember to have been) the Pipes launch party. Everyone involved with the project seemed young and full of optimism, and while they didn't know exactly how Pipes would make money, I remember being as impressed with the UI as I was the forward-thinking ideas that motivated the project. Keep in mind that in 2006, IE 6 was still omnipresent so the fact that they had a usable drag and drop interface running put it in rarified air along with other UX trailblazers like Flickr and DabbleDB.

To this day, I am still bewildered and not just a little angry about how Yahoo's entire M&A machine was really just a catch-and-kill trap for taking the most promising ideas out of the ecosystem and slowly resource starving them into irrelevance.

I remember that ETech. Back when Yahoo was doing a bunch of things that were cool and not profit-driven, like FireEagle.

In a lot of ways that tech was all ahead of its time. It's a shame the monetization schemes weren't similarly evolved. If it were, Yahoo may still be more relevant today.

I met one of the FireEagle devs there, at the same ETech event, and he was a really great person. We proceeded to bump into each other a few more times over the years, specifically at the Ajax Experience conferences. He really tried to convince me that Yahoo wasn't all that bad, although he eventually did have to move on.

As evidenced by the previous paragraph, I've forgotten his name which makes it exceptionally difficult to reconnect.

Consider this a HN "missed connections" attempt. ;)

You're more than likely thinking of Seth Fitzsimmons: https://twitter.com/mojodna

Possibly Tom Coates?

Damn, that is a great guess that is so close it hurts: you've made me confident that I'm mis-remembering enough to think I'm actually talking about the FireEagle launch event. The Pipes demo was a different session, possibly a year or two earlier.

Either way, the person I am doing a laughably poor job of describing was at the FireEagle event but I definitely remember him being a peer/coworker of Tom's - he wasn't the center of attention that day.


Maybe check Tom's leaving-Yahoo post? There's a paragraph listing a bunch of the devs:

"So I want to personally thank Seth Fitzsimmons, Samantha Tripodi, Jeannie Yang, Chris Martin, Ben Ward, Kevin Ryan, Phil Pearson, Rabble, Arnab Nandi, Simon King, Mor Naaman, Ayman Shamma and everyone else who worked on Fire Eagle at any point in its life. I learned an enormous amount from all of you."


> UX trailblazers like Flickr and DabbleDB.

And Meebo! They build an entire windowing system in the browser...and supported IE6.

Thanks for posting and the nice comments. Glad to have built something that people found useful...

It feels like Zapier and IFTTT are the successors to Pipes, but I never quite got the same feeling from them (maybe I haven't checked Zapier in a while). But Pipes was ahead of its time and I'm still not sure of anything that really comes close to what I could do with pipes even back then. Thank you.

Integromat [0] somewhat approximates Pipes. Haven't heard as much about it as IFTTT and Zapier. I found it because they've got one of the few remaining active API integrations with RunKeeper, which since closed the door to new application registrations.

[0] https://www.integromat.com/

Node-RED seems to be the best alternative which is successful and good enough these days. It's selfhosted, and it seems combining it with home assistent seems also very popular to get the best of it.

Sorry, one last note. I was a broke college student at the time. I made tens of thousands of dollars with Pipes doing ethical arbitrage. I still miss it, where's our Kickstarter?

arbitrage of what? Sounds interesting

Sure, I can share. I live abroad now and don't have time to do it anymore. Back in 2008-9? there were 7 major estate sale websites. Nowadays everything has consolidated to estatesales.net. 4 of the 7 had RSS feeds. I drank in all 4 RSS feeds, filtered out sales without pictures, and filtered towards certain keywords. My game was second hand art buying, but only in a 100 mile radius. As I got better at it, my radius started expanding. Note, it wasn't make $300 every sale, it was very hit or miss. But load the car up with some Ed Ruscha originals and Frankenthaler prints and make a killing consigning to auction houses for a morning's work. The old man pickers always wondered how I always got the jump on them. Thanks for paying off my college debt early Yahoo Pipes. Please come back!

Was Caterina Fake involved in any way with Yahoo Pipes? I believe she headed Yahoo Brickhouse at the time, but the timelines are foggy for me. Pipes is so unlike any Yahoo product.

Pipes preceded Brickhouse and Caterina Fake's involvement. Brickhouse was the first step in encrusting the Pipes project in enough layers of management to ensure it would die.

If I recall correctly Pipes was originally a 100%-Sunnyvale product, predating Brickhouse.

Congratulations on having created something that many people seem to have a fond opinion of. That must feel good.

Yahoo pipes was really fun and great, and I was happy that someone besides Google was still innovating in an exciting way. I remember being very sad when Yahoo Pipes was finally unplugged. That day I realised that the web is not about users, not about building fun and useful things, not about building lego-like structures, it's just filter bubbles, echo chambers and troll farms.

I was a big user of Yahoo Pipes for preparing custom RSS feeds.

I have now found that https://feedity.com/ does a good enough job scraping pages to produce feeds. I then run that through https://fivefilters.org/content-only/ to fetch the full content of the page and read it in https://www.inoreader.com/, my favorite Google Reader clone.

Some people are amazed that I am always on top of new content they produce.

Hey Dan, Feedity's founder here. Thank you for using our feeds service! Just so you know, besides HTML webpages you can also create feeds for dynamic webpages with client-scripts, as well as social media profiles.

Oh boy do I remember yahoo pipes. I used it heavily for some time. I always suspected their servers were under massive load running all their jobs so sometimes things were slow.

Pipes made you think the future of the web was going to be technical and brilliant.

>Pipes made you think the future of the web was going to be technical and brilliant.

Oh well, that was at least half right.

Pipes was _awesome_. I built many personal services with it - (and even a few low traffic production services!)

Really tragic they killed it, that's when I knew the Mayer era of Yahoo would be nothing more than a hatchet job.

I wish they made it F/OSS as part of the shutdown. Nothing has filled the void.

As someone who just briefly used Pipes for a couple of things and don't remember the service fully: What unique features of Pipes do you remember that nothing else been able to replicate?

We built apps for Yahoos connected TV platform (it was based on the Konfabulator widget engine) back in the day. Getting API support from clients (broadcast networks) for niche platforms was next to impossible and trying to aggregate all the data via multiple calls on constrained devices wasn’t feasible either.

We used pipes to aggregate multiple services into a single API a few times at Yahoo’s recommendation and it worked pretty well. Also helped get around some CORS issues IIRC.

A drag-and-drop engine for building extract/transform/load machines that ran in the cloud at no cost?

You could make HTTP REST requests to pretty much anything and parse the result and do whatever you wanted with it.

You could use Yahoo pipes to glue APIs together.

If you were the creative sort, the possibilities were endless.

Can you name another product currently on the market that can do that?


Oh wow! You are right! This is very similar. Thanks

This one looks promising from an open source alternative that you can host locally:


Ifttt is a joke compared to Yahoo Pipes. It can only do a very limited number of pre-defined transformations and it’s only 2 steps.

Yahoo Pipes was basically programming with a UI.

whatever your opinion is, it is still a viable alternative for doing simple things. the dude asked for alternatives, not opinions.

Ah pipes. Thanks for the memories of a product that both came and went before it’s time. And technologically an amazing frontend web app. There is an attempt to remake pipes https://pipes.digital

I really wish the pipes.digital project was open source. It could have really gone somewhere. Without it being open source, it seems to have gone the way of yahoo pipes itself. : /

To scratch a similar itch today, I self-host (well, on Heroku) an instance of Huginn (https://github.com/huginn/huginn). It's a bit of a different model, but works nicely. It's also pretty straightforward to extend by adding your own agents (especially if you know ruby/rails, which it is written in).

Surprised to see no mention of Apache NiFi, Quartz Composer, or Simulink:




I feel like every few years GUI based workflow tools make a revival, but then slowly die out. In the 1990s the selling point was that your "business logic" could be written by "the business" instead of those expensive programmers, and we all know how that panned out.

There is obviously something here, however, especially when you consider what people are doing with Simulink. But why do these tools remain niche? Is there some threshold of complexity past which these tools don't scale well, but source code does? Considering these tools are effectively representations of a program's call graph, could there be a future where I toggle between text and graphical versions of my program depending on what works best?

Simulink is hardly niche, it's running right now in hundreds of millions of car engine controllers. And it isn't a programs call graph; Simulink models usually don't have control flow at all. It's all about data flow.

Wow memories. My first startup, grazr.com created a feed processing system that could achieve similar (imho superior) results to pipes and worked in a different way (https://mikepk.com/2007/03/grazrscript-v12-beta/). The idea was to use actual code (rather than graphical drag-drop) to write feed processing and transformation applications.

We used server side javascript (before Node, I like to think we were ahead of our time, our own fork of Spidermonkey with network and other server code!). We used OPML as a container format with <grazr:script> tags in them analogous to html <script> tags. The "pipes" in our system were much more like unix pipes, where the output of one script could be the input to another. In the running script you could access the feed with a customized "Feed Dom" with items and do manipulations.

With grazr's real-time freshness, feed normalization and database it worked amazingly well. We never were able to get people to understand it (it was super nerdy). I built some pretty amazing apps with it, unfortunately none of it exists anymore :).

Yahoo pipes seemed like the sign of the end for Grazr though. Since we had a widget and feed display tech. a _lot_ of people wired their pipes to grazr's widgets for display and publishing. We saw an initial "burst" on pipes announce and then saw the yahoo pipes usage drop pretty fast and never really recover. As a proxy, it looked like there wasn't enough "there" there to continue to pursue our programming environment.

I still miss it though, it was fun to work on.

Pipes where awesome!

The product seemed to have influenced a few products (off the top of my head):

1. IBM's Node Red [1]

2. AWS IoT Things Graph [2]

I'm sure there are others?

[1] https://nodered.org/

[2] https://aws.amazon.com/iot-things-graph/?nc=sn&loc=3&dn=6

Though not exactly the same thing, I like to think of these as spiritual successors:



I agree. When I first heard of it, IFTTT was basically Pipes done right for me.

In many ways Pipes was far too ahead of its time. The Web simply did not have as much computation and reason for integrating data between different web apps at the time.

Apart from those, I’d suggest to take a look to Integromat. Their “http module” (a UI for a curl-like service) is really handy when you want to interact with a 3rd party service that hasn’t been integrated yet.

Azure Logic Apps / Microsoft Flow is yet another that likely owes some direct influence to Pipes.


https://n8n.io/ is another option.


I remember not too many years ago (2011/12?) Yahoo was the best source of free APIs for weather/stock/news information.

It was well documented, fully-featured, had a great web dashboard, and most importantly of all it was super stable.

Not sure if it shut down or if it just became irrelevant in the modern web. But many a engineering student plugged into it to make "hello world" weather apps.

They were shut down. I remember building many a widget that would pull information from Yahoo's simple and thorough APIs, and they were all sunset sometime between 2010 and 2015. Their stock API was unique in that it was one of the few with good data on almost any ticker.

Most stock tracking APIs require licenses to use, or have terrible latency. Similar with weather APIs now.

Pipes was great!

Microsoft has Flow [0] now, but not sure how it compares.

[0] https://flow.microsoft.com/

Mandatory fanboi mention: see also https://luna-lang.org for a product that may hopefully become a worthy successor (and is fully open-source!).

And I also remember its competitor, Microsoft Popfly [1].

[1] https://en.wikipedia.org/wiki/Microsoft_Popfly

I used Pipes for RSS processing as well. I set up a rather convoluted system that let me forward email to SMS for free.

It used a weird mix of free services, including Yahoo Pipes and IFTTT. It went from email to RSS to Google Calendar to SMS. It was nice while it lasted, but now Google Calendar doesn't send SMS notifications so it's no longer possible for free.

I remember I used Pipes to create a new single RSS feed containing dozens of separate, rarely updated RSS feeds. For some reason I preferred seeing that one feed subscribed in my — wait for it — Google Reader, than a long list of feeds that rarely got updated. Well. I managed to survive even after Pipes disappeared.

I also remember using it to create RSS feeds for sites that lacked them.

https://news.ycombinator.com/item?id=9660682 (related discussion from the 2015 end-of-life announcement)

I really liked pipes. I used it to combine the RSS feeds from lots of google code and GitHub repos into a single project feed so I could see all commits across my group in a single, chronological flow.

It let me ignore lots of Designers limitations and work around them.

Although it was possible to code it all directly in many languages it’s an example of how making something more accessible is good, not just making it possible. Also free cloud running.

Pipes was great. I miss it constantly. I used it largely to filter RSS feeds - for instance where I was only interested in posts by certain authors or posts from certain categories of a website.

It was very easy to build these sorts of filters with their visual ui.

Pipes was such a good product and sorely missed. Is there anything equivalent to it today?

I'm in the middle of pushing out v1.0 of Simple scraper[0] this week which will recreate some of the functionality.

Having a lot of fun building the product but with so many of these types of companies shutting down it has me wondering what it forebodes :/

[0] https://simplescraper.io

I think Zapier and IFTTT fill this gap today.

I think there's a lot of projects in this space right now. Especially also fueled by all the people building things the "no code" way.

IFTTT, Zapier, Webflow, integromat etc.

import.io mixed with Zapier / IFTT


You can use modern scripting languages such as Python, Ruby and friends. They will never go out of fashion if you want to put together a pipeline (also known as workflow or script :-).

I really don't think Python qualifies as an equivalent replacement for Pipes.

Of course not :-) Python, Ruby and friends can do so much more (and you can run your pipes on your own computer, for example, ...).

You completly miss the point of what made pipe so useful. Good languages existed back than too, and there were as powerful as today, as they are complicated and timeconsuming. They are simply not an alternative for this.

Yahoo Pipes, Friendfeed were all fun. The world was more open at that point of time.

The comments here are validating. I thought I was going to be one of the few “old guys” who actually remembered/used pipes.

Glad to see there was so much love for one of my favorite things about the internet!

i liked yahoo pipes so much that after decommissioned i wrote this article (in greek sorry) of how much i used and liked the product: https://medium.com/andreas-personal-blog/%CF%80%CE%B1%CE%AF%...

Huge pipes that might cause issueS? that could have be me :P

I feel like it's similar to Airflow https://airflow.apache.org/

It is, but it provided a ton of web-centric APIs in an era where people published useful RSS feeds. Half the value was in publishing modules that spoke RSS in some form, and the other half was a number of online query systems that spoke HTTP and responded with RSS.

There were many technical problems with pipes and one major business problem. The technical problems as usual, were solvable:

- branching. the system really didn't have a clean way to cause branching. if you wanted to only apply a regex on certain items, that would fail hard. I feel like some part of the anime fansub system I built must have been stressing the system because I'd occasionally find it disabled or key components removed from Pipes. (Deluge can consume RSS in the background, but often the fansub RSS feeds have duplicate items for different qualities, or are missing the enclosure field the system keys off of and other field normalization problems)

- types. the system really operated on RSS as a medium of data exchange. Any operation that worked on whole RSS feeds was great. So the typical 'put a ton of blogs into a pipe and union them all into one big feed' hello world app worked great. As soon as you wanted to operate on a specific line of input items and do subprocessing, you were in a sort of hell. It wasn't clear which modules worked on which types, or why.

- composition. a general engineering principle is to build things out of small parts. That's the fundamental reason pipes works, but you never had the ability to treat a pipe you built as a small part itself. You couldn't build pipes that accepted RSS in. The subpipe command they eventually added only allowed RSS out, without even any parameterization. Judging by the comments from support engineers here, the reason is mostly obvious: they couldn't prevent people from building pipes that called themselves indirectly.

But the business problem was most fundamental. The most useful APIs were often monetized via web ads. Yea, someone was collecting amazon referral revenue, but beyond that, no money was changing hands. That was a fundamental challenge to further adoption. Bloggers started publishing truncated or headline blog feeds, and what sort of evolved was a cat and mouse game with content creators: people invent new ways to generate RSS feeds for sites that don't have them while the sites fuzz their UI to prevent it.

I used pipes to build custom RSS feeds to feed into google reader. I miss this product and google reader so very badly.

It was really a good product. I've used it to normalize XML from couple of sources to create a news aggregator

Yahoo Pipes -> Google Reader is now Feedrinse -> Newsblur

Yeah, though given the history of both of the former I'm a bit wary of using a non-open source solution I can't host myself (Newsblur can be self hosted, but AFAIK Feedrinse can't).

pipes looked like the future ... but it was too early, I guess

Alteryx is a 7B dollar company. Their Alteryx Designer product looks like this & does the some of the same things.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact