By the time I turned the west coast back online, the east coast was failing health checks, and the LB failed everything over to the west coast, which then proceeded to be overloaded, and everything flipped back to the east coast... Ping ponged for about 2 hours until they finally settled.
Another fun story: For the longest time the front page had an example pipe that merged search results from various online sites (amazon/ebay/cl). It was made by a former employee and was easily one of the most popular pipes. One day we found out he had his affiliate id in all of those links. We chuckled and moved on.
Wow that must have added up to quite the haul
Just to be clear, I had not idea about that until now.
The backend was an extremely useful tool for munging RSS feeds. With any kind of support, or even benign neglect, the product would have been successful. It took a lot of active mismanagement and folly to keep Pipes from living up to its promise.
Hats off to psadri and the other Pipes creators for a really stellar piece of work.
I feel like this unique blend will be Yahoo's lasting contribution to the field of management theory.
Here in Brazil Elop effect was interesting: it was VERY obvious, yet noone here knows who the hell Elop is.
To put it simple: When Elop released that stupid memo, Nokia had 67% of Brazil smartphone market share, and the share was still rising, and Nokia utterly dominated in the non-smartphone market too. Right after Elop memo, the first thing that happened is that suddenly the communities died (although Nokia store DID suck, there was at least in Brazil a vibrant freeware app community that shared stuff in forums), then just some months later you could not find phones for sale, even new models, because the import companies DID heard of Elop memo, and decided to not risk importing the phones, even if shopkeepers still wanted it.
The end result was funny: Elop nuking of Symbian was so hard, that in Brazil other smartphones could not keep up for a while, and we temporarily had a decrease in smartphone use, and a increase in feature phone use, and that increase came mostly from truly crap chinese companies, because not even Nokia feature phones could be found in stores anymore.
By the time Elop was announced as CEO in late 2010, Kallasvuo had already guaranteed Nokia was toast.
Just go back to the way Nokia was behaving, and what they were saying in 2007/2008 for easy proof of that.
"Nokia CEO calls Apple iPhone ‘niche product’"
> In the same year, he joined Macromedia's Web/IT department [...] from January 2005 for three months before their acquisition by Adobe Systems
> He was then president of worldwide field operations at Adobe, tendering his resignation in June 2006 and leaving in December, after which he was the COO of Juniper Networks for exactly one year from January 2007 – 2008
This is not the CV of a man that I would hire for anything other than running a going concern into the ground.
Pipes's interface is primary the work of Jonathan Trevor, with lots of help from Daniel Raffel, Ed Ho and others.
I would love to read a blog post about how exactly it was mismanaged. It would be great for others to learn lessons from the mistakes.
 I don't have first-hand experience but one of my best friends work there and we regularly vent to each other about work complaints.
> With any kind of support, or even benign neglect, the product would have been successful.
Look at IFTTT services now and it's easy to realize that pipes even in its original incarnation could have been so much more given its brain dead UI ease.
It's a shame that Yahoo is shutting this down rather than pivoting it as an IFTTT or similar. I'd wager that it's lack of ownership that makes it hard to pivot. I also don't recall it having a premium tier which has got to weigh down the resources--talent always gravitates toward revenue centers it seems.
Ie the 'new management can't fire my entire division for being unprofitable' center. Seems like a smart move for the people who wrote Pipes but never found a business model for it.
I'm sad to see it go and it'll be missed by quite a few people who use my plugin.
That faith is dead now, of course, scoured from the earth by walled gardens and VC money. But like Catholics in Elizabethan England, some of us quietly tend our secret shrines and pray for its return.
(Hopefully that will work out better for us than it did for them.)
> Bosworth advocated an open model for data.(...)
That is all dead now.
However, why such pessimism? I hardly think that standard formats and open APIs are dead. Sure, there are some nasty blights upon cyberspace in this regard (ahem, Facebook, Apple), but other players are still keeping the dream alive to some extent (e.g. Google, Yahoo).
But it is possible that it's confirmation bias + uncommon sample on my part.
What do you mean? Channels have an element linking to an RSS feed. In fact, just pasting a channel URL to an RSS reader should work.
Example of a channel feed: https://www.youtube.com/feeds/videos.xml?channel_id=UCr4soU_...
Maybe start a new, open source version of Pipes?
Maybe im dreaming about a free version of Percolate, which I have heard is really cool.
Pipes would often get blocked by 3rd party data providers, which motivated me to roll my own library.
No graphical interface. Personally, I found Y! Pipes UI hard to work with when I needed to build large pipelines.
RssPercolator is a .NET library used for downloading, aggregating, and filtering RSS feeds. Developed out of frustration with Yahoo Pipes.
- RSS and Atom feed formats
- Asynchronous multi-feed download
- Multiple sources and multiple destinations
- Feed filters (string match, wildcards, and Regex)
We (Webflow) are actually considering building a Pipes alternative in the near future, and I'd love to sit down with you and brainstorm some ideas. If you'd be willing to make some time to see a very rough demo, I'd love to get in touch - my email is in my profile.
For another, Yahoo legal prevented us from doing any kind of long-term storage of RSS feeds. That made the service far less useful than it could have been.
Finally, they didn't give it any resources at all. Management was too preoccupied with the failed Brickhouse project (internal startup incubator) and the host of failed ideas that it spawned.
What was Pipes for?
Wait, is there something legally wrong about storing RSS feed data?
I wonder if it would be possible to build something similar using IPV6/some kind of P2P/STUN/TURN/UDP hole punching/swarm/ethereum / any way to make a distributed system that does the same thing. Or maybe just volunteer running Node on their own VPSs.
Seriously, I think the only Yahoo products I use are Pipes, Groups and Flickr. And Flickr doesn't really count.
It definitely inspires a "never forget" kind of mindset, such that I don't trust relying on a SaaS that I don't have a personal exit strategy for.
Their site aesthetic is trying to be Nest-y, but the UX does not communicate simplicity at all.
The only thing Pipes would need is a .io domain and it could be dressed up as the Next Cool Thing which LargeCorp is desperate to get its hands on to prove it's still cool.
Pipes was a basic web "agent". It made basic programming available to the everyman much like HyperCard. Perhaps is just needed a runtime UI that matched the excellence of its design time interface.
Yahoo is focussing on mobile yet here they had this custom agent building tool that could easily be re-purposed to mobile to make a killer platform for Yahoo users.
Hopefully they will opensource the first generation perl version.
Some of my favourite flow based programming links follow.
Surprised nobody has mentioned noflow node.js workflows:
http://www.kamaelia.org/Home.html (BBC research)
Pipes was influenced by unix pipes (hence the name). The UI was influenced by National Instrument's LabView and Apple's Automator.
I wish Pipes was launched in the age of containers (e.g. Docker). We had the idea of one click deployment to what we today call a container.
ConMan: A Visual Programming Language for Interactive Graphics (1988)
Traditionally, interactive applications have been difficult to build, modify and extend. These integrated applications provide bounded functionality, have a single thread of control and a fixed user interface that must anticipate every-thing the user will need. Current workstations allow several processes to share the screen. With proper communication between processes, it is possible to escape previous models for application development and evolution. ConMan is a high-level visual language we use on an IRIS workstation that lets users dynamically build and modify graphics applications. To do this, a system designer disintegrates complex applications into modular components. By interactively connecting simple components, the user constructs a complete graphics application that matches the needs of a task. A connection manager controls the flow of data between individual components. As a result, we replace the usual user-machine dialog with a dynamic live performance that is orchestrated by the user.
Following RSS feeds on a continuing basis takes a lot of RSS polls. Most RSS feeds do not implement RSS in a way that allows getting only new items reliably. The RSS "etag" mechanism is not reliable. Some sites with multiple servers and a load balancer have different etag values on each server. The "guid" field sometimes changes when the content hasn't changed. My experience is that nothing short of full text comparison eliminates duplicates properly. I wrote an RSS reader which does a MD5 of the text of each incoming message to throw out duplicates. Presumably the "pipes" system did something similar.
If RSS feed servers complied with the standard, there'd be less need for feed aggregation services.
I'd sure love to pick the brain of anyone formerly involved in the project - it seems like there is a lot we could learn from the trail it blazed. My email in profile, obligatory beers/coffee/etc. offer.
My only wish would be for a paid tier between 100 and 3000. If I went hog-wild and did everything I want to do, I'd probably use 200-300 a month. I would love to pay for that. But there's some loss-aversion psychology happening that makes me not want to pay for 3000 if I know I'll only use 300.
I'd give you my cc number right now if you did a 1000 zaps at $10/m.
That being said - you know your business better than I do. If it would hurt conversions of $20/m more than it would help, I get that. :)
<3 you guys though.
It seems like that program has since been limited, however, as it now redirects to a referral page only. A combination of a couple referrals around the office and performing the required tasks gave us about ~650 monthly "zaps" at the time and we haven't needed more than that so far. Still, I would have happily paid $5/month or something to avoid the hassle (and be able to pay them for their wonderful product).
> ChangeTracker watches the White House’s web site so you don’t have to. Whenever a page on whitehouse.gov changes, we’ll let you know — via e-mail, Twitter, or RSS. But ChangeTracker is not a piece of software. It’s the output of a series of powerful and mostly free Web-based tools, lovingly connected over the Internet. Here’s how to do it yourself so you can track changes on any Web site on the Internets.
One pipe would grab data from Craigslist, strip out irrelevant items, and send me a text if anything new and interesting appeared. Ditto for Westside Rentals.
This combination worked great. If a lame apartment appeared in CL, I'd edit my Pipes regex to strip the same thing out of WR. In this way I'd get only texts for awesome places.
Vaya con dios, Yahoo Pipes!
I also used it with Freecycle and got a fridge-freezer! (you had to be quick to get them)
Thanks to whoever made it. I guess I'll have to find a way to do it myself now.
And another where I put the pictures inline (though the 3rd party API I was using isn't handling things too well anymore)
I still have 146 subscribers to keep happy, though Feedburner should make any transition transparent. I'm disappointed it's going. Thank you to those who made it!
It was an eye opener for some: they could get, transform and repurpose data, and it could be fun.
Pipes wasn't without its flaws: V1 was buggy and gave us a lot of trouble when 30 students were trying to use it from the same IP, the interface was plagued by the usual issues of visual interfaces (clutter…). Some things were odd, some a little too hard for what they achieved and other were magically easy!
But at a moment when every tech company is saying that "learning to code" is important, it's sad to see a tool that had real educational value disappear.
It was a really effective tool for non-developers to learn about data markup standards, to think in term of data flows, to get introduced to the idea of a data or web API.
For this particular use in education I'm not sure there is a replacement.
I also thought that pipes queries were amazing, but slow, so it was hard to use them as part of an application.
Who is working on a pipes replacement
What would be the best toolset to recreate the Pipes UI?
It's pretty slick, and I could see the underlying UI being useful for modeling a variety of things...
It was great for that - I'm unaware of anything that is even trying to replace it.
Still has the Yahoo problem, but it might be the easiest short term.
Argh! The UI was the innovative part. The whole point of something like this is that you don't have to learn a bunch of syntax - when you add a module, it tells you what sort of data it needs to know (such as a URL or...) and when you connect modules together it tells you what sort of inputs and outputs are available. This allows you to focus on the actual problem domain instead of the morphology of the query language, which is not important to the user.
I keep encountering this attitude among programmers - 'I type code all day, so why don't you want to do the same thing?' If people wanted to type stuff all the time the command line would still rule, but the majority of people prefer graphic user interfaces because they don't want to write code, they want to make selections from available options and use the computer to take care of the plumbing. This is a Good Thing. It's easier, more fun, and yields greater productivity, for the same reason that handing someone a box of Lego is better than handing them a few blocks of wood and a set of carving tools. The existence of Lego hasn't led to a collapse of sculpture or mechanical engineering. On the contrary, the simplicity, consistency, and interoperability of the different components has made a massive creative, educational, and commercial success.
I don't mean this as a hit on you personally, but I find comments like the above frustrating. Imagine you went into a police station to report the disappearance of your bicycle and the person behind the desk shrugged and said 'you're still able to run around on your legs, what's the problem? Your bicycle may have allowed you to go faster down the street but it's not like you could ride it up and down stairs so you should be happy with walking on your feet.' The problem is the loss of a tool that offered substantial efficiencies on some common tasks even though it wasn't ideal for every situation. You'd probably be equally pissed off if your favorite high level language went away and you were told you could just go back to writing assembler. I personally love doing things in assembler because it makes me feel smart but that doesn't mean it's better than high level languages.
One thing not mentioned in this thread is that Pipes 2.0 was a rewrite from the original Pipes which was written in Perl. We ported it to the same engine that powers YQL.
As for why shut down Pipes if it shares infrastructure with YQL? Probably because usage is mostly RSS crawlers that no one looks at anymore.
Still has the Yahoo problem, but it might be the easiest short term.
Any word on whether YQL will live or die?
Netflix does not believe in RSS, even though their Influencer content is not behind a subwall. I use Pipes to parse Netflix's json content and then turn it into a feed. I even created a Pipe that took any influencer ID and automatically created an RSS feed.
The idea was started because Daniel Tunkelang of Netflix was very anti RSS. I showed him, but alas, no more.
I never figured out how to use IFTTT. How can you grab any content from the web, like Netflix's json content? There is no HTTP input as far as I can tell.
For me, I would do the data parsing on Yahoo! Pipes and prepare everything to be triggered, and then the actions would be performed on a service like IFTTT (where the instruction was to just post whatever new item came in the feed).
Um, what? I've never worked at Netflix, and I've never been anti-RSS. And I'm pretty sure there isn't another Daniel Tunkelang. :-)
We're trying to build on this legacy with Weld(.io). See a sneak peek of our programming UI at 0:40 here: http://youtu.be/faG3uuOnqxY
If anyone in the Pipes team would like to be involved somehow, ping me on email@example.com
cloning twit user: https://www.flickr.com/photos/bootload/385441178/
multiple feeds into one: https://www.flickr.com/photos/bootload/385433038/
extracting flickr data: https://www.flickr.com/photos/bootload/385101700/ and flickr backend https://www.flickr.com/photos/bootload/2789198106/
extracting twitter data: https://www.flickr.com/photos/bootload/385194782/
clogged pipes: https://www.flickr.com/photos/bootload/384133421/ and https://www.flickr.com/photos/bootload/384128224/
Read the comments on this link to get an idea of how I used Pipes and the logic behind it: https://www.flickr.com/photos/bootload/385101700
EDIT: added OSS & Commercial alternatives
free or commercial:
I have a emotional connect with Pipes as a tool. Right out of college, and before the whole API thing was pervasive, I had leveraged pipes for so many small projects and hacks. Infact I used it in a Yahoo Hackday hack and it landed me a gig at Y!
More relevant to the present, It has been something I have mentioned to so many people who have come to me discussing UIs for automation/control systems/or to manage workflows, processes. In a way it was what noflow's UI looks like, but years in advance. In a different parallel universe, it may be the way people use APIs, or right big Haddop/Spark/Storm jobs/topologies, and the front end was open-sourced back in the day, with full integration with hadoop for job management.
I had created this years ago to display Twitter data using Pipes, YQL and Google Charts. I haven't had to touch it in years but it's always just quietly worked, and some people seem to like it:
I liked that Pipes allowed services to be chained together driven from the browser.
On a related note, a couple of us have been hacking on a Pipes like user experience around Data Integration and Analytics with Web feeds and APIs -- we had been planning a separate announcement but thought this might be a useful place to post if there are others who are looking for alternatives. Please message me directly - my contact info is in my profile.
I got about halfway through building a "news blend" app where you tap just the categories you want and it would blend all the latest articles from just your categories into one news feed and email that to you daily. But we never finished it.
didn't decide yet what serverside tech i'll be using but quick deployement is a priority for me. If anyone's interested.
EDIT: a lot of useful information here about existing projects, thanks.
I feel like IFTTT is our best bet, while it doesn't do the "Unix pipe" that pipes offered it at least provides some usability to the process.
Showing operators: https://www.flickr.com/photos/bootload/385194782/
I am still using it to repurpose an XKCD feed into a fully formatted feed with images that then is picked up by IFTTT and tweeted out as a tweet with images when new comics go up. I used Pipes just because it was point'n'click and seemed to have all the key components I needed to do the task without worrying about the code-route.
Hats off to all those that worked on the product - great to see it remembered so fondly.
I used it for the first time 2 weeks ago.
Meetup.com has calendar feeds. You can get a calendar feed for all the events you've RSVPed "yes" to. You can get individual calendar feeds for each group's events. But there's no way to get a calendar feed of all the events happening in all of your groups.
I'm a member of something like 40 groups. The thought of manually adding each feed as a separate calendar in Thunderbird - and the thought of Thunderbird then trying to cope with the result - was... unpleasant.
So I set up a pipe to combine all the feeds into one. Very cursory googling suggested that Yahoo Pipes was the easiest way to accomplish this.
Fairly slick UI. Unfortunately it turns out that meetup.com apparently serves 503s to requests coming from Pipes.
That's really interesting. I think it exemplifies a problem that has been bothering me for a while: for many sites, scripting and design are antipatterns designed to corral rather than enable the user and leverage branding.
We all knew it was coming, but the end of Pipes still sucks.