
The Forgotten Yahoo Project That Inspired Two Recently Funded Startups - mikeknoop
http://nickoneill.com/the-forgotten-yahoo-project-that-inspired-two-recently-funded-startups/
======
Cherian
YQL is an amazing piece of technology with robust infrastructure to support
it. I was first introduced to it at a Yahoo Hackday in Bangalore and since
then I’ve used it to to hack up a lot of small scripts. Like this
<http://userscripts.org/scripts/show/62104> \- one that shows torrents links
in Wikipedia movie pages(been couple of years since I updated it)

Christian Heilman was the evangelist behind YQL and it gained a lot of
traction with him travelling all around to spread word -
<http://www.yuiblog.com/blog/2010/02/11/video-heilmann-yql/> . But YQL met
with a quick death once Christian left Yahoo for Mozilla. Christian was also
in the process of writing a book on YQL - <http://icant.co.uk/whyyql/>

The only major downside to YQL is that most robots.txt block the Yahoo Slurp &
Pipes engine; the user agent that powers YQL. E.g.
<http://stackoverflow.com/robots.txt>

YQL has a lot of hidden treasures. Matter of digging the community scripts to
understand.

I continued to use YQL at hackathons in Zynga and stopped it when the primary
author Nagesh Susarla left Yahoo - <http://nagiworld.net/>

Recently a Yahoo engineer told me that YQL is now used internally to power all
services at Yahoo. So I am pretty sure it going very strong.

~~~
drgath
> Christian was also in the process of writing a book on YQL

Work on that has resumed and "YQL: The Definitive Guide" is due out next year.

> So I am pretty sure it going very strong.

Very much so. 100+ billion queries per month.

------
Ogre
Pipes is maybe the only yahoo product I actually use. It's only for one
frivolous thing though. Failblog (part of Cheezeburger I think? Doesn't really
matter) Used to have a sub-blog called Hacked IRL. They shut it down a while
back, but they still occasionally post clearly labelled Hacked IRL stories to
the main Failblog. Since I liked Hacked IRL but really didn't want to have the
whole fail blog feed subscribed, and I remembered having heard of Yahoo Pipes
years before that, I looked it up and it was sure enough still around.

The result was this:
[http://pipes.yahoo.com/pipes/pipe.info?_id=5ead4957874f5a670...](http://pipes.yahoo.com/pipes/pipe.info?_id=5ead4957874f5a670af1232f872b82f1)

Which is simply the fail blog feed filtered down to only posts containing
"Hacked IRL" in the subject. I have the RSS of that subscribed.

Like I said, frivolous. But it only took a few minutes to put together even
having never used pipes before.

------
spullara
Pipes was way ahead of the curve but is also targeted at a moderately more
sophisticated audience than the task triggers IFTTT and Zapier. Pipes focus is
more about data collection, filtering and transformation -- like Unix pipes vs
cron.

BTW, both Pipes and YQL are powered by the same engine and serve billions of
requests per day (for both O&O and 3rd parties).

~~~
malandrew
I'd love to hear more about the engine that powers pipes and YQL. Can you tell
us more or let me know where we can read more about it?

~~~
spullara
I'll encourage them to do a blog post on it.

------
tlrobinson
The Pipes/IFTTT concept screams for a more open solution, it's too important
to be proprietary. I'm always frustrated when I want to use IFTTT but they
don't support the services I need.

A simple protocol based on HTTP, JSON, and WebHooks could be pretty slick.

~~~
sneak
It would be nice if someone would make a painfully simple interface (a la the
"requests" python module) for doing XMPP.

Perhaps though XMPP is just too much of a quagmire.

~~~
lurker14
<https://developers.google.com/appengine/articles/using_xmpp> ?

------
sthatipamala
The problem with Yahoo Pipes was that it was over-optimized for complex
dataflows. It was possible to do things like "find all Flickr photos from
user, filter by tag and display on a map by location taken". The result was
this crazy programming GUI that resembled Labview.

In practice, few people needed such complex querying and chaining. I think
IFTT has a better metaphor ("if this, then that") which is much simpler but
covers 80% of useful things.

~~~
sabat
I agree with you that IFTTT and Zapier are simpler solutions to more common
needs, but man, Pipes is really powerful. It's more like a language than an
automation framework. I don't know if it will ever make any money, but it's a
hell of service.

~~~
teaneedz
Long before IFTTT, Pipes was an excellent solution for various social media
needs.

I recall mashing up RSS feeds and using it to bypass various platform
throttles.

It was awesome for fixing wonky feeds.

Love the UNIX metaphore and interface.

Pipes was the only Yahoo! product that really sang to me.

I think I've been relying on IFTTT too much lately. I may have to revisit
Pipes.

------
mikepk
My first startup played in this space too, Grazr
([http://techcrunch.com/2006/09/18/grazr-10-blasts-off-into-
th...](http://techcrunch.com/2006/09/18/grazr-10-blasts-off-into-the-future-
of-rss/)). We were way ahead of our time too (at least that's what I tell
myself :) ). We were mostly known for an opml display widget but our intent
was for the OPML to be a format for organizing packages of RSS (this allowed
us to have lots of ready sources since feed readers all output OPML).

Instead of a graphical 'pipe building' interface, the way we did mashups was
to have embedded scripts in the XML that ran in a server-side JavaScript
environment (SpiderMonkey) extended with custom syntax and language extensions
we called GrazrScript (and namespaced in the file as <grazr:script> tags). You
could do some pretty amazing stuff with it by writing small amounts of server-
side javascript. We had an interface for http querying, accessing and
manipulating normalized feed representations.

You could have these files call other grazrscript files and create 'pipes' of
functions (more directly like unix pipes).

I like to tell people we were doing embedded, server-side, javascript years
before it was cool. :)

------
lazyjones
There are many good projects and services out there that scare/annoy away
potential users by whacking a "register or login" box in their face just
before things get interesting.

My identity isn't "free" and managing yet another account comes with a burden
(another password to remember or store somewhere, another security risk).

Whoever thought that creating pipes (i.e. experimenting) without logging in
was somehow detrimental to Yahoo's business interests(?), had things
backwards.

~~~
justinhj
Perhaps the login requirement was needed to prevent, or atleast reduce, abuse.

------
jnazario
i've been a pipes user for many years. i mainly use it as a way to merge RSS
feeds into a single bucket (e.g. craigslist items within 50 miles across
multiple craigslists). my most frequently used pipe marries a bunch of news
sources together by company name, a simple way to keep track of my industry
and competitors.

i live in constant worry that yahoo will shut it down. glad to see it's
getting some attention, and getting some clones, too.

------
notlisted
Nobody seems to have mentioned \- Kapow/OpenKapow - <http://kapowsoftware.com>
\- Dapper -
[http://www.readwriteweb.com/archives/dapper_quest_to_unlock_...](http://www.readwriteweb.com/archives/dapper_quest_to_unlock_web_data.php)

I remember giving up on YPipes back then because it was always down (or denied
access). Did more with Dapper and OK

~~~
johnx123-up
Yes, once I used Dapper more than Pipes <http://open.dapper.net/>

------
lop
Does everybody trust into these hosted services? (I do not.) How about to run
your own one? \- <http://pipes.deri.org> \- still alive or dead? \-
<https://github.com/ggaughan/pipe2py> \- looks alive \-
<http://neyric.github.com/webhookit/docs/index.html> \- Alpha and dead? \-
swiftly.org - started as a open source project, but nothing released and dead?

Did I forgot something/somebody?

Just to be complete, GUI as a JS lib: \- <http://jsplumb.org/jquery/demo.html>
\- <http://neyric.github.com/wireit>

And additionally two commercial hosted projects: \-
<http://www.wewiredweb.com> (based on <http://www.mashablelogic.com>) \-
<http://tarpipe.com/>

------
daveman
I found pipes/YQL very useful for doing some ajaxy things with sites and
services that are not on your domain (since most browsers don't allow for
cross-domain ajax calls, you can request a JSON version of a site's markup via
YQL). However I didn't find the service 100% reliable. It will be great to
have more services like this out there.

------
aaronbrethorst
I (and probably sriramk too) am disappointed the article didn't mention
Microsoft Popfly: <http://en.m.wikipedia.org/wiki/Microsoft_Popfly>

Popfly was a more consumer centric mash up tool built by a little skunk works
team inside of Microsoft's Developer Division.

~~~
joshu
Never heard of it. Because it is silverlight, never would have been able to
run it. Sounds like it did a lot less according to
<http://blog.jonudell.net/2008/02/14/popfly-and-pipes/>

Lemme guess, you worked on it?

~~~
aaronbrethorst
Of course! No other reason to mention a Microsoft project that's been dead for
four years otherwise :)

~~~
joshu
Hah. Well, you shipped something. I have to imagine that's incredibly hard.

------
vmialik
The title made me think. Is there a place or archives where companies leave
their forgotten or dead projects that they tried and moved on? Is there any
HN's that dug up a failed project and turned it into a successful
company/project?

------
songrabbit
I don't think I've ever heard of pipes to be able to have forgotten about it.
However, after browsing the website for about 3 seconds I closed the window
out of disgust with no idea what the product did. Compare this to IFTTT (which
is fairly well known in this sphere) or zapier (which I had never heard of,
but could figure it out after a few seconds on the website). It really might
be all about presentation. Unfortunately I'm not willing to evaluate evaluate
this statement in regard to pipes because I'm not willing to look at their
website any longer to see if they actually do what the article claims.

~~~
mikeknoop
There really is some really cool tech "locked up" in these big tech companies.
Yahoo Pipes, Google Fusion Tables...

I think startups are, in a way, almost required to unlock the full potential
of novel tech ideas. A startup can apply an awesome idea to a niche, gain
traction, and expand whereas the defacto launch of a tech product at BigCo X
or Y applies the tech across a broad segment of their user base. Broad
launches are a good way to make your product generally available, but a bad
way to ensure it truly solves some core need your customers have.

------
mehdim
Yahoo pipes is great. And new automation platforms are fun and useful for non
developers. But if you want to be completely freewhen programming the web, let
APIs to developers.

[http://api500.tumblr.com/post/28165124923/your-mashup-is-
so-...](http://api500.tumblr.com/post/28165124923/your-mashup-is-so-2007-let-
apis-to-developers)

------
norswap
Pipes was, and I'd like to emphasis, still _is_ a useful tool. A shame that
Yahoo didn't market it better.

------
nicholassmith
It's so easy to forget that every now and again Yahoo! does something actually
really awesome like Pipes.

------
akg
I remember looking at Yahoo Pipes back in 2007, but perhaps the timing was not
right for it back then. Now as the Web is getting more and more ubiquitous and
the need for services to talk to each other grows, an approach like this could
very well take off.

------
6ren
Love the concept, but I found Pipes got very complex to do anything I wanted -
a common problem with visual tools.

Zapier solves this by pre-configuring tasks that most people want. Also better
times for it, with so many web-apps and web APIs around now.

~~~
drgath
Agreed. Pipes is cool, but as a developer, YQL is so much more intuitive. When
I think of accessing data, I think of SQL & REST APIs, not diagrams. But,
Pipes is good for non-developers who don't know SQL.

------
joestringer
I came across pipes the other day while attempting to link my twitter feed to
a (new) identi.ca account. The concept sounds powerful and I like the idea..
although I'd like to be able to view the source without logging in.

------
nikcub
symbiotic of the problems at the rest of Yahoo. good technology, good ideas,
good people, poorly marketed, poorly sold, poorly managed.

------
lucioscarpa
Pipe is extraordinary, but to complex and with a scary interface. Can't nave
success.

------
sabat
I uses Yahoo Pipes for a project of mine—I _depend_ on it, in fact. It's
really useful if you're doing something service-based, say, if you need to
make an RSS feed out of a website, or otherwise manipulate data. YQL (Yahoo
Query Language) is more than a gimmick; it's useful.

It does seem to be under active development, and I hope it lives.

EDIT: here's an example Pipe that fetches a web page (Pinterest) and turns it
into XPath, and then uses XPath queries to construct an RSS feed. Interesting
stuff you can do!

~~~
AlphaDex
I also depend on pipes. If pipes ever goes away, coding it all by hand would
be a horrible pain...

