Hacker News new | past | comments | ask | show | jobs | submit login
Huginn, an Open Source IFTTT / Yahoo Pipes (github.com/cantino)
374 points by tectonic on March 14, 2013 | hide | past | favorite | 70 comments

Good stuff, although I'm not a Ruby user. Tools like this should be much more widespread. Code Is Bad because you can't write code without a minimum level of vocabulary and syntax, and that excludes a lot of people. True, code offers the most flexibility, but 99% of the time people don't need all that flexibility and want to accomplish fairly straightforward tasks - which is why GUIs are the norm. With GUIs and visual network design, users select from a palette of what's available and can connect components together without wasting time on syntax.

The oft-repeated cry that 'everyone should learn to code' is wrong, wrong, wrong. It's like telling a kid to learn CAD and use a makerbot instead of providing them with a Lego set. Coding is great when you want to make a procedural something generator. If you want to make something specific, then coding often imposes an annoying and unnecessary layer of abstraction. For example, you can write music and/or perform sound synthesis in a superbly powerful language called CSound, but only a tiny number of academic masochists bother to do so. Commercial DSP engineers write in C or assembler, commercial sound designers use Max or Reaktor, much as most electronic engineers use SPICE rather than describe their circuits in code.

In short, keep up the good work!

>> The oft-repeated cry that 'everyone should learn to code' is wrong, wrong, wrong.

It is right, right, right. We don't tell our children, it's ok to stop at crayons and comic books.

Yes, DSPs and CAD are much more productive at their niches than coding in a "normal" language. I find Chrome more productive than "telnet 80" - but knowledge of coding helps me structure my excel spreadsheets more usefully than those produced by intelligent people who cannot code.

Code literacy is important not just because there are faster or easier ways to express ones ideas in a given domain. Literacy is not about using one language - it's about learning to structure thoughts and express them in the best way possible - something learnt cover thousands of years and code literacy is the same - we epxress our thoughts better if we are literate.

Having said that, I cannot wait to try this out,

No one would call crayons and comic books bullshit, because they are artistic mediums where the expression can stand by itself, has the intrinsic value of being expressive.

Yet in CS there are very smart people such as Alan Kay and the Constructivist types such as former-OLPC now-Sugar, all screaming about how we're learning brain-dead languages on machines doing their worst to make it impossible to become useful and literate without a ten year breaking in period until we become journeymen codesmiths. These machines & languages are not trying to help all of us become fluent about what it is that is running: they are there so that those who already know the language and the program can continue advancing it, until it's shippable, at which time it will execute: when we say learn to program, more often than not we mean learn to fit yourself in to that production cycle.

I'm not trying to argue one way or another about the learning to code merits, but it's important not to think of coding as this completed worthwhile thing in the way we do it! OP is an example of systems research to allow us to create agency in systems: that is programming, coding of behaviors, and one who gets good at it has gained a coding literacy. Ditto for spreadsheet wizards!

My point is that we are free to build entirely new forms of literacy both akin and unlike the programming languages we have now, and that the one's we have now are arbitrary. And, judging purely from the size of assumed knowledge passable programmers rely on, I'd contend, as do others, we'd do a lot better at making a literate society with different languages and forms, more tailored to introspection and mirroring their behaviors than towards producing a running application artifact: if we want people to learn systems, make the systems be more about being learnable and meddle-able than about being execution engines. It's our agency we need to see reflected, and IFTTT and OP are great examples of coding where seeing our agency at work at the end is far more of a direct path: it is radically different than how programming today works, in a good, different and fundamentally very important way.

In leaving, I'll point to a perspective on these better machines: these are machines that build agency by way of using APIs of other machines. I'm convinced that machine to machine interface is the key to making programming not suck, not a drain for this planet, for those who would be composers of code. rektide's gentle suggestion- machine to machine systems interfacing is the way to beget human/machine interface.

Thank you.

In the spirit of understanding, can I try to summarise your points as I get them

1. Programming as it is currently practised is still very focused on implementation / procedural.

2. Anyone who uses a DSP / other system to perform a task they want (agency) is doing some form of programming, and that is valid, as it is a step on the path of literacy.

3. The path of literacy might not be towards the current endpoint of neckbeards and Assembler and 10 years to be good.

4. There is likely to be new languages, new systems that will enable people to do programming / achieve agency at different points and with different degrees of abstraction

5. Its Lisp isn't it :-)

I agree with you. Which tells me I may have misunderstood!

But I am perfectly happy to agree that some form of code literacy is a good thing, and that the language of the future has not yet been invented I am also quite prepared to believe.

Aw, hell, it is Lisp :-)

Edit: I like the idea of OLPC etc being about how to meddle. I suspect that the machine you are referencing might be the one in Nand2Tetris - an interesting project that takes undergrads from Nand gates through to compilers and languages on a virtual machine interpreting the CPU built from the interpreted Nands.

2. Anyone who uses a DSP / other system to perform a task they want (agency) is doing some form of programming, and that is valid, as it is a step on the path of literacy.

Well, it's the path to literacy in signal processing (or whatever domain it happens to be). I learned to code >25 years ago and I still sort of like doing small things in assembler or throwing something together quickly in C. But I really don't like dealing with collections of big lengthy text files for projects of any magnitude. It's like reading a description of a painting that begins in the top left corner.

Put another way, text is the problem. Why, I ask myself, don't IDEs automatically generate dynamic flowcharts? Why do I have to spend so much time typing to create structure, when I want to focus on algorithms? Have a look at this software, which is aimed at roboticists but has other applications; it has modules that let you drop into Ruby as needed (and there's another version that allows entry of C or assembler) but doesn't require you to work that way if you don't need it. this is what an IDE ought to look like.

I think I meant DSL.

But, a graph of a novel is not a novel.

We are story telling chimps who have created a tool in our image - a tool to tell stories to other chimps, but based on rigid symbols and calculations.

Or maybe it's just late

Honestly, I think math education fulfills that job of code literacy to a great degree. It's good to understand how computers operate and be aware of code, but 'everybody must code' to me always has a subtext of 'I like coding, why don't you?' I personally don't like the endless reinvention of wheels, the omnipresent long variable names, and the widespread language fetishism of coding. I mean, all code runs on top of electronics but I don't think everyone should spend part of every day wielding a soldering iron and I doubt you do either. I think a lot of people prefer coding to electronic engineering because they don't care for burning themselves and sticking wires into their fingertips on a regular basis.

A service like this needs booth.

Writing processes in a programming language AND a possibility to reuse them for different stuff.

IFTTT is a nice idea, but I can't do stuff like:

1. check rss feed for new stuff 2. generate links to the real data from the links in the rss. (most RSS feeds just suck, because they don't have the full articles and sometimes they link to crazy pages where the article is split in 10 sub-pages) 3. get all the pages you want the data from 4. parse the relevant stuff out of the pages 5. save the stuff to a place where it won't get deleted again from third party

Most of the time the real stuff isn't easily crawleable without writing a pice of software just for the source.

But you're right, there should be a possibility to get such apps to work without any code.

If I wrote a crawler for images from a DA-artist, I didn't want to rewrite it for every other one...

  Most of the time the real stuff isn't easily crawleable without writing a pice of software just for the source.
  But you're right, there should be a possibility to get such apps to work without any code.
  If I wrote a crawler for images from a DA-artist, I didn't want to rewrite it for every other one...
Shameless offtopic plug: that's exactly what we at http://import.io are trying to solve. We're in Developer Preview, check us out!

Do I get this right, this is not a web-service but an application, which runs on my PC?

The application is for building extractors and connectors, which then get published to our platform so you can query using our web service.

Looks very interesting. I like that it's a desktop app!

This has been my side project for the last few months. I'm very interested in your feedback!

I've been building something similar in Erlang/Elixir (OTP's a very good match for an event-driven system which connects a series of inputs to outputs), but never quite managed to get it integrated beyond a few minor inputs/outputs (the fact Erlang is less popular makes finding decent OAuth libraries difficult, for instance). The web interface you've made in particular is impressive; so far my "pipes" are just more scripts.

Well done on this, I'm looking forward to playing with it tonight.

Seeing a Haskell project on github (https://github.com/frio/ifrit) that sounds similar under your handle- post the Elixir one! Would love to see some Elixir code in such great project area, whatever the state!!

Heh, I'd forgotten about that one :D. I've had the idea kicking around for a long time, and have spiked it in a few different languages (the Haskell version was mainly to try out FRP and reactive-banana; I've got the furthest on a Python one that hinges around agents tied to RabbitMQ as my current "production" deployment).

I'll try and get the Elixir stuff cleaned up and published over the next week :). The only source I've implemented so far is inotify (for eventing off Dropbox, so I can rebuild my blog); so far I've spent most of my time digging into the vagaries of the OTP and Erlang as a platform, rather than implementing.

Would you care to open the Python implementation? (and thx for open sourcing the Ruby one)

It's no more than a wrapper around Kombu; it shouldn't be hard to get one going yourself :). It's far from the level of quality of the OP's product.

frio is not op

Awesome project! IFTTT should really be part of every service that has a little bit of interactivity, even at a corner. Too bad I don't know ruby.

Please give us updates on your experience as you use it as well.

This is awesome! We were just talking about building a desktop IFTTT. Can't wait to play with this tonight and learn to make agents.

Please let me know if you have questions. I'm planning to make a screencast tutorial on how to make and link Agents.

I've got a Windows desktop product that does something similar for file activity. The plan is to expand the triggers out to other areas http://www.folderagent.com

Awesome. Thanks so much for releasing the source; this looks like an extremely significant contribution. Kudos!

thanks for sharing, looks useful. i'll have to play with it, i have plenty of uses.

Interesting project. Were you aware there's also an unrelated project called Hugin[1]?

[1] http://en.wikipedia.org/wiki/Hugin_(software)

Nope, I wasn't. :) Do you think I should change the name?

That's a good question. Ultimately, it all comes down to you, but as mentioned by the other commenters, the double ending 'n' (which is uncommon in English, as far as I know) and the fact that there's an established OSS project with a very similar name would make sure that people don't talk about your project as the "Hugin with two n's" or the "Huginn that's not the photography thing." Google will also helpfully redirect queries for "huginn download" to the other project[3], though that's probably because your project doesn't have many links to it yet.

That being said, they're not in a related area, so I doubt anyone would be excessively confused between both, as opposed to say, Go[1] and Go![2].

What I'd do, personally, is try to find another name and only keep the current project name if I can't find a better one that's not already taken. It makes it easier to rank well on Google and is less likely that someone will apt-get the wrong project.

[1] http://en.wikipedia.org/wiki/Go_(programming_language) [2] http://en.wikipedia.org/wiki/Go!_(programming_language) [3] https://www.google.com/search?q=huginn+download

FWIW at first I thought this thread was about Hugin.

Regardless of the hugin conflict, it'd be nice not to have to spell the name out to people when recommending this. Maybe even something ambitious like just calling it "Tubes"? (I'm not aware of anything else with that name.)

I vote yes. It's one consonant different from an existing application/thing. Having said that I'm terrible with naming things and can't offer any alternatives.

Eh, I had never heard of Hugin before.

Don't try Muninn, it's already taken for something sorta similar: http://code.google.com/p/muninn/

Ah just saw the reason for the name on github. I thought you chose it out of the blue. What about a line from Anchorman: "Great Odin's Raven!" or something to that effect :P

@Metapony: you're hellbanned, for quite a while now.

Don't ask me, I just see that your posts are [dead] (so, only someone who has 'showdead' in their profile, like me, can see them), and you don't seem like a troll or spammer, so I decided to tell you.

Vote for "sepip"

Same here.

It's confusing.

Muninn would seem like appropriate alternative.

"Munin" is already very well known as a distributed server monitoring system: http://munin-monitoring.org/

Wow, that's one hell of a side project. Looks really awesome.

Do you have plans on having a hosted version people can use?

I love the idea of hosted open source projects which allow you to easily export your data if you decide later to run it yourself.

Thanks! I'd like to make a version that's trivial to deploy to Heroku, or perhaps a pre-build micro EC2 image? I'm not sure I want to host more than a demo version because it could be used for scraping. The core idea is that this is something that you run yourself because you care about who has your data.

A Heroku or EC2 build would be amazing

Hosted would also be good though, I think there are a whole class of people who don't care all that much who has access to their data so long as they could self host it (or have someone else host it for them) if the service ever stopped running.

I don't understand why those people would be using Flickr, Facebook and Twitter instead of, say, OpenPhoto, Diaspora, and StatusNet.

Because those services are 3-10 orders of magnitude larger than a startup running open-IFTTT as a service? Because they already have critical business operations interacting with these large(r) services?

(Just two reasons I could imagine. But I get your point... especially with my recent sadness over Google Reader.)

Cool. FWIW, I'd use a micro image (or instructions for installing on top of a standard Ubuntu 12.10 image) if it was available.

This is a great project and will surely help a few people down the road starting their own clone of IFTTT :)

Awesome. Great work. Looking forward to digging in.

As a side note, there is another "pipes like" project out there, called DERI Pipes[1]. Development appears to have ended some time back, but I've created a fork[2] of the Github project and am planning to do some work on this.

Anybody else who's interested in this sort of thing, give me a shout.

We also have a demo server setup where you can play with the thing, at: http://demo2.fogbeam.org:8080/pipes/

[1]: http://pipes.deri.org

[2]: https://github.com/mindcrime/semanticwebpipes

Oh nice! Can I use it (create custom agents) if I dont know any ruby?

Also I will need to intall ruby on my debian personal server.. which I'd rather avoid (I already have there python, php, perl..), but having custom IFTTT+pipes is just too tempting :)

This is nicely done - not just the idea and code, but the documentation and examples. Very nice, I look forward to using this, thank you for sharing it!

Thank you for your kind words. I think good documentation is very important. I will also be adding a screencast that explains how to use the UI.

I agree - great looking docs!

Great project, though the name is confusingly similar to an existing open source tool [1] for photographers.

[1] http://hugin.sourceforge.net/

This is great! Since it allows combining and parsing RSS and XML feeds - do you think it could be used to build a personalized RSS / Google Reader replacement? Just a thought.

Well, that's very hot right now. :)

Yes, actually, I use it to scrape a couple of RSS feeds myself since I couldn't be bothered to use a reader when I had this.

Thanks, that is what I was thinking of doing. Maybe even set up a list of the RSS feeds on a simple text / HTML page; then use Huginn to pull the entire list from the URL, combine, sort, remove duplicates and output the HTML. Anytime you wanted to change the list of feeds, you can just edit the HTML page.

This is really fantastic. I've been working on a new UI for something like this and I'd love to compare notes once I dig in a little bit. Great stuff!

Send me an email :)

Yahoo pipes is the bomb. I wish they would have maintained it. Thanks for building this, I can't wait to put it through it's paces!!

Without diving into the code (which I plan to do over coffee tomorrow), what would the User-Agent string of the spider be?

At the moment, it's the curl default I believe.

Really cool! Did you consider using phantomjs thru capybara in order to use webkit to interact with javascript websites?

I looked into letting you write your Agents in sandboxed JS, but didn't end up going very far down that path. If you'd like to add a PhantomJS Agent, that'd be slick.

I'll have a look ;-)

Reminds me of my side-project that I just decided to work full time on, Process Smith https://process-smith.com

Lovely! Is there a ready-hosted version, so I don't have to install it?

Does it also run with postgres?

Great question. I'm not sure. If not, it should be very simple to get it there. Please try it and send a pull request. :)

Will do. P.S. Looks really nice. Good job.

Should have been coded in python :( . Whenever i see a project that i think is cool and then see that it has been coded in languages other than js, c++ or python i consider it just for my usage, and not for my contributions as developer.

hai Google employee!

Applications are open for YC Winter 2022

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact