
Huginn: Create agents that monitor and act on your behalf - daolf
https://github.com/huginn/huginn
======
tectonic
It's fun to wake up to find a project that I started on the top of HN! These
days, I'm no longer very involved with the day-to-day of the project.

Now that it's no longer a young project, here are some musings about Huginn
and responses to people's comments in this thread, in no particular order.

I've found that Huginn excels as a scheduled web scraper with lightweight
filtering. That's what I use it for. On the other hand, while you can write
custom code in it, Huginn is pretty poor at implementing any sort of complex
logic, and is even worse at bidirectional syncing between systems, which is
something people often want it to do, but for which it wasn't designed.

If IFTTT or Zapier meet your needs, awesome! No need to run and monitor your
own service. I personally choose to run Huginn on my own hardware in large
part so that I'm comfortable giving it website cookies and passwords.

Some examples of what I use Huginn for these days:

\- Watching Twitter in realtime for high standard deviation spikes in certain
keywords, such as "san francisco emergency" or "san francisco tsunami
warning", which then sends me a push notification, or "huginn open source",
that goes to a digest email (and I imagine will trigger because of this
thread).

\- Watching Twitter for rare terms and sending me a digest of all tweets that
match them. Also sending me all tweets from a few Twitter users who post
rarely, but that I don't want to miss.

\- Scraping a number of rarely updated blogs that don't have email newsletters
and emailing me when they change. Some use RSS, most are just simple HTML
scraping.

\- Pulling deals from the frontpage and forums of slickdeals and craigslist
and filtering them for certain keywords.

\- Sending an early morning email if it's going to rain today.

\- Watching ebay for some rare items.

\- Sending my wife and me an email on Saturday morning with local yardsales
from craigslist.

\- Watching the HN and producthunt front pages for certain keywords.

Basically, anytime I find myself checking a website more then a few times, I
spend 20min making a Huginn Agent to do it for me.

I think one reason Huginn has worked well for me is that I don't try to make
it do too much. I use it for scraping data and gentle filtering, and that's
about it. It's been super helpful for alerting me to interesting content for
The Orbital Index, my current project, a weekly space-industry newsletter.
(Last issue:
[http://orbitalindex.com/archive/2019-12-10-Issue-42/](http://orbitalindex.com/archive/2019-12-10-Issue-42/))

~~~
wpietri
I'm excited to check this out, but I wanted to congratulate you on a truly
excellent project name. Having spent many, many hours in naming struggles, I
truly appreciate the perfection.

And for those unfamiliar, the (also amazingly named) historian Snori Sturluson
explains: "Two ravens sit on his (Odin’s) shoulders and whisper all the news
which they see and hear into his ear; they are called Huginn and Muninn. He
sends them out in the morning to fly around the whole world, and by breakfast
they are back again. Thus, he finds out many new things and this is why he is
called ‘raven-god’ (hrafnaguð)." [1]

[1] [https://norse-mythology.org/gods-and-
creatures/others/hugin-...](https://norse-mythology.org/gods-and-
creatures/others/hugin-and-munin/)

~~~
fullwaza
Norse Mythology by Neil Giaman is very good if you like such things. The audio
book is especially good as it is narrated by the author.

[https://www.audible.com/pd/Norse-Mythology-
Audiobook/B01LWUJ...](https://www.audible.com/pd/Norse-Mythology-
Audiobook/B01LWUJKQ7?qid=1576180316&sr=1-1&pf_rd_p=e81b7c27-6880-467a-b5a7-13cef5d729fe&pf_rd_r=WTZG8ZN5CEQFDTSC6FAV&ref=a_search_c3_lProduct_1_1)

~~~
monkeywork
Can confirm just finished it on a 8 hr road trip last week. Very entertaining,
found myself laughing out loud in the car at several points

------
traspler
I have been running huginn on my home server for a while. I've mainly used it
to filter RSS feeds and then generate new feeds with the filtered items.
Another use-case for me is ingesting webcomic RSS feeds (or scrape a page) and
post the comics to a private Telegram channel. Once I also had an agent
scraping a page and notifying me if something changed (realestate listing).

I have tried a couple alternatives, e.g. node-red but none really worked the
way I wanted them to for these cases. huginn is incredibly flexible and (at
least for me) the mental model of it's workflow makes a lot of sense. Sadly
more and more pages want you to go through their app/site and make it a bit
difficult to work with, e.g. getting content from an instagram account.

One thing I have not figured out about huginn and which all of these
automation tools seem to lack are loops. E.g I have page an agent scrapes,
from which I want to output the src of an image tag but I also want to check
if a certain condition on the page matches (e.g. a "next page" button exists)
and then firstly output the found src but then also re-invoke the agent with a
new input element. So it would scrape the next page and so on until it does
not find the button anymore.

~~~
robk
Yeah I just use the Javascript agent for v complex stuff like this.

~~~
traspler
Oh, how could I have missed the JS Agent... thanks a lot for the hint! But it
looks like it's not node, so no npm modules and other node APIs.

~~~
tectonic
That's correct. You could shell-out to node with the Shell Command Agent
though.

------
PatrolX
Not just an alternative though.

\- Self-hosted

\- Unlimited and FREE vs $699 / month

\- Create your own agents / integrations

\- Process data through shell scripts or JavaScript

\- Better filtering

\- Liquid templating

\- Completely private

I save thousands of dollars by using Huginn.

It's incredibly powerful and quite frankly I don't trust Zapier with my data
and visibility of what I'm doing because there are commercial implications.

Run it in the cloud on AWS or on an old box at home.

It's very reliable.

~~~
PopeDotNinja
> I save thousands of dollars by using Huginn.

Does that take into account the overhead of operating Huginn?

~~~
PatrolX
Yes, and it's running thousands of requests and posts / minute on AWS. But you
don't even need to put it in the cloud, it runs perfectly at home on an old
box running Ubuntu.

~~~
tectonic
That's awesome that it's working so well for you :)

------
howmayiannoyyou
Zapier has exponentially more integrations than this or anything else, but is
surprisingly difficult to use, and more so since they updated their UI.
Editing Zaps is pure torture because refresh is so difficult. Exception
handling is pure pain. In most instances the breadth of API calls is so
narrow, and so rarely updated by vendors that you end up switching to a custom
integration. I've also noticed vendors rapidly expanding their native
integrations, sidestepping the need for a request broker.

The "retail" integration space remains underserved and if one of the
enterprise players decided to go down-market with a better UI and deeper
integrations - they'd mop the floor clean in 18 months.

~~~
PatrolX
> The "retail" integration space remains underserved

This is accurate, there's a significant opportunity in this space but it won't
stay that way for long.

There's a well-known entrepreneur I know with significant exits and capital
entering this space that's going after Zapier's market and I'm certain he's
not alone.

~~~
toomuchtodo
> There's a well-known entrepreneur I know with significant exits and capital
> entering this space that's going after Zapier's market and I'm certain he's
> not alone.

Having seen how the sausage is made and what it takes to build and support
such a product, I will say that past success is no guarantee of success in
this market. It is a grind, day and day out of dealing with queues, API
integrations that fail or degrade either loudly or silently and you have no
visibility into them, non-technical users who are are not happy when the magic
isn't working, blobs of data that just shouldn't be, spending hours tracking
down single edge cases or failures, and so on.

It's not "write some code, here's your $1B exit" (but you will be writing _a
lot_ of code!). It is constant tending of a complex piece of machinery, and
failures are painful (outages somewhat, data loss exceptionally so).

~~~
PatrolX
I totally agree, it's not easy at all.

Anyone attempting to take on this space is going to have their hands more than
full and they're far more likely to fail than succeed.

Personally I wouldn't even try, mostly because the tools themselves can enable
much easier more likely to succeed lower hanging fruit and that's how I use
them.

If you want to make money "using" these tools then Huginn is your best option
and it gives you a competitive advantage over those you're competing with that
are using Zapier to do similar things.

------
david_draco
"Once a day, ask 5 people for a funny cat photo; send the results to 5 more
people to be rated; send the top-rated photo to 5 people for a funny caption;
send to 5 final people to rate for funniest caption; finally, post the best
captioned photo on my blog."

I'm still laughing :) wth! (My fear is that this might actually be sustainable
with ads.)

~~~
omarhaneef
This is basically the plan of The New Yorker, but with (probably) 1000s of
people.

~~~
Eduard
Can you elaborate?

~~~
omarhaneef
If you don't follow this, you are in for quite a treat. Enjoy:

[https://www.newyorker.com/cartoons/contest#finalists](https://www.newyorker.com/cartoons/contest#finalists)

------
minimaxir
For a more business-friendly-but-still-FOSS approach to task automation
between services, see also Apache Airflow:
[https://airflow.apache.org](https://airflow.apache.org)

Airflow has been a skill that many companies ask for (especially data
engineering), but surprisingly doesn't have many articles written about it.

~~~
codetrotter
The Airflow landing page that you linked to lists many integrations but when
you click on those only a small subset of them are listed in the integrations
section of the docs that is linked to. I guess the docs are in need of some
more work.

~~~
minimaxir
It's easier to look at the APIs:

[https://airflow.apache.org/docs/stable/_api/airflow/operator...](https://airflow.apache.org/docs/stable/_api/airflow/operators/index.html)

[https://airflow.apache.org/docs/stable/_api/airflow/contrib/...](https://airflow.apache.org/docs/stable/_api/airflow/contrib/operators/index.html)

------
esquire_900
This, as well as related projects like n8n & node-red, is a very cool project.
I always wonder what people use it for in real life though. It seems a lot of
trouble (setting up, learning curve, maintaining) for an action that usually
takes a couple of seconds, like checking the weather or opening twitter.

Does anybody have useful workflows going on?

~~~
mcbits
Ever notice when someone criticizes a company on HN, and suddenly the CEO
shows up and engages after 6 months of inactivity? Likely they are using a
tool like this along with Google Alerts, etc, to monitor sites for mentions of
the company and maybe flag negative sentiment. Also, same thing but for
politics.

~~~
janober
Yes, very interesting indeed! Either there is a big conspiracy to uncover or
said CEO reads HN and if something is number 1 and is related to his product
he pays close attention. We will maybe never find out the truth...

~~~
mcbits
From your defensive tone it sounds like you think I was talking about you, but
I really wasn't. It's a phenomenon I've observed here and on Reddit and
elsewhere for many years. And it seems like a perfect fit for a tool such as
yours, is it not?

~~~
janober
Very sorry! As I was the only "CEO" in here which product got criticized I
thought it must have been me ;-)

Yes would actually be perfect for it, but never had the time to set something
like that up. Maybe someday in the future. For now, I only have a simple
Google Alert set and regularly check HN and Google Analytics which is a good
indicator when it got mentioned somewhere.

~~~
csomar
> As I was the only "CEO" in here which product got criticized I thought it
> must have been me ;-)

I don't think so.

------
bitshift
There is also ActiveWorkflow[1], which targets business users and let's you
write custom agents in any programming language[2].

1\.
[https://github.com/automaticmode/active_workflow](https://github.com/automaticmode/active_workflow)
2\.
[https://github.com/automaticmode/active_workflow/blob/master...](https://github.com/automaticmode/active_workflow/blob/master/docs/remote_agent_api.md)

~~~
tectonic
Huh, I did not know about this fork of Huginn.

~~~
bpuvanathasan
[https://tines.io](https://tines.io) (security automation platform) might also
be based on (or inspired by) Huginn

------
rvz
For those using expensive/advanced connectors like Zapier, Tray.io, etc. I
find that [https://n8n.io](https://n8n.io) serves as a far welcoming open-
source alternative that is worth looking at.

~~~
PatrolX
Nowhere near as flexible and powerful as Huginn.

~~~
davidgf
Please elaborate. I'm not questioning what you say is true, but I'm interested
in a tool like this one and would love to know more before choosing one or
another.

------
curo
Something I'm not entirely clear on:

One use case for Zapier (from a developer/company standpoint) is to allow
customers to connect their existing services to actions inside your own app.
For instance, if a customer updates a CRM record, you can have a custom zap
update a record in your own SaaS platform.

To pull that off with huginn, is that as simple as connecting this up to
Singer.io? Or would that require a big marketplace of huginn agents for
popular integrations?

------
anderspitman
If you like this kind of stuff, you might find my patchbay project
interesting:

[https://patchbay.pub/](https://patchbay.pub/)

------
devm0de
I gave this a go today and managed to install huginn on my synology nas by
simply searching for the docker container. I then setup 3 agents to scrape a
Shopify webstore jason endpoint that I’m always checking for inventory, have
huginn parse the json and send me an sms via twilio if inventory changes. Took
about 2 hours, wasn’t too bad. Huginn twilio docs seemed dated.

Used python simplehttp server and ngrok to replicate a json url and play with
the triggers to test it all before pointing it at a real website.

Nice to add a new tool to the belt, thanks!

------
dang
A thread from 2016:
[https://news.ycombinator.com/item?id=11015147](https://news.ycombinator.com/item?id=11015147)

2014:
[https://news.ycombinator.com/item?id=7585605](https://news.ycombinator.com/item?id=7585605)

2013:
[https://news.ycombinator.com/item?id=5377651](https://news.ycombinator.com/item?id=5377651)

------
penagwin
Also checkout Node Red [0], it's fairly popular in the automation space. It's
rather sparse by default, but after adding in some community nodes (or making
some yourself) it's pretty useful.

[https://nodered.org/](https://nodered.org/)

~~~
GordonS
I imagine things have changed, but I tried node-red maybe 3 years ago, and
found it really difficult to setup and get working.

Once I finally had it working, I found it really difficult to debug "tasks" (I
forget the correct, node-red verbiage), for example to find out why an HTTP
call was failing.

I also found the available connectors to be lacking, and a lot I tried were
quite basic.

------
ksrm
Not to be confused with Hugin:
[http://hugin.sourceforge.net/](http://hugin.sourceforge.net/)

~~~
ISL
Agreed. It was a surprise to see Hugin get top billing on HN other than for
awesome panoramic image stitching.

~~~
ISL
I see that this particular horse was already beaten in 2013:
[https://news.ycombinator.com/item?id=5377815](https://news.ycombinator.com/item?id=5377815)

------
j1elo
I've seen the word "rake", so I guess... is this written in Ruby? If so, how's
the performance?

My home server is pretty minimal and lightweight: a Raspberry Pi. Do you think
it will run it fine? (I'm gonna want to try this anyway, didn't know about it
until now and it looks amazing!)

~~~
heavyset_go
A fresh Huginn container idles at ~300MB of memory without any configuration
on my part. A fresh node-red container idles at 30MB.

~~~
floren
I was unable to run the Huginn Docker container on my Pi, I assume because
there are x86 binaries in it. I'm trying the manual install now...

~~~
heavyset_go
Yeah, there's only a x86 Docker image on Docker Hub for Huginn. You could
probably just build an ARM image from whatever Dockerfile the x86 version was
built from.

------
audiometry
Is there any way to implement agents in python rather than JavaScript/ruby?
Looks interesting but I don’t want to invest energy in building fluency in
these other scripting languages.

~~~
oskapt
I run Huginn in a Kubernetes cluster with most of its heavy lifting done by
HTTP calls to OpenFaaS running in the same cluster. This let me write complex
logic in Python but have Huginn handle the execution, triggers, filtering, and
delivery of the data. Among other things I use it to scrape Twitter for
interesting Kubernetes news and follow the retweets back to people who link to
their own websites so that our editorial department can vet them for
invitation into our writing program. Any particular author/account only
appears once. It generates around 40 new candidates each day. I also use it to
scrape the Chilean public alert website (onemi.cl) for alerts in my region (
forest fire, major weather), run those through the translate API and then send
the English translations to my and my wife’s phones via Pushover.

These are things that I want to control and customize, and I vastly prefer
running my own systems over trusting any third party.

------
osprojects
An alternative to webhook automation would be
[https://github.com/adnanh/webhook](https://github.com/adnanh/webhook)

------
elwell
> Create Amazon Mechanical Turk workflows as the inputs, or outputs, of agents
> (the Amazon Turk Agent is called the "HumanTaskAgent"). For example: "Once a
> day, ask 5 people for a funny cat photo; send the results to 5 more people
> to be rated; send the top-rated photo to 5 people for a funny caption; send
> to 5 final people to rate for funniest caption; finally, post the best
> captioned photo on my blog."

Curious how this would turn out.

~~~
DailyHN
You should try it out.

------
m-p-3
Interesting, I should switch to that for some location-based action with my
phone instead of IFTTT and see if I can make it work with Hauk or something
similar.

[https://github.com/bilde2910/Hauk](https://github.com/bilde2910/Hauk)

Disclaimer: I have no link with the Hauk project, just found about it by
browsing the F-Droid store.

------
theshadowmonkey
I’m glad Huginn is on the front page. It’s an awesome project being used for a
long time now. I was testing Huginn to see how skimpy I can be and still run
Huginn on a free tier. Was able to run it on open shift free tier a while ago
when their allowance was generous. But, looks like it’s hard now. Will try
running on a Gcp instance and see if it works.

------
kasbah
Seems like all of these self-hosted automation apps struggle to incorporate
Facebook adapters. Does anyone know of one that would allow me to read Tweets
and post ones containing some particular text to a Facebook page (not my
personal timeline/page but one I created for my project)?

------
DailyHN
I've operated a huginn for 5+ years. AMA

~~~
slightwinder
How does configuration works? Is it just configurable through the webinterface
or can I update/set workflows with some external script?

Is configuration saved in textfiles which I can put under versioncontrol and
backup, or is it some unaccesable binay blob?

How much does it break with each update? And how often does it receive bigger
and smaller updates?

How does this compare to something like home assistant or node-red, if you
know this software.

Anything you consider missing or lacking on some area?

~~~
DailyHN
> How does configuration works? Is it just configurable through the
> webinterface or can I update/set workflows with some external script?

Primarily Web interface. Though, the CommanderAgent can configure and trigger
other agents. You can use that in combination with your external script for
configuration.

> Is configuration saved in textfiles which I can put under versioncontrol and
> backup, or is it some unaccesable binay blob?

Agent configurations are stored in the database and can be exported as text
files for your backups.

> How much does it break with each update? And how often does it receive
> bigger and smaller updates?

Making sure updates don't break existing installations is one of the top
priorities of maintainers. Most of the updates are small/incremental. And I
haven't had any problems with the updates breaking things.

> How does this compare to something like home assistant or node-red, if you
> know this software.

I looks at node-red years ago and determined it wasn't as mature of a project.
Things could've changed since then. And I haven't heard of home assistant.

> Anything you consider missing or lacking on some area?

The UI is a bit basic. Though, I think this can also be looked at as a feature
instead of a flaw.

------
jaequery
This looks worth checking out but I’d love to see a Lambda version of it

~~~
pixiemaster
Yes, that would be the killer feature

~~~
mathnmusic
Related idea that was on HN a few days ago:
[https://gist.github.com/nileshtrivedi/6024434121b073789e6d58...](https://gist.github.com/nileshtrivedi/6024434121b073789e6d580eb84c3ea2)

------
caseysoftware
It's not quite 1:1 but not a bad comparison.

I like to think of Zapier as "wizard mode" \- if you're doing something within
their pattern, planned workflow, capabilities, etc it is a _great_ option.

If you need to do something more advanced or outside that narrow scope, you
need to go to the APIs (or Huginn) itself.

The important questions are: How many people need to get into advanced mode?
How big of an effort/skill/understanding jump is it to go there? (aka will it
hurt too much to switch?)

------
franga2000
I've come across Huginn many times in the past and every time, I tell myself
I'm going to give it a shot. I still haven't even seen it loaded in a web
browser...

After fighting the Docker images for half an hour, I usually realize that I
could've just written a Python script to do whatever I was trying to do by
then and proceed to do just that.

Sorry for being lazy! Maybe I'll get around to it next year...

------
vunie
How well does it support:

\- support remote agents and remote reporting?

\- dead-man triggers for missing reports?

\- is remote reporting properly secured? (encrypted/signed)

~~~
DailyHN
What do you mean remote agents? Can it be accessed over HTTP? Then yes.

One of the great features of huginn is the diagram. That's where there are
green and red indicators. Red indicates both errors and when a longer than
expected period has passed. This makes quickly inspecting your workflows easy.

~~~
vunie
> What do you mean remote agents?

Sorry. I meant running an agent on a machine that is separate from the machine
running huginn. The machine running the agent is not accessible publicly but
has internet access (doesn't accept connections but can initiate them). huginn
would be accessible publicly.

~~~
DailyHN
Ok, so in the remote agent:

Retrieving data from huginn can be done with the DataOutputAgent.

Sending data to huginn can be achieved through the WebhookAgent.

------
binthere
I'm probably missing something here, but what is the value of this when you
could write a script that would do just about the same? (Serious question, I
want to know the answer, not rhetorical)

~~~
bluehatbrit
Huginn is more of a framework for writing and running these scripts. It has
some built in modules to make the process easier. IE: The webhook agent is an
easy want of getting POST data into a script without bootstrapping a webserver
yourself, you can then just raise an event and have another script handle it.
No need to write and manage everything individually, you can run everything in
one place.

~~~
DailyHN
Yes, and this becomes exceedingly important as the number and complexity of
automations increases.

Huginn can assist in organizing complex workflows into manageable components.

------
vorpalhex
I've deployed Huginn a few times but never gotten off the ground with it. I
found agents a bit too cumbersome to configure, but maybe I just need to give
it another shot.

------
moomin
How does this not have a companion technology called Muninn?

------
soared
I would not market this as an alternative to Zapier since it goes after a
different market. The type of people who use Zapier will see a link to
github.com and not even consider it.

I think this should be positioned more like "A high powered Zapier for
technical users" or something similar.

~~~
organsnyder
"Self-hosted" is a growing buzzword these days (for many good reasons, IMHO).

------
e_carra
It really sounds like Son of Anthon from the last season of Silicon Valley.

------
ikeboy
I don't see a list of all supported agents or a way to search.

~~~
davidgf
It's in the wiki [https://github.com/huginn/huginn/wiki/Agent-
Types-&-Descript...](https://github.com/huginn/huginn/wiki/Agent-
Types-&-Descriptions)

------
keyle
This reads like DIY Yahoo Pipes.

