

Webhooks Are Magic - paddyforan
http://blog.iron.io/2013/09/7-reasons-webhooks-are-magic.html

======
e12e
Thanks for summing up, concisely, why webhooks really aren't magic at all, in
the fourth paragraph:

"All [webhooks] are is a promise by an application or API: “when this thing
happens, I’ll send this HTTP request with this data.”"

It's nice to see that all the hype really just boils down to generalized
pingbacks[1]. Now, having a standard is great -- I'm not entirely sure we
_have_ a standard yet, though. It's more like a fancy name for pingbacks that
aren't just for blog posts.

I'm also not entirely sure if I'd prefer webhooks for eg. processing emails.
Emails are naturally "push" \-- they get pushed via SMTP. I know that people
are scared of email/SMTP -- but it seems quite a lot easier to make sure you
don't loose any "events" with SMTP than with HTTP(S).

What if your site is down for a few minutes, just as some service out there
tries to "ping" you with a webhook? Email handles that.

I think what most people seem to mix up is that handling "real" email may be
hard -- but looking for a well formed subject that exactly matches some regex,
then parsing that, isn't that hard. Everything else you can either bounce (a
little dangerous) or just drop (not quite as dangerous).

Mailing-lists work. What you need isn't much more complicated. Maybe allow for
a mime-part with a json or xml, or some other well-formed body. Use gpg or
smime to encrypt data, if you need something more than a short hash/token.

[1]
[http://en.wikipedia.org/wiki/Pingback](http://en.wikipedia.org/wiki/Pingback)

~~~
baudehlo
Part of the problem has traditionally been that email servers haven't been as
easy to "hack" as web servers have. That was a failing on the part of the
developers of those servers.

Thankfully now you have things like Qpsmtpd, James, Lamson and Haraka which
can do those things for you. But not many people install those servers. I'd
love to see that change (for obvious reasons).

~~~
e12e
Partly. But you've been able to do the equivalent of:

    
    
        echo "|magic-program" > .forward
    

For a long, long time. Granted, it's a little unclear what happens if you
manage to overload your server doing that, but in general MTAs are pretty good
at taking care of your mail, either queueing and delivering, or giving up
somewhat gracefully.

The trick is to make sure that the "magic-program" is simple and robust -- and
that isn't quite as hard as most people think, if you don't require it to
parse arbitrary emails -- just accept and parse clearly valid ones, and reject
everything else.

Now, if you require to handle enough incoming requests that forking a process
per mail is a problem ... you probably need to fix your architecture.

~~~
baudehlo
The fragility of parsing emails is a big problem though. But I agree that for
the equivalent of webhooks it is the perfect platform, and simple scripts
work. Except at scale.

------
uptown
I'm new to webhooks - I understand them conceptually, but is there a standard
for authenticating the incoming notification? Take the Stripe example - how do
you know that the HTTP POST is actually coming from Stripe? Do you use a
shared-secret, or some other mechanism to verify the HTTP request is
authentic?

~~~
siong1987
There isn't a standard way of authenticating the incoming notification yet. I
like how Mailgun is doing it, to verify the webhook is originating from
Mailgun you need to:

    
    
      * Concatenate timestamp and token values.
      * Encode the resulting string with the HMAC algorithm (using your API Key as a key and SHA256 digest mode).
      * Compare the resulting hexdigest to the signature.
      * Optionally, you can check if the timestamp is not too far from the current time.
    

So, you essentially get 3 extra parameters from your webhook, `timestamp`,
`token` and `signature`. Obviously, the API key is the shared secret here
between your app and Mailgun.

~~~
ihsw
Grooveshark does it as well.

[http://developers.grooveshark.com/docs/public_api/v3/](http://developers.grooveshark.com/docs/public_api/v3/)

However it depends on a shared secret for generating/verifying signatures, and
some companies ( _cough_ Stripe _cough_ ) have yet to implement that. As
someone else has already mentioned, thankfully each webhook request from
Stripe has an ID in it so you can query their API for verifying a webhook's
authenticity.

~~~
baudehlo
If it's just a shared secret, why not just make a secret (and complicated) URL
for the web hook? It's shared with the deliverer but as long as you're using
SSL it should be just as secure.

Speaking of which, the right way to do this is to validate their SSL client
certificate, but I doubt many places are easily setup to do that - in fact I
would bet the sending endpoint doesn't even use a certificate most of the
time.

------
Sir_Cmpwn
MediaCrush updates itself through GitHub hooks [1]. When we get the POST, it
git pulls and restarts the site automatically. Best thing we ever did. Not
worrying about deployment makes our lives a lot easier.

[1]
[https://github.com/MediaCrush/MediaCrush/blob/master/mediacr...](https://github.com/MediaCrush/MediaCrush/blob/master/mediacrush/views/hook.py)

------
richardlblair
Webhooks are great, until you don't get one... Then they suddenly become a
nightmare.

~~~
paddyforan
That's why logs from the webhook provider are important. :)

~~~
philsnow
That sounds even worse than:

> "I watch people poll APIs or create convoluted connections, and I cry a
> little on the inside."

If the action being sent by the webhook is really important, there seems to be
no way around doing it somewhat like this: Write a polling (or hanging-get)
system, but use webhooks as an unreliable but lower-latency push mechanism.

In this setup, webhooks give you low-latency updates (when they work), and
polling ensures completeness.

~~~
roarroar
Depending on your requirements the remote end can be responsible for ensuring
that you receive and acknowledge the wake up. Once you hit your own timeout
you might do any number of things, not necessarily repoll the other end.

------
huxley
I've started adding web hooks to one of my Django web apps with Zapier's
django-rest-hooks which I'm liking a lot:

[https://github.com/zapier/django-rest-
hooks](https://github.com/zapier/django-rest-hooks)

------
zrail
Stripe's webhooks are really nice. Everything that happens with your account,
be it charges, disputes, transfers, or people changing subscription levels,
all generate events that you can receive and do something with. In my book and
the app that sells it[1] I use them to trigger emails to myself when important
things happen. There's also an iPhone app named Paypad[2] that registers a
webhook on your account that sends you push notifications when things happen.

[1]: [https://www.petekeen.net/mastering-modern-
payments](https://www.petekeen.net/mastering-modern-payments)

[2]: [https://www.pay-pad.com](https://www.pay-pad.com)

~~~
baudehlo
Just don't disable SSL on your server.

[http://baudehlo.wordpress.com/2013/09/06/stripe-webhooks-
don...](http://baudehlo.wordpress.com/2013/09/06/stripe-webhooks-dont-work-
with-tls/)

------
adelevie
Great post, Paddy. One thing I'd love to see in the HUD is a GitHub OAuth +
repo webhook config. That is, you log in with GitHub credentials, you see a
list of repos and can select which workers will get POSTed to after specific
GitHub repo events[1].

Examples:

\- The "adelevie/MyJekyllBlog" repo will POST to the worker "BuildAndDeploy"
after event "push."

\- The "adelevie/MyRailsApp" repo will POST to the worker "TestMyRailsApp"
after events "push" and "pull_request."

[1]
[http://developer.github.com/v3/repos/hooks/](http://developer.github.com/v3/repos/hooks/)

~~~
paddyforan
I love this idea. I've been wanting to do something similar to this, making
webhooks even easier to hook into the Iron.io platform. I don't know if it
will make it into HUD, but I'm definitely going to be trying to setup some
push-button webhook configurations for popular services that support
configuring webhooks from an API.

~~~
adelevie
Will/can there be an Iron.io OAuth API?

~~~
paddyforan
"We have nothing to announce at this time."

I'd like there to be is all I've got for you at the moment. :( Priorities, and
all that.

------
duaneb
Ok, maybe it's just the terminology that's confusing, but how do webhooks
differ from callback requests? Is it that the semantics are standardized, or
is there something that I'm missing? The Author is quite excited about them.

~~~
paddyforan
I'm incredibly excited about them.

As far as I'm concerned, webhooks and callback requests are the same. If we
were going to get pedantic, I suppose we could say that for a request to be a
callback, it would have to be a response to an earlier request (think
asynchronous processing) whereas a webhook is an evented request (it wasn't
prompted by a previous request).

But being pedantic is lame. Let's just be excited about both!

Relevant:
[https://twitter.com/paddyforan/statuses/254375310554968064](https://twitter.com/paddyforan/statuses/254375310554968064)

------
samograd
> It makes the web capable of pushing information to its users, instead of
> waiting for users to ask for information.

[http://www.wired.com/wired/archive/5.03/ff_push.html](http://www.wired.com/wired/archive/5.03/ff_push.html)
[1997]

It seems I'm not the only one to remember the promises of this prophecy:

[http://www.boston.com/business/technology/articles/2004/01/0...](http://www.boston.com/business/technology/articles/2004/01/05/push_technology_gets_a_nudge/)

------
aytekin
If you have an API make sure webhooks can be created using the API.

It is win-win for everyone. It is easier to implement and apps don't have to
poll you constantly.

~~~
zrail
There was a somewhat-confusing attempt at standardizing that pattern a few
weeks ago: [http://resthooks.org](http://resthooks.org)

------
dwb
Great an' all, but what happens when a request fails? There's no (specified)
retry mechanism — and POSTs aren't meant to be idempotent. I guess many
applications don't need strong reliability and so this is cool, but I can't
help but feel HTTP's design is really being stretched with ideas like this. I
totally get that it's really easy to get stuff going if everything just speaks
HTTP, but is it really for the best?

~~~
paddyforan
I can't speak for other providers, but our practice is to let users set the
number of retries they'd like and the space between retries.

We also are close to rolling out error queues, so when a push notification
fails, the message is automatically placed into a queue, to be processed at
the user's convenience.

I agree, the need for safe retries (as this is a process without user
feedback) calls for idempotency, but that's at odds with the need for a
webhook to be possible to be triggered multiple times.

One way around this is to use "tickle" webhooks--webhooks that tell the
destination that data has changed, but require the destination to pull in that
information themselves. Then the webhook call is still idempotent in practice
(at worst, you're refreshing data more than you need to, which is still less
than polling), but multiple webhook calls can be made.

------
mikeknoop
Webhooks are awesome but please don't make your users set them up:
[https://resthooks.org](https://resthooks.org)

~~~
adventured
Firefox is throwing a "This Connection is Untrusted" alert on attempting to
load resthooks.org via ssl. Someone should probably look into that.

~~~
e12e
[https://resthooks.herokuapp.com/](https://resthooks.herokuapp.com/) appears
to be the meat of it.

But that is really just a pointer to:

[https://github.com/net-
engine/resthooks/blob/master/README.m...](https://github.com/net-
engine/resthooks/blob/master/README.md)

------
mwetzler
Love the post... and stripe webhooks! Integrated them with Keen IO in about 60
seconds and even made a (perhaps overly enthusiastic) blog post about it.
[https://keen.io/blog/48628875285/integrate-stripe-and-
keen-i...](https://keen.io/blog/48628875285/integrate-stripe-and-keen-io-
in-60-seconds)

~~~
paddyforan
Awesome!

------
jkarneges
[http://webhookinbox.com/](http://webhookinbox.com/) gives you throwaway URLs
for receiving webhooks. Good for testing.

~~~
paddyforan
John Sheehan, of Runscope.com (which, by the way, is a super rad tool for API
developers) did a great roundup on receiving webhooks for testing:
[http://john-sheehan.com/blog/a-survey-of-the-localhost-proxy...](http://john-
sheehan.com/blog/a-survey-of-the-localhost-proxying-landscape)

------
tootie
This is basically pub/sub messaging, but over the internet?

~~~
dragonwriter
> This is basically pub/sub messaging, but over the internet?

Its basically hubless pub/sub messaging over _HTTP_ ("...over the internet",
sure, but there is _lots_ of _much_ older versions of that); there's also the
with-a-hub version, PubSubHubbub.

------
carimura
The next coming of SOA!

