"All [webhooks] are is a promise by an application or API: “when this thing happens, I’ll send this HTTP request with this data.”"
It's nice to see that all the hype really just boils down to generalized pingbacks. Now, having a standard is great -- I'm not entirely sure we have a standard yet, though. It's more like a fancy name for pingbacks that aren't just for blog posts.
I'm also not entirely sure if I'd prefer webhooks for eg. processing emails. Emails are naturally "push" -- they get pushed via SMTP. I know that people are scared of email/SMTP -- but it seems quite a lot easier to make sure you don't loose any "events" with SMTP than with HTTP(S).
What if your site is down for a few minutes, just as some service out there tries to "ping" you with a webhook? Email handles that.
I think what most people seem to mix up is that handling "real" email may be hard -- but looking for a well formed subject that exactly matches some regex, then parsing that, isn't that hard. Everything else you can either bounce (a little dangerous) or just drop (not quite as dangerous).
Mailing-lists work. What you need isn't much more complicated. Maybe allow for a mime-part with a json or xml, or some other well-formed body. Use gpg or smime to encrypt data, if you need something more than a short hash/token.
Thankfully now you have things like Qpsmtpd, James, Lamson and Haraka which can do those things for you. But not many people install those servers. I'd love to see that change (for obvious reasons).
echo "|magic-program" > .forward
The trick is to make sure that the "magic-program" is simple and robust -- and that isn't quite as hard as most people think, if you don't require it to parse arbitrary emails -- just accept and parse clearly valid ones, and reject everything else.
Now, if you require to handle enough incoming requests that forking a process per mail is a problem ... you probably need to fix your architecture.
1) Receive a webhook and record the ID in your database. If it's a duplicate, stop.
2) Ask Stripe for the event matching the ID they gave you. If they reject it, stop.
3) Process the event.
It requires a roundtrip, but it's the only sane way to validate that the event you got is real and that you haven't seen it before without trying to validate a signature (lots of other APIs make you do that, though)
At the very least they should only provide the event ID over webhook otherwise people will take the lazy route and trust trivially forgeable messages.
Well, as zrail said, you can trivially verify it by fetching the associated event from Stripe and ignoring the webhook body.
We've debated this ourselves, though. While you can theoretically get good security with a combination of SSL and a shared secret in the URL, it's a bit ungainly, and we should perhaps encourage the right thing more straightforwardly by dropping the body from the POST.
I did it the way zrail suggested but laziness typically wins especially if you aren't even aware of the reasons you might want to do it the way zrail suggested.
I'm for making the right way the only way (or at least the default way).
For Iron.io, all our requests are authenticated using an OAuth token. So only people that know your OAuth token (and thus could use our API anyways) can use your webhook endpoints.
But there are other ways, too. The most common I've seen is to provide an API endpoint to verify events with (Stripe does this). If you use HTTPS to receive the webhook, and verify it with a request over HTTPS to the API you expect it to be coming from, you're ensuring the request is authentic.
For some webhook styles, it doesn't actually matter. Some people use webhooks to just say "Something happened", without actually saying what. In this style, the API is still responsible for the data and authentication, the webhook just says "Hey, wake up, the API has something new for you."
* Concatenate timestamp and token values.
* Encode the resulting string with the HMAC algorithm (using your API Key as a key and SHA256 digest mode).
* Compare the resulting hexdigest to the signature.
* Optionally, you can check if the timestamp is not too far from the current time.
However it depends on a shared secret for generating/verifying signatures, and some companies (cough Stripe cough) have yet to implement that. As someone else has already mentioned, thankfully each webhook request from Stripe has an ID in it so you can query their API for verifying a webhook's authenticity.
Speaking of which, the right way to do this is to validate their SSL client certificate, but I doubt many places are easily setup to do that - in fact I would bet the sending endpoint doesn't even use a certificate most of the time.
This is pretty sweet, but it seems like it might be more useful for the baked-in reliability to be on the remote end.
That is, the side sending the POSTs should notice that it didn't get a 200 reply and retry some number of times with backoff. Also, the POSTs need to be idempotent (include a sequence number or nonce or something) so that if the "200 OK" gets lost and the sending side retries, you can discard the duplicates on the receiving side.
That's of course a lot harder since you have to convince everybody whose service you want to use that this is worth the time/effort :)
> "I watch people poll APIs or create convoluted connections, and I cry a little on the inside."
If the action being sent by the webhook is really important, there seems to be no way around doing it somewhat like this: Write a polling (or hanging-get) system, but use webhooks as an unreliable but lower-latency push mechanism.
In this setup, webhooks give you low-latency updates (when they work), and polling ensures completeness.
As far as I'm concerned, webhooks and callback requests are the same. If we were going to get pedantic, I suppose we could say that for a request to be a callback, it would have to be a response to an earlier request (think asynchronous processing) whereas a webhook is an evented request (it wasn't prompted by a previous request).
But being pedantic is lame. Let's just be excited about both!
- The "adelevie/MyJekyllBlog" repo will POST to the worker "BuildAndDeploy" after event "push."
- The "adelevie/MyRailsApp" repo will POST to the worker "TestMyRailsApp" after events "push" and "pull_request."
I'd like there to be is all I've got for you at the moment. :( Priorities, and all that.
It seems I'm not the only one to remember the promises of this prophecy:
We also are close to rolling out error queues, so when a push notification fails, the message is automatically placed into a queue, to be processed at the user's convenience.
I agree, the need for safe retries (as this is a process without user feedback) calls for idempotency, but that's at odds with the need for a webhook to be possible to be triggered multiple times.
One way around this is to use "tickle" webhooks--webhooks that tell the destination that data has changed, but require the destination to pull in that information themselves. Then the webhook call is still idempotent in practice (at worst, you're refreshing data more than you need to, which is still less than polling), but multiple webhook calls can be made.
The semantics of the data exchanged by POST can be such that the request is effectively idempotent even though that's not an HTTP-level expectation (it's not wrong to have a layered-over-HTTP protocol in which methods are safer than HTTP requires, it would be wrong for them to be less safe than HTTP requires -- e.g., non-idempotent GET.)
If you are sending updated notifications over POST with the actual data then needing to be retrieved by a request the other directions, the first POST is effectively idempotent.
If you are sending the content in the POST without a key to prevent duplication, then there is a problem with retries.
But that is really just a pointer to:
It is win-win for everyone. It is easier to implement and apps don't have to poll you constantly.
Its basically hubless pub/sub messaging over HTTP ("...over the internet", sure, but there is lots of much older versions of that); there's also the with-a-hub version, PubSubHubbub.