If you're using a web hook to notify your users of changed content, the WebSub standard (https://www.w3.org/TR/websub/; formerly PubSubHubbub; currently a candidate recommendation) is a good option. It solves a couple (though not all) of the problems of web hooks listed in the article, including:
- Uncertain security: signing notifications is in the spec
- Development and testing: ideally, with libraries that implement the spec, you can assume that the websub part will 'just work' and only need to test your handler function.
- Misbehavior is onerous: subscribers need to actively confirm their subscription every once in a while. This might seem complex, but it allows for periodic checks if the server/client is still functioning as expected, and ideally can be handled automatically by libraries.
- Retries: the spec leaves some room here, but again, a library can handle this.
As I decided to use WebSub as an alternative for a 'plain' webhook for a project recently, I wrote a library for the Flask web framework: https://github.com/marten-de-vries/Flask-Websub. I personally hope libraries for other frameworks will follow now the spec is moving less.
We built Auth0 Extend [1] to address with many of these concerns. Our hypothesis is that the logic that implements these extensibility points should live in the host platform.
For several years now, we've had this sort of hook-via-code in the Rules engine on our Auth0 identity product. It's been such a powerful enabler for many different integration scenarios that we thought that other companies may find the technology useful for their platforms.
Since committing to building Auth0 Extend, we've seen Twilio launch their functions [2] and just today CloudFlare launched its Workers [3]. It seems like the this is an approach that is in its infancy but that is here to stay!
We are just in the process of launching something similar at Converse to allow users to extend our conversational workflow capabilities to add their own modules, events, including extending API endpoints and the core UI.
Definitely a good way of doing things, still looking for more beta testers but feedback is very positive so far.
I've setup a few webhooks over the past few years, and one of the complications that people rarely talk about is the data that is passed to various webhooks.
As the article alludes to APIs change over time, and so you can find the data that is submitted to you changes. In my case I setup an end-point that responded to push events from various code-hosting services.
I found that the three "main" git-hosts sent widely different data in their webhooks, and updates we often painful. (Github, bitbucket, and .. something else. I forget now.) Trying to have one end-point to handle all services was a recipe for frustration and failures. Even parsing out the basic details was hard:
* This is the SSH repository URL of the thing that was committed to
In the end I moved to self-hosting git-repositories, rather than allowing users to point at my webhook. That allowed hooks in the master-repos to fire.
(In my case I trigger DNS updates when git commits are made.)
As using webhooks for git repositories is probably the most common usage, it would be really nice if the different APIs converged here. Now everyone is inventing their own (and I'm in fact guilty to this myself for a side-project).
Good tips! I actually just worked on detecting dead webhook endpoints, as I've had a number of users use ngrok for local testing, create hundreds of events, and then disable ngrok--and much like Stripe, I retry with an exponential backoff over the course of a few days. Needless to say, my logs were filled with failed webhook request errors and I had to figure out how to limit that from happening in the future. Very interesting list of alternatives to webhooks as well.
When it comes to axing dead endpoints, I see two ways to handle this based on the way webhooks are configured.
* If webhooks are configured by UI, then you have to contact the user and tell them their webhook is gone, e.g. via email.
* If webhooks are configured programmatically, then the endpoints ought to be able to be swept up silently, and the receiver should know to periodically refresh the configuration for as long as it cares about it.
Ideally all webhook implementations would work the second way, but unfortunately it is not standard practice.
I guess the section on Retries is closest, but we're looking at moving from a system with straight in-process hooks to one using webhooks, and a complication is we now have to provision a webhook endpoint with very high reliability to accept the callback and (effectively) call the old hook.
The thing is, all the hook does is put a message on a queue…
I would love to see some of these APIs replaced by WebSockets. It would allow building your own client for some services much more easily while maintaining push notifications
Webhooks should not be mentioned without also mentioning SSRF vulns. Because most companies use their firewall as a defense permitter, these attacks can be devastating.
Side note: their music is so good. Paired with their actual tech and vision, I think the music actually makes me want to work at SpaceX more than if paired with something less epic.
- Uncertain security: signing notifications is in the spec
- Development and testing: ideally, with libraries that implement the spec, you can assume that the websub part will 'just work' and only need to test your handler function.
- Misbehavior is onerous: subscribers need to actively confirm their subscription every once in a while. This might seem complex, but it allows for periodic checks if the server/client is still functioning as expected, and ideally can be handled automatically by libraries.
- Retries: the spec leaves some room here, but again, a library can handle this.
As I decided to use WebSub as an alternative for a 'plain' webhook for a project recently, I wrote a library for the Flask web framework: https://github.com/marten-de-vries/Flask-Websub. I personally hope libraries for other frameworks will follow now the spec is moving less.