
REST Hooks - Stop the polling madness - qwertzlcoatl
http://resthooks.org/
======
annnnd
Am I the only one who has trouble sifting through all this marketing? Even
"read more" link leads to an article where half of the text promises what will
be covered and another half talks about some subscriptions and how they are
managed (they might have been mentioned in the first half, I agree - and I
also read text only twice).

If someone has a better understanding of how this works, I would appreciate a
TL;DR version for programmers (which are probably the target audience?).
Thanks!

~~~
bryanh
We are pushing some copy updates to the homepage, most of the technical
details were in docs. TL;DR coming to the homepage soon, but the gist is:

    
    
        REST Hooks itself is not a specification, it is a collection of patterns that
        treat webhooks like subscriptions. These subscriptions are manipulated via a
        REST API just like any other resource. That's it. Really.
    

Thanks for the feedback!

~~~
kristopolous
I have no idea how this solves the fundamental push/pull networking problem.

Is a network connection kept open? Is there an assumption that the user has
some port open that can be contacted?

I see that you somehow reduced your calls, but I don't see how. Please tell me
in actual socket level networking terms how this is done; who establishes a
connection to what, for what, and for how long?

Also how do you deal with disconnects, timeouts, and missed realtime data.

~~~
singlow
I think they are just suggesting that services which are being polled
frequently instead implement a rest-like subscription service which would
accept an endpoint and then push the data to all of the subscribed endpoints
when it changes. In many cases it is more work from the developers of those
services but would reduce their bandwidth and server load immensely.

~~~
acjohnson55
Ohhh, so it's really a server-side technology to better consume third-party
APIs. For whatever reason, this wasn't at all obvious to me. I had initially
assumed that this was technology between client and server sides of a single-
page application.

~~~
kristopolous
As did I. I've built these systems (socket-like systems that work efficiently
across browser in diverse environments). And if you think you can just get
that right the first time, then either you have the hacking skills of which I
have never seen or you are (more likely) wrong.

But this thing here they are talking about is something I've been doing for a
long time. I didn't think about making a brochure site for it though.

Just follow the principle of making the parts as dumb as possible. They should
be the stupidest simplest things ever. Sometimes you can't get to this model
on the first try, and that's ok. But the closer you get to it, the better the
product becomes. I swear this is true and part of the soul of the machine's
ghost.

If you can't make things laterally connect this way, then make them vertically
stack this way and then laterally connect this way. But don't obfuscate the
objective.

------
steveklabnik
> REST Hooks itself is not a specification, it is a collection of patterns
> that treat webhooks like subscriptions.

Bummer. No matter if you like it or not, a collection of patterns with a name
_is_ a specification, just possibly a poorly defined one. See the confusion in
this thread. If it were a link to a spec, nobody would be confused.

> Skip the pedantic arguments about standards and implementation details

This reads to me as "everyone is going to have a slightly incompatible
implementation. One library wont be good enough, I'll need to write a new one
per site that uses this."

Furthermore, what about PubSubHubbub?

Finally, polling is great: Ive found few situations where it doesn't work
well, people just tend to only do the most basic of implementations and blame
polling. See [http://roy.gbiv.com/untangled/2008/paper-tigers-and-
hidden-d...](http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-
dragons) for a really interesting example.

EDIT: numbers were real, added a supporting link

~~~
technoweenie
GitHub implements the PSHB subscription API. It's fine, but doesn't really fit
in well with the rest of our JSON API. For instance, parameters are sent over
as form encoded values like "hub.mode".

[http://developer.github.com/v3/repos/hooks/#pubsubhubbub](http://developer.github.com/v3/repos/hooks/#pubsubhubbub)

We also have a regular JSON endpoint for our hooks resource (which is
essentially a subscription API).

~~~
julien
Let's make it better :) The PubSubHubbub mailing list is still open and would
welcome your comments on how to make things better!

------
tkellogg
This isn't impressive. [MQTT][1] is a much better protocol, and an open
standard. It's a pub/sub that runs over TCP with variants that run over UDP &
TLS. There's only a 2+ byte overhead, so it's very light. It's ideal for
mobile devices because it has a QoS system built-in, so the developer doesn't
have to think through the problem of missing messages where cell service is
sub-optimal. MQTT is even resource-based, so it fits well with REST. I wish
more people would start using MQTT instead of these crazy hacks like REST
Hooks.

[1]: [http://mqtt.org/](http://mqtt.org/)

------
bialecki
If for some reason you can't or don't want to implement webhooks, at least
make sure you the GET endpoint for any object has a query param that supports
fetching the most recently updated or created objects _and_ supports
pagination.

It sounds trivial, but you'd be surprised how many APIs don't support one or
both of those features. When you're writing an API it might seem unnecessary
to start (after all, who could ever have 1000s of <object>?), but if someone
ends up polling your API frequently, having those two features can reduce a
lot of unnecessary load for both you and the poller. And, of course, make sure
you have an index on the created and/or updated dimensions.

That said, webhooks are terrific. Few things to consider when implementing
them:

\- Think carefully about the payload you send to the webhook. It's usually a
good idea to send some related objects/data because many times when someone
gets a webhook payload, that'll trigger calls to your API to get related
information you could've reasonably sent them in the initial payload.

\- You'll likely want to some way to keep track of errors so if an endpoint
starts returning 404s or 500s you have a way to programmatically discard it
after X failed attempts.

\- In your docs, give sample, "real world" payloads developers can test
against. It saves times over creating a RequestBin, pushing there, copying,
cURLing, etc. (Remember, you can't set up a webhook to localhost.)

\- A nice to have is some sort of retry capability with an exponential back-
off. Servers go offline and if they get pushed webhook data then, those
messages are lost. You could say, "tough, it's the consumer's responsibility,"
but if having all the data is important, most people will resort to polling.
(Somewhat related, you'd be surprised how often the APIs of some larger SaaS
companies are "offline" \-- e.g. returning 503 --, so these things do happen.)

~~~
johns
You can send a webhook to localhost, just in a roundabout way using one of the
local tunneling solutions: [http://john-sheehan.com/blog/a-survey-of-the-
localhost-proxy...](http://john-sheehan.com/blog/a-survey-of-the-localhost-
proxying-landscape)

Great points though.

------
aeon10
It is awfully difficult to get a technical TL;DR version from a website, whose
target audience is programmers. I think explaining what this does concisely
and technically, preferably on the front page with code examples etc would be
really useful.

------
bryanh
We also wrote a quick drop-in Django application [0] (that plugs right into
TastyPie if you have an API built on it) . We also have a sample CRM with a
live API running [1]. Our friends at NetEngine wrote a sample Rails app [2],
and our very own James wrote an awesome Node.js Sails app [3].

[0] [https://github.com/zapier/django-rest-
hooks](https://github.com/zapier/django-rest-hooks)

[1] [http://demo.resthooks.org/](http://demo.resthooks.org/)

[2] [https://github.com/net-engine/resthooks](https://github.com/net-
engine/resthooks)

[3] [https://github.com/zapier/node-
resthooksdemo](https://github.com/zapier/node-resthooksdemo)

~~~
aantix
Still scratching my head on the Ruby "reference" implementation. When I
subscribe resource_subscriptions, who is receiving the update notification? Is
the client side js code getting a push? Is the resource_subscriptions a long-
polling endpoint? Not very clear at all.

~~~
bryanh
The server itself will just POST a payload to a defined URL. I'm not wildly
familiar with the internals of that app or Rails in general, otherwise I'd
give more detail!

~~~
onli
So all this is just about the following pattern?

A wants to get updates of B. Instead of polling B for changes, it sends B or C
an URL of A and says: POST to this URL the moment something changes on B.

If that's the case, this really gets lost on all the hooks/subscriptions/load-
mumbojumbo on this site.

~~~
bryanh
That is the case! It's actually a hard concept to get across in words (we've
found). If you have or see an example that really resonates with you, do let
us know (and we're open to pull requests
[https://github.com/zapier/resthooks](https://github.com/zapier/resthooks)
:-).

~~~
onli
I think what is missing is the explanation of a subscription. On the github-
page, there is only one sentence in that regard:

> REST Hooks are a lightweight subscription layer on top of your existing REST
> API.

That is too short. There the explanation is missing, the same way on the
website. maybe in better english if mine is broken, but that addition could
fix it: "...,thus that instead of having to poll that API regularly, the
subscriber is notified with a POST the moment a change occurs."

I'm probably biased, because that is kind of the same way I once tried to
explain the concept, but I think it would be clearer that way.

~~~
bryanh
Thanks! We'll try to work that into the next round of edits!

------
mikeknoop
Oh neat! We're launching resthooks.org today with several partners to promote
an existing but underused real-time API pattern.

Here are more resources:

Intro: [https://zapier.com/engineering/introducing-
resthooksorg/](https://zapier.com/engineering/introducing-resthooksorg/)

Why: [http://resthooks.org/#why](http://resthooks.org/#why)

Demo: [http://demo.resthooks.org/](http://demo.resthooks.org/)

~~~
_lex
I'm looking for a quick response to "How", w/o going through code. Care to
enlighten me?

~~~
rcsorensen
[http://resthooks.org/docs/](http://resthooks.org/docs/) has a section titled
"Minimum Implementation Walkthrough" that looks to be what you're looking for.

------
jalfresi
I hate to be THAT guy but someone was going to say it so it may as well be me
:)

This is a collection of patterns, right? Well why not make this really REST
and rather than list a bunch of URL templates provide REL types for each of
these like so:

REL subscriptions-list -> GET subscription-create -> POST subscription-read ->
GET subscription-update -> PUT subscription-destroy -> DELETE

Now it doesn't matter what the URL structure is, I can pull the <link>
elements from the page and be TOLD the URL, rather than follow a URL template.
That way, this doesnt rely on out of band knowledge i.e this web page and its
(poor) description.

------
rcsorensen
Has anyone use PubSubHubbub (
[http://en.wikipedia.org/wiki/PubSubHubbub](http://en.wikipedia.org/wiki/PubSubHubbub)
) for this pattern in the past? For straight subscriptions, it's always been
compelling to think of the use cases being covered by PSHB and RSS.

~~~
julien
So, PubSubHubbub is not _linked_ with RSS anymore and works with any type of
data format, including JSON obviously :)

The quick win of PubSubHubbub compared to Resthook is that it has a 'security'
mechanism in it, which means that the subscriber can be 100% sure that the
service sending the notification is not someone impersonating it.

Aside from this, I feel like PubSubHubbub and Reshook solve the same problem:
programatically setting up webhooks.

------
mcantrell
This sounds like webhooks, but they have a set payload, so that the server
that is sent the webhook just calls the rest endpoint it is given in the
payload. Interesting idea, but it does sound like a regular webhook to me.
That flow is the same way I use the Stripe webhooks for instance. I use the
event id and make a call to Stripe directly, so that I can make sure the event
is real.

------
zrail
I don't really understand what this is. Is this an attempt at a formalization
of webhooks + JSON?

~~~
atonse
Same here - I keep coming back to "are they just talking about webhooks? or
are they also talking about some kind of persistent http streaming api?"

I'm thinking these are just Webhooks, but the REST part is throwing me off
because I think of it more as a consumption concept (consuming resources,
etc).

Update: Found this, which I think explained it best: "REST Hooks itself is not
a specification, it is a collection of patterns that treat webhooks like
subscriptions. These subscriptions are manipulated via a REST API just like
any other resource. That's it. Really." \- but it was in /docs, not the main
page.

~~~
zrail
Oh, ok. That makes sense. I implemented that for Emma a few years ago but I
don't know if I made it up or saw it somewhere else first. Probably the
latter.

------
bdcravens
The way the info is presented bugs me. The first "case study" is Zapier.
Later, in the "Learn how..." set of links, Zapier is listed as one of many.
Look to the footer, and the Github link, and it's of course evident it's a
Zapier project.

This project (er hmm, "initiative") is core to Zapier's business. If every
service out there had a hook atop their service, it'd make things a lot easier
for Zapier. That's cool. What bugs me is the feeling that Zapier's branding in
the whole thing is less than transparent.

It seems that open source projects are getting more and more marketing driven,
and the way this "initiative" is packaged is a sign of things to come.

~~~
elliotanderson
I don't see how a company being involved with this "initiative" affects the
core concept - it's a net benefit for both API consumer and producer in moving
to a subscribe/push model. Less requests, less load, closer to real time
updates. Swap Zapier with any other consumer and the result is the same

~~~
bdcravens
Disclosure - had their name been at the top somewhere, and said "we" instead
of "Zapier" it would boost the credibility of the project.

------
nkoren
Curious if anybody can answer a question for me. I've recently started
building some apps with MeteorJS, and am having loads of fun with it. At first
I'd been a bit suspicious of its magic pixie dust -- seemed too good to be
true -- but so far it's done everything it promised, and has allowed me to
focus on developing the parts of the app I care about. After checking briefly
to see that my data syncing was genuinely low-latency and low-bandwith (which
under every test I've subjected it to, it appears to be), I've pretty much
forgotten about this level of engineering and have focused on bigger-picture
stuff.

Which brings me to the question: should I care about something like REST
Hooks? Is it a mistake to become complacent and assume that the lower-level
infrastructure will Just Work? Or can MeteorJS (and presumably other high-
level frameworks) be trusted to handle this kind of stuff in a way that makes
it safe for me to forget about it?

Just curious what the esteemed HN denizens think about this, as I'm sure there
are some strong and reasonably well-informed opinions out there...

~~~
1qaz2wsx3edc
I think, if anything, Meteor proves it's possible that it can be the
responsibility of the framework, not the developer.

------
prodigal_erik
The docs talk about "event types" like contact.update which do not exist in
HTTP and require knowing the semantics of a particular group of resources. To
truly be RESTful, I would want "please notify me of any PUT, PATCH, or DELETE
to this list of URLs, or POST creating a URL that matches this pattern." And
if you're going to include a copy of the resource (I don't know what else a
payload would be) you need the media type the subscriber expects as if you
were doing content negotiation for a GET request from them. Or maybe the
payload could just be the current result of a HEAD request so the subscriber
can invalidate their caches, and they can send the appropriate GET request(s)
when and if they care to.

There should be some standard link relation like <link rel="subscription"> so
clients have some hint that this is available and where to request it.

I'd also want some way to manage the freshness/load tradeoff, like "please
notify me within one hour but no more than every five minutes".

------
hcho
Doesn't this assume that the API will only be used from a module living under
a URL?

Most REST apis these days seem to be consumed form client side.

~~~
brian_cooksey
The focus is on server-server communication. A client like a web browser would
need to use a different approach to get notifications, but there are options
in that space besides polling.

------
donpdonp
In addition to webhooks, using a websocket connection to listen for streaming
updates is a great way to eliminate polling.

~~~
dreen
They are also easier to use and much more straightforward than hooks. And if
you use something like Socket.IO the adoption of WS is irrelevant.

------
retrogradeorbit
Give a man a hammer, everything looks like a nail. REST has become a bit of a
hammer. The reason a lot of RESTful designed software I've seen has such poor
performance is because REST is used like a hammer... everywhere! And whole
bunch of places it's not needed and it's not ideal.

------
jalfresi
I hate to be THAT guy but someone was going to say it so it may as well be me
:)

This is a collection of patterns, right? Well why not make this really REST
and rather than list a bunch of URL templates provide REL types for each of
these like so:

REL subscriptions-list

------
derek1800
Are there patterns for client side JavaScript to be notified of changes to
REST resources? I don't believe this particular pattern applies if I
understand correctly as it needs a URL to notify.

~~~
jordanthoms
I think you'd have to have your server receive the hook, and then notify the
client side using websockets/long polling etc. Pusher makes that really easy.

------
lvh
I don't agree with this part:

> In other words, if everyone implemented REST Hooks, server load for both the
> sender and receiver could be reduced by 66x.

No, the number of requests could be reduced by a factor of 66. I'm not saying
that's not impressive, I'm saying that the polling requests that ended up
resulting in no action are cheaper than actionable requests, so, server load
will go down by much less than a factor of 66x. The amount of _work_ is the
same, just _busywork_ is less.

~~~
bryanh
That is a good point. What kills us the most is the latency on network IO when
polling.

------
lojack
While the paradigm of having a webhook makes more sense, I personally hate
working with them. I've yet to see a graceful way of handling local
development that doesn't involve localtunnel (or some alternative) and
constantly having to un-register and re-register endpoints every time
localtunnel gives you a new hostname. I understand the point and would prefer
working with them if there was just a good way to handle local development.

~~~
bryanh
This is a great point, is there a better way to do local dev with webhooks? I
always used localtunnel which is great it not perfect.

~~~
johns
Some other options here: [http://john-sheehan.com/blog/a-survey-of-the-
localhost-proxy...](http://john-sheehan.com/blog/a-survey-of-the-localhost-
proxying-landscape)

I personally think you should be using one that gives you permanent URLs
(ngrok or Passageway) but that also keeps a log when you're client is
disconnected. I'm pretty sure only our Passageway does that.

------
dpweb
Powerful idea, should be made a specification. Simple enough too - what they
describe is almost trivial to implement in /* insert your web framework of
choice */. You wouldn't even need a library. Handle your HTTP verbs, store
subscriber state, and call them out on events. It's nice.

------
tlrobinson
Here's another (work in progress) spec for managing webhook/HTTP callback
subscriptions: [https://github.com/progrium/http-
subscriptions/blob/master/S...](https://github.com/progrium/http-
subscriptions/blob/master/SPEC.md)

------
bdcravens
Why is this a thing? Your API interfaces with a message queue; your end client
uses all the libraries and patterns built up over the years to consume from
the queue. That doesn't sound all startup-y like "REST Hooks", but the pattern
has been around for a long time.

------
geuis
I just want to point out that having read that entire page, I still have
absolutely no idea what this is, how it works, etc.

I have a solution for polling. Its called websockets. Even more, there's a
well-supported library called socket.io that transparently handles it for all
browsers.

------
Kiro
After reading through all the comments here I still can't figure out what this
is. Sorry.

~~~
city41
This blog post should help, it helped me:

[https://zapier.com/engineering/introducing-
resthooksorg/](https://zapier.com/engineering/introducing-resthooksorg/)

------
jedberg
Isn't this like SUP?

[http://blog.friendfeed.com/2008/08/simple-update-protocol-
fe...](http://blog.friendfeed.com/2008/08/simple-update-protocol-fetch-
updates.html)

------
andreineculau
FWIW [http://barelyenough.org/blog/2009/05/push-or-
pull/](http://barelyenough.org/blog/2009/05/push-or-pull/) case closed

------
Hovertruck
> On average, 98.5% of polls are wasted

Erm. Is there a source for that?

~~~
andrewjshults
A bit further down the page, they cite Zapier

"Over a representative time period, Zapier polled for changes 30 million times
but only took action on 460,000 of those polls (1.5% efficient)."

~~~
bdcravens
It's a Zapier project, so they're citing themselves.

[https://github.com/zapier/resthooks](https://github.com/zapier/resthooks)

------
veesahni
TL;DR - REST Hooks is an API pattern where the API provides an endpoint from
which webhooks can be programmatically setup and removed.

------
derefr
This sounds pretty much like the polyfill Socket.IO/SockJS do to create a
socket atop polling-based transports.

------
bradhe
I have no idea what this is.

------
harel
Someone should show this to Facebook.... Just saying...

