
Show HN: Heartbeat – Transform REST Endpoints to Streaming APIs - sidi
https://heartbeat.appbase.io
======
dvcc
Looks cool but a minimum of 1 minute polling intervals? That's a pretty long
time for what should be streaming data, no? If the data has no need to be
real-time then why stream it?

Might want to fix the Heartbeat icon link by the way, it currently links to
[https://heartbeat.appbase.io/heartbeat.appbase.io](https://heartbeat.appbase.io/heartbeat.appbase.io)

~~~
sidi
We would like to enable a lower polling interval, but wanted to see the
general response before doing that.

In fact, we would like to enable:

1\. A lambda function support to transform APIs (diffing to only append new
data, adding meta data, removing unnecessary fields) before they get streamed.

2\. Polling frequencies of up to 1s.

3\. Custom domains for the resulting streaming endpoints.

~~~
mathiasrw
Now - THAT - sounds awesome

------
doublerebel
Well, I can certainly see the usefulness. I just had a need this past week,
but don't mind running my own Node instance. Here's sseries-of-tubes, a Nodejs
library I wrote to turn any Express/Connect route (or anything you can poll)
into a Server Sent Events stream:

[https://github.com/nextorigin/sseries-of-
tubes](https://github.com/nextorigin/sseries-of-tubes)

Pretty fast for you guys to turn it into a service already. What are you using
on the backend? Node makes it really easy.

~~~
jkarneges
Appbase uses Pushpin
([https://github.com/fanout/pushpin](https://github.com/fanout/pushpin)) to
handle streaming.

------
PCaponetti
Why don't more web guys use MQTT for pub/sub live data? I saw the PubSubHubBub
in the comments already, but most devs are still shoe-horning HTTP into
solutions that really aren't request/response based

~~~
jkarneges
In general, the API community has consolidated around HTTP for realtime push,
mainly because it's what developers know and it's super easy to use (you can
just curl a stream for example).

Another thing to know about APIs is that the endpoints tend to be resource-
oriented, and resources are often abstractions. For example, when you interact
with Twitter's streaming API, no doubt there are some pubsub mechanisms going
on behind the scenes, but pubsub concepts such as publishers, subscribers,
brokers, topics, messages, store-and-forward, etc are hidden away by the API.
You just fetch a resource and get tweets.

My feeling is that the most successful protocols used in APIs are going to be
those that expose very little about the inner workings or topology of the
server and other client entities.

~~~
niftich
This; and HTTP has a massive install base, readily available clients on every
platform, casual debuggability from tools many people already possess (browser
address bar, browser js console, browser addons, curl).

Specific protocols have their merits, but they can only really compete in the
backend-to-backend space, and even there they have to compete against HTTP
endpoints. In anything that can be construed as vaguely front-end, HTTP has a
massive first mover advantage with the ecosystem it brings.

~~~
xg15
I wonder though, doesn't SSE technically violate a lot of HTTP's assumptions?
From the point of view of a HTTP middlebox, a SSE response would basically be
a single endless entity, transmitted at a very low bandwidth. I'm no expert at
real-world deployment of SSE, but that they sounds to me like it could break a
lot of proxies that don't have special handling for SSE. (Or at least cause
unnecessary resource consumption for those proxies)

Also I wonder why there are not more websocket-based APIs. WS has the same
widespread support and ready-made debug tools as HTTP and seemed to be
designed exactly for push or non request/response use cases. So, out of
couriosity, why would SSE be preferred over WS?

~~~
niftich
Streamdata.io (linked in a different post) has blogged [1] about why they
chose SSE instead of WS. Additionally, WS drops you down to something very
similar to raw TCP, so you have to bring your own application-level message
framing, while SSE comes with its own minimalistic format. From a middle box's
perspective, SSE isn't that different from streaming video or long polling,
although the SSE recommendation [2] does have a line about older proxies:

"Legacy proxy servers are known to, in certain cases, drop HTTP connections
after a short timeout. To protect against such proxy servers, authors can
include a comment line (one starting with a ':' character) every 15 seconds or
so."

[1] [http://streamdata.io/blog/push-sse-vs-
websockets/](http://streamdata.io/blog/push-sse-vs-websockets/)

[2]
[https://html.spec.whatwg.org/multipage/comms.html#authoring-...](https://html.spec.whatwg.org/multipage/comms.html#authoring-
notes)

~~~
xg15
As far as I know, two of the few things WS does on top of TCP are in fact:
making endpoints URL-addressable - and providing message framing. You can even
have text-only frames with a well-defined encoding.

In any way though, thanks for the links. I haven't given them a deep read yet
but will definitely do so soon.

------
LeonM
Looks neat, but I can't really think of a real world application for it other
than some debugging of API's. It's basically polling-as-a-service. The hashtag
about the real time web is a joke, right?

Please, don't turn it into some kind of SAAS, that would ruin everything. I'd
be willing to pay a license fee on a stand-alone app, not a monthly fee on a
tool I'd use once every few months.

~~~
forgotpwtomain
The socket.io example server is 6 lines:
[https://github.com/socketio/socket.io](https://github.com/socketio/socket.io)

A polling loop is 4-5? I guess that's everything minus the WEB Gui.

------
SmellTheGlove
This is very cool, thank you for sharing. I wish I could demo, but my current
work is such that I really can't expose my endpoints/data to a 3rd party
(financial services/PII type work). That said, pretend I could provided we had
appropriate security - tell me about the security involved.

~~~
sidi
There are two parts:

1\. Source API endpoint - Heartbeat fetches the data using the authentication
supported by the source API (basic auth, security token in headers, URL query
strings for example).

2\. Heartbeat transformed endpoint - This is accessible over an HTTPS endpoint
and is secured with a basic auth username:password key. As long as you use it
within a secure environment (server code, or an internal network), there
should be no problems. If you use it within a web app that's distributed to
other users, they can see the key. However, this key is read-only and we will
allow generating a new key from within the dashboard shortly.

------
simonpure
Reminds me of PubSubHubbub but using streams instead of webhooks for updates -
[https://en.m.wikipedia.org/wiki/PubSubHubbub](https://en.m.wikipedia.org/wiki/PubSubHubbub)

------
altharaz
Hey, that's a cool service.

Is that different from [http://streamdata.io/](http://streamdata.io/) ?

