

Show HN: Deepstream.io – A scalable server for realtime webapps - wolframhempel
http://deepstream.io/

======
imslavko
I looked at the source code and it seems that it mostly glues socket.io with
Redis/Memcached. What did I miss? How exactly is it more scalable than
anything else?

The front-page looks really cool.

~~~
wolframhempel
It goes a bit beyond that :-). Deepstream scales horizontally by allowing to
connect multiple deepstream nodes to a message bus (e.g. AMQP, Redis, Kafka
etc...). The nodes multiplex messages within each other and sync their
client's state.

------
nbrempel
I'm very excited about this! Without reading too much into this, is it safe to
say it's like a self-hosted firebase?

"Due to the way deepstream structures its data (JSON blobs, identified by a
primary key) it is recommendable to use an Object oriented or schema-less
database (e.g. MongoDB or RethinkDB), rather than a relational database like
MySQL."

Is there a way to run complex queries against the data or it designed to
abstract that all away from you to be as fast as possible?

~~~
sirstompsalot
Similarly (without reading too much about it) can I use InfluxDb instead of
RethinkDb? (Disclaimer: I'm a RethinkDB fanboy, but I'd like to know my
options.)

~~~
vidarh
InfluxDB is a time series database (think reporting/analysis of collected
system metrics). It's not intended as a general purpose database.

~~~
sirstompsalot
No, its definitely not. However, I can think of a few instances where I might
use it along with Deepstream.

------
yc1010
Does anyone know with what these style of websites are made with? there must
be hundreds of .io sites here on HN that look and feel the same

I am not a designer so do not know what this latest web design practice is
called

~~~
nathancahill
I've used the Startup Framework Generator[1] for building this style of
website. Highly recommended.

[1] [http://designmodo.com/generator/](http://designmodo.com/generator/)

~~~
kevin
Oh wow, this is great. Thanks for sharing!

------
tristanz
Very necessary. A few questions:

\- How big can records be? Are they stored as a single object in the
persistence layer or using something like materialized paths. The firebase
data model is one huge JSON structure, so this looks different.

\- How are conflicts handled? Last writer on server wins or last writer
anywhere? Do you ensure eventual consistency of all the clients even when you
apply optimistic writes on the client?

\- Can data be larger than the cache size? How is cache eviction handled?

~~~
wolframhempel
Thanks \- Records have no hard size limit - but are limited to the maximum
size of your cache entry or database document. The tcp connection can send a
single record update over multiple packets.

\- Records already have a version that's incremented with every change and
also persisted as part of the record's meta-data. Currently deepstream uses a
simple first-wins strategy and subsequent updates for the same version number
get rejected. However configurable merge strategies are in planning as part of
a "record-options" setting.

\- Currently data has to be explicitly deleted from the cache.

------
applepple
This looks similar to SC [http://socketcluster.io/](http://socketcluster.io/)
\- But I guess SC doesn't take care of the storage layer - You can use your
own database of choice.

------
kevin
It’s clear that you’ve designed something that gets people’s attention (your
animation on the homepage might be one of the best examples of real time apps
I’ve ever seen), but based on the questions in this thread…people are looking
for differentiation.

If I were you, I’d start by answering everyone’s questions here. These are
your potential users and leaving them hanging seems odd. The next step would
be answering those questions on your site.

This is the kind of service that people are going to start using for a new
project (rarely are people going to switch existing infrastructure to
something brand new). But any new framework, language or platform requires
investment and with developer related services and tools, people are going to
research first. Help them do that research efficiently.

Good luck taking this to the next level!

~~~
wolframhempel
Thank you for your feedback. You're absolutely right, but I guess we've missed
the window for Hackernews to an extend now. We're currently putting together a
"how is deepstream different from ..." page and a general "about" section for
the webpage which will be released shortly.

Again, thanks for your advice, much appreciated.

------
tacone
Wow, much needed, I'm downloading the samples right now to take a look.

(fyi: bad link at the bottom of this page
[http://deepstream.io/tutorials/simple-app-using-
react.html](http://deepstream.io/tutorials/simple-app-using-react.html) it
links to the ko demo, not the react's)

~~~
wolframhempel
thanks :-). We're working on fixing it.

------
degenerate
Very nice self-explaining landing page! Although the bulleted list webfont
renders rather poorly in Win7 Chrome:
[http://i.imgur.com/j0a0rAQ.png](http://i.imgur.com/j0a0rAQ.png)

~~~
wolframhempel
oh dear, we'll definitely have to have a look into that. Thanks for posting
the screenshot :-)

~~~
nerdy
Same in Win7 FF38

------
aaggarwal
This looks promising. Does this server architecture also has an embedded API
framework which can utilized to create APIs by just specifying end-points? and
is there any testing framework integrated for node.js applications?

------
seletz
I believe the permissions example with Lisa in the tutorial has it's logic
backwards -- or am I missing something?

[http://deepstream.io/tutorials/permissioning.html](http://deepstream.io/tutorials/permissioning.html)

------
hippich
Link to client api is broken -
[http://i.imgur.com/egcKVyv.jpg](http://i.imgur.com/egcKVyv.jpg)

(case insensitive file system on macos may be? :))

------
asadlionpk
This looks great! I am thinking of writing a Firebase compatible wrapper for
this when I have free time. The main differences are in permissions and auth I
guess.

~~~
wolframhempel
That'd be great :-)

------
harryf
Interesting. How does it differ from
[https://getstream.io](https://getstream.io) ?

~~~
themgt
The GetStream server isn't open source, is it?

~~~
harryf
I believe they Open Sourced a significant part of it here -
[https://github.com/tschellenbach/Stream-
Framework](https://github.com/tschellenbach/Stream-Framework)

------
dvliman
Is there a documentation of transport protocol? i.e http sse, websocket?
...peeking the source code a little bit - tcp-socket?

~~~
wolframhempel
Deepstream is using a TcpPort to allow for connections from other Node-
processes (and hopefully, going forward Java, C++ etc..) and engine.io for
browser communication. Engine.io initially connects using long-polling http
and upgrades to Websocket if available.

A (rudimentary) overview of the message structure can be found here:
[http://deepstream.io/tutorials/message-
structure.html](http://deepstream.io/tutorials/message-structure.html)

~~~
spleeder
Why go through the trouble and reinvent the wheel and not use one of the
available protocols and serialization methods?

------
flippyhead
Very nice. I'd be curious to hear a bit on how this is different than what
Meteor provides when using MongoDB. Thanks!

------
nvartolomei
What about offline first?

