

Why local state is a fundamental primitive in stream processing - datascientist
http://radar.oreilly.com/2014/07/why-local-state-is-a-fundamental-primitive-in-stream-processing.html

======
mamcx
(I read also [http://engineering.linkedin.com/distributed-systems/log-
what...](http://engineering.linkedin.com/distributed-systems/log-what-every-
software-engineer-should-know-about-real-time-datas-unifying)).

I wish to solve the problem of sync data from mobile to a master database, and
is about invoices, orders and that kind of data. Here, is necessary to
preserve the order of the data actions. The data must return back to each
client, and the clients could be offline from time to time. In some cases for
a couple of days. I have total control on the solution, so I could move to any
direction, but my only hard requirement is that the data in the client is on
sqlite.

I struggle to build a reliable solution, but I understand that a log based
could be a good idea. Wonder what way could lead me closer to the solution?

~~~
rakoo
So you need to _process_ your events linearly, but you don't need to _store_
them linearly.

Here's a way you could achieve this:

\- keep sqlite

\- use Pouchdb on top of it to sync your data to your server, typically
Couchdb

\- Since you know the order of events, tag each of them with an application-
level sequence number

\- On your server, when you receive an event, check all events that happened
before, in order. For each event, if it has not been processed, process it

~~~
mamcx
But I need to duplicate the data to pouch, then move it to couch.

My question is how do the sync myself. How build the log so it be usefull to
this task? To be clear, how is the algorithm for this? Any hint or open source
example?

------
capkutay
Shameless plug: at WebAction we have an enterprise-grade (exactly once
processing, scalable, built-in streaming data integration) SQL-based streaming
platform. It's clear real-time analytics is headed in this direction and we
have some interesting problems to solve.

If anyone is interested in chatting about it, feel free to email me
john@webaction.com

------
vosper
Is anyone using Samza outside of LinkedIn? I've been interested in it for a
while, and I'd love to hear your experiences.

