
Stream processing, Event sourcing, Reactive, CEP… and making sense of it all - martinkl
http://blog.confluent.io/2015/01/29/making-sense-of-stream-processing/
======
turtle4
Thanks for posting. There's one piece of this that hasn't quite clicked yet
for me, which is UI refresh coupled with the event stream approach. In the
article's example, for instance, you have the user events going into the
stream, which refreshes the cache, db, data warehouse, etc.

In a normal UI, the user makes some action, which would update the db, clear
the cache for the affected key(s), and display the updated page to the user.
In the event stream processing way, if the front end events are what is being
logged and the cache refresh for example is running via Samza or some other
processor, how does the UI get refreshed appropriately since the processing of
the stream is decoupled? Or does the main app cache remain coupled to the form
submission and there are just 'other' uncoupled caches, etc?

~~~
martinkl
This talk was mostly about backend systems, but it hints at Functional
Reactive Programming at the end. That's a field in which people are working
out how to update the UI based on event streams. Some cool projects to look at
in this area are Elm, React and Meteor, for example.

Once you've got a UI that can be dynamically updated based on event streams,
you can hook the backend streams into the UI using a WebSocket or similar.

~~~
ghuntley
Imagine a simple iPhone application which has the following components:

    
    
            // a text box in which the user types what they are searching for.
    	// a submit button on which the user can optionally press to initiate the search.
    	// a List<T> of search results
    	// a service that takes the search query and executes the search.
    

This is how you can implement the above in a FRP manner:

    
    
            public class SearchViewModel : ISearchViewModel
    	{
    	    public ReactiveList<SearchResults> SearchResults { get; set; }
    
    	    private string searchQuery;
    	    public string SearchQuery {
    	        get { return searchQuery; }
    	        set { this.RaiseAndSetIfChanged(ref searchQuery, value); }
    	 
    	    }
    	 
    	    public ReactiveCommand<List<SearchResults>> Search { get; set; }
    	 
    	    public ISearchService SearchService { get; set; }
    	}
    
    
    	public SearchViewModel(ISearchService searchService = null) : ReactiveObject, IRoutableViewHost
    	{
    	    SearchService = searchService ?? Locator.Current.GetService<ISearchService>();
    	 
    	    // Here we're describing here, in a *declarative way*, the conditions in
    	    // which the Search command is enabled.  Now our Command IsEnabled is
    	    // perfectly efficient, because we're only updating the UI in the scenario
    	    // when it should change.
    	    var canSearch = this.WhenAny(x => x.SearchQuery, x => !String.IsNullOrWhiteSpace(x.Value));
    	 
    	    // ReactiveCommand has built-in support for background operations and
    	    // guarantees that this block will only run exactly once at a time, and
    	    // that the CanExecute will auto-disable and that property IsExecuting will
    	    // be set according whilst it is running.
    	    Search = ReactiveCommand.CreateAsyncTask(canSearch, async _ => {
    	        return await searchService.Search(this.SearchQuery);
    	    });
    	 
    	    // ReactiveCommands are themselves IObservables, whose value are the results
    	    // from the async method, guaranteed to arrive on the UI thread. We're going
    	    // to take the list of search results that the background operation loaded, 
    	    // and them into our SearchResults.
    	    Search.Subscribe(results => {
    	        SearchResults.Clear();
    	        SearchResults.AddRange(results);
    	    });
    	 
    	    // ThrownExceptions is any exception thrown from the CreateAsyncTask piped
    	    // to this Observable. Subscribing to this allows you to handle errors on
    	    // the UI thread. 
    	    Search.ThrownExceptions
    	        .Subscribe(ex => {
    	            UserError.Throw("Potential Network Connectivity Error", ex);
    	        });
    	 
    	    // Whenever the Search query changes, we're going to wait for one second
    	    // of "dead airtime", then automatically invoke the subscribe command.
    	    this.WhenAnyValue(x => x.SearchQuery)
    	        .Throttle(TimeSpan.FromSeconds(1), RxApp.MainThreadScheduler)
    	        .InvokeCommand(this, x => x.Search);
    	}
    

If the above sounds compelling then check out
[http://reactiveui.net/](http://reactiveui.net/) which uses the Reactive
Extensions for .NET to create elegant, testable User Interfaces that run on
any mobile or desktop platform. Supports Xamarin.iOS, Xamarin.Android,
Xamarin.Mac, WPF, Windows Forms, Windows Phone 8, Windows Store apps.

If you want to learn more about FRP then I suggest the academic paper "Out of
the Tar Pit" by Ben Moseley & Peter Marks @
[https://raw.githubusercontent.com/papers-we-love/papers-
we-l...](https://raw.githubusercontent.com/papers-we-love/papers-we-
love/master/design/out-of-the-tar-pit.pdf) then make your way back to the
origin of FRP - [http://elm-lang.org/learn/What-is-FRP.elm](http://elm-
lang.org/learn/What-is-FRP.elm)

As for a good example of a backend designed around Event Streams/Reactive
Extensions then check out
[https://github.com/AdaptiveConsulting/ReactiveTrader](https://github.com/AdaptiveConsulting/ReactiveTrader)

------
rationalthug
I think this is the best article I've read about stream processing yet. Love
how all of the related concepts are tied together and how concepts in one form
of stream processing naturally fall out of the structure preferred in another
form of stream processing.

Great explanations and insights.

~~~
eoincathal
Seconded. I was glued to the article from start to finish.

------
nell
If you've heard of the terms Event Sourcing, Storm, Kafka but don't really
know what it means this write up explains the concept really well. Take the
time to read it.

