
PourOver: A library for simple, fast filtering and sorting in the browser - jsvine
http://nytimes.github.io/pourover/
======
danso
So I visited one of the PourOver examples, this Academy Awards fashion feature
published earlier this year:

[http://www.nytimes.com/interactive/2014/02/02/fashion/red-
ca...](http://www.nytimes.com/interactive/2014/02/02/fashion/red-carpet-
project.html)

I opened the dev tools to inspect the traffic and code, and this pops up in
the console:

    
    
                  0000000                         000        0000000
                111111111      11111111100          000      111111111
                00000        111111111111111111      00000      000000
                000        1111111111111111111111111100000         000
                000        1111       1111111111111111100          000
                000         11       0     1111111100              000
                000          1      00             1               000
                000               00      00       1               000
                000             000    00000       1               000
             00000            0000  00000000       1                00000
           11111            000 00    000000      000                 11111
             00000          0000      000000     00000              00000
                000        10000      000000      000              0000
                000        00000      000000       1               000
                000        000000     10000        1     0         000
                000        1000000 00              1    00         000
                000         1111111                1 0000          000
                000          1111111100           000000           000
                0000          111111111111111110000000            0000
                111111111        111111111111100000          111111111
                  0000000              00000000              0000000
           
           
           NYTimes.com: All the code that's fit to printf()
           We're hiring: http://nytimes.com/careers
           
    
    
    

....You sneaky audience-targeting bastards

~~~
chimeracoder
> NYTimes.com: All the code that's fit to printf()

This gave me a chuckle. I was at a hackathon that the NYTimes sponsored a few
years ago, and my group used almost this exact joke for our project, an
ncurses reader for browsing the New York Times in the terminal[0].

I don't know who thought to put this in the source code, but I'd find it very
amusing if they got the inspiration from hidden slogan from a hackathon
project. Kind of like the old competitions in the 50s/60s to create company
jingles.

[0]
[https://github.com/ChimeraCoder/ncnyt](https://github.com/ChimeraCoder/ncnyt)

~~~
zhemao
I'd almost forgotten we made that joke. It wouldn't surprise me if that's
where they got it from. The NYTimes employees at the hackathon seemed to like
it. At least enough to tweet about it.

------
simonsarris
Half the comments so far are looking for a non-code demo, and I imagine they
mean something visual, so here's a bare-bones visualized version of the Basic
PourOver sample code:

[http://gojs.net/temp/pourover.html](http://gojs.net/temp/pourover.html)

It just takes the queries and uses the resulting data to make some nodes in
GoJS (Disclaimer: a Diagramming library I develop. Not free, but easy to set
up with this kind of data and see stuff fast).

If I had more time I'd make it prettier. The results are just visual
representations of the data results getting filtered. Its very easy to take
PourOver's example collections and data-bind some stuff to them (color of the
nodes is data-bound to monster gender, etc). I'm sure it can't be hard to do
the same in other data-bound visualization libraries.

This is very cool. I'll try to make a much prettier example tonight.

~~~
esmooov
Oh this is great! I really should port something like this to the docs.

------
esmooov
Hi, all.

A lot of folks are asking for a demo and, you're right. I should have included
one. My apologies. I'll get to working on one as soon as I can.

In the meantime, I encourage readers to check out the source for
[http://www.nytimes.com/interactive/2014/02/02/fashion/red-
ca...](http://www.nytimes.com/interactive/2014/02/02/fashion/red-carpet-
project.html) That's probably the clearest "demo" of PourOver at the moment.

More to come!

~~~
j_s
Thanks for doing your 'open source gardening'¹ so exuberantly!

I didn't see any contact info on your profile to let you know that the
proportion of folks here using Ghostery² or Disconnect³ is probably higher
than normal. Installing one will reveal how things break when JavaScript
libraries [usually orthogonal to page functionality] are blocked.

\--

¹[https://news.ycombinator.com/item?id=7593242](https://news.ycombinator.com/item?id=7593242)

²[https://www.ghostery.com/](https://www.ghostery.com/)

³[https://disconnect.me/](https://disconnect.me/)

------
dmix
This page needs a giant "demo" button near the top. The examples are all code.

------
paulcnichols
Reminds me of crossfilter
([http://square.github.io/crossfilter/](http://square.github.io/crossfilter/))
by square. It has a killer demo, however.

~~~
esmooov
Crossfilter was absolutely the inspiration. I just wanted to make a
Crossfilter that worked with the arbitrary boolean composition, could be
dynamically added to and updated, and supported some of the less numerical
patterns I was encountering.

~~~
victorhooi
Coolies =).

Would you ever add some of the Crossfilter functionality, to make it easy for
people to plug in their own data to PourOver?

~~~
esmooov
I would absolutely like to add a Crossfilter Filter type. I'd even accept a
pull request ... ;)

------
kylebrown
Noice! Btw, the docs page is acting funky on an iPad (iOS 7.1). The layout
seems to be alternating between mobile and desktop with each pan/scroll event.
[ps: only happens in landscape orientation. portrait is unaffected]

------
JangoSteve
It seems similar to our Dynatable plugin [1], which is basically the
functionality of this plugin with some additional table-read/write functions
included. The main difference being that this library depends on underscore,
while Dynatable depends on jQuery (which is mainly used for its browser
compatibility functions).

Given both library's emphasis on speed, it looks like I have something to
benchmark against!

[1] [http://www.dynatable.com](http://www.dynatable.com)

~~~
esmooov
This looks really cool. I wouldn't say PourOver is focuses on speed as much as
focused on being fast enough to get 60fps for 100k items. Cheers!

------
nathanhammond
I much prefer a basic construct that makes it simple to do filtering and
sorting that can be simply extended to any complexity. Sorting and filtering
are really nothing more than set manipulation (which they state themselves) so
with simple data binding this becomes a trivial exercise to build an
impressive client-side search.

In Ember that might look like this:

    
    
        Ember.ArrayController.extend({
            filterA: Ember.computed.filter('fieldName1', function comparator() {}),
            filterB: Ember.computed.filter('fieldName2', function comparator() {}),
    
            joined: Ember.computed.union('filterA', 'filterB'),
            
            filtered: Ember.computed.uniq('joined'),
    
            sorted: Ember.computed.sort(function comparator() {})
        });

~~~
esmooov
This is exactly what PourOver offers. You can chain filter results to any
boolean complexity, as well extend the default filter types to optimize
indexing and caching. Indeed, PourOver was trivial to make, an outgrowth of
the very pattern you define above. PourOver is just an attempt to scrap that
boilerplate, allow for the queries to be indexed, combined with sorts, and
automatically rebuild when the collection changes.

~~~
nathanhammond
Thinking about it more, I believe that my visceral reaction was to the
imperative nature of the library. I was imagining trying to code something
that _generated_ a filter (e.g. PourOver.makeExactFilter("mythology",
["greek","norse"]);) and it seems like it would be unnecessarily complex
without data-binding propagation/invalidation.

If I wanted to add "roman" mythology to that filter I would imagine something
like:

    
    
        var mythologies = ["greek", "norse"];
        // ... create a collection with filters
        mythologies.push("roman");
        PourOver.makeExactFilter("mythology", mythologies);
    

But you don't get that last statement for free, nor the one where you join it
into a collection, nor do I see a way to replace the previous mythologies
filter. Any time you need to reprocess a filter you've got to run it through a
series of imperative actions triggered from one of the events, and possibly
throw away the collection and regenerate it (if you can't remove disjoint
filters).

I would be pushing for a Object.observe/dirty checking/get-set version of this
to take PourOver to the next level of utility. (?/Angular/Ember)

~~~
esmooov
I think I'm a little confused. Why wouldn't you have just added the "roman"
possibility from the get go? Could you provide an example of a situation when
you don't know the universe of possibilities in advance?

~~~
rattray
If the user adds one, or new data enters from an outside source (pub/sub,
sockets, etc)

~~~
esmooov
Alright, this is a great point. It would be trivial for me to add
addPossibilities. I think I'll do that! Thanks. This will really help for
using PourOver to power tagging selection fields where you can ... add
possibilities!

~~~
nathanhammond
Exactly what @rattray said. It's something you get nearly for free in data-
binding world and is much harder to accomplish in imperative code. I wish you
luck, and make sure you blog about it when it's done! (I want to see how it's
implemented.)

------
barkingcat
More details at [http://open.blogs.nytimes.com/2014/04/16/introducing-
pourove...](http://open.blogs.nytimes.com/2014/04/16/introducing-pourover-and-
tamper)

There are a few links to projects at the nyt that has used these two
libraries.

------
drv
I wonder if the misspelling is intentional. (The idiom is "pore over".)

~~~
dandelany
I'm sure it is - "pour over" is a way of making coffee, and its sister project
is called "Tamper" which is also coffee related. The "pore over" double
meaning is clever though.

~~~
drv
Indeed, looking at the logo again, it appears to be a pour-over brewer -
clever. (I am a coffee neophyte and wasn't aware of this brewing method.)

------
rpedela
Off-topic, but related.

Does anyone know of any JS libraries that allow for Excel-like interaction?
However I am looking for something that just implements the presentation
layer.

The problem I am running into is that the libraries that do exist conflate the
presentation and data layers and try to do everything for you. In other words,
they make the assumption that all data will be downloaded to the client and
then manipulated on the client. These libraries do not fit my use case.

~~~
cfeduke
I am actually in the middle of an implementation of a grid view myself; I'm
giving Handsontable[1] a whirl and going to look at using pourover for
filtering my dataset. Definitely meets your Excel-like interaction though not
certain that it doesn't try to do everything for you since I haven't made it
that far yet. (Looks promising at least.)

1\. [http://handsontable.com](http://handsontable.com)

------
nashequilibrium
If you going to handle that many obejcts in the browser wouldn't you use
something like pouchdb? [http://pouchdb.com/](http://pouchdb.com/)

Also "Pagination strategies with PouchDB"
[http://pouchdb.com/2014/04/14/pagination-strategies-with-
pou...](http://pouchdb.com/2014/04/14/pagination-strategies-with-pouchdb.html)

------
Neff
And here I thought I would be reading about the App.net PourOver posting
service[0], which the NY Times Opinion account uses for posting[1]

[0]:
[https://directory.app.net/app/255/pourover/](https://directory.app.net/app/255/pourover/)
[1]: [https://alpha.app.net/nytopinion](https://alpha.app.net/nytopinion)

------
earless1
I've used the [https://mixitup.kunkalabs.com/](https://mixitup.kunkalabs.com/)
library for similar functionality in the past. I am not sure how well it works
with hundreds of thousands of items, but it does work well for the 600ish
items that I am using it for.

------
mbesto
Is this basically what Google Refine[0] does with their facet filters? (but
obviously open source and allowing anyone to build their own platform)

[0] - [http://openrefine.org/](http://openrefine.org/)

------
grumblestumble
It would be a great move to isolate the Views/UI part of this out from the
rest of the code base, which would make it more usable inside other MV*
frameworks like Angular or Ember.

------
bestest
Benchmarks comparing PourOver to Backbone would be nice. Anyone?

~~~
esmooov
There are some benchmarks against the base underscore filter methods in the
tests. These are what Backbone uses by default. However, the real utility of
PourOver is less in its raw speed -- I'm sure that it could be improved by
smarter folks than I -- but in the patterns it abstracts. I hope you find it
useful.

------
alixaxel
Having been recently exposed to the benefits (specially performance-wise) of
lodash vs underscore, I wonder why it depends on the later.

~~~
jdd
From a glance of the source it doesn't look like PourOver is doing any heavy
lifting with Underscore. PourOver works lodash.underscore build of Lo-Dash
without issue though because it's not doing any heavy lifting you probably
won't see a perf difference.

------
reshambabble
This is awesome, and exactly the type of thing I've been looking for. Thanks
so much for sharing! Can't wait to try this out.

------
b0z0
I love how New York Times has the coolest devs. Anyone here who works there?
What's it like?

~~~
jashkenas
I do. It's pretty fun.

This might be a bit embarrassing ... but a couple of years ago, Dan Sinker
came by and asked a few of the developers in the newsroom to talk about
exactly that question.

Here's me:
[https://www.youtube.com/watch?v=k3HanxEFFlk](https://www.youtube.com/watch?v=k3HanxEFFlk)

Here's David Nolen (although he won't be around for much longer ;)
[https://www.youtube.com/watch?v=qOdoCqoqXlU](https://www.youtube.com/watch?v=qOdoCqoqXlU)

Here's Tiff Fehr:
[https://www.youtube.com/watch?v=8GOWLNWifxc](https://www.youtube.com/watch?v=8GOWLNWifxc)

~~~
akilism
nice...but yikes that seems like an intimidating place to apply for a
job...you guys are all heavy hitters...

~~~
esmooov
I totally wouldn't be intimidated. We all just love what we do. The best way
to get a job is often to get to know the folks on the team, hack with them on
projects. I'd really hate to work at a place that I felt was exclusive.

~~~
akilism
Oh I didn't mean to imply it was exclusive, just that I'd feel like I was out
of my league. Like I've built a few things with D3 but I still find myself
getting confused sometimes, and I've gone through some of David Nolen's
clojurescript posts / tutorials and I'm still not 100% clear on the advantages
of core.async...but then again I don't use them all that often either.

~~~
esmooov
Well, if you ever wanna chat about core.async or PourOver or jobs or life, I'm
more than happy to do so: erik.hinton@nytimes.com Nolen's also a great guy and
really serious about helping folks out with stuff like core.async. I'm sure he
wouldn't mind jumping in an IRC chat to guide you along!

------
anarchy8
Anyone have a demo?

------
daemonk
Similar to crossfilter?

