

Meteor 0.7.0: Scaling realtime queries using oplog instead of poll-and-diff - debergalis
https://www.meteor.com/blog/2013/12/17/meteor-070-scalable-database-queries-using-mongodb-oplog-instead-of-poll-and-diff

======
rhgraysonii
It seems as if continuing to address these scalability issues is what will be
one of the largest deciders in Meteor's future. It seems as if I near
constantly am perusing posts on the framework here, and after going through
the Discover Meteor book as well as several small projects that were largely
simple prototypes, I've began to have a lot of faith in it for these reasons:

1\. It is EASY. Like, really, really easy. I went through Discover Meteor in
less than two days, and was easily imagining how to implement other
applications after one run through.

2\. It is accessible. The free deployment on the .meteor domain, the open
source sensibilities, the constant growth. It's got a good snowballs path
going on growth.

3\. The developers really believe in this product. Everyone on HN has an
opinion of the best way to develop software. That said, I feel there is a
certain intrinsic respect we share for one another here(MOST of the time ;])
and to ignore developers who are as passionate as these guys building a
functional product that clearly has room to grow, we'd be fools to ignore them
if we didn't have the time and/or interest (and the interest is hard to not
have with what it offers so easily).

This article seems to really further my faith in Meteor. Stopping the "pull
and diff" method is a great first step towards making it truly scalable in the
sense other frameworks have developed the capability.

~~~
rglover
I've been working with the framework almost daily and regarding your comments:

1.) Yep. I've gone from a jQuery-mostly front-end dev to being able to build
full blown apps inside of 6 months. Not only that, the apps run really well.

2.) Incredibly. Even without the meteor.com service, it's super easy to deploy
using things like Demeteorizer
([https://github.com/onmodulus/demeteorizer](https://github.com/onmodulus/demeteorizer))
which strips down a Meteor app to a raw node.js app that can be deployed
anywhere (that supports node, of course) in < 2 minutes.

3.) My favorite part about the framework is that the core devs have
meetups/hackathons frequently to talk about progress on development, show off
demos of pre-releases, and allow community devs to showcase their own work (if
you're curious, I highly recommend checking out the Devshop videos:
[https://www.youtube.com/user/MeteorVideos?feature=watch](https://www.youtube.com/user/MeteorVideos?feature=watch)).

Even if it doesn't sound like your bag of chips, at least do what the OP did
and check out Discover Meteor:
[http://discovermeteor.com](http://discovermeteor.com). It's worth the day or
two it will take you to complete.

~~~
rhgraysonii
The core devs are great with how active they are. Some have even reached out
to me on twitter when I've mentioned them. It's great to see how much passion
they have. What sort of product are you working on with meteor?

~~~
rglover
Right now I have two available: [https://properapp.com](https://properapp.com)
and [http://whatwhenapp.com](http://whatwhenapp.com).

~~~
rhgraysonii
These look great! Need any help with development? I'm about junior-dev skill
level but am actively looking to get into some side projects until I land a
job in the industry now that I'm out of the programming bootcamp I was at. I'd
ton a bunch of C/Python/Ruby before that, but it really solidified my core
software dev skillset and is allowing me to learn at a rapid pace and I'm
loving it :)

------
Tarang
Was 0.6.7-0.6.9 skipped? There was a release candidate for version 0.6.7

The big change with this version is the oplog tailing instead of the poll and
diff which has dramatic performance improvements over the previous version.
More details at the wiki: [https://github.com/meteor/meteor/wiki/Oplog-
Observe-Driver](https://github.com/meteor/meteor/wiki/Oplog-Observe-Driver)

~~~
httpteapot
About the version number:
[https://github.com/meteor/meteor/commit/8bd400e56065b0f2d61e...](https://github.com/meteor/meteor/commit/8bd400e56065b0f2d61e18dd220becb6a2bf7ac8)

I would have expected Meteor UI to be in 0.7, but that not a big deal and that
will probably be in the next version.

~~~
geoffschmidt
Yeah, we have several big projects in flight, and we're shipping them in the
order that they're ready.

We push them out first as prereleases (which you can run by passing a
--release flag to meteor, and it will automatically download and install
everything) and announce them to meteor-core and see what people think. In
this case both oplog tailing and Meteor UI were out as prereleases, but there
was more feedback and more feature requests for Meteor UI, so oplog tailing
ended up winning the race.

------
krebby
I'm working on a project currently that needs this but we're an express /
mongo stack and not a meteor stack. Anyone know of anything similar I can add
to the mongodb node driver? Or can I switch to meteor's node driver without
the rest of meteor?

~~~
akbar501
@krebby, you can create a DDP client that works outside of Meteor, which would
allow you to run a Meteor server + Mongo with your own front end.

The other option is to use Racer from DerbyJS.

If you're interested in adding a real-time data layer to an existing Node app,
then Derby is probably the easier path.

------
sorenbs
Trailing the oplog for events is a great strategy. 10gen use it to generate
point in time snapshots for disaster recovery and various companies have
described systems doing this to react to data changes in mongo. Web Scale
database triggers. Ahrmm.

~~~
reustle
Side note, 10gen has been renamed to MongoDB Inc

~~~
goldenkey
Given the flak MongoDB has been getting lately, that might have been a bad
decision.

------
wbeckler
Can't you scale realtime queries by just timestamping everything in the
database so you know if something comes back in your query where timestamp >
oldtime, then it's new?

------
EGreg
Looks like Meteor is becoming more like the Q platform.

[http://qbixstaging.com/QP](http://qbixstaging.com/QP)

[http://github.com/EGreg/Q](http://github.com/EGreg/Q)

(Not yet released though :)

~~~
lewispb
No.

~~~
EGreg
Why

~~~
goldenkey
Pretty cool framework Greg. Disregard comment above, they lost every key on
their keyboard except for two.

