

Scaling Meteor: The Challenges of Real-time Apps - sgdesign
https://www.discovermeteor.com/blog/scaling-meteor-the-challenges-of-realtime-apps/

======
davedx
Thanks for the article -- always good to know more about how Meteor works.

> Attempting to minimize the number of unique-per-user subscriptions will
> allow Meteor to pool work across users as much as possible.

This kind of thing is why, no matter how much I like Meteor personally (I'm
using it for my two side projects), I disagree with the marketing that Meteor
is for people learning to build web applications. Despite its promises of
simplicity, there are many gotchas and difficult to understand subtleties in
Meteor. It is definitely _not_ an "ultra-simple environment" once you get past
the most basic apps.

What I'd really like to see at this point is some smart packages built on top
of Meteor that make building basic CRUD applications simple again. Something
like scaffolding from Rails. One thing I love about Meteor's accounts-ui is
the "Configure OAuth" in-app wizards -- something this would be fantastic for
scaffolding.

Because let's face it, how much of your average app is exciting realtime stuff
and how much is writing boring forms, validation and CRUD methods to handle
everything?

I'd love to try and build something like this. Now if I could just quit my day
job or steal some time from my other projects.

~~~
akbar501
> What I'd really like to see at this point is some smart packages built on
> top of Meteor that make building basic CRUD applications simple again.

We built what you've described with Exponential.io.

Essentially, we take a spec file and generate an Angular client, an Express
API and a Mongo backend with a focus on CRUD applications.

We're doing a beta / soft launch in 2 weeks on Feb. 28.

In terms of platforms, we're releasing a solution to rapidly build an Angular
+ Node/Express API + Mongo (aka MEAN) on the 28th.

We'll follow-up with a solution for Meteor in short order. Interestingly, we
built the Meteor solution first but are releasing the MEAN solution first as
Meteor 1.0 is close, but not ready yet.

~~~
davedx
Excellent, would love to give your soft launch a try! Building things from
spec files is exactly what I had in mind - I actually did a very short POC for
this in Express too:

[https://github.com/davedx/grindstone](https://github.com/davedx/grindstone)

I think declarative approaches should almost always supercede imperative when
the solutions are so well known. It also makes it easy to build GUI's instead
of having to bang out text files all the time.

Could you email me when you're ready please? davedx@gmail.com

------
mcrider
This is the first I've heard of Meteor APM, looks awesome!

This line confused me:

> Using simpler queries will usually mean there’s less work for Meteor to do
> to decide if a change affects it.

Is this right? Previous Meteor optimization articles I've read have stated the
opposite, and all my years of database development have told me to avoid
'SELECT *' and other simple DB calls like the plague. But maybe I missed
something in the article that explains this.

Thanks Tom and Sacha for your great contributions to the Meteor community!

~~~
tmeasday
Fair question. Obviously that recommendation was supposed to be taken
alongside the other recommendations about not touching too many records, so,
yeah 'SELECT *' is probably not good.

What I meant here is thinking about making your publication queries simpler
when designing your data models.

For instance, sometimes using denormalization you can use a simple equality
selector rather than some more complex mongo selector. Mongo makes it possible
to avoid the denormalization, but it might bite you as it means Meteor can't
do a bunch of short-circuits that it might otherwise do on a simpler query.

------
fiatmoney
With this kind of architecture, do you risk reaching a feedback loop where too
many queries overload the DB and slow it down (especially given how MongoDB
delegates caching to the OS), causing more messages to accumulate, increasing
pressure on both the application server & the DB, leading to a full-on thrashy
death spiral? Or is this mitigated somehow? Basically, what's the failure mode
when usage spikes more than anticipated?

~~~
tmeasday
It's definitely a concern, but I think it's mitigated somewhat by the oplog-
tailing approach. Writes go straight into Mongo, as usual, it's just reads
from the oplog that potentially tie up resources on the webserver.

Those reads are throttled, so _I think_ the net result of too many writes is
just a choked up webserver, and slow "realtime". Potentially a future
architecture moves the oplog processor onto it's own server and the effect is
isolated to the realtime part.

~~~
panarky
So with oplog tailing, I can scale by adding multiple Meteor processes behind
a reverse proxy with sticky sessions.

It seems that since oplog tailing doesn't work with a sharded database, then
each additional Meteor process must also tail the entire oplog and evaluate
all transactions for all Meteor processes.

How does this work for a site with, say, 100 transactions per second and a
dozen subscriptions to observe? Does it saturate the Node event loop, or are
there plenty of ticks left for the application?

And if I use 'skip' and 'limit' to page through large result sets, it looks
like oplog tailing isn't an option. Is there a better way to keep the client
data small and take advantage of oplog tailing?

~~~
imslavko
You can also paginate by `$gt` and `$lt` operators. For example, if you have a
list of posts, they are probably displayed in some order on the screen. Most
likely they are ordered by date. You can maintain the pagination by presenting
all posts older than X but newer than Y.

------
imslavko
What a great article! I especially enjoyed the pictures :)

For those of you who are interested in more in-depth oplog tailing info, there
is a recorded talk from one of the Meteor Devshops:
[https://www.youtube.com/watch?v=_dzX_LEbZyI](https://www.youtube.com/watch?v=_dzX_LEbZyI)

------
rywalker
Wow — very in depth article by Tom Coleman. As we get more growing Meteor apps
out in the wild, scaling become a larger concern. Good to see Meteor
Development Group focusing so much attention on it, and great tools from the
community too.

------
johnrob
At a high level this reads like a description of a real time media server
(e.g. for streaming video). I suppose if you swap data for pixels, and avoid
lossy compression, it's essentially the same problem.

------
joshowens
Wow, Tom really did nail it! You guys really went all out to explain the ways
things did work and how they work now with OpLog.

You guys need to keep these posts coming!

