Meteor's subscription and sync are slow

Question

I have a collection with 10M documents of 6000 stocks, stock name is indexed. When I subscribe to a new stock, meteor hangs more than 10 seconds to get about 3000 documents of this stock. Also after several stocks are subscribed, meteor hangs with 100% cpu usage. Meteor looks really slow with syncing "big" collection. Actually my app just read only. I am wondering if there is way to speed up meteor for read-only client? I am also wondering if creating a separate collection for each stock helps?

Answer 1

Meteor is pushing the entire dataset to your client.

You can turn off autopublish by removing the autopublish package:

meteor remove autopublish

Then create specific a specific subscription for your client.

When you subscribe you can pass a session variable as an argument, so on the client you do something like:

sub = new Meteor.autosubscribe(function(){ Meteor.subscribe('channelname', getSession('filterval')); }

On the server you use the argument to filter the result set sent to the client, so that you are not piping everything all at once. You segment the data in some fashion using a filter.

Meteor.publish('channelname', function(filter){ return Collection.find({field: filter}); }

Now, whenever you change the filterval on the client using setSession('filterval', 'newvalue'); the subscription will be automatically changed, and the new dataset will sent to the client.

You can use this as a means of controlling how much and what data is sent to the client.

As another poster said, you really have to ask if this is the best tool for this job. Meteor is meant for relatively small datasets that are updated in real-time in (potentially) two directions. It is heavily optimised and has a ton of scaffolding for that use case.

For another use case (such as the read-only huge dataset) it may not make sense. It has a lot of overhead that provides functionality that you are not going to use, and you'll be coding to get the functionality that you need.