Implement cursor population in batches, only when batchSize is passed

[biomorgoth]

Summary
This adds a new feature that allows to optimize QueryCursor usage when populate is passed to the query. It requires a batchSize to be defined on the query, otherwise it will work as usual.

Motivation
Actually population on cursors iterates one doc at a time, running N population queries per doc which is not the best scenario for large datasets. By using a custom batchSize, the Mongo driver is already getting a batch of documents into memory, which can be safely be passed into Mongoose QueryCursor without triggering a new getMore command to the server. Then those documents can be populated together, reducing the amount of extra queries to the DB.

Examples

const cursor = Model.find().populate('someRef anotherRef').batchSize(5).cursor()

// The first .next() called on the iterator will populate the 5 elements that already were on memory
// but still passing one at a time to the iterator.
// Later on the 6th, 11th, 15th .next() call, Mongo driver will issue a getMore command to the server
// and the received docs will be populated per batch again, until cursor is exhausted.
for await(const doc of cursor) {
  console.log(doc);
}

[biomorgoth]

I'm willing to hear comments on this feature, i noted the current cursor population queries all populate fields per document which is very slow when multiple populate fields are requested and the main query total size is large.

I turned on logging for the Mongo driver while testing the new feature and effectively the amount of queries lowers drastically when specifying batchSize with this approach. Populate queries are requested at the same rate as getMore commands.

[vkarpov15]

Overall I think this PR is great! Made a couple comments.

Could we put this in the 5.11 release? This change might be a bit too big for a patch release.

[vkarpov15]

I'd prefer we not use bind(), especially with a custom object as this.

[biomorgoth]

Yeah, that was a quick approach to make ESLint happy (you can see the previous commit). Take in mind callback needs to be preserved through the internal cursor iteration. There probably could be a cleaner way to succeed in this. Any suggestions?

I also did not wanted to expose onNext, onPopulate and nextDoc in the QueryCursor API, as they are purely internal methods. That forces me to pass somehow the ctx object in any way. And thus i ended with the bind() approach.

[vkarpov15]

Is there a reason to have both .batchDocs and ._batchDocs? Model.populate(docs) modifies docs in place so they point to the same object.

[biomorgoth]

Note this.batchDocs belongs to the custom object initially bound to _onNext, while this.ctx._batchDocs belongs to the QueryCursor instance. So this assignation is basically passing the Array from one object to another.

Sorry i did not researched that deep to know that Model.populate(docs) modified the Array in place, so maybe i could even reset ctx._batchDocs just before line 312 and always keep one Array. Let me see if that works as expected.

[biomorgoth]

No problem with adding this to the 5.11 release. When is it expected to be released?

[biomorgoth]

Done with the single batchDocs array @vkarpov15 . Still not sure how to remove the .bind() and at the same time pass the necessary data to _onNext/_populateBatch (specially the callback function).

My first version ( 3ec17eb ) did not use .bind(), but inner functions that relied on the closure to have access to ctx/callback. ESLint rules for the project did not liked that style ☹️

[vkarpov15]

We're planning on releasing 5.11 in October or November 2020. Does that work for you or do you need this feature out sooner?

[biomorgoth]

Great! I can manage to use my fork as temporary dependency for my projects right now until the 5.11 release. The current expectative is to not use the fork dependency as a long term solution.