Batch size and Serializing big indexes on modification

[h0lg]

I've run into an issue with serializing larger index files when modifications occur as described here. If I use this pattern while committing batches of changes to an index in a loop and serialize it into a large enough index file, it will occur that the file is still being written to by the serialization triggered by the previous index modification, resulting in an IOException because the file is in use.

I am absolutely able to work around that by using e.g. a SemaphoreSlim(1, 1) to coordinate access to the index file - but don't know whether waiting for each snapshot to be saved unnecessarily slows down processing. Either way, it raises a few questions:

1. How do I decide on a good index size? When is it too big and I should think about partitioning my index file?
2. How do I choose a good batch size before committing? What does that depend on? The Item size? The index size?
3. Could I safely debounce the index serialization on modification? I.e. could I safely serialize only the latest snapshot - will it contain the changes contained in the previous snapshots?

[mikegoatly]

That's surprising - I would have expected that the LIFTI indexing logic would wait for the index modification actions to complete before returning control back to your code to continue indexing the next batch.

It could be a bug I guess, but are you missing an await somewhere, e.g. when calling CommitBatchChangeAsync or AddRangeAsync?

[h0lg]

I think I found the issue. Unlike the example I was reusing the same instance of FullTextIndexBuilder and attached different modification actions (specific to each index) using WithIndexModificationAction to it before calling Build on it to build an index. That resulted in all index modifications being written to the file of last built index.

[mikegoatly]

Thanks for following up - yeah, a builder should only really be used once. It might be more helpful if the an exception was thrown if it's done multiple times - I'm not sure if that would cause any problems elsewhere though. I'll create an issue for it for investigation later.