Additional improvements to the Batching algorithm

[AlexGhiondea]

No description provided.

[Mohit-Chakraborty]

When submitting documents for indexing, we split batches on unique key of the documents, so that any exception from the service side can be correctly mapped and our document submission retry mechanism is stable. When designing a solution, we should aim to -

1. Try to fill the entire batch
2. Maintain the order of document submission

For .NET, we modified the batching algorithm via Azure/azure-sdk-for-net#18469
This helps with 2 above (maintain the order of document submission), but we can do better regarding 1 (try to fill the batch to the maximum extent).

I tried another round of improvement with Azure/azure-sdk-for-net#18603, but there were concerns about semantic change to the operation.

The change is adding a “flush duplicate actions immediately after the batch, regardless of size” behavior that might result in us sending a lot of extra batches. Ideally we’d just update the existing pending/retry queues and keep the rest of the logic the same (since it’s already so much to wrap your head around). That’s probably a nontrivial ask with .NET’s Queue though. I think we might need to switch from Queue to something else.

The remaining work is to modify the batching algorithm further, so that the above concerns are alleviated.

For other languages, we need to evaluate where we sit with regards to their implementation and finish the rest of the work.