module loader: Avoid deadlock

[radeksimko]

This fixes a number of bugs I found while testing this both manually in VS Code and by running tests with high enough -count.

1. Deadlock could occur when retry of dispatch is attempted, because while run() is executing, another goroutine (e.g. one executing a different operation) could have finished and triggered another tryDispatchingModuleOp which then filled the channel and the next tryDispatchingModuleOp triggered from run() as a result of retry would block, therefore blocking the main loop.
   • This was resolved by running retry tryDispatchingModuleOp in an additional goroutine
2. tryDispatchingModuleOp can get triggered by many threads at the same time and its original implementation left an opportunity for race condition due to queue.Peek() relying on queue.Len() but the reliance was not guaranteed in any way. Additionally Peek could have "peeked" at the exact same operation more than once, which then resulted in that duplicate operation being dispatched.
   • This was resolved by getting rid of Peek method entirely and ensuring we never pop from an empty queue by utilizing the internal mutex. Retries should be rare enough that (especially with the 100ms delay) shouldn't cause concern wrt resource consumption by repeated push/pop.
3. tryDispatchingModuleOp was triggered from inside of executeModuleOp, but loading counters were being decremented via defer after, which could've caused the dispatch attempt to fail when the queue was full, just because the two operations were done in the wrong order.
   • This was resolved by getting rid of defer and calling tryDispatchingModuleOp from the main loop in run() where we can better control the order.

[aeschright]

👍

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the context necessary to investigate further.