Relay client should limit max concurrency

[cody-littley]

Why are these changes needed?

In preprod, we observe relays rate limiting concurrent GetChunks() requests. This is non-ideal in an environment where the relays are not under intentional attack or high load.

The following screenshot is a histogram of the logs reporting concurrency throttling.

This PR introduces two primary changes:

• increases the maximum permissible concurrency per relay client from 1 to 2
• adds logic to relay clients that causes them to avoid exceeding maximum permissible concurrency

[ian-shim]

can we increase this further like 8 or 16? (obviously this is just default value and real value will be configured in the cluster, but wondering what the most accommodating value should be for clients calling relay)

[cody-littley]

increased to 8

[ian-shim]

Why don't we return the error immediately (in case of rate limits at the relay, it will return ResourceExhausted error) instead of blocking and waiting? This would allow the user of this client to handle the error however it wants

[cody-littley]

Currently we don't have retry logic when fetching from a relay. The idea here was that it was better to slightly delay a request to a relay than it was to fail to fetch data (and potentially not sign a batch).

[ian-shim]

I think this choice should be made not inside the client but by the users of the client, and client being a thin wrapper for interacting with the relay.
Should we add this functionality at the node side if we really do want this kind of limiting logic from the client side?
I'm fine with not putting in this limiting mechanism at all from the client side (node) in favor of potentially retrying with backoff in case of ResourceExhausted error. Because the users of this relay client don't know the rate limiting configuration set in the relay, it's highly likely the client side accounting is not going to be correct.