[exporterhelper] fix deadlock when initializing persistent queue

[swiatekm]

Description:
Fixing a potential deadlock in persistent queue initialization.

The queue maintains a channel of dummy items whose size should be equal to the queue size at all times. This makes it easier for consumers to wait when the queue is empty. During initialization, we simply add a dummy item to the channel for each queue item.

However, we also attempt to requeue items which were previously dispatched, but not sent, before this step, which actually does add dummy items to the channel as well. As a result, we could have more dummy items in the channel than there are actual items in the storage. This is normally not a big problem, as the queue will quietly discard the extraneous dummy items when it's empty.

It is, however, a problem, if this causes the dummy item channel to go over capacity during initialization. If number_of_dispatched_items + queue_size > queue_capacity, then we try to put more dummy items in the channel than we have capacity, resulting in a deadlock.

The reason this problem doesn't appear in practice very often, is that if the queue is full, the dispatched items are simply discarded. I have seen it happen in combination with other storage-related problems, where it muddies the waters and makes troubleshooting more difficult.

Testing: Added a test that deadlocks without the change.

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.01 ⚠️

Comparison is base (2399471) 91.05% compared to head (209d3c9) 91.04%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7400      +/-   ##
==========================================
- Coverage   91.05%   91.04%   -0.01%     
==========================================
  Files         292      292              
  Lines       14234    14233       -1     
==========================================
- Hits        12961    12959       -2     
- Misses       1006     1007       +1     
  Partials      267      267              

Impacted Files	Coverage Δ
...rter/exporterhelper/internal/persistent_storage.go	87.86% <100.00%> (-0.47%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.