Loki: Fix bug where items are returned to a sync.Pool incorrectly

[slim-bean]

#4160 correctly fixed an issue to make sure all iterators are properly closed when nonOverlappingSampleIterator's Close() method is called.

However, there was an unexpected side effect with this change.

It introduced occurences of the iterators close method being called more than once.

Loki has a mix of iterators that will error if closed more than once and others that need to be idempotent, there are some situations where an iterator wrapping another iterator can't always know if the underlying iterator(s) have been closed already so it does result in Close() being called twice.

We discovered a case where the SampleIteratorWithClose was not properly idempotent resulting in a somewhat obscure and difficult to find problem as Cyril described in #4512

The memchunk.go and unordered.go's use of the close function both return elements to a sync.Pool

Our understanding here is that the first call would return the items to the pool correctly, however when any subsequent calls to the iterators close were made, the iterator still holds references to the same items already returned to the pool. These same items could possibly have been lended out by the pool again between calls to the close funtion. This results in items actively in use being put back into the pool where they could be lended out again. Making it possible for multiple separate iterators to all have references to the same items and memory from the pool.

a sync.Once{} has been added to the iterator to make sure the close function is idempotent.

Fixes #4512

[dannykopping]

Can't comment too much on the code but I think we should add a test for the scenario that we're trying to fix.

[slim-bean]

yeah i pushed this a bit early, there is a better way to apply the sync.Once which is testable

[cyriltovena]

you could keep the multi error as an error that would simplify the code.

[cyriltovena]

LGTM