Failed TestStoreGateway_BlocksSyncWithDefaultSharding_RingTopologyChangedAfterScaleUp

[pracucci]

I've just seen this test failing (CI). We should investigate it:

--- FAIL: TestStoreGateway_BlocksSyncWithDefaultSharding_RingTopologyChangedAfterScaleUp (17.94s)
    gateway_test.go:406: random generator seed: 1635936443186330968
    gateway_test.go:496: expected 40, got 16
FAIL

[bboreham]

Another one https://github.com/cortexproject/cortex/runs/4103618650

--- FAIL: TestStoreGateway_BlocksSyncWithDefaultSharding_RingTopologyChangedAfterScaleUp (18.75s)
    gateway_test.go:406: random generator seed: 1636021349283089808
    gateway_test.go:496: expected 40, got 16
FAIL

[pedro-stanaka]

Had the same running on my local:

?       github.com/cortexproject/cortex/pkg/storage/tsdb/testutil       [no test files]
--- FAIL: TestStoreGateway_BlocksSyncWithDefaultSharding_RingTopologyChangedAfterScaleUp (10.34s)
    gateway_test.go:406: random generator seed: 1638348556975475935
    gateway_test.go:496: expected 40, got 0
FAIL
FAIL    github.com/cortexproject/cortex/pkg/storegateway        170.002s

[bboreham]

I believe this is caused by the issue identified in grafana/dskit#108 - the test is polling for metrics to update, but they won't update until the 10-second timer. Since it only waits 5 seconds this may not work.

cortex/pkg/storegateway/gateway_test.go

Line 496 in dd698bc

test.Poll(t, 5*time.Second, float64(numAllGateways*numInitialGateways), func() interface{} {

So when we update dskit with the change in grafana/dskit#107 it should be ok.
Update is proposed in #4601.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.