Compact Panic: Won't run compaction - versions 0.13.0 & 0.14.0

[0xVox]

Thanos: 0.14.0, 0.13.0
Prometheus: 2.19.0
Go: Official images from quay.io

Object Storage Provider:
S3

What happened:
Since upgrading our stack to 0.14.0 from 0.12.2, I found that Thanos compact would sit in this crash loop and not do anything.
I rolled compact back to 0.13.0 to see if it was present, and the exact error below still occurred.

In order to get life out of compact, I've started using it at version 0.12.2 again.

How to reproduce it (as minimally and precisely as possible):
I notice it fails in this function ReplicaLabelRemover - The replica label in my blocks has changed at a couple of points - I wonder if this could play into it?

Full logs to relevant components:

Logs

level=debug ts=2020-07-23T15:55:53.498871778Z caller=main.go:102 msg="maxprocs: Updating GOMAXPROCS=[1]: determined from CPU quota"
level=info ts=2020-07-23T15:55:53.500228374Z caller=main.go:151 msg="Tracing will be disabled"
level=info ts=2020-07-23T15:55:53.500349269Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2020-07-23T15:55:53.505913921Z caller=compact.go:420 msg="starting compact node"
level=info ts=2020-07-23T15:55:53.505955803Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2020-07-23T15:55:53.507187007Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2020-07-23T15:55:53.507211189Z caller=http.go:56 service=http/server component=compact msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2020-07-23T15:55:53.509149805Z caller=compact.go:943 msg="start sync of metas"
level=debug ts=2020-07-23T15:56:00.109247068Z caller=fetcher.go:739 msg="block is too fresh for now" block=01EDY4QA63AYY7AA4P54DWQMFJ
level=debug ts=2020-07-23T15:56:00.109665608Z caller=fetcher.go:739 msg="block is too fresh for now" block=01EDY4QA6BK4J3NMKXHCKNBKY3
level=info ts=2020-07-23T15:56:00.110715665Z caller=fetcher.go:452 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.592143536s cached=4120 returned=4120 partial=0
level=info ts=2020-07-23T15:56:02.876532265Z caller=fetcher.go:452 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.721299669s cached=4120 returned=4120 partial=0
panic: runtime error: index out of range [0] with length 0

goroutine 155 [running]:
github.com/thanos-io/thanos/pkg/block.(*ReplicaLabelRemover).Modify(0xc00085cde0, 0x266a460, 0xc0002e7240, 0xc001297080, 0xc00084bf60, 0x0, 0x0)
        /go/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:695 +0x656
github.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch(0xc000103f80, 0x266a460, 0xc0002e7240, 0xc0002e6f40, 0xc0002e6f00, 0x4, 0x4, 0xc0005307f0, 0x1, 0x1, ...)
        /go/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:440 +0x626
github.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch(0xc00064c3c0, 0x266a460, 0xc0002e7240, 0xc, 0x0, 0x0, 0x2)
        /go/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:473 +0x9f
github.com/thanos-io/thanos/pkg/compact.(*Syncer).SyncMetas(0xc0007efc80, 0x266a460, 0xc0002e7240, 0x0, 0x0)
        /go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:126 +0x9e
github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact(0xc00064cde0, 0x266a460, 0xc0002e7240, 0x0, 0x0)
        /go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:944 +0x2c2
main.runCompact.func5(0xc0002e6b40, 0x6)
        /go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:294 +0x15a
main.runCompact.func6.1(0x45d964b800, 0xc00021cdc0)
        /go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:348 +0x99
github.com/thanos-io/thanos/pkg/runutil.Repeat(0x45d964b800, 0xc000117ec0, 0xc0003fdee0, 0x0, 0x0)
        /go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:72 +0x8e
main.runCompact.func6(0x0, 0x0)
        /go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:347 +0x29e
github.com/oklog/run.(*Group).Run.func1(0xc00064dda0, 0xc0007efd00, 0xc0008660f0)
        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb

[bwplotka]

Ack, thanks for reporting. It should be straightforward to fix thanks to your stack trace 👍

[0xVox]

Thanks for your timely response!

Should be straightforward to fix

Always good to hear!

[bwplotka]

Fixed on master (:

[bwplotka]

Thanks to @yeya24 ❤️

[0xVox]

So quick! Thank you guys!

[0xVox]

Wicked work guys - running smoothly again :)