Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TSDB shipper + WAL #6049

Merged
merged 89 commits into from
May 5, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
7231181
begins speccing out TSDB Head
owen-d Apr 8, 2022
5e20970
auto incrementing series ref + mempostings
owen-d Apr 8, 2022
a04a2fd
mintime/maxtime methods
owen-d Apr 8, 2022
0b67b76
tsdb head IndexReader impl
owen-d Apr 9, 2022
9019d10
head correctly populates ref lookup
owen-d Apr 11, 2022
35402a0
tsdb head tests
owen-d Apr 11, 2022
e69729e
adds prometheus license to tsdb head
owen-d Apr 11, 2022
8de2ee6
Merge remote-tracking branch 'upstream/main' into tsdb/head
owen-d Apr 11, 2022
968e617
linting
owen-d Apr 11, 2022
e051eee
[WIP] speccing out tsdb head wal
owen-d Apr 12, 2022
7f207c7
fix length check and adds tsdb wal encoding tests
owen-d Apr 12, 2022
3da9db8
exposes wal structs & removes closed semantics
owen-d Apr 12, 2022
734a106
logs start time in the tsdb wal
owen-d Apr 12, 2022
dbb3067
wal interface + testing
owen-d Apr 12, 2022
cab1020
exports walrecord + returns ref when appending
owen-d Apr 13, 2022
60bfe0b
specs out head manager
owen-d Apr 13, 2022
5a07e53
tsdb head manager wal initialization
owen-d Apr 14, 2022
d83b86e
tsdb wal rotation
owen-d Apr 14, 2022
c629a2b
wals dont use node name, but tsdb files do
owen-d Apr 14, 2022
c115627
cleans up fn signature
owen-d Apr 14, 2022
4eaa1f3
multi tsdb idx now just wraps Index interfaces
owen-d Apr 15, 2022
9dc7234
no longer sorts indices when creating multi-idx
owen-d Apr 15, 2022
bf4b125
tenantHeads & HeadManger index impls
owen-d Apr 15, 2022
5d9e66c
head mgr tests
owen-d Apr 15, 2022
3ccd7ee
bugfixes & head manager tests
owen-d Apr 15, 2022
cd39afb
tsdb dir selection now helper fns
owen-d Apr 15, 2022
b556d90
period utility
owen-d Apr 15, 2022
ce6a27c
pulls out more code to helpers, fixes some var races
owen-d Apr 15, 2022
2fd813e
head recovery is more generic
owen-d Apr 15, 2022
4750079
tsdb manager builds from wals
owen-d Apr 15, 2022
c5eda36
pulls more helpers out of headmanager
owen-d Apr 16, 2022
c6b0747
lockedIdx, Close() on idx, tsdbManager update
owen-d Apr 16, 2022
f6859d1
Merge remote-tracking branch 'upstream/main' into tsdb/head-wal
owen-d Apr 18, 2022
3ece142
removes mmap from index reader implementation
owen-d Apr 18, 2022
8b4c2b4
tsdb file
owen-d Apr 18, 2022
ee6eda9
adds tsdb shipper config and refactors initStore
owen-d Apr 18, 2022
b844d85
removes unused tsdbManager code
owen-d Apr 18, 2022
0be6483
Merge remote-tracking branch 'upstream/main' into tsdb/head-wal
owen-d Apr 19, 2022
eb31b32
implements stores.Index and stores.ChunkWriter for tsdb
owen-d Apr 19, 2022
32218b9
chunk.Data now supports an Entries() method
owen-d Apr 19, 2022
7d20b5e
moves walreader to new util/wal pkg to avoid circular dep + tsdb stor…
owen-d Apr 19, 2022
f6f557e
tsdb store
owen-d Apr 19, 2022
3fbf316
passes indexWriter to chunkWriter
owen-d Apr 21, 2022
5ad1cf5
build a tsdb per index bucket in according with shipper conventions
owen-d Apr 21, 2022
52c4ea3
dont open tsdb files until necessary for indexshipper
owen-d Apr 21, 2022
e28b941
tsdbManager Index impl
owen-d Apr 21, 2022
15667a7
tsdb defaults + initStore fix for invalid looping
owen-d Apr 22, 2022
4775798
fixes UsingTSDB helper
owen-d Apr 22, 2022
7a51eda
disables deleteRequestStore when using TSDB
owen-d Apr 22, 2022
5fa3b4a
pass limits to tsdb store
owen-d Apr 22, 2022
c88d742
always start headmanager for tsdb
owen-d Apr 25, 2022
ed60710
fixes copy bug
owen-d Apr 25, 2022
cacbea8
more logging
owen-d Apr 25, 2022
afecd21
fixes duplicate tenant label bug
owen-d Apr 25, 2022
1f84ce0
debug logs, uses label builder, removes __name__=logs for tsdb
owen-d Apr 26, 2022
3b734cf
tsdb fixes labels at earlier pt
owen-d Apr 26, 2022
b88df5c
account for setting tenant label in head manager test
owen-d Apr 27, 2022
cdb9cfd
changing tsdb dir names
owen-d Apr 27, 2022
2c2dbb7
identifier interface, builder to tsdb pkg
owen-d Apr 27, 2022
9024210
tsdb version path prefix
owen-d Apr 27, 2022
40cc48c
fixes buildfromwals identifier
owen-d Apr 27, 2022
8d2d033
fixes tsdb shipper paths
owen-d Apr 28, 2022
08500ed
split buckets once per user set
owen-d Apr 28, 2022
f84af6a
refactors combining single and multi tenant tsdb indices on shipper r…
owen-d Apr 28, 2022
aa46336
indexshipper ignores old gzip logic
owen-d Apr 28, 2022
f388e24
method name refactor
owen-d Apr 28, 2022
4690dff
remove unused record type
owen-d May 3, 2022
a31c203
removes v1 prefix in tsdb paths and refactores indices method
owen-d May 3, 2022
8e6956f
ignores double optimization in tsdb looking for multitenant idx, ship…
owen-d May 3, 2022
e928519
removes 5-ln requirement on shipper tablename regexp
owen-d May 3, 2022
bc0fe93
groups identifiers, begins removing multitenant prefix in shipped files
owen-d May 3, 2022
9fc3385
passses open fn to indexshipper
owen-d May 3, 2022
3ce9110
exposes RealByteSlice
owen-d May 3, 2022
725abf8
TSDBFile no longer needs a file descriptor, parses gzip extensions
owen-d May 3, 2022
8da79b8
method signature fixing
owen-d May 3, 2022
bce8c52
stop masquerading as compressed indices post-download in indexshipper
owen-d May 3, 2022
e229160
variable bucket regexp
owen-d May 3, 2022
0b2125e
removes accidental configs committed
owen-d May 4, 2022
d1ff6f7
Merge remote-tracking branch 'upstream/main' into tsdb/head-wal
owen-d May 4, 2022
cc34584
label matcher handling for multitenancy and metricname in tsdb
owen-d May 4, 2022
a850e42
explicitly require fingerprint when creating tsdb index
owen-d May 4, 2022
14dcb1d
only add tenant label when creating multitenant tsdb
owen-d May 4, 2022
a853035
linting + unused removal
owen-d May 4, 2022
4871228
more linting :(
owen-d May 4, 2022
0a0aed8
goimports
owen-d May 4, 2022
74ccc09
removes uploadername from indexshipper
owen-d May 4, 2022
65aac18
maxuint32 for arm32 builds
owen-d May 4, 2022
eafaf1a
tsdb chunk filterer support
owen-d May 4, 2022
c6e7cf6
always set ingester name when using object storage index
sandeepsukhani May 5, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
linting + unused removal
  • Loading branch information
owen-d committed May 4, 2022
commit a853035941fcf88a9fe4d70f5dbb073e5acd9596
7 changes: 4 additions & 3 deletions pkg/storage/stores/tsdb/chunkwriter.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,16 @@ import (
"context"

"github.com/go-kit/log/level"
"github.com/pkg/errors"
"github.com/prometheus/common/model"
"github.com/prometheus/prometheus/model/labels"

"github.com/grafana/loki/pkg/storage/chunk"
"github.com/grafana/loki/pkg/storage/chunk/fetcher"
"github.com/grafana/loki/pkg/storage/config"
"github.com/grafana/loki/pkg/storage/stores/series"
"github.com/grafana/loki/pkg/storage/stores/tsdb/index"
"github.com/grafana/loki/pkg/util/spanlogger"
"github.com/pkg/errors"
"github.com/prometheus/common/model"
"github.com/prometheus/prometheus/model/labels"
)

type IndexWriter interface {
Expand Down
2 changes: 1 addition & 1 deletion pkg/storage/stores/tsdb/head_wal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"testing"
"time"

"github.com/go-kit/kit/log"
"github.com/go-kit/log"
"github.com/grafana/loki/pkg/storage/stores/tsdb/index"
"github.com/prometheus/prometheus/tsdb/chunks"
"github.com/prometheus/prometheus/tsdb/record"
Expand Down
3 changes: 2 additions & 1 deletion pkg/storage/stores/tsdb/identifier.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@ import (
"strings"
"time"

"github.com/grafana/loki/pkg/storage/stores/tsdb/index"
"github.com/prometheus/common/model"

"github.com/grafana/loki/pkg/storage/stores/tsdb/index"
)

// Identifier can resolve an index to a name (in object storage)
Expand Down
43 changes: 0 additions & 43 deletions pkg/storage/stores/tsdb/index.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package tsdb

import (
"context"
"sync"

"github.com/prometheus/common/model"
"github.com/prometheus/prometheus/model/labels"
Expand Down Expand Up @@ -69,45 +68,3 @@ func (NoopIndex) LabelNames(ctx context.Context, userID string, from, through mo
func (NoopIndex) LabelValues(ctx context.Context, userID string, from, through model.Time, name string, matchers ...*labels.Matcher) ([]string, error) {
return nil, nil
}

// LockedIndex wraps an index+RWMutex and only calls index methods under rlock
type LockedIndex struct {
mtx *sync.RWMutex
idx Index
}

func NewLockedMutex(mtx *sync.RWMutex, idx Index) *LockedIndex {
return &LockedIndex{
mtx: mtx,
idx: idx,
}
}

func (i *LockedIndex) Close() error { return i.Close() }
func (i *LockedIndex) Bounds() (from, through model.Time) {
i.mtx.RLock()
defer i.mtx.RUnlock()
return i.idx.Bounds()
}
func (i *LockedIndex) GetChunkRefs(ctx context.Context, userID string, from, through model.Time, res []ChunkRef, shard *index.ShardAnnotation, matchers ...*labels.Matcher) ([]ChunkRef, error) {
i.mtx.RLock()
defer i.mtx.RUnlock()
return i.idx.GetChunkRefs(ctx, userID, from, through, res, shard, matchers...)
}

func (i *LockedIndex) Series(ctx context.Context, userID string, from, through model.Time, res []Series, shard *index.ShardAnnotation, matchers ...*labels.Matcher) ([]Series, error) {
i.mtx.RLock()
defer i.mtx.RUnlock()
return i.idx.Series(ctx, userID, from, through, res, shard, matchers...)
}
func (i *LockedIndex) LabelNames(ctx context.Context, userID string, from, through model.Time, matchers ...*labels.Matcher) ([]string, error) {
i.mtx.RLock()
defer i.mtx.RUnlock()
return i.idx.LabelNames(ctx, userID, from, through, matchers...)
}
func (i *LockedIndex) LabelValues(ctx context.Context, userID string, from, through model.Time, name string, matchers ...*labels.Matcher) ([]string, error) {
i.mtx.RLock()
defer i.mtx.RUnlock()
return i.idx.LabelValues(ctx, userID, from, through, name, matchers...)

}
5 changes: 3 additions & 2 deletions pkg/storage/stores/tsdb/index_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,14 @@ package tsdb
import (
"context"

"github.com/prometheus/common/model"
"github.com/prometheus/prometheus/model/labels"

"github.com/grafana/loki/pkg/logproto"
"github.com/grafana/loki/pkg/querier/astmapper"
"github.com/grafana/loki/pkg/storage/chunk"
"github.com/grafana/loki/pkg/storage/config"
"github.com/grafana/loki/pkg/storage/stores/tsdb/index"
"github.com/prometheus/common/model"
"github.com/prometheus/prometheus/model/labels"
)

// implements stores.Index
Expand Down
6 changes: 4 additions & 2 deletions tools/tsdb/tsdb-map/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ import (
"github.com/grafana/loki/pkg/storage/config"
"github.com/grafana/loki/pkg/storage/stores/shipper/compactor/retention"
shipper_util "github.com/grafana/loki/pkg/storage/stores/shipper/util"
"github.com/grafana/loki/pkg/storage/stores/tsdb"
"github.com/grafana/loki/pkg/storage/stores/tsdb/index"
"github.com/prometheus/common/model"
)

var (
Expand Down Expand Up @@ -64,7 +66,7 @@ func main() {
panic(err)
}

builder := index.NewBuilder()
builder := tsdb.NewBuilder()

log.Println("Loading index into memory")

Expand All @@ -80,7 +82,7 @@ func main() {
return it.Err()
}
entry := it.Entry()
builder.AddSeries(entry.Labels, []index.ChunkMeta{{
builder.AddSeries(entry.Labels, model.Fingerprint(entry.Labels.Hash()), []index.ChunkMeta{{
Checksum: extractChecksumFromChunkID(entry.ChunkID),
MinTime: int64(entry.From),
MaxTime: int64(entry.Through),
Expand Down