Code

TSDB shipper + WAL#6049

Code

Merged

sandeepsukhani merged 89 commits intografana:maingrafana/loki:mainfrom owen-d:tsdb/head-walowen-d/loki:tsdb/head-walCopy head branch name to clipboardMay 5, 2022

+2,926-297

Commits

Commits on Apr 8, 2022

• begins speccing out TSDB Head

  owen-d

  committed
  7231181

  Copy full SHA for 7231181

• auto incrementing series ref + mempostings

  owen-d

  committed
  5e20970

  Copy full SHA for 5e20970

• mintime/maxtime methods

  owen-d

  committed
  a04a2fd

  Copy full SHA for a04a2fd

Commits on Apr 9, 2022

• tsdb head IndexReader impl

  owen-d

  committed
  0b67b76

  Copy full SHA for 0b67b76

Commits on Apr 11, 2022

• head correctly populates ref lookup

  owen-d

  committed
  9019d10

  Copy full SHA for 9019d10

• tsdb head tests

  owen-d

  committed
  35402a0

  Copy full SHA for 35402a0

• adds prometheus license to tsdb head

  owen-d

  committed
  e69729e

  Copy full SHA for e69729e

• Merge remote-tracking branch 'upstream/main' into tsdb/head

  owen-d

  committed
  8de2ee6

  Copy full SHA for 8de2ee6

• linting

  owen-d

  committed
  968e617

  Copy full SHA for 968e617

Commits on Apr 12, 2022

• [WIP] speccing out tsdb head wal

  owen-d

  committed
  e051eee

  Copy full SHA for e051eee

• fix length check and adds tsdb wal encoding tests

  owen-d

  committed
  7f207c7

  Copy full SHA for 7f207c7

• exposes wal structs & removes closed semantics

  owen-d

  committed
  3da9db8

  Copy full SHA for 3da9db8

• logs start time in the tsdb wal

  owen-d

  committed
  734a106

  Copy full SHA for 734a106

• wal interface + testing

  owen-d

  committed
  dbb3067

  Copy full SHA for dbb3067

Commits on Apr 13, 2022

• exports walrecord + returns ref when appending

  owen-d

  committed
  cab1020

  Copy full SHA for cab1020

• specs out head manager

  owen-d

  committed
  60bfe0b

  Copy full SHA for 60bfe0b

Commits on Apr 14, 2022

• tsdb head manager wal initialization

  owen-d

  committed
  5a07e53

  Copy full SHA for 5a07e53

• tsdb wal rotation

  owen-d

  committed
  d83b86e

  Copy full SHA for d83b86e

• wals dont use node name, but tsdb files do

  owen-d

  committed
  c629a2b

  Copy full SHA for c629a2b

• cleans up fn signature

  owen-d

  committed
  c115627

  Copy full SHA for c115627

Commits on Apr 15, 2022

• multi tsdb idx now just wraps Index interfaces

  owen-d

  committed
  4eaa1f3

  Copy full SHA for 4eaa1f3

• no longer sorts indices when creating multi-idx

  owen-d

  committed
  9dc7234

  Copy full SHA for 9dc7234

• tenantHeads & HeadManger index impls

  owen-d

  committed
  bf4b125

  Copy full SHA for bf4b125

• head mgr tests

  owen-d

  committed
  5d9e66c

  Copy full SHA for 5d9e66c

• bugfixes & head manager tests

  owen-d

  committed
  3ccd7ee

  Copy full SHA for 3ccd7ee

• tsdb dir selection now helper fns

  owen-d

  committed
  cd39afb

  Copy full SHA for cd39afb

• period utility

  owen-d

  committed
  b556d90

  Copy full SHA for b556d90

• pulls out more code to helpers, fixes some var races

  owen-d

  committed
  ce6a27c

  Copy full SHA for ce6a27c

• head recovery is more generic

  owen-d

  committed
  2fd813e

  Copy full SHA for 2fd813e

• tsdb manager builds from wals

  owen-d

  committed
  4750079

  Copy full SHA for 4750079

Commits on Apr 16, 2022

• pulls more helpers out of headmanager

  owen-d

  committed
  c5eda36

  Copy full SHA for c5eda36

• lockedIdx, Close() on idx, tsdbManager update

  owen-d

  committed
  c6b0747

  Copy full SHA for c6b0747

Commits on Apr 18, 2022

• Merge remote-tracking branch 'upstream/main' into tsdb/head-wal

  owen-d

  committed
  f6859d1

  Copy full SHA for f6859d1

• removes mmap from index reader implementation

  owen-d

  committed
  3ece142

  Copy full SHA for 3ece142

• tsdb file

  owen-d

  committed
  8b4c2b4

  Copy full SHA for 8b4c2b4

• adds tsdb shipper config and refactors initStore

  owen-d

  committed
  ee6eda9

  Copy full SHA for ee6eda9

• removes unused tsdbManager code

  owen-d

  committed
  b844d85

  Copy full SHA for b844d85

Commits on Apr 19, 2022

• Merge remote-tracking branch 'upstream/main' into tsdb/head-wal

  owen-d

  committed
  0be6483

  Copy full SHA for 0be6483

• implements stores.Index and stores.ChunkWriter for tsdb

  owen-d

  committed
  eb31b32

  Copy full SHA for eb31b32

• chunk.Data now supports an Entries() method

  owen-d

  committed
  32218b9

  Copy full SHA for 32218b9

• moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment

  owen-d

  committed
  7d20b5e

  Copy full SHA for 7d20b5e

• tsdb store

  owen-d

  committed
  f6f557e

  Copy full SHA for f6f557e

Commits on Apr 21, 2022

• passes indexWriter to chunkWriter

  owen-d

  committed
  3fbf316

  Copy full SHA for 3fbf316

• build a tsdb per index bucket in according with shipper conventions

  owen-d

  committed
  5ad1cf5

  Copy full SHA for 5ad1cf5

• dont open tsdb files until necessary for indexshipper

  owen-d

  committed
  52c4ea3

  Copy full SHA for 52c4ea3

• tsdbManager Index impl

  owen-d

  committed
  e28b941

  Copy full SHA for e28b941

Commits on Apr 22, 2022

• tsdb defaults + initStore fix for invalid looping

  owen-d

  committed
  15667a7

  Copy full SHA for 15667a7

• fixes UsingTSDB helper

  owen-d

  committed
  4775798

  Copy full SHA for 4775798

• disables deleteRequestStore when using TSDB

  owen-d

  committed
  7a51eda

  Copy full SHA for 7a51eda

• pass limits to tsdb store

  owen-d

  committed
  5fa3b4a

  Copy full SHA for 5fa3b4a

Commits on Apr 25, 2022

• always start headmanager for tsdb

  Show description for c88d742
  owen-d

  committed
  c88d742

  Copy full SHA for c88d742

• fixes copy bug

  Show description for ed60710
  owen-d

  committed
  ed60710

  Copy full SHA for ed60710

• more logging

  Show description for cacbea8
  owen-d

  committed
  cacbea8

  Copy full SHA for cacbea8

• fixes duplicate tenant label bug

  Show description for afecd21
  owen-d

  committed
  afecd21

  Copy full SHA for afecd21

Commits on Apr 26, 2022

• debug logs, uses label builder, removes __name__=logs for tsdb

  Show description for 1f84ce0
  owen-d

  committed
  1f84ce0

  Copy full SHA for 1f84ce0

• tsdb fixes labels at earlier pt

  Show description for 3b734cf
  owen-d

  committed
  3b734cf

  Copy full SHA for 3b734cf

Commits on Apr 27, 2022

• account for setting tenant label in head manager test

  owen-d

  committed
  b88df5c

  Copy full SHA for b88df5c

• changing tsdb dir names

  owen-d

  committed
  cdb9cfd

  Copy full SHA for cdb9cfd

• identifier interface, builder to tsdb pkg

  owen-d

  committed
  2c2dbb7

  Copy full SHA for 2c2dbb7

• tsdb version path prefix

  owen-d

  committed
  9024210

  Copy full SHA for 9024210

• fixes buildfromwals identifier

  owen-d

  committed
  40cc48c

  Copy full SHA for 40cc48c

Commits on Apr 28, 2022

• fixes tsdb shipper paths

  owen-d

  committed
  8d2d033

  Copy full SHA for 8d2d033

• split buckets once per user set

  owen-d

  committed
  08500ed

  Copy full SHA for 08500ed

• refactors combining single and multi tenant tsdb indices on shipper reads

  owen-d

  committed
  f84af6a

  Copy full SHA for f84af6a

• indexshipper ignores old gzip logic

  owen-d

  committed
  aa46336

  Copy full SHA for aa46336

• method name refactor

  owen-d

  committed
  f388e24

  Copy full SHA for f388e24

Commits on May 3, 2022

• remove unused record type

  owen-d

  committed
  4690dff

  Copy full SHA for 4690dff

• removes v1 prefix in tsdb paths and refactores indices method

  owen-d

  committed
  a31c203

  Copy full SHA for a31c203

• ignores double optimization in tsdb looking for multitenant idx, shipper handles this

  owen-d

  committed
  8e6956f

  Copy full SHA for 8e6956f

• removes 5-ln requirement on shipper tablename regexp

  owen-d

  committed
  e928519

  Copy full SHA for e928519

• groups identifiers, begins removing multitenant prefix in shipped files

  owen-d

  committed
  bc0fe93

  Copy full SHA for bc0fe93

• passses open fn to indexshipper

  owen-d

  committed
  9fc3385

  Copy full SHA for 9fc3385

• exposes RealByteSlice

  owen-d

  committed
  3ce9110

  Copy full SHA for 3ce9110

• TSDBFile no longer needs a file descriptor, parses gzip extensions

  owen-d

  committed
  725abf8

  Copy full SHA for 725abf8

• method signature fixing

  owen-d

  committed
  8da79b8

  Copy full SHA for 8da79b8

• stop masquerading as compressed indices post-download in indexshipper

  owen-d

  committed
  bce8c52

  Copy full SHA for bce8c52

• variable bucket regexp

  owen-d

  committed
  e229160

  Copy full SHA for e229160

Commits on May 4, 2022

• removes accidental configs committed

  owen-d

  committed
  0b2125e

  Copy full SHA for 0b2125e

• Merge remote-tracking branch 'upstream/main' into tsdb/head-wal

  owen-d

  committed
  d1ff6f7

  Copy full SHA for d1ff6f7

• label matcher handling for multitenancy and metricname in tsdb

  owen-d

  committed
  cc34584

  Copy full SHA for cc34584

• explicitly require fingerprint when creating tsdb index

  owen-d

  committed
  a850e42

  Copy full SHA for a850e42

• only add tenant label when creating multitenant tsdb

  Show description for 14dcb1d
  owen-d

  committed
  14dcb1d

  Copy full SHA for 14dcb1d

• linting + unused removal

  owen-d

  committed
  a853035

  Copy full SHA for a853035

• more linting :(

  owen-d

  committed
  4871228

  Copy full SHA for 4871228

• goimports

  owen-d

  committed
  0a0aed8

  Copy full SHA for 0a0aed8

• removes uploadername from indexshipper

  owen-d

  committed
  74ccc09

  Copy full SHA for 74ccc09

• maxuint32 for arm32 builds

  owen-d

  committed
  65aac18

  Copy full SHA for 65aac18

• tsdb chunk filterer support

  owen-d

  committed
  eafaf1a

  Copy full SHA for eafaf1a

Commits on May 5, 2022

• always set ingester name when using object storage index

  sandeepsukhani

  committed
  c6e7cf6

  Copy full SHA for c6e7cf6