Skip to content

Commit

Permalink
implementing string interning to optimize resource usage for storing …
Browse files Browse the repository at this point in the history
…and processing non-indexed labels (#10044)

**What this PR does / why we need it**:
In PR #9700, we added support for storing non-indexed labels in chunks.
This PR optimizes resource usage for storing and processing non-indexed
labels by doing string interning. We will store deduped label names and
values as a list in chunks
in the newly added non-indexed labels section. The labels would then be
referenced in blocks by their index(called symbols).

Additionally, I have started the convention of writing lengths of
sections with their offsets within chunks, making it easier to introduce
new sections. The section offsets and lengths would be stored at the end
of the chunk, similar to
[TOC](https://ganeshvernekar.com/blog/prometheus-tsdb-persistent-block-and-its-index/#a-toc)
in TSDB.

**Checklist**
- [x] Tests updated
  • Loading branch information
sandeepsukhani authored Jul 26, 2023
1 parent 122313c commit 9b554bb
Show file tree
Hide file tree
Showing 7 changed files with 1,025 additions and 309 deletions.
Loading

0 comments on commit 9b554bb

Please sign in to comment.