Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove bloom compactor component #13957

Closed
4 tasks done
chaudum opened this issue Aug 26, 2024 · 2 comments
Closed
4 tasks done

Remove bloom compactor component #13957

chaudum opened this issue Aug 26, 2024 · 2 comments
Assignees
@chaudum
Labels
feature/blooms

Comments

@chaudum
Copy link
Contributor

chaudum commented Aug 26, 2024 •
edited
Loading

Background

Some time ago, a two new components for building bloom filters were introduced: the bloom planner and the bloom builder.

These new components fulfil the same functionality as the the compactor, however, they come with certain advantages:

  • The consistent hash ring for the compactors is removed. The ring is hard to operate and configure and causes problems with the ownership of fingerprint ranges when instances are added and removed regularly (e.g. due to OOMing)
  • The planner/builder approach can distribute load better, because builders get new tasks from the queue whenever their previous task is completed. This reduces idle time of instances.
  • Only the planner is "stateful", since it keeps the queue. Builders are stateless and can be scaled horizontally depending on the needs, without changing the fingerprint ranges of the tasks (which was the case when running compactors). This can enable huge cost savings, because planners can be run on spot instances.

List of PRs

Change request

⚠️ Note
Bloom filters are an experimental feature and therefore subject to rapid changes.

Because of these new components, the bloom compactor should not be considered any more and existing deployments should be migrated as soon as possible.

TODOs

  • Update documentation to reference bloom planner and builders
  • Integrate bloom building into SSD deployment
  • Remove all bloom compactor related code
  • Update Helm chart (distributed) to use the new components
@chaudum chaudum self-assigned this Aug 26, 2024
@chaudum chaudum mentioned this issue Aug 26, 2024
Merged
7 tasks
chaudum added a commit that referenced this issue Aug 27, 2024
@chaudum
chore(blooms): Remove bloom compactor component
eae3643 
This commit removes the code related to the bloom compactor which is
superseded by the bloom planner and builders.

A handful of CLI arguments changed their prefixes from
`-bloom-compactor.*` to `-bloom-build.*`.

Part of #13957

Documentation update #13965

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
@chaudum chaudum mentioned this issue Aug 27, 2024
Merged
7 tasks
chaudum added a commit that referenced this issue Aug 27, 2024
@chaudum
chore(blooms): Remove bloom compactor component
7b303e6 
This commit removes the code related to the bloom compactor which is
superseded by the bloom planner and builders.

A handful of CLI arguments changed their prefixes from
`-bloom-compactor.*` to `-bloom-build.*`.

Part of #13957

Documentation update #13965

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this issue Aug 29, 2024
@chaudum
chore(blooms): Remove bloom compactor component
a7a973e 
This commit removes the code related to the bloom compactor which is
superseded by the bloom planner and builders.

A handful of CLI arguments changed their prefixes from
`-bloom-compactor.*` to `-bloom-build.*`.

Part of #13957

Documentation update #13965

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this issue Aug 29, 2024
@chaudum
chore(blooms)!: Remove bloom compactor component (#13969)
b75eacc 
**What this PR does / why we need it**:

This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders.

A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information.

**Special notes for your reviewer**:

:heavy_check_mark: Part of #13957

📔 Documentation update #13965

⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR.

---

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this issue Aug 29, 2024
@chaudum
Replace bloom compactor in Helm chart with bloom planner and builder
e341d53 
This is PR is part of the process of removing the bloom compactor and
replace it with bloom planner and bloom builder.

See also #13957

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
@chaudum chaudum mentioned this issue Aug 29, 2024
Merged
7 tasks
pascal-sochacki pushed a commit to pascal-sochacki/loki that referenced this issue Aug 29, 2024
@chaudum @pascal-sochacki
chore(blooms)!: Remove bloom compactor component (grafana#13969)
a9bd19a 
**What this PR does / why we need it**:

This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders.

A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information.

**Special notes for your reviewer**:

:heavy_check_mark: Part of grafana#13957

📔 Documentation update grafana#13965

⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR.

---

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
pascal-sochacki pushed a commit to pascal-sochacki/loki that referenced this issue Aug 29, 2024
@chaudum @pascal-sochacki
chore(blooms)!: Remove bloom compactor component (grafana#13969)
7727141 
**What this PR does / why we need it**:

This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders.

A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information.

**Special notes for your reviewer**:

:heavy_check_mark: Part of grafana#13957

📔 Documentation update grafana#13965

⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR.

---

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
pascal-sochacki pushed a commit to pascal-sochacki/loki that referenced this issue Aug 29, 2024
@chaudum @pascal-sochacki
chore(blooms)!: Remove bloom compactor component (grafana#13969)
a9612fb 
**What this PR does / why we need it**:

This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders.

A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information.

**Special notes for your reviewer**:

:heavy_check_mark: Part of grafana#13957

📔 Documentation update grafana#13965

⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR.

---

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this issue Aug 30, 2024
@chaudum
Replace bloom compactor in Helm chart with bloom planner and builder
10f67f0 
This is PR is part of the process of removing the bloom compactor and
replace it with bloom planner and bloom builder.

See also #13957

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this issue Sep 2, 2024
@chaudum
Replace bloom compactor in Helm chart with bloom planner and builder
f76df73 
This is PR is part of the process of removing the bloom compactor and
replace it with bloom planner and bloom builder.

See also #13957

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
chaudum added a commit that referenced this issue Sep 3, 2024
@chaudum
feat(helm): Replace bloom compactor with bloom planner and builder (#…
08e61ca 
…14003)

This is PR is part of the process of removing the bloom compactor and replace it with bloom planner and bloom builder.

Part of #13957

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
@LukoJy3D
Copy link
Contributor

Love this initiative!

@chaudum chaudum mentioned this issue Sep 27, 2024
Open
7 tasks
mraboosk pushed a commit to mraboosk/loki that referenced this issue Oct 7, 2024
@chaudum
feat(helm): Replace bloom compactor with bloom planner and builder (g…
694757b 
…rafana#14003)

This is PR is part of the process of removing the bloom compactor and replace it with bloom planner and bloom builder.

Part of grafana#13957

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
@chaudum
Copy link
Contributor Author

chaudum commented Jan 29, 2025

This can be considered as complete.

@chaudum chaudum closed this as completed Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chaudum chaudum

Labels
feature/blooms
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@chaudum @LukoJy3D and others