Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimise calls to iter_objects (repo v2) #6567

Closed
ThomasWaldmann opened this issue Apr 11, 2022 · 1 comment · Fixed by #8332
Closed

optimise calls to iter_objects (repo v2) #6567

ThomasWaldmann opened this issue Apr 11, 2022 · 1 comment · Fixed by #8332
Milestone

Comments

@ThomasWaldmann
Copy link
Member

ThomasWaldmann commented Apr 11, 2022

if a repo only has PUT2 (and no PUT any more), e.g. replay_segments should call iter_objects(..., read_data=False).

with PUT tags, a CRC32 check was only possible when reading the data (even if we didn't really need the data).
with PUT2, the CRC32 check is now possible without reading the data, so we can get much faster.

replay_segments is called e.g. if there is:

  • an index_transaction N
  • a segments_transaction M
  • N > M
  • that means: some segment files at the end were deleted (I just did that for a repo on a disk with I/O errors - in that case it will completely read ALL the segment files..................................).

besides replay_segments also check all other callers of iter_objects which do not already give read_data=False - maybe they could!

@ThomasWaldmann ThomasWaldmann added this to the 1.3.0b1 milestone Apr 11, 2022
@ThomasWaldmann ThomasWaldmann changed the title repo v2 perf optimize replay_segments optimise calls to iter_objects (repo v2) Apr 11, 2022
@ThomasWaldmann
Copy link
Member Author

ThomasWaldmann commented Jul 27, 2022

Guess we need to shift that to N+1 release (not 2.0), because 2.0 still deals with old repos (that do not have PUT2 yet).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant