interrupted borg compact / compact_segments

[ThomasWaldmann]

Discussed in #6283

^{Originally posted by sophie-h February 11, 2022}

Will borg compact in borg 1.2 be able to continue where it was left off?

I'm currently trying to prune hundreds of archives but there does not seem to be an easy way to complete this.

[ThomasWaldmann]

compaction copies valid entries from old source segment files to new target segment files, skipping over the logically deleted stuff.

i had a look at the code and complete_xfer is called when a target segment is full and then adds an intermediate commit to it and also gets rid of all (at that point) completely unused source segment files.

at the end of the compaction complete_xfer is called unconditionally and will add a final commit.

the index files are only written at the very end of the whole process, guess this needs some review whether that is ok.

[ThomasWaldmann]

for cases when borg compact could take rather long (e.g. after deleting lots of archives in one borg delete/prune command), it might be a good idea to slowly step-by-step lower the compaction threshold, so it has less to do per compaction run.

an alternative approach is to avoid mass-deletion of archives and only delete/prune a few archives, then borg compact, then delete next few, etc.

in general, users are advised to run borg prune frequently (like daily or weekly) - it is a rather bad idea to first accumulate hundreds or thousands of archives in a repo and then wanting to get rid of most of them all at once.