Auto enable Periodic Compactions if a Compaction Filter is used #5865
Conversation
There was a problem hiding this comment.
@sagar0 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@sagar0 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@sagar0 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@sagar0 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@sagar0 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
0c16cbf to
c1760c0
Compare
|
@sagar0 has updated the pull request. Re-import the pull request |
1 similar comment
|
@sagar0 has updated the pull request. Re-import the pull request |
c2e6abe to
c926bc9
Compare
|
@sagar0 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@sagar0 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@sagar0 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@sagar0 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
…book#5865) Summary: - Periodic compactions are auto-enabled if a compaction filter or a compaction filter factory is set, in Level Compaction. - The default value of `periodic_compaction_seconds` is changed to UINT64_MAX, which lets RocksDB auto-tune periodic compactions as needed. An explicit value of 0 will still work as before ie. to disable periodic compactions completely. For now, on seeing a compaction filter along with a UINT64_MAX value for `periodic_compaction_seconds`, RocksDB will make SST files older than 30 days to go through periodic copmactions. Some RocksDB users make use of compaction filters to control when their data can be deleted, usually with a custom TTL logic. But it is occasionally possible that the compactions get delayed by considerable time due to factors like low writes to a key range, data reaching bottom level, etc before the TTL expiry. Periodic Compactions feature was originally built to help such cases. Now periodic compactions are auto enabled by default when compaction filters or compaction filter factories are used, as it is generally helpful to all cases to collect garbage. `periodic_compaction_seconds` is set to a large value, 30 days, in `SanitizeOptions` when RocksDB sees that a `compaction_filter` or `compaction_filter_factory` is used. This is done only for Level Compaction style. Pull Request resolved: facebook#5865 Test Plan: - Added a new test `DBCompactionTest.LevelPeriodicCompactionWithCompactionFilters` to make sure that `periodic_compaction_seconds` is set if either `compaction_filter` or `compaction_filter_factory` options are set. - `COMPILE_WITH_ASAN=1 make check` Differential Revision: D17659180 Pulled By: sagar0 fbshipit-source-id: 4887b9cf2e53cf2dc93a7b658c6b15e1181217ee
periodic_compaction_secondsis changed to UINT64_MAX, which lets RocksDB auto-tune periodic compactions as needed. An explicit value of 0 will still work as before ie. to disable periodic compactions completely. For now, on seeing a compaction filter along with a UINT64_MAX value forperiodic_compaction_seconds, RocksDB will make SST files older than 30 days to go through periodic copmactions.Some RocksDB users make use of compaction filters to control when their data can be deleted, usually with a custom TTL logic. But it is occasionally possible that the compactions get delayed by considerable time due to factors like low writes to a key range, data reaching bottom level, etc before the TTL expiry. Periodic Compactions feature was originally built to help such cases. Now periodic compactions are auto enabled by default when compaction filters or compaction filter factories are used, as it is generally helpful to all cases to collect garbage.
periodic_compaction_secondsis set to a large value, 30 days, inSanitizeOptionswhen RocksDB sees that acompaction_filterorcompaction_filter_factoryis used.This is done only for Level Compaction style.
Test Plan:
DBCompactionTest.LevelPeriodicCompactionWithCompactionFiltersto make sure thatperiodic_compaction_secondsis set if eithercompaction_filterorcompaction_filter_factoryoptions are set.COMPILE_WITH_ASAN=1 make check