Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update store and fix benchmarks #1

Merged
merged 100 commits into from
Mar 7, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
a549522
[WIP] refactor multistore => root store
roysc Aug 31, 2021
ec5d41e
baseapp - disable snapshot tests
roysc Oct 27, 2021
9db21a5
store updates, fixes
roysc Jan 20, 2022
ceabd08
updates, rf
roysc Jan 18, 2022
ab531ff
changelog + docs
roysc Jan 20, 2022
47330c5
Merge remote-tracking branch 'upstream/master' into refactor-multistore
roysc Jan 20, 2022
929f5ad
fixes
roysc Jan 20, 2022
3876113
Merge branch 'master' into roysc/adr-040-refactor-multistore
roysc Jan 20, 2022
25d0a9f
changelog fix
roysc Jan 21, 2022
355f181
cleanup
roysc Jan 21, 2022
8cbf4b2
nit, StoreConfig =>StoreParams
roysc Jan 21, 2022
4e431eb
store doc
roysc Jan 21, 2022
861b24c
store rearrange
roysc Jan 21, 2022
f45ce0f
godoc nits
roysc Jan 27, 2022
79bea1d
rm StoreConstructor
roysc Jan 28, 2022
e912111
compatibility wrapper for original MultiStore interface
roysc Jan 27, 2022
704570b
rename BasicMultiStore -> MultiStore
roysc Feb 7, 2022
05266d7
docs: proto generation scripts (#11133)
robert-zaremba Feb 9, 2022
ee42db7
build(deps): Bump amannn/action-semantic-pull-request (#11153)
dependabot[bot] Feb 10, 2022
acf7115
build(deps): Bump actions/setup-go from 2.1.5 to 2.2.0 (#11154)
dependabot[bot] Feb 10, 2022
b1f9a11
build(deps): Bump github.com/jhump/protoreflect from 1.10.2 to 1.10.3…
dependabot[bot] Feb 10, 2022
5859713
docs: Improve markdownlint configuration (#11104)
julienrbrt Feb 10, 2022
fa8099d
feat: add grants by grantee authz query (#10944)
cmwaters Feb 10, 2022
43b7e83
fix(cosmovisor): fix version when 'go install ...cosmovisor' (#10460)
robert-zaremba Feb 10, 2022
89739a5
cosmovisor v1.1 udpates (#11160)
robert-zaremba Feb 10, 2022
2c184d2
feat: Add percentage decision policy to x/group (#11072)
likhita-809 Feb 11, 2022
2e1c328
perf: add inplace decimal operations (#11004)
sunnya97 Feb 11, 2022
2d1686e
chore: add cosmovisor v1.1 release notes (#11162)
robert-zaremba Feb 11, 2022
a200bdf
docs: add mention of modes from 0.35 (#11167)
tac0turtle Feb 11, 2022
5fbd225
fix proto generation (#11169)
tac0turtle Feb 11, 2022
d98503b
feat: supply by denom as param (#11170)
tac0turtle Feb 11, 2022
94e6a37
fix(orm): ValidateJSON fails when tables are missing (#11174)
aaronc Feb 11, 2022
774e333
refactor!: remove 'keep-every' from pruning (#11152)
alexanderbez Feb 12, 2022
0a7b523
fix: Fix `raw_log` when ABCIMessageLog.MsgIndex is 0 (#11147)
assafmo Feb 12, 2022
9ff6d54
feat: Add debug pubkey-raw cli command (#11006)
sunnya97 Feb 14, 2022
b36c8f5
patch multistore
roysc Feb 7, 2022
d035fb8
patch compat store
roysc Feb 7, 2022
875378b
refactor: Align on gov/group Proposals and Vote syntax (#11097)
amaury1093 Feb 14, 2022
0d4cb92
refactor(orm)!: rename table interfaces from Store -> Table in codege…
aaronc Feb 14, 2022
81e9d0b
fix: failing tests in x/group (#11184)
atheeshp Feb 14, 2022
e7066c4
docs: Code blocks in SDK docs are broken (#11189)
julienrbrt Feb 14, 2022
bfa7b88
multi store- use StoreKey instances, not strings
roysc Feb 14, 2022
ccb908d
wrap v1 MultiStore in v2 interface; use in baseapp
roysc Feb 7, 2022
532a6ad
viewstore.CacheStore() replaces CacheMultiStoreWithVersion
roysc Feb 15, 2022
5f6fe12
test cleanup
roysc Feb 15, 2022
b200867
trace context fence
roysc Feb 15, 2022
9cdd990
get all versions
roysc Feb 15, 2022
5099c15
fix: non consistent keyring (#10844)
swelf19 Feb 15, 2022
2c59d34
Merge branch 'master' of github.com:cosmos/cosmos-sdk into refactor-m…
roysc Feb 15, 2022
2627ce3
Merge remote-tracking branch 'upstream/master' into refactor-multistore
roysc Feb 15, 2022
df87459
build(deps): Bump github.com/jhump/protoreflect from 1.10.3 to 1.11.0…
dependabot[bot] Feb 15, 2022
d10034f
docs: ADR-049 state sync hooks (#10976)
adu-web3 Feb 15, 2022
7869d38
feat(orm): support nil PageRequest (#11171)
aaronc Feb 15, 2022
cb6fea9
disable url parameter in swagger-ui page (#11202)
yihuang Feb 16, 2022
1338438
fix: gov 0.46 migration (#11206)
robert-zaremba Feb 16, 2022
6d1525f
feat(authz)!: pruning expired authorizations (#10714)
aleem1314 Feb 16, 2022
f2fe1d6
chore: rename migrations v045 -> v046 (#11210)
aleem1314 Feb 17, 2022
9ce0844
feat(baseapp): support pulsar gRPC query servers (#11192)
aaronc Feb 17, 2022
555be73
patches
roysc Feb 18, 2022
b631a9a
Merge branch 'master' of github.com:cosmos/cosmos-sdk into refactor-m…
roysc Feb 18, 2022
39a6f44
build(deps): Bump url-parse from 1.5.3 to 1.5.7 in /docs (#11221)
dependabot[bot] Feb 18, 2022
24e5a0d
docs: ADR 047: Extend Upgrade Plan - Initial Draft (#10602)
dwedul-figure Feb 18, 2022
83cd109
patches
roysc Feb 18, 2022
71a9bb9
fix v1asv2 commit
roysc Feb 18, 2022
ea67659
feat: Allow to restrict MintCoins from app.go (#10771)
mattverse Feb 18, 2022
fcd02d0
docs: improve RegisterMigration docs (#11225)
robert-zaremba Feb 18, 2022
20e17ea
feat: Add `MsgCreateGroupWithPolicy` to x/group (#10963)
likhita-809 Feb 18, 2022
ee7c86f
chore: retract v0.43.0 from usage (#11211)
tac0turtle Feb 20, 2022
850a093
build(deps): Bump bufbuild/buf-setup-action from 0.7.0 to 1.0.0 (#11231)
dependabot[bot] Feb 20, 2022
2adb1cb
chore: move from relative links to git links (#11238)
tac0turtle Feb 21, 2022
5c8021e
build(deps): Bump github.com/tendermint/tm-db in /orm (#11244)
dependabot[bot] Feb 21, 2022
a8f31c9
build(deps): Bump github.com/jhump/protoreflect from 1.11.0 to 1.12.0…
dependabot[bot] Feb 21, 2022
81cfc6c
fix: implement Amino serialization for x/authz and x/feegrant (#11224)
RiccardoM Feb 22, 2022
56af6a7
feat: Add ics23 proof tools: `ics23-{iavl,tendermint,smt}` (#10802)
roysc Feb 22, 2022
e66b8ef
feat: ADR-040: ICS-23 proofs for SMT store (#10015)
roysc Feb 22, 2022
dd65ef8
feat: include transactions in QueryBlockByHeight (#10880)
technicallyty Feb 22, 2022
afbb0bd
feat: min and max operators on coins (#11200)
JimLarson Feb 22, 2022
78b0a12
PR revisions
roysc Feb 23, 2022
4334200
build(deps): Bump github.com/google/go-cmp from 0.5.6 to 0.5.7 in /or…
dependabot[bot] Feb 23, 2022
ff31490
feat: replace all ModuleCdc instances with legacy.Cdc (#11240)
RiccardoM Feb 23, 2022
ddf8d0f
fix simd export
roysc Feb 23, 2022
3ddf675
open db at data/application/
roysc Feb 23, 2022
def4107
db cleanup
roysc Feb 23, 2022
880a4ee
badgerdb fix
roysc Feb 23, 2022
75bcf47
refactor: prune everything (#11177)
alexanderbez Feb 23, 2022
cf82b66
docs: add module migration order guidance (#11255)
yaruwangway Feb 23, 2022
40190fb
build(deps): Bump github.com/magiconair/properties from 1.8.5 to 1.8.…
dependabot[bot] Feb 23, 2022
6324b5a
fix: reject query with block height in the future (#11222)
yihuang Feb 24, 2022
ef6fed6
fix: case unauthorized message (#11229)
atheeshp Feb 24, 2022
c9a37fe
updates (#11263)
alexanderbez Feb 24, 2022
7e18e9f
feat!: Add hooks to allow app modules to add things to state-sync (#1…
yihuang Feb 24, 2022
4de7d40
feat(orm)!: add orm app wiring proto definitions (#11119)
aaronc Feb 24, 2022
8f58686
Merge branch 'master' of github.com:cosmos/cosmos-sdk into refactor-m…
roysc Feb 25, 2022
c686634
fix(crypto): Multisig verification doesn't work for multisig sizes of…
julienrbrt Feb 25, 2022
fcb9ff9
Merge remote-tracking branch 'upstream/master' into refactor-multistore
roysc Feb 25, 2022
c622a24
Merge branch 'refactor-multistore' into store_benchmarks
roysc Mar 1, 2022
b72b734
fix benchmarks
roysc Mar 1, 2022
b6592f2
change rocksdb build tag to "rocksdb"
roysc Mar 2, 2022
6ac110f
badgerdb - handle ErrTxnTooBig
roysc Mar 2, 2022
75d5e43
use tempdir
roysc Mar 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor!: remove 'keep-every' from pruning (cosmos#11152)
  • Loading branch information
alexanderbez authored Feb 12, 2022
commit 774e333ccb9ca6399ce56421107ef697df9449e4
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ Ref: https://keepachangelog.com/en/1.0.0/

### API Breaking Changes

* (store)[\#11152](https://github.com/cosmos/cosmos-sdk/pull/11152) Remove `keep-every` from pruning options.
* [\#10950](https://github.com/cosmos/cosmos-sdk/pull/10950) Add `envPrefix` parameter to `cmd.Execute`.
* (x/mint) [\#10441](https://github.com/cosmos/cosmos-sdk/pull/10441) The `NewAppModule` function now accepts an inflation calculation function as an argument.
* [\#10295](https://github.com/cosmos/cosmos-sdk/pull/10295) Remove store type aliases from /types
Expand Down
16 changes: 0 additions & 16 deletions baseapp/abci.go
Original file line number Diff line number Diff line change
Expand Up @@ -709,22 +709,6 @@ func (app *BaseApp) GetBlockRetentionHeight(commitHeight int64) int64 {
retentionHeight = commitHeight - cp.Evidence.MaxAgeNumBlocks
}

// Define the state pruning offset, i.e. the block offset at which the
// underlying logical database is persisted to disk.
statePruningOffset := int64(app.cms.GetPruning().KeepEvery)
if statePruningOffset > 0 {
if commitHeight > statePruningOffset {
v := commitHeight - (commitHeight % statePruningOffset)
retentionHeight = minNonZero(retentionHeight, v)
} else {
// Hitting this case means we have persisting enabled but have yet to reach
// a height in which we persist state, so we return zero regardless of other
// conditions. Otherwise, we could end up pruning blocks without having
// any state committed to disk.
return 0
}
}

if app.snapshotInterval > 0 && app.snapshotKeepRecent > 0 {
v := commitHeight - int64((app.snapshotInterval * uint64(app.snapshotKeepRecent)))
retentionHeight = minNonZero(retentionHeight, v)
Expand Down
6 changes: 1 addition & 5 deletions baseapp/abci_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,11 @@ func TestGetBlockRentionHeight(t *testing.T) {
"pruning iavl snapshot only": {
bapp: baseapp.NewBaseApp(
name, logger, db,
baseapp.SetPruning(sdk.PruningOptions{KeepEvery: 10000}),
baseapp.SetMinRetainBlocks(1),
),
maxAgeBlocks: 0,
commitHeight: 499000,
expected: 490000,
expected: 498999,
},
"pruning state sync snapshot only": {
bapp: baseapp.NewBaseApp(
Expand All @@ -69,7 +68,6 @@ func TestGetBlockRentionHeight(t *testing.T) {
"pruning all conditions": {
bapp: baseapp.NewBaseApp(
name, logger, db,
baseapp.SetPruning(sdk.PruningOptions{KeepEvery: 10000}),
baseapp.SetMinRetainBlocks(400000),
baseapp.SetSnapshotInterval(50000), baseapp.SetSnapshotKeepRecent(3),
),
Expand All @@ -80,7 +78,6 @@ func TestGetBlockRentionHeight(t *testing.T) {
"no pruning due to no persisted state": {
bapp: baseapp.NewBaseApp(
name, logger, db,
baseapp.SetPruning(sdk.PruningOptions{KeepEvery: 10000}),
baseapp.SetMinRetainBlocks(400000),
baseapp.SetSnapshotInterval(50000), baseapp.SetSnapshotKeepRecent(3),
),
Expand All @@ -91,7 +88,6 @@ func TestGetBlockRentionHeight(t *testing.T) {
"disable pruning": {
bapp: baseapp.NewBaseApp(
name, logger, db,
baseapp.SetPruning(sdk.PruningOptions{KeepEvery: 10000}),
baseapp.SetMinRetainBlocks(0),
baseapp.SetSnapshotInterval(50000), baseapp.SetSnapshotKeepRecent(3),
),
Expand Down
9 changes: 1 addition & 8 deletions baseapp/baseapp.go
Original file line number Diff line number Diff line change
Expand Up @@ -292,16 +292,9 @@ func (app *BaseApp) init() error {

// make sure the snapshot interval is a multiple of the pruning KeepEvery interval
if app.snapshotManager != nil && app.snapshotInterval > 0 {
rms, ok := app.cms.(*rootmulti.Store)
if !ok {
if _, ok := app.cms.(*rootmulti.Store); !ok {
return errors.New("state sync snapshots require a rootmulti store")
}
pruningOpts := rms.GetPruning()
if pruningOpts.KeepEvery > 0 && app.snapshotInterval%pruningOpts.KeepEvery != 0 {
return fmt.Errorf(
"state sync snapshot interval %v must be a multiple of pruning keep every interval %v",
app.snapshotInterval, pruningOpts.KeepEvery)
}
}

return nil
Expand Down
2 changes: 0 additions & 2 deletions baseapp/baseapp_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,6 @@ func setupBaseAppWithSnapshots(t *testing.T, blocks uint, blockTxs int, options
app := setupBaseApp(t, append(options,
baseapp.SetSnapshotStore(snapshotStore),
baseapp.SetSnapshotInterval(snapshotInterval),
baseapp.SetPruning(sdk.PruningOptions{KeepEvery: 1}),
routerOpt)...)

app.InitChain(abci.RequestInitChain{})
Expand Down Expand Up @@ -480,7 +479,6 @@ func TestLoadVersionPruning(t *testing.T) {
logger := log.NewNopLogger()
pruningOptions := storetypes.PruningOptions{
KeepRecent: 2,
KeepEvery: 3,
Interval: 1,
}
pruningOpt := baseapp.SetPruning(pruningOptions)
Expand Down
5 changes: 0 additions & 5 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,6 @@ github.com/cosmos/cosmos-sdk/errors v1.0.0-beta.2/go.mod h1:Gi7pzVRnvZ1N16JAXpLA
github.com/cosmos/go-bip39 v0.0.0-20180819234021-555e2067c45d/go.mod h1:tSxLoYXyBmiFeKpvmq4dzayMdCjCnu8uqmCysIGBT2Y=
github.com/cosmos/go-bip39 v1.0.0 h1:pcomnQdrdH22njcAatO0yWojsUnCO3y2tNoV1cb6hHY=
github.com/cosmos/go-bip39 v1.0.0/go.mod h1:RNJv0H/pOIVgxw6KS7QeX2a0Uo0aKUlfhZ4xuwvCdJw=
github.com/cosmos/gorocksdb v1.2.0/go.mod h1:aaKvKItm514hKfNJpUJXnnOWeBnk2GL4+Qw9NHizILw=
github.com/cosmos/iavl v0.17.3 h1:s2N819a2olOmiauVa0WAhoIJq9EhSXE9HDBAoR9k+8Y=
github.com/cosmos/iavl v0.17.3/go.mod h1:prJoErZFABYZGDHka1R6Oay4z9PrNeFFiMKHDAMOi4w=
github.com/cosmos/keyring v1.1.7-0.20210622111912-ef00f8ac3d76 h1:DdzS1m6o/pCqeZ8VOAit/gyATedRgjvkVI+UCrLpyuU=
Expand Down Expand Up @@ -324,7 +323,6 @@ github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f/go.mod h1:xH/i4TFM
github.com/dgraph-io/badger/v2 v2.2007.2/go.mod h1:26P/7fbL4kUZVEVKLAKXkBXKOydDmM2p1e+NhhnBCAE=
github.com/dgraph-io/badger/v2 v2.2007.4 h1:TRWBQg8UrlUhaFdco01nO2uXwzKS7zd+HVdwV/GHc4o=
github.com/dgraph-io/badger/v2 v2.2007.4/go.mod h1:vSw/ax2qojzbN6eXHIx6KPKtCSHJN/Uz0X0VPruTIhk=
github.com/dgraph-io/badger/v3 v3.2103.2/go.mod h1:RHo4/GmYcKKh5Lxu63wLEMHJ70Pac2JqZRYGhlyAo2M=
github.com/dgraph-io/ristretto v0.0.3-0.20200630154024-f66de99634de/go.mod h1:KPxhHT9ZxKefz+PCeOGsrHpl1qZ7i70dGTu2u+Ahh6E=
github.com/dgraph-io/ristretto v0.0.3/go.mod h1:KPxhHT9ZxKefz+PCeOGsrHpl1qZ7i70dGTu2u+Ahh6E=
github.com/dgraph-io/ristretto v0.1.0 h1:Jv3CGQHp9OjuMBSne1485aDpUkTKEcUqF+jm/LuerPI=
Expand Down Expand Up @@ -537,8 +535,6 @@ github.com/google/btree v1.0.1/go.mod h1:xXMiIv4Fb/0kKde4SpL7qlzvu5cMJDRkFDxJfI9
github.com/google/certificate-transparency-go v1.0.21/go.mod h1:QeJfpSbVSfYc7RgB3gJFj9cbuQMMchQxrWXz8Ruopmg=
github.com/google/certificate-transparency-go v1.1.1/go.mod h1:FDKqPvSXawb2ecErVRrD+nfy23RCzyl7eqVCEmlT1Zs=
github.com/google/flatbuffers v1.11.0/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/flatbuffers v1.12.1/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/flatbuffers v2.0.0+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
Expand Down Expand Up @@ -1657,7 +1653,6 @@ golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20211013075003-97ac67df715c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211105183446-c75c47738b0c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211113001501-0c823b97ae02/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211205182925-97ca703d548d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211210111614-af8b64212486/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
Expand Down
3 changes: 0 additions & 3 deletions server/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ type BaseConfig struct {

Pruning string `mapstructure:"pruning"`
PruningKeepRecent string `mapstructure:"pruning-keep-recent"`
PruningKeepEvery string `mapstructure:"pruning-keep-every"`
PruningInterval string `mapstructure:"pruning-interval"`

// HaltHeight contains a non-zero block height at which a node will gracefully
Expand Down Expand Up @@ -207,7 +206,6 @@ func DefaultConfig() *Config {
InterBlockCache: true,
Pruning: storetypes.PruningOptionDefault,
PruningKeepRecent: "0",
PruningKeepEvery: "0",
PruningInterval: "0",
MinRetainBlocks: 0,
IndexEvents: make([]string, 0),
Expand Down Expand Up @@ -265,7 +263,6 @@ func GetConfig(v *viper.Viper) Config {
InterBlockCache: v.GetBool("inter-block-cache"),
Pruning: v.GetString("pruning"),
PruningKeepRecent: v.GetString("pruning-keep-recent"),
PruningKeepEvery: v.GetString("pruning-keep-every"),
PruningInterval: v.GetString("pruning-interval"),
HaltHeight: v.GetUint64("halt-height"),
HaltTime: v.GetUint64("halt-time"),
Expand Down
7 changes: 3 additions & 4 deletions server/config/toml.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,14 @@ const DefaultConfigTemplate = `# This is a TOML config file.
# specified in this config (e.g. 0.25token1;0.0001token2).
minimum-gas-prices = "{{ .BaseConfig.MinGasPrices }}"

# default: the last 100 states are kept in addition to every 500th state; pruning at 10 block intervals
# default: the last 362880 states are kept, pruning at 10 block intervals
# nothing: all historic states will be saved, nothing will be deleted (i.e. archiving node)
# everything: all saved states will be deleted, storing only the current state; pruning at 10 block intervals
# custom: allow pruning options to be manually specified through 'pruning-keep-recent', 'pruning-keep-every', and 'pruning-interval'
# custom: allow pruning options to be manually specified through 'pruning-keep-recent' and 'pruning-interval'
pruning = "{{ .BaseConfig.Pruning }}"

# These are applied if and only if the pruning strategy is custom.
pruning-keep-recent = "{{ .BaseConfig.PruningKeepRecent }}"
pruning-keep-every = "{{ .BaseConfig.PruningKeepEvery }}"
pruning-interval = "{{ .BaseConfig.PruningInterval }}"

# HaltHeight contains a non-zero block height at which a node will gracefully
Expand Down Expand Up @@ -202,7 +201,7 @@ enable-unsafe-cors = {{ .GRPCWeb.EnableUnsafeCORS }}
[state-sync]

# snapshot-interval specifies the block interval at which local state sync snapshots are
# taken (0 to disable). Must be a multiple of pruning-keep-every.
# taken (0 to disable).
snapshot-interval = {{ .StateSync.SnapshotInterval }}

# snapshot-keep-recent specifies the number of recent snapshots to keep and serve (0 to keep all).
Expand Down
1 change: 0 additions & 1 deletion server/pruning.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ func GetPruningOptionsFromFlags(appOpts types.AppOptions) (storetypes.PruningOpt
case storetypes.PruningOptionCustom:
opts := storetypes.NewPruningOptions(
cast.ToUint64(appOpts.Get(FlagPruningKeepRecent)),
cast.ToUint64(appOpts.Get(FlagPruningKeepEvery)),
cast.ToUint64(appOpts.Get(FlagPruningInterval)),
)

Expand Down
2 changes: 0 additions & 2 deletions server/pruning_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,12 @@ func TestGetPruningOptionsFromFlags(t *testing.T) {
v := viper.New()
v.Set(FlagPruning, types.PruningOptionCustom)
v.Set(FlagPruningKeepRecent, 1234)
v.Set(FlagPruningKeepEvery, 4321)
v.Set(FlagPruningInterval, 10)

return v
},
expectedOptions: types.PruningOptions{
KeepRecent: 1234,
KeepEvery: 4321,
Interval: 10,
},
},
Expand Down
10 changes: 4 additions & 6 deletions server/start.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,6 @@ const (

FlagPruning = "pruning"
FlagPruningKeepRecent = "pruning-keep-recent"
FlagPruningKeepEvery = "pruning-keep-every"
FlagPruningInterval = "pruning-interval"
FlagIndexEvents = "index-events"
FlagMinRetainBlocks = "min-retain-blocks"
Expand Down Expand Up @@ -77,15 +76,15 @@ func StartCmd(appCreator types.AppCreator, defaultNodeHome string) *cobra.Comman
Long: `Run the full node application with Tendermint in or out of process. By
default, the application will run with Tendermint in process.

Pruning options can be provided via the '--pruning' flag or alternatively with '--pruning-keep-recent',
'pruning-keep-every', and 'pruning-interval' together.
Pruning options can be provided via the '--pruning' flag or alternatively with '--pruning-keep-recent'
and 'pruning-interval' together.

For '--pruning' the options are as follows:

default: the last 100 states are kept in addition to every 500th state; pruning at 10 block intervals
default: the last 362880 states are kept, pruning at 10 block intervals
nothing: all historic states will be saved, nothing will be deleted (i.e. archiving node)
everything: all saved states will be deleted, storing only the current state; pruning at 10 block intervals
custom: allow pruning options to be manually specified through 'pruning-keep-recent', 'pruning-keep-every', and 'pruning-interval'
custom: allow pruning options to be manually specified through 'pruning-keep-recent' and 'pruning-interval'

Node halting configurations exist in the form of two flags: '--halt-height' and '--halt-time'. During
the ABCI Commit phase, the node will check if the current block height is greater than or equal to
Expand Down Expand Up @@ -147,7 +146,6 @@ which accepts a path for the resulting pprof file.
cmd.Flags().Bool(FlagTrace, false, "Provide full stack traces for errors in ABCI Log")
cmd.Flags().String(FlagPruning, storetypes.PruningOptionDefault, "Pruning strategy (default|nothing|everything|custom)")
cmd.Flags().Uint64(FlagPruningKeepRecent, 0, "Number of recent heights to keep on disk (ignored if pruning is not 'custom')")
cmd.Flags().Uint64(FlagPruningKeepEvery, 0, "Offset heights to keep on disk after 'keep-every' (ignored if pruning is not 'custom')")
cmd.Flags().Uint64(FlagPruningInterval, 0, "Height interval at which pruned heights are removed from disk (ignored if pruning is not 'custom')")
cmd.Flags().Uint(FlagInvCheckPeriod, 0, "Assert registered invariants every N blocks")
cmd.Flags().Uint64(FlagMinRetainBlocks, 0, "Minimum block height offset during ABCI commit to prune Tendermint blocks")
Expand Down
4 changes: 1 addition & 3 deletions snapshots/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ filesystem under `<node_home>/data/snapshots/`, with metadata in a LevelDB datab

Snapshots are taken asynchronously, i.e. new blocks will be applied concurrently
with snapshots being taken. This is possible because IAVL supports querying
immutable historical heights. However, this requires `state-sync.snapshot-interval`
to be a multiple of `pruning-keep-every`, to prevent a height from being removed
while it is being snapshotted.
immutable historical heights.

When a remote node is state syncing, Tendermint calls the ABCI method
`ListSnapshots` to list available local snapshots and `LoadSnapshotChunk` to
Expand Down
9 changes: 1 addition & 8 deletions store/rootmulti/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -428,14 +428,7 @@ func (rs *Store) Commit() types.CommitID {
// be pruned, where pruneHeight = (commitHeight - 1) - KeepRecent.
if int64(rs.pruningOpts.KeepRecent) < previousHeight {
pruneHeight := previousHeight - int64(rs.pruningOpts.KeepRecent)
// We consider this height to be pruned iff:
//
// - KeepEvery is zero as that means that all heights should be pruned.
// - KeepEvery % (height - KeepRecent) != 0 as that means the height is not
// a 'snapshot' height.
if rs.pruningOpts.KeepEvery == 0 || pruneHeight%int64(rs.pruningOpts.KeepEvery) != 0 {
rs.pruneHeights = append(rs.pruneHeights, pruneHeight)
}
rs.pruneHeights = append(rs.pruneHeights, pruneHeight)
}

// batch prune if the current height is a pruning interval height
Expand Down
15 changes: 7 additions & 8 deletions store/rootmulti/store_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,6 @@ func TestMultiStoreRestart(t *testing.T) {
db := dbm.NewMemDB()
pruning := types.PruningOptions{
KeepRecent: 2,
KeepEvery: 3,
Interval: 1,
}
multi := newMultiStoreWithMounts(db, pruning)
Expand Down Expand Up @@ -488,9 +487,9 @@ func TestMultiStore_Pruning(t *testing.T) {
}{
{"prune nothing", 10, types.PruneNothing, nil, []int64{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}},
{"prune everything", 10, types.PruneEverything, []int64{1, 2, 3, 4, 5, 6, 7, 8, 9}, []int64{10}},
{"prune some; no batch", 10, types.NewPruningOptions(2, 3, 1), []int64{1, 2, 4, 5, 7}, []int64{3, 6, 8, 9, 10}},
{"prune some; small batch", 10, types.NewPruningOptions(2, 3, 3), []int64{1, 2, 4, 5}, []int64{3, 6, 7, 8, 9, 10}},
{"prune some; large batch", 10, types.NewPruningOptions(2, 3, 11), nil, []int64{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}},
{"prune some; no batch", 10, types.NewPruningOptions(2, 1), []int64{1, 2, 4, 5, 7}, []int64{3, 6, 8, 9, 10}},
{"prune some; small batch", 10, types.NewPruningOptions(2, 3), []int64{1, 2, 4, 5}, []int64{3, 6, 7, 8, 9, 10}},
{"prune some; large batch", 10, types.NewPruningOptions(2, 11), nil, []int64{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}},
}

for _, tc := range testCases {
Expand Down Expand Up @@ -520,7 +519,7 @@ func TestMultiStore_Pruning(t *testing.T) {

func TestMultiStore_PruningRestart(t *testing.T) {
db := dbm.NewMemDB()
ms := newMultiStoreWithMounts(db, types.NewPruningOptions(2, 3, 11))
ms := newMultiStoreWithMounts(db, types.NewPruningOptions(2, 11))
require.NoError(t, ms.LoadLatestVersion())

// Commit enough to build up heights to prune, where on the next block we should
Expand All @@ -534,13 +533,13 @@ func TestMultiStore_PruningRestart(t *testing.T) {
// ensure we've persisted the current batch of heights to prune to the store's DB
ph, err := getPruningHeights(ms.db)
require.NoError(t, err)
require.Equal(t, pruneHeights, ph)
require.Equal(t, []int64{1, 2, 3, 4, 5, 6, 7}, ph)

// "restart"
ms = newMultiStoreWithMounts(db, types.NewPruningOptions(2, 3, 11))
ms = newMultiStoreWithMounts(db, types.NewPruningOptions(2, 11))
err = ms.LoadLatestVersion()
require.NoError(t, err)
require.Equal(t, pruneHeights, ms.pruneHeights)
require.Equal(t, []int64{1, 2, 3, 4, 5, 6, 7}, ms.pruneHeights)

// commit one more block and ensure the heights have been pruned
ms.Commit()
Expand Down
26 changes: 9 additions & 17 deletions store/types/pruning.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package types

import "fmt"
import (
"fmt"
)

// Pruning option string constants
const (
Expand All @@ -16,15 +18,15 @@ var (
// at every 10th height. The last 362880 heights are kept assuming the typical
// block time is 5s and typical unbonding period is 21 days. If these values
// do not match the applications' requirements, use the "custom" option.
PruneDefault = NewPruningOptions(362880, 100, 10)
PruneDefault = NewPruningOptions(362880, 10)

// PruneEverything defines a pruning strategy where all committed heights are
// deleted, storing only the current height and where to-be pruned heights are
// pruned at every 10th height.
PruneEverything = NewPruningOptions(0, 0, 10)
PruneEverything = NewPruningOptions(0, 10)

// PruneNothing defines a pruning strategy where all heights are kept on disk.
PruneNothing = NewPruningOptions(0, 1, 0)
PruneNothing = NewPruningOptions(0, 0)
)

// PruningOptions defines the pruning strategy used when determining which
Expand All @@ -33,30 +35,20 @@ type PruningOptions struct {
// KeepRecent defines how many recent heights to keep on disk.
KeepRecent uint64

// KeepEvery defines how many offset heights are kept on disk past KeepRecent.
KeepEvery uint64

// Interval defines when the pruned heights are removed from disk.
Interval uint64
}

func NewPruningOptions(keepRecent, keepEvery, interval uint64) PruningOptions {
func NewPruningOptions(keepRecent, interval uint64) PruningOptions {
return PruningOptions{
KeepRecent: keepRecent,
KeepEvery: keepEvery,
Interval: interval,
}
}

func (po PruningOptions) Validate() error {
if po.KeepEvery == 0 && po.Interval == 0 {
return fmt.Errorf("invalid 'Interval' when pruning everything: %d", po.Interval)
}
if po.KeepEvery == 1 && po.Interval != 0 { // prune nothing
return fmt.Errorf("invalid 'Interval' when pruning nothing: %d", po.Interval)
}
if po.KeepEvery > 1 && po.Interval == 0 {
return fmt.Errorf("invalid 'Interval' when pruning: %d", po.Interval)
if po.KeepRecent > 0 && po.Interval == 0 {
return fmt.Errorf("invalid 'Interval' when pruning recent heights: %d", po.Interval)
}

return nil
Expand Down
Loading