chore: add v0.27 migration guide

.tool-versions

@@ -1,2 +1,2 @@
-golang 1.22.8
+golang 1.22.9
 nodejs 20.16.0

migrate/v0_26/iavl-v1.md → migrate/v0_27/iavl-v1.md

@@ -4,21 +4,37 @@ IAVL V1 is an updated data format for the low-level storage of application data
 The change in format results in much more performant syncing of the chain, and greatly reduces the
 storage footprint required for Kava nodes.
 
-The first version of Kava using IAVL V1 is `v0.26.2-iavl-v1`. Future major versions will use IAVL V1
-unless otherwise specified.
+As of `v0.27.0`, IAVL V1 is the data storage format for the Kava blockchain.
 
-Steps for using IAVL V1:
+# Configuration
 
-**For `goleveldb` nodes:**
+> [!IMPORTANT]
+> Nodes running IAVL V1 should set `iavl-disable-fastnode = true` in their `app.toml` file.
+
+> [!IMPORTANT]
+> Before starting kava on existing IAVL V0 data, please read the note about pruning changes.
+
+### `goleveldb` nodes
 1. Replace or recreate your node data with IAVL V1: see [Data](#data).
 2. Install an IAVL V1 binary:
 ```sh
-git checkout v0.26.2-iavl-v1
+git checkout v0.27.0
 make install
 ```
-1. Start kava as usual: `kava start`
+3. Start kava as usual: `kava start`
+
+### `rocksdb` nodes
+
+> [!TIP]
+> Since v0.26.2, databases are opened with [opendb](https://github.com/Kava-Labs/opendb/) which
+> makes rocksdb more configurable. For best memory performance, node operators are encouraged to add
+> these settings to their `app.toml` when running with `db_backend = rocksdb`:
+> ```toml
+> [rocksdb]
+> max_open_files = 16384 # increase default max # of open files from 4096
+> block_size = 16384 # decreases block index memory by 4x!
+> ```
 
-**For `rocksdb` nodes:**
 1. Replace or recreate your node data with IAVL V1: see [Data](#data).
 2. Update your default memory allocator: see [default memory allocator](#default-memory-allocator)
 3. Install an IAVL V1 binary:
@@ -66,3 +82,30 @@ To update your node to use `tcmalloc`:
     * from source: see [tcmalloc Quickstart](https://google.github.io/tcmalloc/quickstart.html)
 2. whenever running kava, do so with the new memory allocator: `LD_PRELOAD=/path/to/tcmalloc kava start`
     * replace `/path/to/tcmalloc` above with correct path
+
+## IAVL V1 Pruning Changes
+
+There is an important difference in how changes to pruning settings are handled in IAVL V1.
+
+In IAVL V0, a change to pruning settings takes effect only for new blocks. In IAVL V1, all blocks are
+pruned to the new settings the moment the change is made.
+
+When switching to IAVL V1 in this upgrade, you will experience a long startup time if your node has
+had a change to pruning settings. If your node has changed its pruning settings during its runtime,
+you are encouraged to upgrade with `pruning = "nothing"`.
+
+If you experience a long start time or an out of memory error on startup, try again with `pruning = "nothing"`.
+For optimal performance, node operators are encouraged to reinitialize their node with fully IAVL V1
+data.
+
+**Example:**
+A node is started with `pruning = "nothing"` on IAVL V0 (any release prior to `kava@v0.26.2-iavl-v1`).
+
+The node runs for awhile, is stopped, and then switched to `pruning = "everything"`.
+
+On startup, the node does not prune the existing blocks. Only new blocks are processed with the current
+pruning setting.
+
+In IAVL V1 (when you start v0.27 on top of the above data), the `pruning = "everything"` setting is
+applied to the entire state. Depending on how many blocks there are that should be pruned, the intial
+start up can take a long time and may fail due to a lack of memory.

migrate/v0_27/migrate.md

@@ -0,0 +1,88 @@
+# Kava 17 Upgrade Instructions
+
+## Software Version and Key Dates
+
+- The version of `kava` for Kava 17 is `v0.27.0`
+- The Kava 17 chain will be shutdown with a `SoftwareUpgradeProposal` that
+  activates at approximately 15:00 UTC on November 21, 2024.
+
+## Data Storage Changes
+
+Kava 17 uses IAVL V1, an improved storage format for the low-level data storage used by the Kava blockchain.
+
+Using IAVL V1 brings performance speedups for syncing and massively reduces the data stored on disk (~2.4x less data storage required for a full-archive node).
+
+For full-archive node operators, an IAVL V1 snapshot will be linked when available. We are working with partners to host IAVL V1 full historical data.
+For validators & operators of pruning nodes, it is recommended that node data is recreated from scratch via statesync.
+
+Node operators using rocksdb are encouraged to [use `tcmalloc` as their memory allocator](./iavl-v1.md#default-memory-allocator).
+
+**See the [IAVL V1 migration guide](./iavl-v1.md).**
+
+## Dependency Changes
+
+### For validators using RocksDB
+
+> [!NOTE]
+> If you use goleveldb or other database backends, this is not required.
+
+If you use RocksDB as your database backend, you will need to update RocksDB if you are using `< v9.3.1`. The tested and recommended RocksDB version is `v9.3.1`.
+Please reference the [RocksDB repository](https://github.com/facebook/rocksdb/tree/v9.3.1) to update your installation before building the RocksDB kava binary.
+
+Node operators using rocksdb are encouraged to [use `tcmalloc` as their memory allocator](https://github.com/Kava-Labs/kava/blob/v0.26.2-iavl-v1/migrate/v0_26/iavl-v1.md#default-memory-allocator).
+
+### On the day of the upgrade
+
+The kava chain is expected to halt at block height **TO BE DETERMINED**. **Do not stop your node and begin the upgrade before the upgrade height**, or you may go offline and be unable to recover until after the upgrade!
+
+**Make sure the kava process is stopped before proceeding and that you have backed up your validator**. Failure to backup your validator could make it impossible to restart your node if the upgrade fails.
+
+**Ensure you are using golang 1.22.x and not a different version.** Golang 1.21 and below may cause app hash mismatches!
+
+To update to v0.27.0
+
+```sh
+# check go version - look for 1.22!
+go version
+# go version go1.22.9 linux/amd64
+
+# in the `kava` folder
+git fetch
+git checkout v0.27.0
+
+# Note: Golang 1.22 must be installed before this step
+make install
+
+# verify versions
+kava version --long
+# name: kava
+# server_name: kava
+# version: 0.27.0
+# commit: TODO
+# build_tags: netgo ledger,
+# go: go version go1.22.9 linux/amd64
+# build_deps:
+#  ...
+# cosmos_sdk_version: v0.47.10
+
+# Restart node -
+kava start
+```
+
+### Risks
+
+As a validator, performing the upgrade procedure on your consensus nodes carries a heightened risk of double-signing and being slashed. The most important piece of this procedure is verifying your software version and genesis file hash before starting your validator and signing.
+
+The riskiest thing a validator can do is discover that they made a mistake and repeat the upgrade procedure again during the network startup. If you discover a mistake in the process, the best thing to do is wait for the network to start before correcting it. If the network is halted and you have started with a different genesis file than the expected one, seek advice from a Kava developer before resetting your validator.
+
+### Recovery
+
+Prior to applying the Kava 17 upgrade, validators are encouraged to take a full data snapshot at the upgrade height before proceeding. Snap-shotting depends heavily on infrastructure, but generally this can be done by backing up the .kava directory.
+
+It is critically important to back-up the .kava/data/priv_validator_state.json file after stopping your kava process. This file is updated every block as your validator participates in consensus rounds. It is a critical file needed to prevent double-signing, in case the upgrade fails and the previous chain needs to be restarted.
+
+In the event that the upgrade does not succeed, validators and operators must downgrade back to v0.26.x of the Kava software and restore to their latest snapshot before restarting their nodes.
+
+### Coordination
+
+If the Kava 17 chain does not launch by November 22nd at 00:00 UTC, the launch should be considered a failure. In the event of launch failure, coordination will occur in the [Kava discord](https://discord.com/invite/kQzh3Uv).

tests/e2e/e2e_community_update_params_test.go

@@ -51,6 +51,8 @@ func (suite *IntegrationTestSuite) TestCommunityUpdateParams_NonAuthority() {
 }
 
 func (suite *IntegrationTestSuite) TestCommunityUpdateParams_Authority() {
+	suite.SkipIfKvtoolDisabled() // probably not testing w/ an account that has 1000 KAVA
+
 	// ARRANGE
 	govParamsRes, err := suite.Kava.Grpc.Query.Gov.Params(context.Background(), &govv1.QueryParamsRequest{
 		ParamsType: govv1.ParamDeposit,