Unmanageable near/res/testnet.json

[maxzaver]

near/res/testnet.json is 39MiB and it is currently really hard to work with it especially when there is a merge conflict. We should remove the state from config and use state dump instead. Good thing this functionality is aleady implemented.

[bowenwang1996]

How do we dump state?

[maxzaver]

You can call genesis populate tool with 0 additional accounts: https://github.com/nearprotocol/nearcore/blob/96f3c599dbebe6d3d500ddc8e70c75a254630840/genesis-tools/README.md

[evgenykuzyakov]

I'm going to split testnet.json into 2 files. One for config and one for records.

[ilblackdragon]

@nearmax I want to reiterate:

• There should be genesis state in the repo, so people can start nodes. So state dumping doesn't work.
• Our genesis state does contain all of this data as we keep hard forking the network.
• There should not be merged conflicts, because you should not update that file unless the network was actually hard forked. Instead there should be script that upgrades the previous state into the new state.
• Every breaking change should pretty much invalidate the start_testnet script until such hard fork happens.
• Staging script should actually apply most recent upgrade script on prev TestNet.json and start with that

[maxzaver]

I suggest we unify the config migration discussion that we had in Slack and #1922 here.

Here is the list of workflows that we want to cover:

• S1. A person makes a change in a separate PR that is not backward compatible with the state format of the code in the staging branch and they want to submit this PR to staging;
• S2. A person makes a change in a separate PR that is not backward compatible with the state format that is currently used by the currently running version of the testnet and they want to submit this PR to staging;
• C1. A person makes a change in a separate PR that is not backward compatible with the config format of the code in the staging branch and they want to submit this PR to staging;
• C2. A person makes a change in a separate PR that is not backward compatible with the config format that is currently used by the currently running version of the testnet and they want to submit this PR to staging;

The current workflow

The current workflow, for all the above cases, is to write a new python script that given the previous config.json file generates a new config.json file with a modified state/format. This approach has the following issues:

1. The node version needs to be bumped for every S1 and C1 which means we might have several versions between two merges of staging branch into master. Since the merge of staging to master corresponds to a new release it will mean our release versions will jump through several versions at once;

   • Note the following approach suggested by @ilblackdragon has issues:

     Only time when this file changes is on the release with hard fork - which we discussed we should have only one between now and Release Candidate version.

     We cannot write the python script only during the release. If it is C1 scenario, then the change of the config value would need to be reflected in the python script immediately in the PR that is being merged to staging. If it is S1 scenario then release manager would have to talk to each person that did the breaking change to understand how to write the migration script that takes all breaking changes into account.

2. If we keep a python script for each version then we will have a proliferation of python scripts that are only different by one or two lines. If we do not keep old python scripts than two developers working concurrently will have to merge their scripts;

3. The script can be run only once (unless we maintain old_config.json alongside config.json);

4. Since we cannot verify the changes to the config.json file and should trust that it was generated by the python script it becomes not secure, as mentioned by @evgenykuzyakov .

We also store the state as state records together with the rest of the config in one file. It has the following issues:
5. Many editing tools cannot work with such large config. E.g. we cannot open it in CLion or diff it against the previous version, so config.json becomes a black box that would rarely read;
6. If we ever observe a regression and decide to debug it by diffing our current config with all the versions of the config for the past month we won't be able to do it.

[bowenwang1996]

@nearmax C1 and C2 are the same :)
I think in the case of S1 we should merge the migration scripts and have one per merge to master.

[maxzaver]

@nearmax C1 and C2 are the same :)

Typo. Fixed.

[evgenykuzyakov]

As part of #1913, I need to make some changes to the testnet.json file. I introduces scripts/state/mega-migrate.py for continuous migrations of testnet.json. It does the following:

• introduces a config_version field, that is used for config migrations
• based on the current config_version in the testnet.json upgrades the file to move it to the latest config_version version.
• saves the testnet.json inplace with the latest version of config and sorts keys.
• saves a file testnet.json.config along-side to testnet.json for git diff reviews. This file is unused by the nearcore codebase.

[bowenwang1996]

@evgenykuzyakov why not testnet.config.json so that it is easier to view in IDE.

[evgenykuzyakov]

@bowenwang1996 This issue is obsolete right? We've solved it differently

[bowenwang1996]

it's fixed.