ci job to detect divergence (e.g. multi-node local chain)

[dckc]

To start a multi-node test chain, we have a scenario1, but it involves more infrastructure than I expected (terraform, ansible) and seemed to take more disk space than I expected.

gaia has a docker-compose based approach that is about the size and shape of what I expected. I'd like to try porting this to ag-chain-cosmos.

https://github.com/cosmos/gaia/blob/main/docs/gaia-tutorials/installation.md and https://github.com/cosmos/gaia/blob/main/docs/gaia-tutorials/deploy-testnet.md got me a 4 node cosmos network:

git clone ...
make install
gaiad version --long  # after cleaning up an old ~/.gaia
make build-linux
make build-docker-gaiadnode
make localnet-start

[dckc]

@mhofman you noted lately that you weren't sure about automated techniques for a multi-node chain. This looks pretty interesting, to me. It's different from the way @michaelfig usually does it, but I don't understand some of the motivation for his approach (terraform, ...).

cc @warner

[michaelfig]

It's different from the way @michaelfig usually does it, but I don't understand some of the motivation for his approach (terraform, ...).

The motivation is that I want to use the same process for CI testing that we actually use to deploy the devnet, just on Docker containers instead of cloud machines.

The Gaia style of testing is reasonable, too, though. Probably some combination of the two will be useful.

[Tartuffo]

Simple case will be done as part of CI. We don't currently do node restarts, but should be straightforward with Michael's new tools.

[Tartuffo]

@mhofman Please create new issue for longer term stop/restart multiple validator node test, then close this.

[Tartuffo]

Done and merged.

[mhofman]

Opened #4557 for a stop/restart based test