R4R: Slashing, validator set, and governance simulation #1783

cwgoes · 2018-07-20T22:10:02Z

Continuation of #1620.

This PR does the following:

Adds simulated messages for x/gov and x/slashing
Simulates validators signing/missing blocks (with a simple Markov chain written by @ValarDragon)
Simulates Tendermint validator set update logic
Simulates enough time for unbonding to happen
Simulates double-signatures (occaisionally)
Adds in several methods to make long-running simulations faster

Will include:

Simulated validator set & simulated validator signatures (or missed signatures!) on each block
Simulated random amount of time elapsed per block
Simulated slashing msgs
Simulated governance msgs

Want to complete before R4R/merge:

Clearer logs for slashing and governance tally events (ensure that we've actually tested these)
Simplify simulation parameter choices (~~or interactive CLI!~~)
Cleanup the app.Commit() logic (avoid the actual multistore commit)

To be completed in a followup PR:

edit: Wishlist moved to #1924.

Standard checklist:

Updated all relevant documentation (docs/)
Updated all relevant code comments
Wrote tests
Added entries in PENDING.md
Updated cmd/gaia and examples/

For Admin Use:

Added appropriate labels to PR (ex. wip, ready-for-review, docs)
Reviewers Assigned
Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)

codecov · 2018-07-20T22:16:37Z

Codecov Report

Merging #1783 into release/v0.24.0 will decrease coverage by 1.06%.
The diff coverage is 40.74%.

@@                 Coverage Diff                 @@
##           release/v0.24.0    #1783      +/-   ##
===================================================
- Coverage            64.86%   63.79%   -1.07%     
===================================================
  Files                  115      113       -2     
  Lines                 6862     6671     -191     
===================================================
- Hits                  4451     4256     -195     
- Misses                2127     2132       +5     
+ Partials               284      283       -1

ValarDragon · 2018-07-25T17:49:27Z

cmd/gaia/app/sim_test.go

@@ -125,7 +125,7 @@ func TestFullGaiaSimulation(t *testing.T) {
 		},
 		[]simulation.RandSetup{},
 		[]simulation.Invariant{
-			allInvariants,
+			simulation.PeriodicInvariant(allInvariants, 100, 0),


Would it make sense to break up these invariants more? i.e. bank invariant every 100, staking/slashing every time?

Possibly! Currently optimized for ease of changing periodicity...

In addition to what I previously suggested, maybe we should have a separate convenient debug full simulation function which takes in the seed and then run the invariants every time. (i.e. I'd prefer if we had a very easy way to see exactly what caused that randomized error once we've found an error)

We can search by bisection.

(and also have a way to run the invariants every time, sure)

ValarDragon · 2018-08-15T18:03:48Z

Makefile

@@ -136,7 +136,7 @@ test_sim_modules:

 test_sim_gaia_fast:
 	@echo "Running full Gaia simulation. This may take several minutes..."
-	@go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=200 -timeout 2h
+	@go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=200 -timeout 24h


This shouldn't be fast. Perhaps make a test_sim_gaia_release, and have that run only on merges to master

alexanderbez

Thanks @cwgoes! Left some feedback on UX and non-determinism.

alexanderbez · 2018-08-16T02:49:46Z

Makefile

+	@echo "Running individual module simulations..."
+	@go test $(PACKAGES_SIMTEST)
+
+test_sim_gaia_fast:


I'm not able to send a successful SIGINT to this command.

Good point! We should resolve this, but I strongly think we should get this PR merged, and fix this once its merged. Reviewing / rebasing / etc. is becoming a pain, and it will be much easier to parallelize work once its on develop. Mentioned your point about SIGINT #1924

Just send SIGKILL!

alexanderbez · 2018-08-16T02:49:52Z

Makefile

+	@echo "Running full Gaia simulation. This may take several minutes..."
+	@go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=1000 -v -timeout 24h
+
+test_sim_gaia_slow:


alexanderbez · 2018-08-16T02:51:49Z

x/mock/simulation/random_simulate_blocks.go

 ) {
 	log := fmt.Sprintf("Starting SimulateFromSeed with randomness created with seed %d", int(seed))
-	keys, addrs := mock.GeneratePrivKeyAddressPairs(numKeys)
+	fmt.Printf("%s\n", log)
+	keys, accs := mock.GeneratePrivKeyAddressPairs(numKeys)


This is non-deterministic regardless of seed which results in different state hashes in each iteration.

Reference:

func GenPrivKey() PrivKeyEd25519 { return genPrivKey(crypto.CReader()) }

/cc @ValarDragon

Fixed in #2059 , but lets get this PR merged in first.

ValarDragon · 2018-08-16T04:17:56Z

Can we remove testing this from CLI, and get this merged in? We can open a new PR for bug fixes in the interrim. I think just getting the code here merged is valuable so that we can work on this in a way that is easier to review.

I'll write the fix for the non-determinism in a separate PR aimed at this one (with the intent of being rebased onto develop once this is merged in)

ValarDragon · 2018-08-16T05:10:42Z

appState, validators, err := app.ExportAppStateAndValidators() fails on randomized testing gaia instances, but lets save debugging that for a separate PR.

cwgoes · 2018-08-16T11:14:10Z

appState, validators, err := app.ExportAppStateAndValidators() fails on randomized testing gaia instances, but lets save debugging that for a separate PR.

Did you call InitGenesis?

cwgoes added 4 commits July 20, 2018 21:20

Simulate time passing

15a5cd5

Print out final height & time

2a04c04

Simulate validator set changes

79e870a

Simulate different block loads

a1ea7cc

cwgoes requested review from ebuchman and rigelrozanski as code owners July 20, 2018 22:10

cwgoes added the wip label Jul 20, 2018

cwgoes added 15 commits July 23, 2018 20:35

Merge branch 'develop' into cwgoes/slashing-simulation

727b7bd

Random SigningValidators

9c55ec5

Simple random Byzantine evidence

782fcd7

Option to skip RequestBeginBlock randomization

e4d0b17

Initial genesis validator in simulation

0f85d6c

Helpful printouts

d2709e2

Better load simulation

6c07f4a

Nicer logs

01264b9

Linter fixes; 'make format'

d7f1553

Delay RequestBeginBlock.SigningValidators one block; nicer formatting

738009c

Period invariant checks

b032b8e

SimulateMsgUnrevoke

2bf2c9a

Simulated x/gov msgs, more initially bonded

e69f7fc

Merge branch 'develop' into cwgoes/slashing-simulation

3a4aed5

Linter fixes; 'make format'

9a07c07

cwgoes requested review from jbibla and zramsay as code owners July 25, 2018 01:47

cwgoes added 4 commits July 25, 2018 05:31

Somehow nondeterministic

be70a21

Debugging contd.

d151988

Merge branch 'develop' into cwgoes/slashing-simulation

f0cff46

Fix nondeterminism but now we found a bug!

5058379

ValarDragon reviewed Jul 25, 2018

View reviewed changes

cwgoes added 6 commits August 13, 2018 14:07

Merge branch 'develop' into cwgoes/slashing-simulation

2b703c9

Remove simulation from 'make test_cover'

b40c9e8

.ABCIValidator() includes address

198f0f3

Minor fixes

f172dd6

Remove from CircleCI test_cover (why is this different)

63762e9

Merge branch 'release/v0.24.0' into cwgoes/slashing-simulation

11ae0f3

cwgoes changed the base branch from develop to release/v0.24.0 August 15, 2018 12:15

cwgoes added 2 commits August 15, 2018 14:37

Update Gopkg.lock

70cc9bd

Change timeout on 'make test_sim_gaia_fast'

44c5ae7

ValarDragon reviewed Aug 15, 2018

View reviewed changes

cwgoes added 4 commits August 15, 2018 23:05

testing...

4bd01b0

Merge branch 'release/v0.24.0' into cwgoes/slashing-simulation

6e58500

Some logs for stake bug

1e1f337

Add more debugging

c13dd76

alexanderbez suggested changes Aug 16, 2018

View reviewed changes

ValarDragon mentioned this pull request Aug 16, 2018

Randomized Simulation wishlist #1924

Closed

Make randomized simulation deterministic from a given seed (#2059)

93060d7

cwgoes added 3 commits August 16, 2018 13:27

Update Gopkg.lock

54ee496

Zero inflation on stake module test; additional event

617a503

Remove accidental change from develop

169260f

This was referenced Aug 16, 2018

Panic in staking caught by randomized simulation #2063

Closed

Release/v0.24.0 #1991

Merged

Remove printfs for merge into release/v0.24.0

eb3e623

cwgoes merged commit 3d50567 into release/v0.24.0 Aug 16, 2018

cwgoes deleted the cwgoes/slashing-simulation branch August 16, 2018 15:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

R4R: Slashing, validator set, and governance simulation #1783

R4R: Slashing, validator set, and governance simulation #1783

cwgoes commented Jul 20, 2018 •

edited

Loading

codecov bot commented Jul 20, 2018 •

edited

Loading

ValarDragon Jul 25, 2018

cwgoes Jul 25, 2018

ValarDragon Jul 25, 2018 •

edited

Loading

cwgoes Jul 25, 2018

cwgoes Jul 25, 2018

ValarDragon Aug 15, 2018

alexanderbez left a comment •

edited

Loading

alexanderbez Aug 16, 2018

ValarDragon Aug 16, 2018 •

edited

Loading

cwgoes Aug 16, 2018

alexanderbez Aug 16, 2018

alexanderbez Aug 16, 2018 •

edited

Loading

ValarDragon Aug 16, 2018

ValarDragon commented Aug 16, 2018 •

edited

Loading

ValarDragon commented Aug 16, 2018

cwgoes commented Aug 16, 2018

R4R: Slashing, validator set, and governance simulation #1783

R4R: Slashing, validator set, and governance simulation #1783

Conversation

cwgoes commented Jul 20, 2018 • edited Loading

codecov bot commented Jul 20, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValarDragon Jul 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexanderbez left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValarDragon Aug 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexanderbez Aug 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValarDragon commented Aug 16, 2018 • edited Loading

ValarDragon commented Aug 16, 2018

cwgoes commented Aug 16, 2018

cwgoes commented Jul 20, 2018 •

edited

Loading

codecov bot commented Jul 20, 2018 •

edited

Loading

ValarDragon Jul 25, 2018 •

edited

Loading

alexanderbez left a comment •

edited

Loading

ValarDragon Aug 16, 2018 •

edited

Loading

alexanderbez Aug 16, 2018 •

edited

Loading

ValarDragon commented Aug 16, 2018 •

edited

Loading