storage: enable pre-vote by default

[irfansharif]

Tested on blue under heavy chaos for half a week, tested on ultramarine for a week. Part of #16950.

crontab for blue:

0  0-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200
48 0-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200
36 1-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200
24 2-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200
12 3-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200
0  4-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200
48 4-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200
36 5-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200
24 6-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200
12 7-23/8 * * * /home/cockroach/chaos_runner.sh 60 1200

crontab for ultramarine:

0  * * * * /home/cockroach/chaos_runner.sh 60 600
14 * * * * /home/cockroach/chaos_runner.sh 60 600
29 * * * * /home/cockroach/chaos_runner.sh 60 600
44 * * * * /home/cockroach/chaos_runner.sh 60 600

Last ultramarine chaos run can be found here. Nothing jumps out to me. Ditto for the logs on ultramarine, still present (blue was wiped out for 1.0.5 testing).

+cc @bdarnell

[cockroach-teamcity]

This change is 

[bdarnell]

There are big swings in raft append and heartbeat traffic every time a node goes down.

We don't see swings that severe on blue:

Is this because of differing configs (loadgens, chaos, etc) between blue and ultramarine? Can you run ultramarine for a while with this change reverted so we can have apples-to-apples comparisons?

[irfansharif]

Here's a run from last night: link

The first half of the graphs (between 20:28:30 - 3:43:30) is running with the pre-vote changes enabled, the second half is without.

The larger variations noted above was owed to differences in workload.

0  0-23/2 * * * /home/cockroach/chaos_runner.sh 60 600
29 0-23/2 * * * /home/cockroach/chaos_runner.sh 60 600
0 1-23/2 * * * /home/cockroach/chaos_runner.sh 60 600
29 1-23/2 * * * /home/cockroach/chaos_runner.sh 60 600

This run is running kv --concurrency 128 --tolerate-errors --min-block-bytes=8 --max-block-bytes=128 --read-percent=25 on each node.

[irfansharif]

🤞