fix(ci): fix hangs in lightwalletd tests by checking concurrent process output in different threads

[teor2345]

Motivation

lightwalletd is unstable in our CI, because zebrad hangs when its log buffer fills up.

Close #4820
Depends-On: #4826

Solution

• Clear the zebrad and lightwalletd logs in different threads, to avoid deadlocks due to the log buffer filling up
• Increase test timeouts

This increases the sync test time, but we can fix that in later PRs.

Related fixes:

• Only copy the state once in the send transactions test, to avoid a "disk full" error
• Wait longer for lightwalletd gRPC server startup, to avoid a "connection refused" error
• Look for cached state disks for this commit and branch, then the main branch, then any branch
• Do ZK parameter preloads in the lightwalletd tests that need them
• Make code execution time logs shorter

Review

This is a high-priority sync fix.

Reviewer Checklist

• Code implements Specs and Designs
• CI passes

Follow Up Tasks

• Switch zebrad to a non-blocking tracing logger #4834

[codecov]

Codecov Report

Merging #4828 (3cb3798) into main (404f682) will increase coverage by 0.07%.
The diff coverage is 57.89%.

@@            Coverage Diff             @@
##             main    #4828      +/-   ##
==========================================
+ Coverage   78.80%   78.88%   +0.07%     
==========================================
  Files         305      305              
  Lines       38733    38730       -3     
==========================================
+ Hits        30524    30551      +27     
+ Misses       8209     8179      -30     

[teor2345]

I ran 77 of these tests locally overnight, and they all succeeded.
(CI could be different, but it's still a good sign the restart seems to be working reliably.)

I just need to fix a bug in the GitHub actions where we're using an old cached state.

[teor2345]

This should all work now, we're just waiting for CI.

[teor2345]

I needed to re-do the fix, because the problem was actually deadlocks from not reading the zebrad logs.

[teor2345]

There was a google cloud SSH failure:

Timeout, server 35.222.36.92 not responding.

Recommendation: To check for possible causes of SSH connectivity issues and get
recommendations, rerun the ssh command with the --troubleshoot option. Example:

gcloud None compute ssh example-instance --zone=us-central1-a --troubleshoot

Or, to investigate an IAP tunneling issue:

gcloud None compute ssh example-instance --zone=us-central1-a --tunnel-through-iap --troubleshoot

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

https://github.com/ZcashFoundation/zebra/runs/7553151823?check_suite_focus=true#step:6:332

I've never seen one of these before, so I restarted those jobs.

[teor2345]

Admin-merging because the one failing test is going to be fixed soon.

There's just one issue with a regex in the send transactions test, I'll fix it in PR #4840.