[3.2] Add logs to artifacts

[heifner]

Store off logs as Github Workflow artifacts.
This does not address everything we would like to see archived, but it is a good first step.

[linh2931]

I went to the artifacts page https://github.com/eosnetworkfoundation/mandel/actions/runs/2580037524#artifacts for this PR's CICD and downloaded forked-chain-logs and irreversible-logs. I only saw var/subprocess_results.log and etc/eosio/launcher/testnet.template. I did not see directories like var/lib/node_00, etc/eosio/node_00, ...

Not sure you need to specify all the directory levels like build/var/lib/*/* to get build/var/lib/node_00, or just build to get the whole directory. Or those node directories were somehow deleted.

[heifner]

They are cleaned up by the tests on successful runs. We might choose to add --keep-logs to our tests in the future so they are preserved for successful runs, but we do not currently do that.

[heifner]

Just kicked off: #567 so we can verify logs are saved on failed tests.

[linh2931]

I did not see logs for the failed tests in https://github.com/eosnetworkfoundation/mandel/actions/runs/2586917690#artifacts.

[spoonincode]

It looks like you need to set continue-on-error: true for the test step so that the upload step still executes when tests fail

[heifner]

It looks like you need to set continue-on-error: true for the test step so that the upload step still executes when tests fail

The continue-on-error: true would mark the test as passing when it failed. Used if: always() instead. See #567 for example run with test failures.

[spoonincode]

It seems like the upload was removed from some jobs? Like parallel and wasm tests.

[heifner]

It seems like the upload was removed from some jobs? Like parallel and wasm tests.

Yes. Those jobs do not log anything to var/* or etc/*. They only have the output from ctest. I did add -V to ctest.

[linh2931]

I saw logs now in the test PR. Just one thing. Those saved files are very big, 4GB for a test. Will the copying slow down in successful tests? I think we don't need to pay for the storage.

[heifner]

I saw logs now in the test PR. Just one thing. Those saved files are very big, 4GB for a test. Will the copying slow down in successful tests? I think we don't need to pay for the storage.

My original thought was that it might be useful to compare a successful run output, but since we are not currently passing --keep-logs to the tests, it is actually of little value. I'll switch this to if: failure()

[spoonincode]

one sec

[spoonincode]

I think you should instead tarball up (w/ zstd) these logs in to a single artifact upload (per job, of course) Based on

https://github.com/actions/upload-artifact#too-many-uploads-resulting-in-429-responses

and how long it took for my download of this zip file to start, I suspect you're really actually uploading 10+GB when the logs are uploaded

[spoonincode]

okay good news is that it is gzip compressing each file but it's still going to be a lot more efficient if we just tarball these up before sending them. In particular tar will handle those shared_memory.bin sparse files a lot better.

[spoonincode]

ugh
https://github.com/actions/upload-artifact/blob/3cea5372237819ed00197afe530f5a7ea3e805c8/dist/index.js#L7998-L8005
maybe not zstd then. or lie about the extension 🤷 Too bad they don't have a parameter to skip compression attempt