misc: api: Annotate lotus tests according to listed behaviors

[brdji]

Related Issues

filecoin-project/system-test-matrix#279
filecoin-project/system-test-matrix#226
filecoin-project/system-test-matrix#225

Proposed Changes

Annotate all tests according to behaviors specified in the system-test-behavior and system-test-matrix. Collect these annotations so that we can gain an overview of lotus test coverage.
The System Test Matrix is a dashboard with a detailed list of FIL features and behaviors and a mapping between those features
and test suites, systems and subsystems that those features are related to.

Files and tests are annotated using //stm: comments.
Every FIL test file is marked with a special annotation at the top of the file (e.g. #unit, #integration, #cli, #api ... )
For example, //stm: #unit would mark the file as a unit test file.
The test crawler will gather all test files, including those that are not annotated using a file-level annotation. However, test coverage will be collected only from files bearing a file-level annotation.

Single tests are annotated using //stm: @ comments, with each behavior preceded by the @ character.
For example, a test with the following comment
//stm: @MARKET_RESERVE_FUNDS_001, @MINER_CREATE_BLOCK_001
is assumed to test:

• MARKET_RESERVE_FUNDS_001 behavior market system
• MINER_CREATE_BLOCK_001 behavior of the miner system

The system-test-matrix crawler will go through all lotus tests, gather information from all unit and integration tests present,
and document:

• Which behaviors are tested
• Which behaviors are untested
• Which tests cover which behaviors
• Which test files lack annotations
• etc.

This information is then compiled into an output json file, and shown in the system-test-matrix dashboard. The dashboard offers visualization and querying of test and behavior coverage of the entire lotus repository.

Additional Info

See https://github.com/filecoin-project/system-test-matrix for additional information on test annotations.

[codecov]

Codecov Report

Merging #7835 (6b8f526) into master (3e6c482) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #7835      +/-   ##
==========================================
- Coverage   39.16%   39.16%   -0.01%     
==========================================
  Files         660      660              
  Lines       71419    71419              
==========================================
- Hits        27973    27968       -5     
+ Misses      38628    38626       -2     
- Partials     4818     4825       +7     

Impacted Files	Coverage Δ
cli/util.go	41.66% <0.00%> (-8.34%)	⬇️
markets/retrievaladapter/client_blockstore.go	62.50% <0.00%> (-6.25%)	⬇️
miner/miner.go	54.75% <0.00%> (-4.27%)	⬇️
storage/wdpost_sched.go	81.37% <0.00%> (-3.93%)	⬇️
extern/sector-storage/sched_worker.go	78.19% <0.00%> (-0.88%)	⬇️
extern/sector-storage/sched.go	83.95% <0.00%> (-0.83%)	⬇️
chain/sync_manager.go	66.77% <0.00%> (-0.63%)	⬇️
chain/messagepool/messagepool.go	58.75% <0.00%> (-0.62%)	⬇️
chain/sync.go	65.00% <0.00%> (-0.34%)	⬇️
chain/stmgr/call.go	71.51% <0.00%> (ø)
... and 9 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e6c482...6b8f526. Read the comment docs.

[magik6k]

Can we deduce unit/integration based on directory?

[brdji]

I think it's possible to do so (ie. tests in the itests folder are integration tests), though I'm not sure if anything more than that is too fragile and inflexible.
I would suggest something different: the crawler can add an annotation stm: #TODO to the top of each test file that does not already have a specified annotation. These TODO's can then be shown as warnings in the dashboard, or in the CI, etc. This way, the developer is forced to fill in the correct type.