[Improvement] Test with onnx models and TensorRT engines.

[irvingzhang0512]

Fix #614

Description

• Run tools/test.py with onnx models and TesnorRT engines instead of pytorch models.
• For now, only support fixed shape and single gpu

Usage

• export onnx models by tools/pytorch2onnx.py.
• generate tensorrt engines by official tensorrt tools.
• Make sure the input tensors of onnx/tensor models and the output tensors of dataset share the same shape.
  • onnx/tensorrt model input shape is set by [--shape ${SHAPE}] in tools/pytorch2onnx.py.
  • The shape of dataset is set by videos_per_gpu(batch size) and test pipelines(all kinds of crop/resize, and num_clips for dense sampling, test augmentation like ThreeCrop/TenCrop/twice_sample, etc. )
  • It is recommended to remove all test augmentations.
    • The input shapes of onnx/tensorrt models should be 1, num_segments, 3, height, width for 2D Recognizers and 1, 1, 3, num_segments, height, width for 3D recognizers.
    • set videos_per_gpu=1 and remove test augmentation pipelines(ThreeCrop/TenCrop/twice_sample, etc.)

data = dict(
    videos_per_gpu=8,  # default value for train/val/test
    workers_per_gpu=4,
    test_dataloader=dict(videos_per_gpu=1), # specific value for test
    train=dict(...),
    val=dict(...),
    test=dict(...),
)

• Run test script

python tools/test.py /path/to/config.py /path/to/model.onnx --onnx --out test.json

python tools/test.py /path/to/config.py /path/to/model.trt --tensorrt --out test.json

TODO

• codes in test.py
• docs in getting started.
• test all kinds of models
  • tsm
  • i3d
  • tsn
  • slowonly
  • slowfast

[codecov]

Codecov Report

Merging #758 (513bfc3) into master (a2cbd11) will increase coverage by 0.04%.
The diff coverage is n/a.

❗ Current head 513bfc3 differs from pull request most recent head 8be7a9b. Consider uploading reports for the commit 8be7a9b to get more accurate results

@@            Coverage Diff             @@
##           master     #758      +/-   ##
==========================================
+ Coverage   85.15%   85.19%   +0.04%     
==========================================
  Files         130      130              
  Lines        9418     9418              
  Branches     1591     1591              
==========================================
+ Hits         8020     8024       +4     
+ Misses       1000      997       -3     
+ Partials      398      397       -1     

Flag	Coverage Δ
unittests	85.18% <ø> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmaction/datasets/pipelines/augmentations.py	94.85% <0.00%> (+0.50%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a2cbd11...8be7a9b. Read the comment docs.

[irvingzhang0512]

Any comments about this pr? @innerlee

I have no idea how to add unittest about onnx/tensorrt inference.

[dreamerlin]

cc. @RunningLeon

[RunningLeon]

@dreamerlin Currently, we only add unit tests for plugins of TensorRT.

[innerlee]

@congee524 Could you have a try? Will merge this if you can run it successfully :)

[congee524]

OKK

[congee524]

Hey, I have tested the script and passed.
Since we don't support doing average_clip after inferring the class scores in ONNX models, I think it's necessary to add notes in the docstring to remind users.
i.e. don't use ThreeCrops, TenCrops, Twice_sample etc. to test ONNX models that were transformed with our tools/pytorch2onnx.py.

[irvingzhang0512]

1. For the clip_average issue, I think we should add docs for pytorch2onnx.py, not add docs for this pr.

Maybe we should add an option to export onnx with softmax. I'll look into it.

2. I'll add some docs for the ThreeCrop/TenCrop issue.

[congee524]

Happy April Fool's Day :p

[innerlee]

Thanks!