Skip to content

Commit

Permalink
Update "acceleration/TensorRT_inference_acceleration.ipynb" (Project-…
Browse files Browse the repository at this point in the history
…MONAI#1668)

Fixes Project-MONAI#1667 

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

---------

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
KumoLiu and pre-commit-ci[bot] authored Apr 1, 2024
1 parent ed8854f commit bc7f27e
Showing 1 changed file with 4 additions and 14 deletions.
18 changes: 4 additions & 14 deletions acceleration/TensorRT_inference_acceleration.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,7 @@
" config_file=inference_config,\n",
" meta_file=meta_config,\n",
" logging_file=os.path.join(bundle_path, \"configs\", \"logging.conf\"),\n",
" bundle_root=bundle_path,\n",
")\n",
"\n",
"workflow.initialize()\n",
Expand Down Expand Up @@ -622,7 +623,7 @@
"metadata": {},
"source": [
"### Benchmark the TensorRT fp32 and fp16 models\n",
"In this part, the `trt_fp32_model` and `trt_fp16_model` are loaded to the `workflow`. The updated `workflow` runs the same iterations as before to benchmark the latency difference. Since the `trt_fp32_model` and `trt_fp16_model` cannot be loaded through the `CheckpointLoader` and don't have `amp` mode, the `CheckpointLoader` in the `handlers` of the `workflow` needs to be removed and the `amp` parameter in the `evaluator` of the `workflow` needs to be set to `False`.\n",
"In this part, the `trt_fp32_model` and `trt_fp16_model` are loaded to the `workflow`. The updated `workflow` runs the same iterations as before to benchmark the latency difference. Since the `trt_fp32_model` and `trt_fp16_model` cannot be loaded through the `CheckpointLoader` and don't have `amp` mode, disable the `CheckpointLoader` in the `initialize` of the `workflow` and the `amp` parameter in the `evaluator` of the `workflow` needs to be set to `False`.\n",
"\n",
"The `POST_PROCESS` and `PREPARE_BATCH` stages require a considerable amount of time. Although the model forward time is much improved, there is still room for acceleration in reducing the end-to-end latency on this particular MONAI bundle."
]
Expand All @@ -633,20 +634,9 @@
"metadata": {},
"outputs": [],
"source": [
"def pop_checkpoint_loader(handlers):\n",
" pop_index = -1\n",
" for cnt, obj in enumerate(handlers):\n",
" if isinstance(obj, monai.handlers.CheckpointLoader):\n",
" pop_index = cnt\n",
" break\n",
" if pop_index >= 0:\n",
" handlers.pop(pop_index)\n",
"\n",
"\n",
"workflow.initialize()\n",
"inference_handlers = workflow.handlers\n",
"pop_checkpoint_loader(inference_handlers)\n",
"workflow.handlers = inference_handlers\n",
"workflow.add_property(\"load_pretrain\", True, \"load_pretrain\")\n",
"workflow.load_pretrain = False\n",
"workflow.network_def = trt_fp32_model\n",
"\n",
"workflow.initialize()\n",
Expand Down

0 comments on commit bc7f27e

Please sign in to comment.