Skip to content

Latest commit

 

History

History
63 lines (43 loc) · 5.35 KB

usage.md

File metadata and controls

63 lines (43 loc) · 5.35 KB

Usage

Compiling a DNN model is the process of quantizing and converting the model into a format that can be inferred with TIDL. TIDL (and its open source front ends) provides utilities to compile models. The imported artifacts can then be used to run inference.

In addition to what is provided with TIDL, this repository provides igh level utilities for compiling DNN models. These tools include dataset loaders, pre-processing utilities, post-processing utilities and metric computation utilities.

Components of this repository

This repository is generic and can be used with a variety of runtimes and models supported by TIDL. This repository contains the following parts:

  • edgeai_benchmark: Core scritps for core model compilation, inference and accuracy benchmark scripts provided as a python package (that can be imported using: import edgeai_benchmark or using: from edgeai_benchmark import)
  • scripts: These are the top level scripts - to compile models, to infer and do accuracy & performance benchmark, to collect accuracy report and to package the generated artifacts.
  • configs: These are the actual configurations of to run the models in the model zoo. These configurations define the Dataset parameters, pre-processing, inference session parameters and post-processing.

Running inference / benchmark on PC using pre-compiled model artifacts

run_benchmarks_pc.sh is the main script in this repository that does compilation of models and benchmark. Usage is

run_benchmarks_pc.sh <SOC>

modelartifacts_path in the settings_base.yaml file indicates the location where the artifacts are generated or expected. It currently points to work_dirs/modelartifacts//

We have a symbolic link in this repository that points to the corresponding folder in edgeai-modelzoo.

If this is present, compilation will be skipped, those model artifacts will be downloaded automatically and inference/benchmark will be performed.

Compile models in the model zoo

run_benchmarks_pc.sh is the main script in this repository that does compilation of models and benchmark. However, there are a few things to note:

  • Model compilation can be run only on PC. The device does not support model compilation.
  • However, the inference of a compiled model can be run on PC or on device.
  • For the models in the model zoo, we have provided (link to) pre-compiled artifacts in edgeai-modelzoo and this repository has a symbolic link to that. If you would like to use the latest compiled artifacts, then pull the edgeai-modelzoo repository.
  • If you like to do the compilation of these models yourself, you can either rename the folder work_dirs/modelartifacts or change the value of modelartifacts_path in settings_base.yaml.
  • While running this script, compilation of models in the model zoo will be performed as the first step before the inference. But if the pre-compiled model artifacts are present, model compilation will be skipped. param.yaml file present in each model artifacts folder indicates that the model compilation is complete.
  • result.yaml file, if present in each model artifacts folder indicates that the model inference is complete. If result.yaml is present, inference is also skipped. Manually delete result.yaml if it is present (i.e. if you have done it once already) to do the inference - otherwise, the script will merely print the result information from result.yaml.
  • Multiple models can be compiled in parallel by setting the parameter parallel_processes in settings_import_on_pc.yaml to a higher value (for example 4 or 8)

Generate report

A CSV report containing all your benchmarking results is generated by running run_generate_report.sh

run_generate_report.sh <SOC>

Packing the artifacts to be used for inferece in the EVM

  • The imported artifacts can be used to run inference on the target device (eg. EVM)
  • If you have compiled models yourself and would like to run those model artifacts in the target device, run the script run_package_artifacts_for_evm.sh to package the artifacts for use in the target device.
run_package_artifacts_for_evm.sh <SOC>
  • These packaged artifacts can be copied to the device to run inference there.
  • Please note the above comment about result.yaml. These result files are removed while packaing the artifact to be used in EVM (as we actually want to run the inference in EVM). Instead of using the packaging script, you can use the compiled model artifact directory as well, but be sure to remove the result.yaml if you actually want the inference to run.
  • Change the modelartifacts_path in settings.yaml to point to the correct modelartifacts path. For example, after packaging, the packaged folder name will be _package. To use this packaged artifacts, set modelartifacts_path accordingly.

Sharing files between PC and EVM

  • Copying datasets and compiled artifacts to EVM can be time consuming and can use significant disk space (which may not be available on the EVM).
  • To run on EVM, we recommend to share your the datasets and work_dirs folders in your PC to the EVM and using NFS. exportfs can be used for this NFS sharing.