Skip to content

Commit

Permalink
build(tvm_utility): remove download logic from CMake and update docum…
Browse files Browse the repository at this point in the history
…entation (#4923)

* add include tier4_autoware_utils and dependency

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* remove downloading logic from Cmake, update documentation 

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): remove downloading logic from Cmake, update documentation 

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* build(tvm_utility): fix lint_cmake error

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): format warning message 

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): add logic to work with autoware_data folder, add nn config header and test image 

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* style(pre-commit): autofix

* build(tvm_utility): refactor, update InferenceEngineTVM constructor

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* build(tvm_utility): add lightweight model and test with it

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): make building yolo_v2_tiny disable by default

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): remove test artifact for yolo_v2_tiny

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): update docs 

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): update docs

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* build(tvm_utility): update namespace in abs_model test

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): rewrite yolo_v2_tiny as example 

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): clean comments in yolo_v2_tiny example main.cpp

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): add launch file for yolo_v2_tiny example

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): update yolo_v2_tiny example readme

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* build(tvm_utility): add model for arm based systems, need to be tested on device

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* style(pre-commit): autofix

* build(tvm_utility): update config header for arm

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* build(tvm_utility): remove debug output 

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): add find_package conditional section

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): fix lint_cmake errors

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): remove coping model files during build

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): update readme with new data folder structure

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): fix spell check warnings 

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* build(tvm_utility): add no model files guard to get_neural_network 

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* build(tvm_utility): set back default paths in config headers

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): add param file, update launch file

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): add schema file, update node name

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* build(tvm_utility): fix json-schema-check

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): fix json-schema-check

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* build(tvm_utility): add parameter table to example readme

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* build(tvm_utility): fix typo-error in description of schema.json

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

* style(pre-commit): autofix

* buiild(tvm_utility): fix spell-check warning and typo

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>

---------

Signed-off-by: Alexey Panferov <lexavtanke@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
lexavtanke and pre-commit-ci[bot] authored Oct 26, 2023
1 parent 27f28d6 commit e16ebd8
Show file tree
Hide file tree
Showing 21 changed files with 649 additions and 144 deletions.
2 changes: 1 addition & 1 deletion .cspell-partial.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@
"perception/bytetrack/lib/**"
],
"ignoreRegExpList": [],
"words": []
"words": ["dltype", "tvmgen"]
}
2 changes: 0 additions & 2 deletions common/tvm_utility/.gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +0,0 @@
artifacts/**/*.jpg
data/
80 changes: 67 additions & 13 deletions common/tvm_utility/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,19 @@ set(TVM_UTILITY_NODE_LIB_HEADERS
ament_auto_add_library(${PROJECT_NAME} SHARED ${TVM_UTILITY_NODE_LIB_HEADERS})
set_target_properties(${PROJECT_NAME} PROPERTIES LINKER_LANGUAGE CXX)

if(BUILD_TESTING)
set(BUILD_EXAMPLE OFF CACHE BOOL "enable build yolo_v2_tiny")

if(BUILD_TESTING OR BUILD_EXAMPLE)
find_package(OpenCV REQUIRED)
set(tvm_runtime_DIR ${tvm_vendor_DIR})
find_package(tvm_runtime CONFIG REQUIRED)
# Get target backend
set(${PROJECT_NAME}_BACKEND llvm CACHE STRING "${PROJECT_NAME} neural network backend")
endif()

if(BUILD_TESTING)
# compile each folder inside test/ as a test case
find_package(ament_cmake_gtest REQUIRED)
find_package(OpenCV REQUIRED)
set(tvm_runtime_DIR ${tvm_vendor_DIR})
find_package(tvm_runtime CONFIG REQUIRED)

set(TEST_ARTIFACTS "${CMAKE_CURRENT_LIST_DIR}/artifacts")
file(GLOB TEST_CASES test/*)
Expand All @@ -47,17 +51,11 @@ if(BUILD_TESTING)
endif()
# the folder name becomes the test case name
file(RELATIVE_PATH TEST_CASE_NAME ${CMAKE_CURRENT_LIST_DIR}/test ${TEST_FOLDER})

# Get neural network.
set(NN_DEPENDENCY "")
get_neural_network(${TEST_CASE_NAME} ${${PROJECT_NAME}_BACKEND} NN_DEPENDENCY)
get_neural_network(${TEST_CASE_NAME}_${CMAKE_SYSTEM_PROCESSOR} ${${PROJECT_NAME}_BACKEND} NN_DEPENDENCY)

if(NOT NN_DEPENDENCY STREQUAL "")
if(TEST_CASE_NAME STREQUAL "yolo_v2_tiny" AND
NOT EXISTS ${TEST_ARTIFACTS}/yolo_v2_tiny/test_image_0.jpg)
message(WARNING "Missing image artifact for yolo_v2_tiny, skipping test")
continue()
endif()
# add all cpp files in the folder to the target
file(GLOB TEST_CASE_SOURCES ${TEST_FOLDER}/*.cpp)
ament_add_gtest(${TEST_CASE_NAME} ${TEST_CASE_SOURCES})
Expand All @@ -75,7 +73,7 @@ if(BUILD_TESTING)
target_include_directories("${TEST_CASE_NAME}" SYSTEM PUBLIC
"${OpenCV_INCLUDE_DIRS}"
"${tvm_utility_FOUND_INCLUDE_DIRS}"
"data/models"
"data/models/${TEST_CASE_NAME}_${CMAKE_SYSTEM_PROCESSOR}"
"include"
)

Expand All @@ -93,5 +91,61 @@ if(BUILD_TESTING)
endforeach()
endif()

if(BUILD_EXAMPLE)
# compile each folder inside example/ as an example
find_package(rclcpp REQUIRED)

set(EXAMPLE_ARTIFACTS "${CMAKE_CURRENT_LIST_DIR}/artifacts")
file(GLOB EXAMPLE_CASES example/*)
foreach(EXAMPLE_FOLDER ${EXAMPLE_CASES})
if(NOT IS_DIRECTORY ${EXAMPLE_FOLDER})
continue()
endif()
# the folder name becomes the example name
file(RELATIVE_PATH EXAMPLE_NAME ${CMAKE_CURRENT_LIST_DIR}/example ${EXAMPLE_FOLDER})
# Get neural network.
set(NN_DEPENDENCY "")
get_neural_network(${EXAMPLE_NAME} ${${PROJECT_NAME}_BACKEND} NN_DEPENDENCY)

if(NOT NN_DEPENDENCY STREQUAL "")
if(EXAMPLE_NAME STREQUAL "yolo_v2_tiny" AND
NOT EXISTS ${EXAMPLE_ARTIFACTS}/yolo_v2_tiny/test_image_0.jpg)
message(WARNING "Missing image artifact for yolo_v2_tiny, skipping example")
continue()
endif()
# add all cpp files in the folder to the target
file(GLOB EXAMPLE_SOURCES ${EXAMPLE_FOLDER}/*.cpp)
ament_auto_add_executable(${EXAMPLE_NAME} ${EXAMPLE_SOURCES})
ament_target_dependencies(${EXAMPLE_NAME}
"ament_index_cpp"
"tvm_vendor"
"rclcpp"
)
add_dependencies(${EXAMPLE_NAME} ${NN_DEPENDENCY})

target_link_libraries("${EXAMPLE_NAME}"
"${OpenCV_LIBRARIES}"
"${tvm_runtime_LIBRARIES}"
)

target_include_directories("${EXAMPLE_NAME}" SYSTEM PUBLIC
"${OpenCV_INCLUDE_DIRS}"
"${tvm_utility_FOUND_INCLUDE_DIRS}"
"data/models"
"include"
)

else()
message(WARNING "No model is generated for ${EXAMPLE_FOLDER} example")
endif()

endforeach()
endif()

list(APPEND ${PROJECT_NAME}_CONFIG_EXTRAS "${PROJECT_NAME}-extras.cmake")
ament_auto_package()
# ament_auto_package()
ament_auto_package(
INSTALL_TO_SHARE
launch
config
artifacts)
59 changes: 19 additions & 40 deletions common/tvm_utility/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ The earliest supported version depends on each package making use of the inferen

#### Models

Dependent packages are expected to use the `get_neural_network` cmake function from this package in order to get the compiled TVM models.
Dependent packages are expected to use the `get_neural_network` cmake function from this package in order to build proper external dependency.

### Error detection and handling

Expand All @@ -50,76 +50,55 @@ error description.

### Neural Networks Provider

This package also provides a utility to get pre-compiled neural networks to packages using them for their inference.

The neural networks are compiled as part of the
[Model Zoo](https://github.com/autowarefoundation/modelzoo/) CI pipeline and saved to an S3 bucket.
This package exports cmake variables and functions for ease of access to those neural networks.

The `get_neural_network` function creates an abstraction for the artifact management.
The artifacts are saved under the source directory of the package making use of the function; under "data/".
Priority is given to user-provided files, under "data/user/${MODEL_NAME}/".
If there are no user-provided files, the function tries to reuse previously-downloaded artifacts.
If there are no previously-downloaded artifacts, and if the `DOWNLOAD_ARTIFACTS` cmake variable is set, they will be downloaded from the bucket.
Otherwise, nothing happens.
Users should check if model configuration header file is under "data/user/${MODEL_NAME}/". Otherwise, nothing happens and compilation of the package will be skipped.

The structure inside of the source directory of the package making use of the function is as follow:

```{text}
.
├── data
│ ├── downloads
│ │ ├── ${MODEL 1}-${ARCH 1}-{BACKEND 1}-{VERSION 1}.tar.gz
│ │ ├── ...
│ │ └── ${MODEL ...}-${ARCH ...}-{BACKEND ...}-{VERSION ...}.tar.gz
│ ├── models
│ │ ├── ${MODEL 1}
│ │ │ ├── ...
│ │ │ └── inference_engine_tvm_config.hpp
│ │ ├── ...
│ │ └── ${MODEL ...}
│ │ └── ...
│ └── user
│ └── models
│ ├── ${MODEL 1}
│ │ ├── deploy_graph.json
│ │ ├── deploy_lib.so
│ │ ├── deploy_param.params
│ │ └── inference_engine_tvm_config.hpp
│ ├── ...
│ └── ${MODEL ...}
│ └── ...
```

The `inference_engine_tvm_config.hpp` file needed for compilation by dependent packages is made available under "data/models/${MODEL_NAME}/inference_engine_tvm_config.hpp".
The `inference_engine_tvm_config.hpp` file needed for compilation by dependent packages should be available under "data/models/${MODEL_NAME}/inference_engine_tvm_config.hpp".
Dependent packages can use the cmake `add_dependencies` function with the name provided in the `DEPENDENCY` output parameter of `get_neural_network` to ensure this file is created before it gets used.

The other `deploy_*` files are installed to "models/${MODEL_NAME}/" under the `share` directory of the package.

The target version to be downloaded can be overwritten by setting the `MODELZOO_VERSION` cmake variable.

#### Assumptions / Known limits

If several packages make use of the same neural network, it will be downloaded once per package.

In case a requested artifact doesn't exist in the S3 bucket, the error message from ExternalProject is not explicit enough for the user to understand what went wrong.
The other model files should be stored in autoware_data folder under package folder with the structure:

In case the user manually sets `MODELZOO_VERSION` to "latest", the archive will not be re-downloaded when it gets updated in the S3 bucket (it is not a problem for tagged versions as they are not expected to be updated).
```{text}
$HOME/autoware_data
| └──${package}
| └──models
| ├── ${MODEL 1}
| | ├── deploy_graph.json
| | ├── deploy_lib.so
| | └── deploy_param.params
| ├── ...
| └── ${MODEL ...}
| └── ...
```

#### Inputs / Outputs

Inputs:

- `DOWNLOAD_ARTIFACTS` cmake variable; needs to be set to enable downloading the artifacts
- `MODELZOO_VERSION` cmake variable; can be used to overwrite the default target version of downloads

Outputs:

- `get_neural_network` cmake function; can be used to get a neural network compiled for a specific backend
- `get_neural_network` cmake function; create proper external dependency for a package with use of the model provided by the user.

In/Out:

- The `DEPENDENCY` argument of `get_neural_network` can be checked for the outcome of the function.
It is an empty string when the neural network couldn't be made available.
It is an empty string when the neural network wasn't provided by the user.

## Security considerations

Expand Down
6 changes: 6 additions & 0 deletions common/tvm_utility/config/yolo_v2_tiny_example.param.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
/**:
ros__parameters:
image_filename: $(find-pkg-share tvm_utility)/artifacts/yolo_v2_tiny/test_image_0.jpg
label_filename: $(find-pkg-share tvm_utility)/artifacts/yolo_v2_tiny/labels.txt
anchor_filename: $(find-pkg-share tvm_utility)/artifacts/yolo_v2_tiny/anchors.csv
data_path: $(env HOME)/autoware_data
36 changes: 36 additions & 0 deletions common/tvm_utility/data/models/abs_model_aarch64/deploy_graph.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"nodes": [
{
"op": "null",
"name": "a",
"inputs": []
},
{
"op": "tvm_op",
"name": "tvmgen_default_fused_abs",
"attrs": {
"num_outputs": "1",
"num_inputs": "1",
"flatten_data": "0",
"func_name": "tvmgen_default_fused_abs",
"hash": "1be44995aa501758"
},
"inputs": [[0, 0, 0]]
}
],
"arg_nodes": [0],
"heads": [[1, 0, 0]],
"attrs": {
"dltype": ["list_str", ["float32", "float32"]],
"device_index": ["list_int", [1, 1]],
"storage_id": ["list_int", [0, 1]],
"shape": [
"list_shape",
[
[2, 2],
[2, 2]
]
]
},
"node_row_ptr": [0, 1, 2]
}
Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
// Copyright 2021 Arm Limited and Contributors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

#include "tvm_utility/pipeline.hpp"

#ifndef COMMON__TVM_UTILITY__DATA__MODELS__ABS_MODEL_AARCH64__INFERENCE_ENGINE_TVM_CONFIG_HPP_ // NOLINT
#define COMMON__TVM_UTILITY__DATA__MODELS__ABS_MODEL_AARCH64__INFERENCE_ENGINE_TVM_CONFIG_HPP_

namespace model_zoo
{
namespace inf_test
{
namespace engine_load
{
namespace abs_model
{

static const tvm_utility::pipeline::InferenceEngineTVMConfig config{
{0, 0, 0}, // modelzoo_version

// cspell: ignore mtriple
"abs_model_aarch64", // network_name
"llvm -mtriple=aarch64-linux-gnu", // network_backend

"deploy_lib.so", // network_module_path
"deploy_graph.json", // network_graph_path
"deploy_param.params", // network_params_path

// cspell: ignore DLCPU
kDLCPU, // tvm_device_type
0, // tvm_device_id

{{"a", kDLFloat, 32, 1, {2, 2}}}, // network_inputs

{{"output", kDLFloat, 32, 1, {2, 2}}} // network_outputs
};

} // namespace abs_model
} // namespace engine_load
} // namespace inf_test
} // namespace model_zoo
#endif // COMMON__TVM_UTILITY__DATA__MODELS__ABS_MODEL_AARCH64__INFERENCE_ENGINE_TVM_CONFIG_HPP_
// NOLINT
36 changes: 36 additions & 0 deletions common/tvm_utility/data/models/abs_model_x86_64/deploy_graph.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"nodes": [
{
"op": "null",
"name": "a",
"inputs": []
},
{
"op": "tvm_op",
"name": "tvmgen_default_fused_abs",
"attrs": {
"num_outputs": "1",
"num_inputs": "1",
"flatten_data": "0",
"func_name": "tvmgen_default_fused_abs",
"hash": "1be44995aa501758"
},
"inputs": [[0, 0, 0]]
}
],
"arg_nodes": [0],
"heads": [[1, 0, 0]],
"attrs": {
"dltype": ["list_str", ["float32", "float32"]],
"device_index": ["list_int", [1, 1]],
"storage_id": ["list_int", [0, 1]],
"shape": [
"list_shape",
[
[2, 2],
[2, 2]
]
]
},
"node_row_ptr": [0, 1, 2]
}
Binary file not shown.
Binary file not shown.
Loading

0 comments on commit e16ebd8

Please sign in to comment.