From f15df2e4d92836b3a75e5e0ed8f748bd8f644f93 Mon Sep 17 00:00:00 2001 From: azaytsev Date: Mon, 14 Mar 2022 14:33:18 +0300 Subject: [PATCH 1/2] Minor fixes --- docs/OV_Runtime_UG/migration_ov_2_0/intro.md | 30 ++++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/OV_Runtime_UG/migration_ov_2_0/intro.md b/docs/OV_Runtime_UG/migration_ov_2_0/intro.md index b8f886087d63c7..a54a8f61dc0048 100644 --- a/docs/OV_Runtime_UG/migration_ov_2_0/intro.md +++ b/docs/OV_Runtime_UG/migration_ov_2_0/intro.md @@ -11,50 +11,50 @@ openvino_2_0_configure_devices openvino_2_0_preprocessing openvino_2_0_model_creation - + @endsphinxdirective ### Introduction Older versions of OpenVINO (prior to 2022.1) required to change the logic of applications when an user migrates from the frameworks like TensorFlow, ONNX Runtime, PyTorch, PaddlePaddle, etc. The change of application's logic is connected with: -- Model Optimizer changed input precisions for some inputs. For example, neural langauge processing models with `I64` input are becoming to have `I32` input element type. +- Model Optimizer changed input precisions for some inputs. For example, neural language processing models with `I64` input are becoming to have `I32` input element type. - Model Optimizer changed layouts for TensorFlow models (see [Layouts in OpenVINO](../layout_overview.md)). It leads to unexpected user behavior that a user needs to use a different layout for its input data with compare to the framework: ![tf_openvino] - Inference Engine API (`InferenceEngine::CNNNetwork`) also applied some conversion rules for input and output precisions because of device plugins limitations. - Users need to specify input shapes during model conversions in Model Optimizer and work with static shapes in the application. -OpenVINO Runtime API 2.0 is introduced to align logic of working with model as it is done in the frameworks - no layout and precision changes, operates with tensor names and indeces to address inputs and outputs. OpenVINO Runtime is composed of Inference Engine API used for inference and ngraph API targeted to work with models, operations. The OpenVINO API 2.0 has common structure, naming convention styles, namespaces, removes duplicated structures. See [How to migrate to OpenVINO 2.0 API](./common_inference_pipeline.md) for details. +OpenVINO Runtime API 2.0 is introduced to align logic of working with model as it is done in the frameworks - no layout and precision changes, operates with tensor names and indeces to address inputs and outputs. OpenVINO Runtime is composed of Inference Engine API used for inference and nGraph API targeted to work with models, operations. The OpenVINO API 2.0 has common structure, naming convention styles, namespaces, removes duplicated structures. See [How to migrate to OpenVINO 2.0 API](./common_inference_pipeline.md) for details. > **NOTE**: Most important is that your existing application can continue working with OpenVINO Runtime 2.0 as it used to be, but we recommend migration to new API to unlock additional features like [Preprocessing](../preprocessing_overview.md) and [Dynamic shapes support](../ov_dynamic_shapes.md). ### Introduce IR v11 -To support these features, OpenVINO introduced IR v11 which is generated by Model Optimizer by default since 2022.1. The model represented in IR v11 fully matches the original model in a original framework format in terms of inputs and outputs. Also, a user does not have to specify input shapes during the conversion, so the resulting IR v11 contains `-1` to denote undefined dimensions (see [Working with dynamic shapes](../ov_dynamic_shapes.md) to fully utilize this feature; or [Changning input shapes](../ShapeInference.md) to reshape to static shapes in the application). +To support these features, OpenVINO introduced IR v11 which is generated by Model Optimizer by default since 2022.1. The model represented in IR v11 fully matches the original model in a original framework format in terms of inputs and outputs. Also, a user does not have to specify input shapes during the conversion, so the resulting IR v11 contains `-1` to denote undefined dimensions (see [Working with dynamic shapes](../ov_dynamic_shapes.md) to fully utilize this feature; or [Changing input shapes](../ShapeInference.md) to reshape to static shapes in the application). -What is also important to mention - the IR v11 is fully compatible with old applications written with Inference Engine API from older versions of OpenVINO. This is achieved by adding additional runtime information to the IR v11 which is responsible for backwark compatible behavior. So, once the IR v11 is read by the old Inference Engine based application, it's internally converted to IR v10 to provide backward-compatible behavior. +What is also important to mention - the IR v11 is fully compatible with old applications written with Inference Engine API from older versions of OpenVINO. This is achieved by adding additional runtime information to the IR v11 which is responsible for backward compatible behavior. So, once the IR v11 is read by the old Inference Engine based application, it's internally converted to IR v10 to provide backward-compatible behavior. The IR v11 is supported by all OpenVINO Development tools including Post Training Optimization tool, Benchmark app, etc. ### IR v10 compatibility -OpenVINO Runtime API 2.0 also supports model in IR v10 for backward compatibility. So, if a user has an IR v10, such IR v10 can be fed to OpenVINO Runtime as well (see [migration steps](./common_inference_pipeline.md)). +OpenVINO Runtime API 2.0 also supports models in IR v10 for backward compatibility. So, if a user has an IR v10, it can be fed to OpenVINO Runtime as well (see [migration steps](common_inference_pipeline.md)). Some OpenVINO Development Tools also support both IR v10 and IR v11 as an input: - Accuracy checker also supports IR v10, but requires an additional option to denote which API is used underneath. - [Compile tool](../../../tools/compile_tool/README.md) compiles the model to be used in OpenVINO 2.0 API by default. If a user wants to use the resulting compiled blob in Inference Engine API, the additional `ov_api_1_0` option should be passed. -But the following OpenVINO tools don't support IR v10 as an input, they require to regenerate an IR v11 from the original model with latest Model Optimizer: -- Post Training Optimization tool -- Deep Learning WorkBench +The following OpenVINO tools don't support IR v10 as an input, and require to generate an IR v11 from the original model with the latest version of Model Optimizer: +- Post-Training Optimization tool +- Deep Learning Workbench -> **NOTE**: If IR v10 models need to be quantized, the recommendation is to download and use Post Training Optimization tool from OpenVINO 2021.4 release. +> **NOTE**: If IR v10 models need to be quantized to run with OpenVINO 2.0, the recommendation is to download and use Post Training Optimization tool from OpenVINO 2021.4 release. ### Differences between Inference Engine and OpenVINO Runtime 2.0 -Inference Engine and ngraph APIs are not deprecated, they are fully functional and can be used in applications. But OpenVINO recommends users to migrate to new OpenVINO Runtime API 2.0, because it already has additional features and this list will be extended later. The following list of additional features is supported by new API: -- [Working with dynamic shapes](../ov_dynamic_shapes.md). The feature is quite usefull for best performance for NLP (Neural Language Processing) models, super resolution models and other which accepts dynamic input shapes. -- [Preprocessing of the model](../preprocessing_overview.md) to add preprocessing operations to the inference models and fully ocupay the accelerator and free CPU resources. +Inference Engine and nGraph APIs are not deprecated, they are fully functional and can be used in applications. But OpenVINO recommends users to migrate to new OpenVINO Runtime API 2.0, because it already has additional features and this list will be extended later. The following list of additional features is supported by the new API: +- [Working with dynamic shapes](../ov_dynamic_shapes.md). The feature is quite useful for best performance for NLP (Neural Language Processing) models, super resolution models and other which accepts dynamic input shapes. +- [Preprocessing of the model](../preprocessing_overview.md) to add preprocessing operations to the inference models and fully occupy the accelerator and free CPU resources. To define a difference on the API level between Inference Engine and OpenVINO RUntime API 2.0, let's define two types of behaviors: - **Old behavior** of OpenVINO supposes: @@ -62,11 +62,11 @@ To define a difference on the API level between Inference Engine and OpenVINO RU - Inference Engine can override input and output element types. - Inference Engine API operates with operation names to address inputs and outputs (e.g. InferenceEngine::InferRequest::GetBlob). - Does not support compiling of models with dynamic input shapes. -- **New behavior** assumes full model aligment with the framework and is implemented in OpenVINO 2.0: +- **New behavior** assumes full model alignment with the framework and is implemented in OpenVINO 2.0: - Model Optimizer preserves the input element types, order of dimensions (layouts) and stores tensor names from the original models. - OpenVINO Runtime 2.0 reads models in any formats (IR v10, IR v11, ONNX, PaddlePaddle, etc) as is. - OpenVINO Runtime API 2.0 operates with tensor names. Note, the difference between tensor names and operations names is that in case if a single operation has several output tensors, such tensors cannot identified in a unique manner, so tensor names are used for addressing as it's usually done in the frameworks. - - OpenVINO Runtime API 2.0 can address input and outputs tensors also by its index. Some model formats like ONNX are sensitive to order of inputs, outputs and its preserved by OpenVINO Runtime 2.0. + - OpenVINO Runtime API 2.0 can address input and outputs tensors also by its index. Some model formats like ONNX are sensitive to order of inputs, outputs and its preserved by OpenVINO Runtime 2.0. The table below demonstrates which behavior **old** or **new** is used depending on a model source, used APIs. From e6709cc22025c609def7cd6a6625ad6ddbfefa44 Mon Sep 17 00:00:00 2001 From: azaytsev Date: Tue, 15 Mar 2022 17:24:35 +0300 Subject: [PATCH 2/2] Grammar fixes --- docs/OV_Runtime_UG/migration_ov_2_0/intro.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/OV_Runtime_UG/migration_ov_2_0/intro.md b/docs/OV_Runtime_UG/migration_ov_2_0/intro.md index d0a0c6581e60df..8b20ce0f3c19b7 100644 --- a/docs/OV_Runtime_UG/migration_ov_2_0/intro.md +++ b/docs/OV_Runtime_UG/migration_ov_2_0/intro.md @@ -24,7 +24,7 @@ Older versions of OpenVINO™ (prior to 2022.1) required to change the logic of - Inference Engine API (`InferenceEngine::CNNNetwork`) also applied some conversion rules for input and output precisions because of device plugins limitations. - Users need to specify input shapes during model conversions in Model Optimizer and work with static shapes in the application. -OpenVINO™ introduces API 2.0 to align logic of working with model as it is done in the frameworks - no layout and precision changes, operates with tensor names and indices to address inputs and outputs. OpenVINO Runtime is composed of Inference Engine API used for inference and nGraph API targeted to work with models, operations. The OpenVINO API 2.0 has common structure, naming convention styles, namespaces, removes duplicated structures. See [How to migrate to OpenVINO API v2](common_inference_pipeline.md) for details. +OpenVINO™ introduces API 2.0 to align logic of working with model as it is done in the frameworks - no layout and precision changes, operates with tensor names and indices to address inputs and outputs. OpenVINO Runtime is composed of Inference Engine API used for inference and nGraph API targeted to work with models, operations. The API 2.0 has common structure, naming convention styles, namespaces, removes duplicated structures. See [How to migrate to OpenVINO API v2](common_inference_pipeline.md) for details. > **NOTE**: Most important is that your existing application can continue working with OpenVINO Runtime 2022.1 as it used to be, but we recommend migration to API 2.0 to unlock additional features like [Preprocessing](../preprocessing_overview.md) and [Dynamic shapes support](../ov_dynamic_shapes.md). @@ -48,7 +48,7 @@ The following OpenVINO tools don't support IR v10 as an input, and require to ge - Post-Training Optimization tool - Deep Learning Workbench -> **NOTE**: If IR v10 models need to be quantized to run with OpenVINO 2.0, the recommendation is to download and use Post-Training Optimization tool from OpenVINO 2021.4 release. +> **NOTE**: If you need to quantize your IR v10 models to run with OpenVINO 2022.1, it's recommended to download and use Post-Training Optimization tool from OpenVINO 2021.4 release. ### Differences between Inference Engine and OpenVINO Runtime 2022.1 @@ -62,7 +62,7 @@ To define a difference on the API level between Inference Engine and API 2.0, le - Inference Engine can override input and output element types. - Inference Engine API operates with operation names to address inputs and outputs (e.g. InferenceEngine::InferRequest::GetBlob). - Does not support compiling of models with dynamic input shapes. -- **New behavior** assumes full model aligment with the framework and is implemented in OpenVINO 2022.1: +- **New behavior** assumes full model alignment with the framework and is implemented in OpenVINO 2022.1: - Model Optimizer preserves the input element types, order of dimensions (layouts) and stores tensor names from the original models. - OpenVINO Runtime 2022.1 reads models in any formats (IR v10, IR v11, ONNX, PaddlePaddle, etc) as is. - API 2.0 operates with tensor names. Note, the difference between tensor names and operations names is that in case if a single operation has several output tensors, such tensors cannot identified in a unique manner, so tensor names are used for addressing as it's usually done in the frameworks. @@ -72,7 +72,7 @@ The table below demonstrates which behavior **old** or **new** is used depending | API | IR v10 | IR v11 | ONNX file | Model created in code | |-------------------------------|---------|---------|-----------|-----------------------| -|Inference Engine / ngraph APIs | Old | Old | Old | Old | +|Inference Engine / nGraph APIs | Old | Old | Old | Old | |API 2.0 | Old | New | New | New | Please look at next transition guides to understand how migrate Inference Engine-based application to API 2.0: