bug: onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_4 node.

[saikrishna2893]

Describe the bug

I was trying to get inference for stable diffusion onnx {CompVis/stable-diffusion-v1-4} model, I got stuck at this error
image = pipe(prompt).images[0] File "onnx_openvpy39\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_onnx_stable_diffusion.py", line 274, in call noise_pred = self.unet(sample=latent_model_input, timestep=timestep, encoder_hidden_states=text_embeddings) File "onnx_openvpy39\lib\site-packages\diffusers\onnx_utils.py", line 62, in call return self.model.run(None, inputs) File "onnx_openvpy39\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 192, in run return self._sess.run(output_names, input_feed, run_options) onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_4 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_4_0' Status Message: C:\Users\sfatima\source\repos\onnxruntime_newmodel\onnxruntime\onnxruntime\core\providers\openvino\ov_interface.cc:36 class std::shared_ptr __cdecl onnxruntime::openvino_ep::OVCore::ReadModel(const class std::basic_string<char,struct std::char_traits,class std::allocator > &) const [OpenVINO-EP] [OpenVINO-EP] Exception while Reading network: invalid external data: ExternalDataInfo(data_full_path: weights.pb, offset: 1738007040, data_length: 13107200, sha1_digest: 0)

Trying to decode and fix the issue, any thoughts on fixing issue would be helpful.

System information

Windows 11
pip install onnxruntime-openvino=1.11.0
pip install openvino==2022.1
Python version: 3.9
onnx model from stable-diffusion : https://huggingface.co/CompVis/stable-diffusion-v1-4/tree/onnx

Trying to get inference using onnxruntime using Openvino execution provider.

Reproduction

from diffusers import OnnxStableDiffusionPipeline
import onnxruntime as rt
import openvino.utils as utils

pipe = OnnxStableDiffusionPipeline.from_pretrained(
"CompVis/stable-diffusion-v1-4",
revision="onnx",
provider="OpenVINOExecutionProvider",
provider_options=[{'device_type' : 'CPU_FP32'}]
)
prompt = "a photo of an astronaut riding a horse on mars"
#Running the session by passing in the input data of the model
image = pipe(prompt).images[0]

#Note: need to pip install diffusers transformers onnxruntime onnx torch ftfy scipy spacy in virtualenv.

Logs

No response

System Info

Windows 11

ONNX Runtime Installation : ONNX Runtime Version 1.13

ONNX Runtime API
Python 3.9

Architecture
X64

Execution Provider :
OpenVINO
Installations in virtualenv:
pip install onnxruntime-openvino=1.11.0
pip install openvino==2022.1

[saikrishna2893]

@pcuenca @bes-dev, can you suggest how to convert the stable diffusion v1.4 {CompVis} pytorch model to onnx model or openvino ir. Thanks

[averad]

How to convert the stable diffusion v1.4 {CompVis} pytorch model to onnx model: https://github.com/huggingface/diffusers/blob/main/scripts/convert_stable_diffusion_checkpoint_to_onnx.py
python convert_stable_diffusion_checkpoint_to_onnx.py --model_path="./diffusers_model" --output_path="./onnx_model"

How to Clone stable diffusion v1.4 Onnx Model from HuggingFace:
git clone https://huggingface.co/CompVis/stable-diffusion-v1-4 --branch onnx --single-branch stable_diffusion_v1-4_onnx

[anton-l]

@saikrishna2893 sorry, the OpenVINO runtime is not fully supported for Stable Diffusion yet, only CPUExecutionProvider is tested for CPU inference. For better support we will soon switch to an optimum model wrapper: huggingface/optimum-intel#122

But in case you find an openvino workaround for the current ONNX exporter linked by @averad, please open a PR! 🤗

[saikrishna2893]

How to convert the stable diffusion v1.4 {CompVis} pytorch model to onnx model: https://github.com/huggingface/diffusers/blob/main/scripts/convert_stable_diffusion_checkpoint_to_onnx.py python convert_stable_diffusion_checkpoint_to_onnx.py --model_path="./diffusers_model" --output_path="./onnx_model"

How to Clone stable diffusion v1.4 Onnx Model from HuggingFace: git clone https://huggingface.co/CompVis/stable-diffusion-v1-4 --branch onnx --single-branch stable_diffusion_v1-4_onnx

Thank you for quick response @averad, We actually tried to convert the existing onnx model on onnx tree of CompVis\stablediffusion to OpenVINO IR, but we faced issues while conversion with respect to dimensions. I will share the conversion output log soon. But yes will try to use the script provided to convert from pytorch to onnx and then from onnx to IR. I will update on the conversion.

[saikrishna2893]

@saikrishna2893 sorry, the OpenVINO runtime is not fully supported for Stable Diffusion yet, only CPUExecutionProvider is tested for CPU inference. For better support we will soon switch to an optimum model wrapper: huggingface/optimum-intel#122

But in case you find an openvino workaround for the current ONNX exporter linked by @averad, please open a PR! 🤗

Thanks for your response @anton-l. Will try to get the OpenVINO running for the ONNX model and will update. Will refer to the link provided.

[kleiti]

@saikrishna2893 sorry, the OpenVINO runtime is not fully supported for Stable Diffusion yet, only CPUExecutionProvider is tested for CPU inference. For better support we will soon switch to an optimum model wrapper: huggingface/optimum-intel#122
But in case you find an openvino workaround for the current ONNX exporter linked by @averad, please open a PR! 🤗

Thanks for your response @anton-l. Will try to get the OpenVINO running for the ONNX model and will update. Will refer to the link provided.

@saikrishna2893 There might be a bug in the converter. Try copying the unet weights.pb to the folder where you run the diffusers script from.

[saikrishna2893]

@saikrishna2893 sorry, the OpenVINO runtime is not fully supported for Stable Diffusion yet, only CPUExecutionProvider is tested for CPU inference. For better support we will soon switch to an optimum model wrapper: huggingface/optimum-intel#122
But in case you find an openvino workaround for the current ONNX exporter linked by @averad, please open a PR! 🤗

Thanks for your response @anton-l. Will try to get the OpenVINO running for the ONNX model and will update. Will refer to the link provided.

@saikrishna2893 There might be a bug in the converter. Try copying the unet weights.pb to the folder where you run the diffusers script from.

Will check and let you know. Thanks

[saikrishna2893]

@kleiti I have tried copying the weights.pb file to the folder where the script is run from, still observing the same issue.

[kleiti]

@saikrishna2893 I gave this a try in a clean virtualenv and was able to run it fairly quickly. There are updates to below modules:
pip install onnxruntime-openvino=1.13.1
pip install openvino=2022.3.0

After that, I was seeing the same issue as you with OpenVINO EP reading the network, but got around that by copying unet weights.pb to the same folder where I have the diffusers script.

[kleiti]

It even runs on my Intel integrated GPU with "device_type": "GPU.0_FP32" or "GPU.1_FP32" if you have discrete Intel card, but you might need to convert with the below command to get a smaller FP16 model:
convert_stable_diffusion_checkpoint_to_onnx.py --model_path stable-diffusion-v1-4 --output_path fp16_sd14 --fp16

[saikrishna2893]

@saikrishna2893 I gave this a try in a clean virtualenv and was able to run it fairly quickly. There are updates to below modules: pip install onnxruntime-openvino=1.13.1 pip install openvino=2022.3.0

After that, I was seeing the same issue as you with OpenVINO EP reading the network, but got around that by copying unet weights.pb to the same folder where I have the diffusers script.

@kleiti did you install older versions as mentioned in the steps above? I will also try to install newer versions in a clean virtual environment and will try running the inference. Did you change anything specific for this to be working or is it possible to share any log of the output. Actually diffusers doesnt take directly Openvino Execution provider as input, I have modified file onnx_utils.py in diffuser library to pass the specific execution provider. Kindly let me know if any other modifications have been made. Thank you in advance. I will try out your steps and will share details.

[saikrishna2893]

@kleiti In new environment with above mentioned versions when I tested out, I observe this following error. Using python 3.9.

[kleiti]

@saikrishna2893 I saw that too. For some reason, I needed to run the setupvars.bat in the below package - the import openvino.utils part in Python script didn't seem to work.

https://storage.openvinotoolkit.org/repositories/openvino/packages/2022.3/windows

[saikrishna2893]

@saikrishna2893 I saw that too. For some reason, I needed to run the setupvars.bat in the below package - the import openvino.utils part in Python script didn't seem to work.

https://storage.openvinotoolkit.org/repositories/openvino/packages/2022.3/windows

Will check and let you know.

[saikrishna2893]

@saikrishna2893 I saw that too. For some reason, I needed to run the setupvars.bat in the below package - the import openvino.utils part in Python script didn't seem to work.
https://storage.openvinotoolkit.org/repositories/openvino/packages/2022.3/windows

Will check and let you know.

@kleiti thank you very much, with above steps, using setupvars.bat file, I have activated the environment and placed weights.pb file in directory of inference script. I could successfully run the inference now.

[saikrishna2893]

@saikrishna2893 I gave this a try in a clean virtualenv and was able to run it fairly quickly. There are updates to below modules: pip install onnxruntime-openvino=1.13.1 pip install openvino=2022.3.0

After that, I was seeing the same issue as you with OpenVINO EP reading the network, but got around that by copying unet weights.pb to the same folder where I have the diffusers script.

We need to copy unet weights.pb file outside because, the unet model is saved using use_external_data flag, which saves the original model to external data file named as weights.pb, model.onnx should contain the relative path for weights.pb, but I guess its not properly getting attached. Thanks for your support.

[kleiti]

@saikrishna2893 I gave this a try in a clean virtualenv and was able to run it fairly quickly. There are updates to below modules: pip install onnxruntime-openvino=1.13.1 pip install openvino=2022.3.0
After that, I was seeing the same issue as you with OpenVINO EP reading the network, but got around that by copying unet weights.pb to the same folder where I have the diffusers script.

We need to copy unet weights.pb file outside because, the unet model is saved using use_external_data flag, which saves the original model to external data file named as weights.pb, model.onnx should contain the relative path for weights.pb, but I guess its not properly getting attached. Thanks for your support.

@anton-l Is this something that could be fixed in the ONNX converter?

[patrickvonplaten]

Also cc @echarlaix

[saikrishna2893]

@kleiti @anton-l I would like to convert the stable-diffusion v1.4 onnx model with fp32 precision to fp16 and run on CPU using openvino execution provider. Is the conversion for onnx model with fp32 precision to fp16 possible to be run on CPU? I have used conversion script link with flag --fp16, it throws error that requires gpu support. In documentation of the onnxrumtime-openvino execution provider, saw that CPU_FP16, is being supported. Just curious on this conversion of FP32 to FP16 for CPU. Checked few options of onnxruntime.transformers.optimizer link1 link2 to modify to fp16. Not successful yet. Any thoughts?

[patrickvonplaten]

cc @anton-l @echarlaix here

[saikrishna2893]

@anton-l in issue link mentioned that CPU based conversion script development is in progress. Is it possible to get inference of fp16 onnx models on onnxruntime using openvino execution provider with device_type set to CPU_FP16?

[anton-l]

@saikrishna2893 as we move the exporter and model classes to Optimum, the development of new ONNX features in diffusers is frozen. For now the exporter is in active delopment and works with fp32:
python -m optimum.exporters.onnx --model runwayml/stable-diffusion-v1-5 exported_model_dir
And the conversion to fp16 and OpenVINO should be possible with the Optimum stack, but @echarlaix will be able to comment more on that :)

[echarlaix]

Hi @saikrishna2893,

Conversion to fp16 is enabled in optimum through the ORTOptimizer but stable diffusion models are not yet supported. This could be additionally integrated in the ONNX export CLI in the future.

Concerning OpenVINO, we plan to integrate the export of stable diffusion models to the OpenVINO IR along with pipelines allowing inference using OpenVINO Runtime to optimum, you can see the progress in #122.

[echarlaix]

Conversion to fp16 is enabled in optimum through the ORTOptimizer but stable diffusion models are not yet supported. This could be additionally integrated in the ONNX export CLI in the future.

fp16 will soon be enabled in optimum ONNX export (huggingface/optimum#749)