MBart model cannot be loaded

[thies1006]

System Info

optimum @ git+https://github.com/huggingface/optimum.git@3347a0a75f18b854979dd7e9f78d4c3ebb92852a
transformers==4.21.3
onnxruntime==1.12.1

Who can help?

@lewtun, @michaelbenayoun

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer, pipeline

import os
os.makedirs("/tmp/onnx_model")

model = ORTModelForSeq2SeqLM.from_pretrained("facebook/mbart-large-en-ro",from_transformers=True)
model.save_pretrained("/tmp/onnx_model")
model = ORTModelForSeq2SeqLM.from_pretrained("/tmp/onnx_model")

Error:

Traceback (most recent call last):
  File "convert.py", line 12, in <module>
    model = ORTModelForSeq2SeqLM.from_pretrained("/tmp/onnx_model")
  File "/secondary/thies/.virtualenvs/onnx/lib/python3.8/site-packages/optimum/modeling_base.py", line 237, in from_pretrained
    return cls._from_pretrained(
  File "/secondary/thies/.virtualenvs/onnx/lib/python3.8/site-packages/optimum/onnxruntime/modeling_seq2seq.py", line 316, in _from_pretrained
    model = cls.load_model(
  File "/secondary/thies/.virtualenvs/onnx/lib/python3.8/site-packages/optimum/onnxruntime/modeling_seq2seq.py", line 213, in load_model
    decoder_session = onnxruntime.InferenceSession(str(decoder_path), providers=[provider])
  File "/secondary/thies/.virtualenvs/onnx/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 347, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/secondary/thies/.virtualenvs/onnx/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 395, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Deserialize tensor onnx::MatMul_3788 failed.Invalid fd was supplied: -1

Expected behavior

.

[JingyaHuang]

By reproducing your code, I got the following output ONNX models. It seems that the decoders are only model proto and the external data files are missing.

ORTModels might need to improve the export when the model size exceeds 2GB. Will ask internally and make a fix ASAP.

[JingyaHuang]

Gently tagging @mht-sharma as you are working on the encoder-decoder new exporter #497. FYI there was a bug when exporting the seq2seq model exceeding 2GB, external files seem not have been correctly stored.

[mht-sharma]

Thanks @JingyaHuang. The issue is that the external data files are not copied when the save_pretrained method is called. There is a PR 255 open from @NouamaneTazi which tries to tackle the issue.

[fxmarty]

Fixed in #586