[Bug/Model Request]: ONNX runtime exception

[elcapo]

What happened?

I've been getting a ONNX runtime error message when I attempt to call:

from fastembed import TextEmbedding
model = TextEmbedding()
embeddings = list(model.embed(["hello world"]))

The error message is:

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running SkipLayerNormalization node. Name:'SkipLayerNorm_AddBias_0' Status Message: /onnxruntime_src/include/onnxruntime/core/framework/op_kernel_context.h:42 const T* onnxruntime::OpKernelContext::Input(int) const [with T = onnxruntime::Tensor] Missing Input: encoder.layer.0.attention.output.LayerNorm.weight

Reproducible Environment

I've been obtaining the same result in different contexts, the following one being the easiest to reproduce:

# Dockerfile
FROM python:3.11-slim
RUN pip install fastembed

# docker-compose.yml
services:
  fastembed:
    build:
      context: .

With this environment, I'm using this:

docker compose run -it --rm fastembed python

... to execute the three lines of Python code described at the top of this message.

from fastembed import TextEmbedding

# when I execute this line, the model_optimized.onnx file (and others) is downloaded
model = TextEmbedding()

# when I execut this other line, I see the error message from above
embeddings = list(model.embed(["hello world"]))

What Python version are you on? e.g. python --version

Python 3.11 (via Docker)
Fastembed 0.4.1

Version

0.4.1 (Latest)

What os are you seeing the problem on?

Linux

Relevant stack traces and/or logs

Python 3.11.10 (main, Oct 19 2024, 01:04:28) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastembed import TextEmbedding
>>> model = TextEmbedding()
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 706/706 [00:00<00:00, 4.66MB/s]
special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████| 695/695 [00:00<00:00, 4.08MB/s]
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████| 1.24k/1.24k [00:00<00:00, 8.03MB/s]
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 711k/711k [00:00<00:00, 988kB/s]
model_optimized.onnx: 100%|██████████████████████████████████████████████████████████████████████████████████| 66.5M/66.5M [00:33<00:00, 1.96MB/s]
Fetching 5 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:34<00:00,  6.98s/it]
>>> embeddings = list(model.embed(["hello world"]))██████████████████████████████████| 66.5M/66.5M [00:33<00:00, 2.32MB/s]
2024-11-01 19:55:01.219086112 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running SkipLayerNormalization node. Name:'SkipLayerNorm_AddBias_0' Status Message: /onnxruntime_src/include/onnxruntime/core/framework/op_kernel_context.h:42 const T* onnxruntime::OpKernelContext::Input(int) const [with T = onnxruntime::Tensor] Missing Input: encoder.layer.0.attention.output.LayerNorm.weight

[ahaimo]

Hi,

I fixed this issue by downgrading onnxruntime to 0.19.2

[elcapo]

I fixed this issue by downgrading onnxruntime to 0.19.2

Hi @ahaimo,

Downgrading onnxruntime fixed the issue in both my Docker and Poetry environments.

Thanks!

PS: I think that you meant to refer to this onnxruntime's version: 1.19.2.

[anakin87]

I'm experiencing the same issue.

It seems that fastembed is not compatible with ONNX Runtime v1.20.0, released two days ago.

@joein could you take a look?

[joein]

Hi, it seems that onnxruntime v1.20.0 has deprecated python <3.10 and also brought in new requirements:
The list of new dependencies:

• libcudnn_adv.so.9
• libcudnn_cnn.so.9
• libcudnn_engines_precompiled.so.9
• libcudnn_engines_runtime_compiled.so.9
• libcudnn_graph.so.9
• libcudnn_heuristic.so.9
• libcudnn_ops.so.9
• libnvrtc.so.12
• libz.so.1

Source:
https://github.com/Microsoft/onnxruntime/releases/tag/v1.20.0

We'll investigate it and update the instructions how to install fastembed with the latest onnxruntime

[joein]

We'll limit onnxruntime to <v1.20 for now, there is also an issue regarding the problem in onnx repository microsoft/onnxruntime#22704

[joein]

The fix should be available in v0.4.2, if the issue persists, please re-open the issue

[raminmardani]

I receive this error.