Error when flashing LSTM model onto ESP32 (TFMIC-20)

[ChicchiPhD]

Hi Developers, hi all!

I'm making some experiments with LSTM networks using Keras, TensorFlow (specifically, tf-nightly 2.11.0-dev20220911), and TensorFlowLite. On the model authoring side, I'm able to successfully build the model, apply compression techniques and get the equivalent Lite model, and the .cc file.

However, when it's time to flash the model onto the ESP32, I always get an error, typically of the kind "Failed to get registration from op code CUSTOM". Therefore, I've decided to do something simpler, such as 1) using an existing example from Keras LSTM fusion Codelab, 2) avoiding the adoption of compression techniques.

Despite such a try, I still face an error when flashing onto ESP32, which is the following:

Didn't find op for builtin opcode 'UNIDIRECTIONAL_SEQUENCE_LSTM' version '1'. An older version of this builtin might be supported. Are you using an old TFLite binary with a newer model?
Failed to get registration from op code UNIDIRECTIONAL_SEQUENCE_LSTM

To recap:

• the existing example I'm using to generate the very simple model is from Keras LSTM fusion Codelab, which uses TensorFlow version 2.4;
• I use espressif, toolchain version esp-idf-v4.4
• my target is esp32;
• my specific HW onto which I try to flash the model is ESP32-WROOM-32.

I'm stuck on trying to understand the source of the problem, i.e., if it is a matter of TensorFlow version, or espressif version, or something that is not yet supported on the espressif side. Can you please help me try to shed a light on this?

Here attached you can find the TFLite model and the .cc generated with xxd starting from the simple Keras LSTM fusion Codelab example if this can be of help to reproduce the error.

Please, let me know if you need any further information.

Chiara
material.zip

[vikramdattu]

Hi @ChicchiPhD UNIDIRECTIONAL_SEQUENCE_LSTM is supported on current version.

Can you please make sure you have registered this OP in your code?

Something like this:

  static tflite::MicroMutableOpResolver<5> micro_op_resolver;
  micro_op_resolver.AddReshape();
  micro_op_resolver.AddSoftmax();
  micro_op_resolver.AddFullyConnected();
  micro_op_resolver.AddUnidirectionalSequenceLSTM();

and check if this works?
Reference from example: https://github.com/espressif/tflite-micro-esp-examples/blob/2a93aa3106f181768d75f64bcac629f344a2ca22/examples/person_detection/main/main_functions.cc#L94

[ChicchiPhD]

Hi Vikram, thanks for your reply! In the code there is actually static tflite::AllOpsResolver resolver; Therefore it should include all supported operOP, am I right? Il giorno mar 13 set 2022 alle ore 09:17 Vikram Dattu < ***@***.***> ha scritto:

…

Hi @ChicchiPhD <https://github.com/ChicchiPhD> UNIDIRECTIONAL_SEQUENCE_LSTM is supported on current version. Can you please make sure you have registered this OP in your code? Something like this: static tflite::MicroMutableOpResolver<5> micro_op_resolver; micro_op_resolver.AddReshape(); micro_op_resolver.AddSoftmax(); micro_op_resolver.AddFullyConnected(); micro_op_resolver.AddUnidirectionalSequenceLSTM(); and check if this works? Reference from example: https://github.com/espressif/tflite-micro-esp-examples/blob/2a93aa3106f181768d75f64bcac629f344a2ca22/examples/person_detection/main/main_functions.cc#L94 — Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYFUMNCAUXVDBMJFWGBZGZTV6AS7DANCNFSM6AAAAAAQKHYJHY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Chiara Contoli, Ph.D. Junior Assistant Professor (fixed-term) Department of Pure and Applied Sciences (DiSPeA) University of Urbino "Carlo Bo" Piazza della Repubblica 13, 61029 Urbino, Italy

[vikramdattu]

Hi @ChicchiPhD yes, you're right. AllOpsResolver should just work fine but code pulled in will be larger!

I used model from your model.cc with AllOpsResolver and this works fine.

I think you need to update your local code. Just checkout latest master from espressif/tflite-micro-esp-examples. This should solve your problem.

[ChicchiPhD]

Hi Vikram, my bad, I had misunderstood the guidelines ((For ESP32-S3 target, IDF version release/v4.4 is needed)), therefore I thought that for the ESP32 an older version was needed. I checked out the latest version, the simple example worked. I switched back to my experiment, so I've built my LSTM network, applied both pruning and quantization (specifically, fully integer) and, as I've said in my very first message, I got the equivalent Lite models, and the .cc file. When I try to flash the model onto the ESP32, I get this error also with the latest espressif version, "Failed to get registration from op code CUSTOM Allocate Tensors() Failed". Is this related to the fact that the network model contains something not supported by espressif? is there a way to discover which is, or are, the things contained in the model that are not supported? Here attached you can find two .tflite files (one it's related to the model not quantized yet and the other to the quantized one), the third one is the .cc file of the quantized model. Thanks for your support! Il giorno mar 13 set 2022 alle ore 11:17 Vikram Dattu < ***@***.***> ha scritto:

…

Hi @ChicchiPhD <https://github.com/ChicchiPhD> yes, you're right. AllOpsResolver should just work fine but code pulled in will be larger! I used model from your model.cc with AllOpsResolver and this works fine. I think you need to update your local code. Just checkout latest master from espressif/tflite-micro-esp-examples. This should solve your problem. — Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYFUMNFAPXXGM27NITWQFFLV6BBBJANCNFSM6AAAAAAQKHYJHY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Chiara Contoli, Ph.D. Junior Assistant Professor (fixed-term) Department of Pure and Applied Sciences (DiSPeA) University of Urbino "Carlo Bo" Piazza della Repubblica 13, 61029 Urbino, Italy

[vikramdattu]

Hi @ChicchiPhD I was referring to updating tflite-micro-esp-examples repo.
If this is the problem with CUSTOM ops, it is not Espressif specific as it concerns core tflite-micro framework!

Previous model you attached (with UnidirectionalSequenceLSTM) works fine for me. I cannot find attachments in your recent message. I could take a look.

[ChicchiPhD]

Hi @vikramdattu ,

Sorry, maybe uploading the attachment from the email failed, had to switch to github page to upload the zip folder again. Can you please check if you incur the same error with this file?

Thank you so much for your help.
material (1).zip

[vikramdattu]

Hey @ChicchiPhD sorry for late reply. Did you figure out yourself how to add custom ops? Do you still want me to try the model?

[chakib67100]

hi @vikramdattu sorry to disturb you. I have an issue like that when i try an implementation of my LSTM model on ESP32 : Failed to get registration from op code CUSTOM

AllocateTensors() failed
Model begin: ERROR
Guru Meditation Error: Core 0 panic'ed (Load access fault). Exception was unhandled.

There is the tflite model that ive convert into a char array for the implementation, can you help me with this please ? :
modele_sans_quantification.zip