TF Lite support #1224

farwayer · 2018-02-11T00:50:38Z

I tried to convert pb to tflite format but look like current model uses some operations unsupported in TF Lite:

bazel-bin/tensorflow/contrib/lite/toco/toco --input_file=/mnt/data/dev/tf-data/output_graph.pb --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --output_file=/tmp/out.lite --output_arrays=logits --input_arrays=input_lengths,input_node

Some of the operators in the model are not supported by the standard TensorFlow Lite runtime. If you have a custom implementation for them you can disable this error with --allow_custom_ops. Here is a list of operators for which you will need custom implementations: CAST, Enter, Exit, ExpandDims, Fill, LoopCond, Range, ReverseSequence, SPLIT, Select, Stack, TensorArrayGatherV3, TensorArrayReadV3, TensorArrayScatterV3, TensorArraySizeV3, TensorArrayV3, TensorArrayWriteV3, TensorFlowGreaterEqual, TensorFlowLess, TensorFlowMerge, TensorFlowMinimum, TensorFlowShape, TensorFlowSwitch.

It can be hard but IMO adaptation model to TF Lite is necessary for using DeepSpeech on low resource devices such as mobile devices.

The text was updated successfully, but these errors were encountered:

lissyx · 2018-02-11T09:18:30Z

Working on TF Lite is actually on the roadmap, as you can infer from #1213, but still, I have not yet had time to actually to it, so thanks for the feedback. Model changes can be hard, I agree.

lissyx · 2018-03-05T18:23:18Z

@reuben So, working with toco from master (~ 1.6.0~ish) on the streaming model, only those ops are problem:

VariableV2,
Assign,
Select

reuben · 2018-03-05T18:54:09Z

Sounds like the same problem as XLA. Variable sequence lengths cause us to depend on Select, and we use VariableV2 and Assign to store and update the RNN state. Both are can be worked around with some effort.

lissyx · 2018-04-30T12:24:50Z

During latest experiments, we endup being only blocked by the lack of TensorFlowMinimum. According to tensorflow/tensorflow#17289 this was fixed upstream. We should give it a new try.

JaviBonilla · 2018-05-15T11:27:44Z

So, are the Stack, ResizeNearestNeighbor, and TensorFlowShape operators supported by TensorFlow Lite? or were they removed from the streaming model?

lissyx · 2018-05-15T11:58:24Z

@JaviBonilla Looks like they were removed by the streaming model :)

sshabana · 2018-08-30T11:12:14Z

Porting Deepspeech model to android

I am trying to convert the deepspeech model output_graph.pb to tflite using toco using the below command

bazel-bin/tensorflow/contrib/lite/toco/toco --input_file=output_graph.pb --output_file=output_graph_fr.tflite --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --inference_type=QUANTIZED_UINT8 --input_array=input_node --output_array=output_node --allow_custom_ops

i am using the deepspeech model from the link https://github.com/mozilla/DeepSpeech

i get the following error

2018-08-30 16:52:09.853405: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Equal
2018-08-30 16:52:09.853572: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayV3
2018-08-30 16:52:09.853600: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayV3
2018-08-30 16:52:09.853661: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayScatterV3
2018-08-30 16:52:09.853685: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:09.853704: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:09.853722: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:09.853739: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:09.853769: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:09.853791: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: LoopCond
2018-08-30 16:52:09.853839: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:09.853860: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:09.853879: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayReadV3
2018-08-30 16:52:10.027703: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.027749: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.027812: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.027848: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.027903: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.027932: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayWriteV3
2018-08-30 16:52:10.027979: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Exit
2018-08-30 16:52:10.028003: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArraySizeV3
2018-08-30 16:52:10.028044: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayGatherV3
2018-08-30 16:52:10.028071: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: ReverseSequence
2018-08-30 16:52:10.028189: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Equal
2018-08-30 16:52:10.028325: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayV3
2018-08-30 16:52:10.028357: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayV3
2018-08-30 16:52:10.028421: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayScatterV3
2018-08-30 16:52:10.028449: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.028473: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.028495: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.028516: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.028550: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.028576: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: LoopCond
2018-08-30 16:52:10.028627: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.028652: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.028674: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayReadV3
2018-08-30 16:52:10.198055: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.198113: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.198182: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.198214: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.198265: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Enter
2018-08-30 16:52:10.198294: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayWriteV3
2018-08-30 16:52:10.198336: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: Exit
2018-08-30 16:52:10.198355: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArraySizeV3
2018-08-30 16:52:10.198394: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: TensorArrayGatherV3
2018-08-30 16:52:10.198422: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: ReverseSequence
2018-08-30 16:52:10.227836: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1366] Converting unsupported operation: CTCBeamSearchDecoder
2018-08-30 16:52:10.270343: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 213 operators, 348 arrays (0 quantized)
2018-08-30 16:52:10.272146: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After Removing unused ops pass 1: 203 operators, 332 arrays (0 quantized)
2018-08-30 16:52:10.274541: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 203 operators, 332 arrays (0 quantized)
2018-08-30 16:52:10.429472: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 173 operators, 299 arrays (1 quantized)
2018-08-30 16:52:10.543914: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 2: 171 operators, 295 arrays (1 quantized)
2018-08-30 16:52:10.546528: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 3: 169 operators, 291 arrays (1 quantized)
2018-08-30 16:52:10.549369: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before pre-quantization graph transformations: 169 operators, 291 arrays (1 quantized)
2018-08-30 16:52:10.550791: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After pre-quantization graph transformations pass 1: 169 operators, 291 arrays (1 quantized)
2018-08-30 16:52:10.552826: F tensorflow/contrib/lite/toco/tooling_util.cc:1589] Array Relu, which is an input to the TensorFlowMinimum operator producing the output array Minimum, is lacking min/max data, which is necessary for quantization. Either target a non-quantized output format, or change the input graph to contain min/max information, or pass --default_ranges_min= and --default_ranges_max= if you do not care about the accuracy of results.
Aborted (core dumped)

i have few questions and i have tried the following experiments:

a)Error thrown for operator relu is supposed to be coming form tooling_util.cc from the function "CheckIsReadyForQuantization", line 1590. the relu operator is not getting the input from the graph. is there any default min max value we should specify or enable some datatype for the operator.

b)Some of the operators are not supported by the TensorFlowLiteRuntime Like CTCBeamSearchDecoder,Enter ,TensorArrayV3 etc. I suppose all these have to be supported by the tensorflow so that the tflite model is converted & inferenced successfully. Is there a reference on how to add these operators, or any guideline or help possible to add a custom operator.

c) i also tried this experiment. i used the quantize_weights and allow_custom_ops flags and built the tflite model for deepspeech using the below command :

bazel-bin/tensorflow/contrib/lite/toco/toco --input_file=output_graph.pb --output_file=output_graph_fr.tflite --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --inference_type=FLOAT --input_array=input_node --output_array=output_node --allow_custom_ops --quantize_weights

This went ahead and build the tflite model for the deepseech even though it showed the log "converting unsupported operation". After i port the tflite model in java classifier and run, i get the below error: Mapped Bytebuffer is not a valid flatbuffer model. Hence we are back to enabling these operators.

Any help or direction in this regard will be appreciated.

reuben · 2018-08-30T11:13:55Z

The streaming model with a few modifications only has supported OPs, but it runs into this problem: tensorflow/tensorflow#21924

We're actively looking into TFLite support.

sshabana · 2018-08-31T06:13:55Z

Thanks reuben. what is the difference between the streaming model and the original ds model ?? is it just that the unsupported ops have been removed or are they handled in any other way??

kdavis-mozilla · 2018-08-31T06:18:14Z

@sshabana The core difference is that the streaming model is now a RNN and not a BRNN. Thus it no longer requires the entire audio clip to start processing.

sshabana · 2018-08-31T06:21:49Z

Thanks kdavis. Does this change of model to RNN imply that the operators which were not supported does not have to enabled any more ?? And thus making it suitable to implement the Tflite model easily ??we are really interested to get this working on our Android based development kit.

kdavis-mozilla · 2018-08-31T06:32:56Z

@sshabana Yes this exactly what it implies. However, it runs in to the problem @reuben mentioned above.

We are also very interested in making this work on our Android based development kit too. If you have any insight on tensorflow/tensorflow#21924 it will help us both!

sshabana · 2018-10-16T05:02:39Z

hey kdavis, even with the tflite deepspeech streaming model, i am still seeing the issue in java classifier i get the below error: Mapped Bytebuffer is not a valid flatbuffer model. Now that all operators are enabled on a streaming model, why shd we have such issues. Any insights in this will be helpful. Thanks

lissyx · 2018-10-22T14:24:07Z

hey kdavis, even with the tflite deepspeech streaming model, i am still seeing the issue in java classifier i get the below error: Mapped Bytebuffer is not a valid flatbuffer model. Now that all operators are enabled on a streaming model, why shd we have such issues. Any insights in this will be helpful. Thanks

You are passing the wrong kind of model, according to the error. TFLite requires a flatbuffer model, processed by toco.

lissyx · 2018-10-22T14:24:43Z

Test run on Pixel 2 device:

$ adb shell /data/local/tmp/lite_benchmark_model --graph=/data/local/tmp/output_graph_tflite_2048.pb --show_flops --input_layer=input_node,previous_state_c,previous_state_h --input_layer_type=float,float,float --input_layer_shape=1,16,19,26:1,2048:1,2048 --output_layer=logits --max_num_runs=10000 --max_time="10.0" --show_sizes=true --show_time=true
STARTING!
Num runs: [50]
Inter-run delay (seconds): [-1]
Num threads: [1]
Benchmark name: []
Output prefix: []
Warmup runs: [1]
Graph: [/data/local/tmp/output_graph_tflite_2048.pb]
Input layers: [input_node,previous_state_c,previous_state_h]
Input shapes: [1,16,19,26:1,2048:1,2048]
Use nnapi : [0]
Loaded model /data/local/tmp/output_graph_tflite_2048.pb
resolved reporter
Initialized session in 35.283ms
Running benchmark for 1 iterations
count=1 curr=559370

Running benchmark for 50 iterations
count=50 first=495628 curr=504729 min=493421 max=511619 avg=502164 std=4554

Average inference timings in us: Warmup: 559370, Init: 35283, no stats: 502164

lissyx · 2018-10-22T14:25:14Z

But NNAPI is still not yet for us:

$ adb shell /data/local/tmp/lite_benchmark_model --graph=/data/local/tmp/output_graph_tflite_2048.pb --show_flops --input_layer=input_node,previous_state_c,previous_state_h --input_layer_type=float,float,float --input_layer_shape=1,16,
19,26:1,2048:1,2048 --output_layer=logits --max_num_runs=10000 --max_time="10.0" --show_sizes=true --show_time=true --use_nnapi=true
STARTING!
Num runs: [50]
Inter-run delay (seconds): [-1]
Num threads: [1]
Benchmark name: []
Output prefix: []
Warmup runs: [1]
Graph: [/data/local/tmp/output_graph_tflite_2048.pb]
Input layers: [input_node,previous_state_c,previous_state_h]
Input shapes: [1,16,19,26:1,2048:1,2048]
Use nnapi : [1]
Loaded model /data/local/tmp/output_graph_tflite_2048.pb
resolved reporter
Initialized session in 33.27ms
Running benchmark for 1 iterations
NNAPI doesn't support tensors with rank 0 (index 7 name Minimum/y)
Returning error since NNAPI returned failure nnapi_delegate.cc:716.
Failed to build graph for NNAPI
Failed to invoke!
Aborted

lissyx · 2018-10-23T08:44:07Z

Lots of roadblocks on NNAPI path. So far:

tf.minimum() triggers NNAPI doesn't support tensors with rank 0 (index 7 name Minimum/y)
tf.unstack() triggers Unpack op that is not supported
manually removing unstack generated StridedSlice ops that are also not supported

lissyx · 2018-10-30T18:24:39Z

It's not yet complete enough to close that bug, but #1687 will allow to directly export from training a model that is ready for TF Lite. Still have to run it through toco etc, but we will add that later.

lissyx · 2018-10-31T15:07:38Z

With #1690 it will directly export as tflite format. Requires a bump to TensorFlow r1.12

lissyx · 2018-11-01T09:59:30Z

@farwayer You can now just train and --export_dir ... --export_tflite, this should get you something :)

lissyx · 2018-12-13T18:50:03Z

We have tooling to export, and we have TF Lite available for Android builds now. Support is still early, no CI or anything, this will come later. But the feature is here.

lock · 2019-01-13T05:22:12Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

farwayer mentioned this issue Feb 11, 2018

Building Native Client for Android #1215

Closed

lissyx added enhancement Priority: P1 labels Feb 11, 2018

lissyx mentioned this issue Mar 19, 2018

DeepSpeech exporting INT64 instead of expected INT32 into SparseToDence opp, not friendly to mobile TensorFlow. #1309

Closed

lissyx closed this as completed Dec 13, 2018

lock bot locked and limited conversation to collaborators Jan 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TF Lite support #1224

TF Lite support #1224

farwayer commented Feb 11, 2018

lissyx commented Feb 11, 2018

lissyx commented Mar 5, 2018 •

edited

Loading

reuben commented Mar 5, 2018

lissyx commented Apr 30, 2018

JaviBonilla commented May 15, 2018

lissyx commented May 15, 2018

sshabana commented Aug 30, 2018

reuben commented Aug 30, 2018

sshabana commented Aug 31, 2018

kdavis-mozilla commented Aug 31, 2018

sshabana commented Aug 31, 2018

kdavis-mozilla commented Aug 31, 2018

sshabana commented Oct 16, 2018

lissyx commented Oct 22, 2018

lissyx commented Oct 22, 2018

lissyx commented Oct 22, 2018

lissyx commented Oct 23, 2018

lissyx commented Oct 30, 2018

lissyx commented Oct 31, 2018

lissyx commented Nov 1, 2018

lissyx commented Dec 13, 2018

lock bot commented Jan 13, 2019

TF Lite support #1224

TF Lite support #1224

Comments

farwayer commented Feb 11, 2018

lissyx commented Feb 11, 2018

lissyx commented Mar 5, 2018 • edited Loading

reuben commented Mar 5, 2018

lissyx commented Apr 30, 2018

JaviBonilla commented May 15, 2018

lissyx commented May 15, 2018

sshabana commented Aug 30, 2018

reuben commented Aug 30, 2018

sshabana commented Aug 31, 2018

kdavis-mozilla commented Aug 31, 2018

sshabana commented Aug 31, 2018

kdavis-mozilla commented Aug 31, 2018

sshabana commented Oct 16, 2018

lissyx commented Oct 22, 2018

lissyx commented Oct 22, 2018

lissyx commented Oct 22, 2018

lissyx commented Oct 23, 2018

lissyx commented Oct 30, 2018

lissyx commented Oct 31, 2018

lissyx commented Nov 1, 2018

lissyx commented Dec 13, 2018

lock bot commented Jan 13, 2019

lissyx commented Mar 5, 2018 •

edited

Loading