JRNN batch size to speed up inference?

[max-markov]

It seems like the out-of-the box version provided for JRNN (building on the original published model) is very slow, and inference for a single sentence takes quite a while, even when using a GPU (around 1 minute).

Increasing the batch size doesn't help since the inference code is buggy and only supports single samples. Changing the dimensionality in the graph definition file (graph-2016-09-10.pbtxt) leads to dimensionality mismatches downstream.

Has anyone been able to make inference faster for this model?

Thank you :)