torch.compile compatibility

[anijain2305]

I think HuggingFace team made some excellent progress in making generate more torch.compile compatible - huggingface/transformers#29114

It might be worth revisiting torch.compile issues and see if those can be resolved now.

[fumiama]

That PR has been merged on Feb. The version restriction in ChatTTS is

transformers>=4.41.1

which has included this patch, but nothing changed.

[medemi68]

I would love to see if we could get this working or if it is possible to do. On Parler-TTS for example, compilation with reduce overhead brings 4x speed improvement. Right now even with a 4090 I’ve struggled to get good TTFB for realtime streaming. It’d be great if that could be improved somehow.

[fumiama]

Now we're implementing vLLM to speed it up.