Multiple questions about how to use StyleTTS2.

[harmlessman]

Hello, I am very interested in StyleTTS2 repo.
I'd like to ask you a question about the usage.

1. The text of Data/val_list.txt is IPA text (ex 'o ʊ v ɚ w ˈɛ lm ə m ɪŋd'), so should the user prepare the text data by phonemizing the text and transforming it into IPA?

2. Are we going to phonemize the text, transform it into IPA, put it in Bert to make it embedded, and put it in the styletts2 model to learn? (Question about sequence flow)

3. Looking at examples and other people's cases, the batch size was as small as 2 or 4. When I used the vits, the batch size could be up to 64 even if the vram was around 24 to 40, but why is the batch size so small here?

4. I saw a discussion that PL-BERT, which only has English learning, should be multilingualized to support multilingualism. Is there anything else I need to do to support multilingualism?

5. Why do you use Bert for the tts model instead of word2vec that you used during the embedding process?

You can also give me a paper or link with the answers to your questions!
It may be a bothersome question, but if you know the answer, please answer it.
Thank you.

[Kreevoz]

1. Yes. Preprocess the text for your training datasets using the same phonemization + tokenization code that the inference code uses. You could use code like this: Notes on Finetuning #81 (reply in thread) and combine that with this here: https://github.com/jaywalnut310/vits/blob/main/preprocess.py as a general structure so you can feed it your list files.
2. Just provide the phonemized text, StyleTTS2 will take care of doing the rest for you.
3. The batch size is so small because the architecture is different from VITS, with a lot more complex parts, and takes a lot of VRAM to train. VITS is using big batch sizes because its component are essentially strung together bits from older TTS architectures, which by comparison each had fairly modest VRAM requirements.
4. Keep an eye on the discussions and issues related to multilanguage support. Patience. These are the early days of StyleTTS2.
5. Not qualified to answer that one, check the paper.

The paper is linked on the main page of the repo, read it here: https://arxiv.org/abs/2306.07691

[harmlessman]

Thank you so much for your kind reply.
I'll ask you if there's anything else I don't know..

[yl4579]

For 5, I didn't use word2vec at all. Could you explain your question more clearly?

[harmlessman]

It was a question about why you use Bert in the embedding process.
I found out that Bert is using it simply because it performs better than word2vec.
It was translated through a translator, so there was room for misunderstanding.

I have one more question.
Is there a big difference between the learning results when the batch size is 2 and when the batch size is 32?
In the case of other TTS models, there was an opinion that the result was unstable when the batch size was less than 16.

[yl4579]

The use of BERT is already well-established in TTS. I would recommend you read the PL-BERT paper that this project uses for more details: https://arxiv.org/pdf/2301.08810.pdf
As for the batch size, since this project uses GAN training scheme, it is expected that a small batch size actually helps with the stability.

[harmlessman]

Thank you for your reply.
If there's something I don't know, I'll ask you again!!