Fixing m4t. #27240
Fixing m4t. #27240
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
| self.shared = nn.Embedding(config.vocab_size, config.hidden_size, config.pad_token_id) | ||
| self.speech_encoder = SeamlessM4TSpeechEncoder(config) | ||
| self.text_decoder = SeamlessM4TDecoder(config) | ||
| self.text_decoder = SeamlessM4TDecoder(config, self.shared) |
There was a problem hiding this comment.
If I understand it correctly, this should also appears on the SeamlessM4TForSpeechToSpeech class right ?
There was a problem hiding this comment.
Oddly enough, the shared part is not defined in the Speech Encoder/Decoder
There was a problem hiding this comment.
I'm not familiar with the models, but Iḿ guessing the text vocabulary is not needed there.
There was a problem hiding this comment.
The speech encoder doesn't have an embedding layer so need for a shared embeddings with it. The text decoder still needs it though
There was a problem hiding this comment.
You were correct I didn't read properly that model I Supposed the SpeechDecoder decoded directly to voice, but it's actually being vocoded from the text, you are correct. However my tests do pass locally... Very odd.
There was a problem hiding this comment.
OMG the test passes on CUDA but not CPU..... Witchcraft
There was a problem hiding this comment.
Let me know if I can help you here
|
Thanks @Narsil! |
* Fixing m4t. * Trying to remove comparison ? Odd test failure. * Adding shared. But why on earth does it hang ???? * Putting back the model weights checks the test is silently failing on cuda. * Fix style + unremoved comment.
What does this PR do?
Fixed m4t which had a disparity in configuration for SpeechToText.
Fixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.