Multilingual multitoken ASR tokens mixup during inference

[stalevna]

Hi everyone!
Did anyone try an aggregate tokenizer with two or more different tokenizers using different alphabets?
For some reason when I do an inference on such a model, I get some words that are constructed from a mix of tokens from different tokenizers. I tested my tokenizers and they don't have an intersection alphabet or token-wise.
Did anyone else have such problems? How did you fix it?