Confusing characters after transform emoji using text-transformation

[elszczepano]

Steps to reproduce:

1. Add custom text transformation e.g.:

typing: {
    transformations: {
        extra: [
	    { from: ':)', to: '🙂' },
	    { from: ':+1:', to: '👍' },
	    { from: ':tada:', to: '🎉' }
        ]
    }
}

2. Type e.g :tada: and then push space a few times.

[pjasiun]

You need very specific content to reproduce it. You can reproduce it with:

far from the truth Even the studies that conducted amost half a century show that the language you speak has more effects on you then you realise.

One of the very first experiments conducted on this topic dates back to 1964.

But it enough to remove a letter (any letter) form the first sentence and the bug does not appear anymore. Magic.

[Mgsy]

It looks like it occurs when a paragraph contains 148 or more characters.

[elszczepano]

And bug doesn't occur when you add an emoji in the middle of a paragraph. It also occurs for headings.

[pjasiun]

Also, you need to add an emoji at the end of the first paragraph, after the bolded text. It does not occur when you add it in the middle of the paragraph.

[Reinmar]

If it starts to happen at 148 characters, my guess will be the diff() function. At some threshold, it starts to use the fastDiff() function and my guess is that it doesn't handle two-byte unicode characters too well.

[mlewand]

BTW it's a cross-browser bug.

[oskarwrobel]

Looks like multi-byte characters may be a problem here. When the paragraph is long enough and you start to type after a multi-byte character (It doesn't have to be directly after but in the same paragraph) then a character before the selection is inserted instead of a character you type. This may explain this weird � character because it's only one byte from a multi-byte character.

[elszczepano]

It may be somehow related. If you add a lot of emojis strange characters are shown (I've shown it in a comment for simplicity - it occurs in the main editor too). Also, the editor inserts content in the other place than a marker position.

[farthinker]

I found a working solution:
mycolorway/ckeditor5-typing@90cfb1c
mycolorway/ckeditor5-utils@835ac1f

These two commits make sure both diff and fastDiff correctly handle two-byte characters and the results are correctly normalized.