Most emojis don't work on Windows

[mrcdk]

Godot version:

Godot 3.1 master 0d8284d

OS/device including version:

Window 10

Issue description:

In theory, emoji support was added with this PR #15780 but I've been testing it and I can't get it to work. I tested it with the NotoColorEmoji.ttf font. That's the font I think @volzhs used for both that PR and this issue #10873 but I couldn't make it work. The font doesn't work on windows (opening it with the font viewer throws an error) because Windows doesn't support the type of color font googlefonts/noto-emoji#43 But I'm not sure if it has anything to do with the issue.

Steps to reproduce:

1. Download the font https://www.google.com/get/noto/#emoji-zsye-color
2. Use it in a label with an emoji like 😄

[volzhs]

did you compile with builtin_freetype option?
scons platform=windows builtin_freetype=yes

[mrcdk]

I didn't specify builtin_freetype=yes explicitly but it's on by default. I tried just in case and still nothing. When I c&p an emoji from the internet this is how it shows in the inspector but nothing shows in the actual label.

I tried a couple of other emojis and some did work like ✔

I also tried using a PoolByteArray and push the utf-8 bytes in there but no luck either. I tried:

func _ready():
	var bytes = PoolByteArray()
	bytes.push_back(0xF0)
	bytes.push_back(0x9F)
	bytes.push_back(0x98)
	bytes.push_back(0x84)
	text = bytes.get_string_from_utf8()

Which should be this one https://apps.timwhitlock.info/unicode/inspect?s=%F0%9F%98%84

[bruvzg]

Most emotes have char codes > 0xFFFF.

Godot string is based on wchar_t which is 16-bit on Windows, convention from UTF-8 replaces chars with codes above 0xFFFF with spaces, ustring.cpp#L1497 and Godot doesn't support UTF-16 surrogate pairs (and Windows copy-paste probably use UTF-16).

On Linux an macOS wchar_t is 32-bit and all emotes should work fine.

[mrcdk]

Oh well, I updated the title of the issue to reflect that information. Thanks @bruvzg

[bruvzg]

To be precise Unicode supplementary planes have much more stuff, not just emojis.

BiDi/Shaping PR (#21791) I'm working on, should fix rendering of > 0xFFFF chars on all platforms, but wchar_t size difference still cause some input problems on Windows and excessive encoding conversions on Linux/macOS.

[xpol]

How a bout switch to char32_t?

[bruvzg]

How a bout switch to char32_t?

char32_t and char16_t are C++11 types and Godot currently targets C++03.

And it's probably better to switch to UTF-16 instead for several reasons:

• Windows APIs (and JavaScript APIs IIRC) use UTF-16 (input, copy-paste, filesystem), Linux/macOS APIs use UTF-8, none use UTF-32.
• ICU required for [WIP] BiDi and shaping support for Font::draw/Font::get_string_size #21791 use UTF-16.
• UTF-32 is space inefficient, non BMP chars are rare and in average UTF-16 is using two times less memory than UTF-32.

[ConnorBP]

I wrote some code to handle utf code conversion in c++ somewhere. I can take a look for it. wchar on windows is really quite a pain in the butt, but it can be worked with.

update: I would love to be able to 😏 people in game :^)

[Anutrix]

Godot is gonna support C++11 soon so what's the status on this? Are we using UTF-16?

[Anutrix]

It's time to switch to char16_t or something else, as Godot now targets C++11.

[MrMrBlackDragon]

The same goes for Godots ord() where nearly all emojis return 32 (32 is actually the space " "), meaning that the emojis aren't recognized at all. ☹

[Calinou]

Note: I changed the milestone to 3.5 as #54625 addresses the main problem described here in 3.5beta1 and later. Try 3.5beta3 for instance.