Support for RTL input and display

[xsoh]

The editor doesn't recognize the non-latin characters (it writes nothing) even with changing the font of the editor. Also trying to paste a word of non-latin will print the Unicode representation (e.g. \u2320\u3433...etc) not the real word (e.g. سلام).

[reduz]

the default font does not, but you can set any custom font you like in the
editor, that supports your character set.

https://github.com/okamstudio/godot/wiki/editor_font

On Wed, Dec 17, 2014 at 1:53 AM, Suhail Alkowaileet <
notifications@github.com> wrote:

The editor doesn't recognize the non-latin characters (it writes nothing)
even with changing the font of the editor. Also trying to paste a word of
non-latin will print the Unicode representation (e.g. \u2320\u3433...etc)
not the real word (e.g. سلام).

—
Reply to this email directly or view it on GitHub
#982.

[xsoh]

Yes I have changed the font but no luck with that. The editor doesn't write anything and if paste it, it writes it as Unicode (decoded) chars.

[reduz]

oh when you change a font, you have to specify it to import unicode,
otherwise it only imports latin.
Still it's weird that it shows those characters, sounds like the text was
not escaped (or more like, label will not unescape ext)

On Wed, Dec 17, 2014 at 1:12 PM, Suhail Alkowaileet <
notifications@github.com> wrote:

Yes I have changed the font but no luck with that. The editor doesn't
write anything and if paste it, it writes it as Unicode (decoded) chars.

[image: screenshot from 2014-12-17 18 53 12]
https://cloud.githubusercontent.com/assets/871079/5474205/a39a8276-8620-11e4-8e78-56bf06bd9774.png

—
Reply to this email directly or view it on GitHub
#982 (comment).

[xsoh]

Oh sorry didn't see that.

OK. Now the characters are shown but we have two challenges :

• the characters are written in LTR instead of RTL.
• the characters are not connected to each other (It should be like "سلام" not "س ل ا م").
  Fribidi is a solution for that if its not using TrueType or Harfbuzz for TrueType.

Also still Ctrl + V didn't work for Unicode (E.g. سلام).

Thanks reduz.

[reduz]

this is opengl, so we can't use truetype. Do you know where documentation
for supporting unicode RTL can be found?

On Wed, Dec 17, 2014 at 1:40 PM, Suhail Alkowaileet <
notifications@github.com> wrote:

Oh sorry didn't see that.
[image: screenshot from 2014-12-17 19 29 52]
https://cloud.githubusercontent.com/assets/871079/5474710/79fc8884-8624-11e4-8eb9-1d259590212d.png
OK. Now the characters are shown but we have two challenges :

• the characters are written in LTR instead of RTL.
• the characters are not connected to each other (It should be like
  "سلام" not "س ل ا م"). Fribidi https://github.com/behdad/fribidi is
  a solution for that if its not using TrueType or Harfbuzz
  https://github.com/behdad/harfbuzz for TrueType.

Also still Ctrl + V didn't work for Unicode (E.g. سلام).

Thanks reduz.

—
Reply to this email directly or view it on GitHub
#982 (comment).

[xsoh]

What about Fribidi? it doesn't need truetype. Its just apply the Bidi-Algorithim.

For that you can find it here Unicode Bidirectional Algorithm

[reduz]

fribidi is LGPL, can't be used on IOS, Android, consoles etc. will probably
have to implement this myself.

On Wed, Dec 17, 2014 at 2:01 PM, Suhail Alkowaileet <
notifications@github.com> wrote:

What about Fribidi? it doesn't need truetype. Its just apply the
Bidi-Algorithim.

For that you can find it here Unicode Bidirectional Algorithm
http://www.unicode.org/reports/tr9/

—
Reply to this email directly or view it on GitHub
#982 (comment).

[Masood-Lapeh]

Hi!
[http://www.godotengine.org/forum/viewtopic.php?f=11&t=1232]
I've the same problem with Farsi (Persian) characters...
So I've to use some images instead... but It won't work everywhere... for example in text entry boxes...
I'd appreciate it if you could fix it...
Thanks for you attention!

[xsoh]

@Masood-Lapeh Yes, all the languages that written in the Arabic script language system will face same problem.

[reduz]

by the way, what do you guys use for text input? do you need a special IME popup or the keyboard just inserts the right characters directly?

[Evi1M4chine]

@reduz: Arabs have just normal keyboard layouts. The font rendering is a bit special though, as the characters combine into something that looks different.
But I don’t think it matters, because I have an advanced German keyboard layout, and for all the unusual characters, when I press the key, Godot simply does nothing.

[reduz]

what platform are you on? maybe it has something to do with that, for me it
works with english and spanish

On Fri, Jan 2, 2015 at 4:22 PM, Evi1M4chine notifications@github.com
wrote:

@reduz https://github.com/reduz: Arabs have just normal keyboard layouts
http://en.wikipedia.org/wiki/Arabic_keyboard. The font rendering is a
bit special though, as the characters combine into something that looks
different.
But I don’t think it matters, because I have an advanced German keyboard
layout http://www.neo-layout.org/, and for all the unusual characters,
when I press the key, Godot simply does nothing.

—
Reply to this email directly or view it on GitHub
#982 (comment).

[Evi1M4chine]

@reduz: Might be. This is

• amd64 platform,
• Linux 3.18.1,
• Xorg 7.4 (server 1.16.2.901, but with nVidia drivers 340.65),
• KDE 4.14.3,
• GCC 4.8.3 (LLVM 3.5.0 is installed too)
• Python 2.7.8 (3.3.5 is installed too, but disabled via eselect, as Godot doesn’t work with it)
• SCons 2.3.4
• Mesa 10.3.5
• ALSA libs 1.0.28
• Freetype 2.5.4
• OpenSSL 1.0.1j
• pkgconfig 0.28
• and Godot is obviously self-compiled from the latest git sources.

I just tested it a bit (trying to enter special characters in a label text field and found at least one character for each mod key combination that worked. It seems to be a case of a limited character set.
It also apparently seems to slow down, because the invisible characters still count. So when using the cursor or pressing backspace, it may look like nothing happened, or as if it was hanging. This was really hard to find out. As for rendering them in the scene: I could not find a character that showed up in the text field but not in the scene. Probably because I imported and used my own full Unicode font. (I know that font works, because I use it for the rest of the system.)

I could not test advanced layout things though (like Arab or Thai scripts)…

Anyway, this is not that important to me. I’m just interested in having GDscript identifiers with Unicode characters in them, and the basic German letters. Everything special can be done by making my own images. :)

[godotengine]

question, what do you do with the NEO keyboard layout with shortcuts such
as ctrl-z, ctrl-s, etc?

On Fri, Jan 2, 2015 at 7:02 PM, Evi1M4chine notifications@github.com
wrote:

@reduz https://github.com/reduz: Might be. This is

• amd64 platform,
• Linux 3.18.1,
• Xorg 7.4 (server 1.16.2.901, but with nVidia drivers 340.65),
• KDE 4.14.3,
• GCC 4.8.3 (LLVM 3.5.0 is installed too)
• Python 2.7.8 (3.3.5 is installed too, but disabled via eselect, as
  Godot doesn’t work with it)
• SCons 2.3.4
• Mesa 10.3.5
• ALSA libs 1.0.28
• Freetype 2.5.4
• OpenSSL 1.0.1j
• pkgconfig 0.28
• and Godot is obviously self-compiled from the latest git sources.

I just tested it a bit (trying to enter special characters in a label text
field and found at least one character for each mod key combination that
worked. It seems to be a case of a limited character set.
It also apparently seems to slow down, because the invisible characters
still count. So when using the cursor or pressing backspace, it may look
like nothing happened, or as if it was hanging. This was really hard to
find out. As for rendering them in the scene: I could not find a character
that showed up in the text field but not in the scene. Probably because I
imported and used my own full Unicode font. (I know that font works,
because I use it for the rest of the system.)

I could not test advanced layout things though (like Arab or Thai scripts)…

Anyway, this is not that important to me. I’m just interested in having
GDscript identifiers with Unicode characters in them, and the basic German
letters. Everything special can be done by making my own images. :)

—
Reply to this email directly or view it on GitHub
#982 (comment).

OkamStudio

[Evi1M4chine]

Well, the Ctrl, Alt and Win keys are still available. Maybe the fact that in Germany the Ctrl key is called “Strg” was confusing.

So I just press them like normal.

[Just that of course I have more other combinations available, as I could set shortcuts to Ctrl-Alt-Mod3-Mod4-(someletter) if I was crazy. ;) Even KDE, and Gnome sometimes fail to handle those correctly though. E.g. because Mod3-A is actually {, and so Shift-{ is actually Mod3-Shift-A… but that is also the letter α, and so is ambiguous.]

[mohaalak]

@reduz I contacted Behdad programmer of Fribidi for licensing issue on Fribidl , I am waiting for an answer.

[reduz]

That looks o,k, though I don't really understand how it's supposed to work.
As far as I understand, the font engine must support the following?

1. RTL and LTR characters, for switching direction? (as far as I understand
   it's like characters are pushed, then retrieved in the opposite direction?)
2. Ligatures between different types of character combinations

Is there anything else?

On Mon, Mar 9, 2015 at 9:05 AM, Mohammad Hadi Aliakbar <
notifications@github.com> wrote:

@reduz https://github.com/reduz I contacted for licensing issue on
Fribidl , I am waiting for an answer.

—
Reply to this email directly or view it on GitHub
#982 (comment).

[mohaalak]

1 - in RTL langauges all character go from left to right but digits and LTR characters should go from left to right (it's not like align left)
2- yeah correct the combination of characters should print something else.

you think we can create from ground up this feature without using any library?

if you point me in right direction I think I can make it work at least for Farsi and Arabic.

should I change
https://github.com/okamstudio/godot/blob/master/scene/resources/font.cpp

draw function?

.

[reduz]

Ok, but as far as I understand Unicode by default is LTR, so if unicode
supports RTL it must have some sort of character that tells you "ok from
now on, following text is RTL" and then something to also tell "ok from now
on, following text is LTR again".

On Mon, Mar 9, 2015 at 11:07 AM, Mohammad Hadi Aliakbar <
notifications@github.com> wrote:

1 - in RTL langauges all character go from left to right but digits and
LTR characters should go from left to right (it's not like align left)
2- yeah correct the combination of characters should print something else.

you think we can create from ground up this feature without using any
library?

if you point me in right direction I think I can make it work at least for
Farsi and Arabic.

should I change
https://github.com/okamstudio/godot/blob/master/scene/resources/font.cpp

draw function?

.

—
Reply to this email directly or view it on GitHub
#982 (comment).

[mohaalak]

every text editor that support RTL has a button to make text RTL or LTR.
we should add a property that make control is on RTL or LTR.

[reduz]

oh so this is external to the text.. you mean that label should have a
button that says "Use RTL Text"

However, if we were dealing with game translations and we wished to support
RTL languages, how is this handled?

On Mon, Mar 9, 2015 at 3:51 PM, Mohammad Hadi Aliakbar <
notifications@github.com> wrote:

every text editor that support RTL has a button to make text RTL or LTR.
we should add a property that make control is on RTL or LTR.

—
Reply to this email directly or view it on GitHub
#982 (comment).

[xsoh]

This project might be good for understanding Bidi
https://github.com/salshaaban/BidiRenderer

[mohaalak]

it's a little tricky , when I want to use it on web , after translation we should set direction to RTL , here we should have a system that if this language is RTL or not , if it is RTL the Controller should change from LTR to RTL and every object that inherit this should become RTL.

in some softwares when you change your layout from LTR to RTL everything change from left to right,
like when in a form, labels is left and inputs are in front , in right of screen , when change to RTL mode in some softwares it change all things , labels goest to right of screen, inputs go to left of screen .

It's living in hell to write multi-language software when RTL language is concerned :D

I think if labels and text edits can support RTL is fine.

@xsoh this code comes from a library called Unicode bidirectional algorithm , and this algorithm is GPL. I don't know much about licensing but @reduz say we can't use it cause in android , ios , consoles you can't use GPL

[lostdj]

this is opengl, so we can't use truetype.

Yes you can. :)

[xsoh]

@mohaalak Does the algorithm it self (I don't mean the code. I mean the algorithim) has licensing? What I meant for BidiRenderer is just for understanding not using the code it self.

[mohaalak]

I read in fribidI that algorithm has license too. I don't get these
licensing issues and after a little research Fribidi might not be our best
option . There is another library named harbuzz it use true type and it has
'old MIT' License
with Cairo library we could support true type fonts but I think it's too
much work to change how fonts draws now and there will be lots of bugs and
compatibility issue with older version of godot.
after reading source code of harfbuzz I think I could change it to use
BMfonts .I think it gets only size of each glyph from ttf . I could pass
size from BMfonts to harfbuzz . it should be interesting.
On Mar 14, 2015 11:21 AM, "Suhail Alkowaileet" notifications@github.com
wrote:

@mohaalak https://github.com/mohaalak Does the algorithm it self (I
don't mean the code. I mean the algorithim) has licensing? What I meant for
BidiRenderer is just for understanding not using the code it self.

—
Reply to this email directly or view it on GitHub
#982 (comment).

[mohaalak]

I have good news and bad news.

Good News :

• implement RTL support for Godot in this gui interfaces:
  • label
  • text edit
  • line edit
  • button
• another good news is that if you write RTL language , it will automatically change to RTL mode. @reduz so I think in translation everything is safe

Bad News :

I could not work with harfbuzz , it needs true type support and for that we should use cairo library and font is gonna be change fundamentally. I think @reduz and other main developers should discuss about this change (I could do it but first let's discuss it ) , a post about cairo and bmfonts

I used Fribidi ,it's license is LGPL, it means that you can't release it to Android , iOS or anywhere else unless , users can change just Fribidi library ,link it to your code and everything else works again. it can be done.

1. you can open source your game ( not a great idea for everyone)
2. you can export a release mode from godot and let's anyone that wants, compile godot from repository with their changes and use your released datapack( I don't know if it's doable or not)
3. you can change scons to build a Static Library for Fribidi and ship your game with this library I know it works in android , windows , linux but for ios I just found this link, short answer is that you should ask permission of Fribidi to use it under iOS, I don't know anything about other consoles.

you can clone my repository and work with it if you want , I don't open a pull request I don't think @reduz wants this changes into main repository.

[mohaalak]

by the way we can use BIDI algorithm but I don't know much about licensing , it does not mention anything here about any license but I think I read somewhere in fribidi doc that algorithm is LGPL also.
I found this source codes for bidi algorithm , I will work with this sample code too , maybe I can make something out , but if we use truetype it's gonna be better.

[mohaalak]

above algorithm work for RTL support but we need Ligatures for other languages. I can make something for arabic and persian but I don't know about other languages.

[reduz]

@mohaalak Gandhi once said "Be the change you want to see in the world". If we don't get help from someone who understands how RTL is used, we simply can't do it.

[khaledhosny]

Proper Arabic/Indic/etc and RTL support will require a punch of extra dependencies as it is a rather huge amount of work to do from scratch or fork and bundle existing libraries. Are such dependencies something the maintainers are open to?

[akien-mga]

@khaledhosny Up to now the general feeling is that the less additional dependencies, the better. We were under the impression that it should be possible to reimplement the BiDi algorithm directly (or via a lightweight library) - but actually the more time passes, the more I think there might be a reason why RTL libraries are so big (especially as Arabic letters need to support mutating based on their context).

So if a RTL-aware dev could do some research about what would be a minimal set of dependencies/features needed to support RTL and text shaping properly on all platforms, that would help bringing this issue closer to a fix.

[khaledhosny]

Minimal dependencies would be FriBiDi for bidirectional support, HarfBuzz for font shaping, FreeType for font rasterization (not a hard requirement) and some code to glue everything together (Raqm can be that glue but it is not absolutely needed). If FriBiDi license is an issue then ICU is another replacement, but otherwise I recommend against just using the sample code from Unicode unless someone has the time an interest in actively maintaining it in this code base.

This should give you support for Arabic, Hebrew, lots of Indic and South Asian scripts and more.

[akien-mga]

FriBiDi is licensed under the LGPL, which is (AFAIU) a bit tricky when you link it statically as we do in Godot for our thirdparty libraries. I think it may be fine, but we'd have to double check with a lawyer:

A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of the Library, and therefore falls outside the scope of this License.

The problem of ICU is that it's huge and provides a lot more than just the BiDi algorithm, so it would bloat the engine unnecessarily.

An alternative could be the Apache-licensed SheenBidi.

For FreeType, it's included already. HarfBuzz could be added, it seems to be the go-to solution for font shaping.

[khaledhosny]

I have no experience with using SheenBidi and it seems relatively new, but in principle any conforming UBA implementation should be fine.

[khaledhosny]

I looked into the text layout code briefly and I’m afraid it is not an easy task since the code seem to be written with the assumption that text is laid sequentially from left to right, which is obviously not the case for most complex scripts. Fixing that will require quite some refactoring and deeper understanding of the code.

[mohaalak]

ICU implmentation and Fribidi was pull reqeuested before but denied cause of license and huge library,
I don't know if it's possible to add ICU as extensioin using GDNative or not?
I didnot read GDNative docs yet

I used fribidi and ICU implementation and even released games in Iran, maybe after version 3 release I fork and add ICU to it so every major version have a RTL Support.

[akien-mga]

Just mentioning that this is still a wanted feature, but it's a complex one to implement properly (especially font shaping). Current WIP in #10546 with discussion on how it should be done with the minimal impact for languages which can print chars one by one as done currently.

[nightblade9]

I am also interested in this. Just tried a custom font with Arabic letter support and it shows up broken and left-to-right. (I didn't read this entire thread, just the top few and bottom few posts.)

If anyone's interested, this is how it looks in Godot 3.0.4 (letters are left-to-right and broken):

This is how it should appear:

القرآن الكريم

I can definitely help with testing, but probably not much help with implementation. I've heard HarfBuzz is a very good solution (used in HaxeFlixel/OpenFL).

[nightblade9]

I just realized that the ICU-support WIP sample here sort of works. Is there any possibility of merging it in and swapping it out later?

If we wait for proper support of this feature, we may end up waiting a long time; this WIP would at least get us something we can use for now (even if it's not well-supported or ideal).

[g--o]

Hopping on this issue, other RTL languages suffer here as well with the same original flipped text problem.
So there's localization to the editor but it's just useless when the entire thing is flipped!

[golddotasksquestions]

Maybe these guys can help: https://www.arabeyes.org/Translation_requests
Also this technical dictionary might be of use: https://www.arabeyes.org/techdict/techdict.pdf

[akien-mga]

As we're moving all feature requests to the new godot-proposals repository, I'll close this as superseded by godotengine/godot-proposals#4.

Note that fixing these issues is one of our priorities for Godot 4.0, as we can take the opportunity of a compatibility breakage to properly rethink our text layout controls to accommodate the requirements of RTL display and font shaping.