Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Figure out what we're doing with ICU tokenisation and locales #15124

Open
matrixbot opened this issue Dec 20, 2023 · 0 comments
Open

Figure out what we're doing with ICU tokenisation and locales #15124

matrixbot opened this issue Dec 20, 2023 · 0 comments

Comments

@matrixbot
Copy link
Collaborator

matrixbot commented Dec 20, 2023

This issue has been migrated from #15124.


The ICU tokenisation rules seem to vary on different platforms.

Is it the ICU version? The locale? (How does ICU even get a default locale? I had a quick spelunk in the source code and couldn't find it!)

We need to figure out:

  • what we actually want from the ICU library
  • how we get that
  • how we get consistent results.

It feels like we want a 'universal' locale independent of the host's settings, so that Synapse works well with all languages. (This may be a pie in the sky goal!)
What's the best we can do?

This issue was originally dug up in #15079, but e.g. Patrick's machine generates another tokenisation yet again. I'm not satisfied with the current solution..

@matrixbot matrixbot changed the title Dummy issue Figure out what we're doing with ICU tokenisation and locales Dec 21, 2023
@matrixbot matrixbot reopened this Dec 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant