AO3-6087 Stop removing non-printable Unicode characters #4798
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
especially, zero-width joiner, zero-width non-joiner and word joiner
brianjaustin
requested changes
May 8, 2024
brianjaustin
approved these changes
May 9, 2024
sarken
added a commit
that referenced
this pull request
May 31, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
as the first thing in your pull request title (e.g.
AO3-1234 Fix thing)until they are reviewed and merged before creating new pull requests.
Issue
https://otwarchive.atlassian.net/browse/AO3-6087
Purpose
This will retain non-printable characters entered by the user in various places including the chapter text, comment, etc. They are necessary for representing characters in some languages.
Apparently the special characters caused problems to the parser in the past. While the details are unclear, we tentatively believe that is not the case any more.
Testing Instructions
Try entering text with zero-width joiner, zero-width non-joiner, and word joiner into the main text of a work, or into a comment. They should remain.
References
I believe this will also affect emojis with gender variants (and maybe other variants), as reported in https://old.reddit.com/r/AO3/comments/1cbpoh8/commenting_issue/ .
Credit
Potpotkettle (they/them)