Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make most emojis full-width #5795

Merged
9 commits merged into from
May 8, 2020
Merged

Make most emojis full-width #5795

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
19e4c53
adding the rest of the emoji codepoints in to see
leonMSFT May 5, 2020
e0eac63
comment
leonMSFT May 5, 2020
e8e2beb
comments and microsoft emoji additions
leonMSFT May 7, 2020
7ae5d20
marking the overridden ones
leonMSFT May 8, 2020
ee2b59f
removed MS emoji codepoints
leonMSFT May 8, 2020
224f880
added a few missing
leonMSFT May 8, 2020
56000d4
hot springs, female sign
leonMSFT May 8, 2020
fc0fe03
spell check
leonMSFT May 8, 2020
9e2ee25
wow i can't do math
leonMSFT May 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 120 additions & 55 deletions src/types/CodepointWidthDetector.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,18 @@ namespace
return range.upperBound < searchTerm;
}

static constexpr std::array<UnicodeRange, 285> s_wideAndAmbiguousTable{
static constexpr std::array<UnicodeRange, 351> s_wideAndAmbiguousTable{
// generated from http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt
// anything not present here is presumed to be Narrow.
//
// GH #900 - Supplemented with emoji codepoints from https://www.unicode.org/Public/13.0.0/ucd/emoji/emoji-data.txt
// Emojis in 0x2010 - 0x2B59 used to be marked as Ambiguous in GetQuickCharWidth() in order to force a font lookup,
// but since we default all Ambiguous width to Narrow, those emojis always came out looking squished/tiny. They've been
// moved into this table and marked as Wide.
// There are also some codepoints that Microsoft has given an emoji presentation where it isn't specified
// as emoji in the unicode standard or are specified as Narrow/Ambiguous in the EastAsianWidth.txt.
// Some of them are included in here, but there is no easy way to get a full list
// of Microsoft specific emoji, so others may be missing.
UnicodeRange{ 0xa1, 0xa1, CodepointWidth::Ambiguous },
UnicodeRange{ 0xa4, 0xa4, CodepointWidth::Ambiguous },
UnicodeRange{ 0xa7, 0xa8, CodepointWidth::Ambiguous },
Expand Down Expand Up @@ -147,17 +156,21 @@ namespace
UnicodeRange{ 0x22bf, 0x22bf, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2312, 0x2312, CodepointWidth::Ambiguous },
UnicodeRange{ 0x231a, 0x231b, CodepointWidth::Wide },
UnicodeRange{ 0x2329, 0x232a, CodepointWidth::Wide },
UnicodeRange{ 0x23e9, 0x23ec, CodepointWidth::Wide },
UnicodeRange{ 0x23f0, 0x23f0, CodepointWidth::Wide },
UnicodeRange{ 0x23f3, 0x23f3, CodepointWidth::Wide },
UnicodeRange{ 0x2460, 0x24e9, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2328, 0x232a, CodepointWidth::Wide },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible -- and I know this is a lot of work -- i'd like a comment on each line that refers to a footnote about us overriding these. Just so we can come find it later.

UnicodeRange{ 0x23cf, 0x23cf, CodepointWidth::Wide },
UnicodeRange{ 0x23e9, 0x23ef, CodepointWidth::Wide },
UnicodeRange{ 0x23f0, 0x23f3, CodepointWidth::Wide },
UnicodeRange{ 0x23f8, 0x23fa, CodepointWidth::Wide },
UnicodeRange{ 0x2460, 0x24c1, CodepointWidth::Ambiguous },
UnicodeRange{ 0x24c2, 0x24c2, CodepointWidth::Wide },
UnicodeRange{ 0x24c3, 0x24e9, CodepointWidth::Ambiguous },
UnicodeRange{ 0x24eb, 0x254b, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2550, 0x2573, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2580, 0x258f, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2592, 0x2595, CodepointWidth::Ambiguous },
UnicodeRange{ 0x25a0, 0x25a1, CodepointWidth::Ambiguous },
UnicodeRange{ 0x25a3, 0x25a9, CodepointWidth::Ambiguous },
UnicodeRange{ 0x25aa, 0x25ab, CodepointWidth::Wide },
UnicodeRange{ 0x25b2, 0x25b3, CodepointWidth::Ambiguous },
UnicodeRange{ 0x25b6, 0x25b7, CodepointWidth::Ambiguous },
UnicodeRange{ 0x25bc, 0x25bd, CodepointWidth::Ambiguous },
Expand All @@ -167,56 +180,86 @@ namespace
UnicodeRange{ 0x25ce, 0x25d1, CodepointWidth::Ambiguous },
UnicodeRange{ 0x25e2, 0x25e5, CodepointWidth::Ambiguous },
UnicodeRange{ 0x25ef, 0x25ef, CodepointWidth::Ambiguous },
UnicodeRange{ 0x25fd, 0x25fe, CodepointWidth::Wide },
UnicodeRange{ 0x25fb, 0x25fe, CodepointWidth::Wide },
UnicodeRange{ 0x2600, 0x2604, CodepointWidth::Wide },
UnicodeRange{ 0x2605, 0x2606, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2609, 0x2609, CodepointWidth::Ambiguous },
UnicodeRange{ 0x260e, 0x260e, CodepointWidth::Wide },
UnicodeRange{ 0x260e, 0x260f, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2611, 0x2612, CodepointWidth::Wide }, // MS addition
UnicodeRange{ 0x2614, 0x2615, CodepointWidth::Wide },
UnicodeRange{ 0x261c, 0x261c, CodepointWidth::Ambiguous },
UnicodeRange{ 0x261e, 0x261e, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2618, 0x2618, CodepointWidth::Wide },
UnicodeRange{ 0x261a, 0x261f, CodepointWidth::Wide }, // 0x261A-0x261C, 0x261E-0x261F MS addition
UnicodeRange{ 0x2620, 0x2620, CodepointWidth::Wide },
UnicodeRange{ 0x2622, 0x2623, CodepointWidth::Wide },
UnicodeRange{ 0x2626, 0x2626, CodepointWidth::Wide },
UnicodeRange{ 0x262a, 0x262a, CodepointWidth::Wide },
UnicodeRange{ 0x262e, 0x262f, CodepointWidth::Wide },
UnicodeRange{ 0x2638, 0x263a, CodepointWidth::Wide },
UnicodeRange{ 0x2640, 0x2640, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2642, 0x2642, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2648, 0x2653, CodepointWidth::Wide },
UnicodeRange{ 0x2660, 0x2661, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2663, 0x2665, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2667, 0x266a, CodepointWidth::Ambiguous },
UnicodeRange{ 0x265f, 0x265f, CodepointWidth::Wide },
UnicodeRange{ 0x2660, 0x2660, CodepointWidth::Wide },
UnicodeRange{ 0x2661, 0x2667, CodepointWidth::Wide }, // 0x2661, 0x2662, 0x2664, 0x2666 MS addition
UnicodeRange{ 0x2668, 0x2668, CodepointWidth::Wide },
UnicodeRange{ 0x2669, 0x266a, CodepointWidth::Ambiguous },
UnicodeRange{ 0x266c, 0x266d, CodepointWidth::Ambiguous },
UnicodeRange{ 0x266f, 0x266f, CodepointWidth::Ambiguous },
UnicodeRange{ 0x267f, 0x267f, CodepointWidth::Wide },
UnicodeRange{ 0x2693, 0x2693, CodepointWidth::Wide },
UnicodeRange{ 0x267b, 0x267b, CodepointWidth::Wide },
UnicodeRange{ 0x267e, 0x267f, CodepointWidth::Wide },
UnicodeRange{ 0x2692, 0x2697, CodepointWidth::Wide },
UnicodeRange{ 0x2699, 0x2699, CodepointWidth::Wide },
UnicodeRange{ 0x269b, 0x269c, CodepointWidth::Wide },
UnicodeRange{ 0x269e, 0x269f, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26a1, 0x26a1, CodepointWidth::Wide },
UnicodeRange{ 0x26a0, 0x26a1, CodepointWidth::Wide },
UnicodeRange{ 0x26a7, 0x26a7, CodepointWidth::Wide },
UnicodeRange{ 0x26aa, 0x26ab, CodepointWidth::Wide },
UnicodeRange{ 0x26b0, 0x26b1, CodepointWidth::Wide },
UnicodeRange{ 0x26bd, 0x26be, CodepointWidth::Wide },
UnicodeRange{ 0x26bf, 0x26bf, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26c4, 0x26c5, CodepointWidth::Wide },
UnicodeRange{ 0x26c6, 0x26cd, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26ce, 0x26ce, CodepointWidth::Wide },
UnicodeRange{ 0x26cf, 0x26d3, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26d4, 0x26d4, CodepointWidth::Wide },
UnicodeRange{ 0x26c4, 0x26c5, CodepointWidth::Wide },
UnicodeRange{ 0x26c6, 0x26c7, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26c8, 0x26c8, CodepointWidth::Wide },
UnicodeRange{ 0x26c9, 0x26cd, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26ce, 0x26cf, CodepointWidth::Wide },
UnicodeRange{ 0x26d0, 0x26d0, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26d1, 0x26d1, CodepointWidth::Wide },
UnicodeRange{ 0x26d2, 0x26d2, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26d3, 0x26d4, CodepointWidth::Wide },
UnicodeRange{ 0x26d5, 0x26e1, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26e3, 0x26e3, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26e8, 0x26e9, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26ea, 0x26ea, CodepointWidth::Wide },
UnicodeRange{ 0x26eb, 0x26f1, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26f2, 0x26f3, CodepointWidth::Wide },
UnicodeRange{ 0x26f4, 0x26f4, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26f5, 0x26f5, CodepointWidth::Wide },
UnicodeRange{ 0x26f6, 0x26f9, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26fa, 0x26fa, CodepointWidth::Wide },
UnicodeRange{ 0x26e8, 0x26e8, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26e9, 0x26ea, CodepointWidth::Wide },
UnicodeRange{ 0x26eb, 0x26ef, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26f0, 0x26f5, CodepointWidth::Wide },
UnicodeRange{ 0x26f6, 0x26f6, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26f7, 0x26fa, CodepointWidth::Wide },
UnicodeRange{ 0x26fb, 0x26fc, CodepointWidth::Ambiguous },
UnicodeRange{ 0x26fd, 0x26fd, CodepointWidth::Wide },
UnicodeRange{ 0x26fe, 0x26ff, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2702, 0x2702, CodepointWidth::Wide },
UnicodeRange{ 0x2705, 0x2705, CodepointWidth::Wide },
UnicodeRange{ 0x270a, 0x270b, CodepointWidth::Wide },
UnicodeRange{ 0x2708, 0x2710, CodepointWidth::Wide }, // 0x2710, 0x270E MS addition
UnicodeRange{ 0x2712, 0x2712, CodepointWidth::Wide },
UnicodeRange{ 0x2714, 0x2714, CodepointWidth::Wide },
UnicodeRange{ 0x2716, 0x2716, CodepointWidth::Wide },
UnicodeRange{ 0x271d, 0x271d, CodepointWidth::Wide },
UnicodeRange{ 0x2721, 0x2721, CodepointWidth::Wide },
UnicodeRange{ 0x2728, 0x2728, CodepointWidth::Wide },
UnicodeRange{ 0x2733, 0x2734, CodepointWidth::Wide },
UnicodeRange{ 0x273d, 0x273d, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2744, 0x2744, CodepointWidth::Wide },
UnicodeRange{ 0x2747, 0x2747, CodepointWidth::Wide },
UnicodeRange{ 0x274c, 0x274c, CodepointWidth::Wide },
UnicodeRange{ 0x274e, 0x274e, CodepointWidth::Wide },
UnicodeRange{ 0x2753, 0x2755, CodepointWidth::Wide },
UnicodeRange{ 0x2757, 0x2757, CodepointWidth::Wide },
UnicodeRange{ 0x2763, 0x2765, CodepointWidth::Wide }, // 0x2765 MS addition
UnicodeRange{ 0x2776, 0x277f, CodepointWidth::Ambiguous },
UnicodeRange{ 0x2795, 0x2797, CodepointWidth::Wide },
UnicodeRange{ 0x27a1, 0x27a1, CodepointWidth::Wide },
UnicodeRange{ 0x27b0, 0x27b0, CodepointWidth::Wide },
UnicodeRange{ 0x27bf, 0x27bf, CodepointWidth::Wide },
UnicodeRange{ 0x2b1b, 0x2b1c, CodepointWidth::Wide },
Expand Down Expand Up @@ -258,50 +301,72 @@ namespace
UnicodeRange{ 0x18800, 0x18af2, CodepointWidth::Wide },
UnicodeRange{ 0x1b000, 0x1b11e, CodepointWidth::Wide },
UnicodeRange{ 0x1b170, 0x1b2fb, CodepointWidth::Wide },
UnicodeRange{ 0x1f004, 0x1f004, CodepointWidth::Wide },
UnicodeRange{ 0x1f000, 0x1f02b, CodepointWidth::Wide }, // All mahjong tiles except 0x1f004 are MS addition
UnicodeRange{ 0x1f0cf, 0x1f0cf, CodepointWidth::Wide },
UnicodeRange{ 0x1f100, 0x1f10a, CodepointWidth::Ambiguous },
UnicodeRange{ 0x1f110, 0x1f12d, CodepointWidth::Ambiguous },
UnicodeRange{ 0x1f130, 0x1f169, CodepointWidth::Ambiguous },
UnicodeRange{ 0x1f170, 0x1f18d, CodepointWidth::Ambiguous },
UnicodeRange{ 0x1f170, 0x1f171, CodepointWidth::Wide },
UnicodeRange{ 0x1f172, 0x1f17d, CodepointWidth::Ambiguous },
UnicodeRange{ 0x1f17e, 0x1f17f, CodepointWidth::Wide },
UnicodeRange{ 0x1f180, 0x1f18d, CodepointWidth::Ambiguous },
UnicodeRange{ 0x1f18e, 0x1f18e, CodepointWidth::Wide },
UnicodeRange{ 0x1f18f, 0x1f190, CodepointWidth::Ambiguous },
UnicodeRange{ 0x1f191, 0x1f19a, CodepointWidth::Wide },
UnicodeRange{ 0x1f19b, 0x1f1ac, CodepointWidth::Ambiguous },
UnicodeRange{ 0x1f1e6, 0x1f1ff, CodepointWidth::Wide },
UnicodeRange{ 0x1f200, 0x1f202, CodepointWidth::Wide },
UnicodeRange{ 0x1f210, 0x1f23b, CodepointWidth::Wide },
UnicodeRange{ 0x1f240, 0x1f248, CodepointWidth::Wide },
UnicodeRange{ 0x1f250, 0x1f251, CodepointWidth::Wide },
UnicodeRange{ 0x1f260, 0x1f265, CodepointWidth::Wide },
UnicodeRange{ 0x1f300, 0x1f320, CodepointWidth::Wide },
UnicodeRange{ 0x1f32d, 0x1f335, CodepointWidth::Wide },
UnicodeRange{ 0x1f337, 0x1f37c, CodepointWidth::Wide },
UnicodeRange{ 0x1f37e, 0x1f393, CodepointWidth::Wide },
UnicodeRange{ 0x1f3a0, 0x1f3ca, CodepointWidth::Wide },
UnicodeRange{ 0x1f3cf, 0x1f3d3, CodepointWidth::Wide },
UnicodeRange{ 0x1f3e0, 0x1f3f0, CodepointWidth::Wide },
UnicodeRange{ 0x1f3f4, 0x1f3f4, CodepointWidth::Wide },
UnicodeRange{ 0x1f3f8, 0x1f43e, CodepointWidth::Wide },
UnicodeRange{ 0x1f440, 0x1f440, CodepointWidth::Wide },
UnicodeRange{ 0x1f442, 0x1f4fc, CodepointWidth::Wide },
UnicodeRange{ 0x1f300, 0x1f321, CodepointWidth::Wide },
UnicodeRange{ 0x1f324, 0x1f393, CodepointWidth::Wide },
UnicodeRange{ 0x1f396, 0x1f397, CodepointWidth::Wide },
UnicodeRange{ 0x1f399, 0x1f39b, CodepointWidth::Wide },
UnicodeRange{ 0x1f39e, 0x1f39f, CodepointWidth::Wide },
UnicodeRange{ 0x1f3a0, 0x1f3f0, CodepointWidth::Wide },
UnicodeRange{ 0x1f3f3, 0x1f3f5, CodepointWidth::Wide },
UnicodeRange{ 0x1f3f7, 0x1f4fd, CodepointWidth::Wide },
UnicodeRange{ 0x1f4ff, 0x1f53d, CodepointWidth::Wide },
UnicodeRange{ 0x1f54b, 0x1f54e, CodepointWidth::Wide },
UnicodeRange{ 0x1f549, 0x1f54e, CodepointWidth::Wide },
UnicodeRange{ 0x1f550, 0x1f567, CodepointWidth::Wide },
UnicodeRange{ 0x1f57a, 0x1f57a, CodepointWidth::Wide },
UnicodeRange{ 0x1f595, 0x1f596, CodepointWidth::Wide },
UnicodeRange{ 0x1f5a4, 0x1f5a4, CodepointWidth::Wide },
UnicodeRange{ 0x1f5fb, 0x1f64f, CodepointWidth::Wide },
UnicodeRange{ 0x1f56f, 0x1f570, CodepointWidth::Wide },
UnicodeRange{ 0x1f573, 0x1f57a, CodepointWidth::Wide },
UnicodeRange{ 0x1f587, 0x1f587, CodepointWidth::Wide },
UnicodeRange{ 0x1f58a, 0x1f58d, CodepointWidth::Wide },
UnicodeRange{ 0x1f590, 0x1f590, CodepointWidth::Wide },
UnicodeRange{ 0x1f594, 0x1f596, CodepointWidth::Wide }, // 0x1f594 MS addition
UnicodeRange{ 0x1f5a4, 0x1f5a5, CodepointWidth::Wide },
UnicodeRange{ 0x1f5a8, 0x1f5a8, CodepointWidth::Wide },
UnicodeRange{ 0x1f5b1, 0x1f5b2, CodepointWidth::Wide },
UnicodeRange{ 0x1f5bc, 0x1f5bc, CodepointWidth::Wide },
UnicodeRange{ 0x1f5c2, 0x1f5c4, CodepointWidth::Wide },
UnicodeRange{ 0x1f5d1, 0x1f5d2, CodepointWidth::Wide },
UnicodeRange{ 0x1f5dc, 0x1f5de, CodepointWidth::Wide },
UnicodeRange{ 0x1f5e1, 0x1f5e1, CodepointWidth::Wide },
UnicodeRange{ 0x1f5e3, 0x1f5e3, CodepointWidth::Wide },
UnicodeRange{ 0x1f5e8, 0x1f5e8, CodepointWidth::Wide },
UnicodeRange{ 0x1f5ef, 0x1f5ef, CodepointWidth::Wide },
UnicodeRange{ 0x1f5f3, 0x1f5f3, CodepointWidth::Wide },
UnicodeRange{ 0x1f5fa, 0x1f64f, CodepointWidth::Wide },
UnicodeRange{ 0x1f680, 0x1f6c5, CodepointWidth::Wide },
UnicodeRange{ 0x1f6cc, 0x1f6cc, CodepointWidth::Wide },
UnicodeRange{ 0x1f6d0, 0x1f6d2, CodepointWidth::Wide },
UnicodeRange{ 0x1f6cb, 0x1f6d2, CodepointWidth::Wide },
UnicodeRange{ 0x1f6d5, 0x1f6d7, CodepointWidth::Wide },
UnicodeRange{ 0x1f6e0, 0x1f6e5, CodepointWidth::Wide },
UnicodeRange{ 0x1f6e9, 0x1f6e9, CodepointWidth::Wide },
UnicodeRange{ 0x1f6eb, 0x1f6ec, CodepointWidth::Wide },
UnicodeRange{ 0x1f6f4, 0x1f6f8, CodepointWidth::Wide },
UnicodeRange{ 0x1f910, 0x1f93e, CodepointWidth::Wide },
UnicodeRange{ 0x1f940, 0x1f94c, CodepointWidth::Wide },
UnicodeRange{ 0x1f950, 0x1f96b, CodepointWidth::Wide },
UnicodeRange{ 0x1f980, 0x1f997, CodepointWidth::Wide },
UnicodeRange{ 0x1f9c0, 0x1f9c0, CodepointWidth::Wide },
UnicodeRange{ 0x1f9d0, 0x1f9e6, CodepointWidth::Wide },
UnicodeRange{ 0x1f6f0, 0x1f6f0, CodepointWidth::Wide },
UnicodeRange{ 0x1f6f3, 0x1f6fc, CodepointWidth::Wide },
UnicodeRange{ 0x1f7e0, 0x1f7eb, CodepointWidth::Wide },
UnicodeRange{ 0x1f90c, 0x1f9ff, CodepointWidth::Wide },
UnicodeRange{ 0x1fa70, 0x1fa74, CodepointWidth::Wide },
UnicodeRange{ 0x1fa78, 0x1fa7A, CodepointWidth::Wide },
UnicodeRange{ 0x1fa80, 0x1fa86, CodepointWidth::Wide },
UnicodeRange{ 0x1fa90, 0x1faa8, CodepointWidth::Wide },
UnicodeRange{ 0x1fab0, 0x1fab6, CodepointWidth::Wide },
UnicodeRange{ 0x1fac0, 0x1fac2, CodepointWidth::Wide },
UnicodeRange{ 0x1fad0, 0x1fad6, CodepointWidth::Wide },
UnicodeRange{ 0x20000, 0x2fffd, CodepointWidth::Wide },
UnicodeRange{ 0x30000, 0x3fffd, CodepointWidth::Wide },
UnicodeRange{ 0xe0100, 0xe01ef, CodepointWidth::Ambiguous },
Expand Down
Loading