CLDR-16836 kbd: add EBNF to spec for transforms #4261

srl295 · 2025-01-01T23:54:48Z

CLDR-16836

This PR completes the ticket.

ALLOW_MANY_COMMITS=true

jira-pull-request-webhook · 2025-01-02T20:04:07Z

Notice: the branch changed across the force-push!

docs/ldml/tr35-keyboards.md is different
keyboards/abnf/transform-from-required.abnf is different
tools/scripts/keyboard-abnf-tests/check-keyboard-abnf.sh is different
tools/scripts/keyboard-abnf-tests/transform-from-required.d/from-match.fail.txt is now changed in the branch
tools/scripts/keyboard-abnf-tests/transform-from-required.d/from-match.pass.txt is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

srl295 · 2025-01-02T20:10:37Z

fyi @stasm and @aphillips

srl295 · 2025-01-02T20:13:19Z

NMTOKEN is a little incomplete - had a tool problem with high values. (worked around it)
Need to do the to= side (done)

jira-pull-request-webhook · 2025-01-03T18:14:07Z

Notice: the branch changed across the force-push!

docs/ldml/tr35-keyboards.md is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

- add keyboard abnf and sample files and automated tests Temporarily skip non-BMP chars, see hildjj/node-abnf#25 which is being fixed.

jira-pull-request-webhook · 2025-01-03T21:31:29Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

macchiati

Looks good!

The EBNF links should point to https://unicode.org/reports/tr35/#ebnf. (Since the LDML EBNF is a superset of the vanilla EBNF that should work.
The validity / well-formedness constraints (eg no more than 9 capture groups), should use the w3c syntax.

srl295 · 2025-01-03T21:50:42Z

Looks good!

Thanks!

The EBNF links should point to https://unicode.org/reports/tr35/#ebnf. (Since the LDML EBNF is a superset of the vanilla EBNF that should work.

I will add this:

The following is the [LDML EBNF](./tr35.md#ebnf) format for the grammar:

The validity / well-formedness constraints (eg no more than 9 capture groups), should use the w3c syntax.

The 9 capture groups is 9 inner capture groups. So for example, valid:

/(first)(second)(third)(fourth)(fifth)(sixth)(seventh)(eighth)(?:And possibly, (ninth))?/

but invalid:

/(first)(second)(third)(fourth)(fifth)(sixth)(seventh)(eighth)(?:And possibly, (ninth))?(?:But not, (tenth)!)?/

because it's a nested group it may be challenging to express in the grammar, it's more like a resource (slot) limit.

jira-pull-request-webhook · 2025-01-03T21:51:47Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

srl295 · 2025-01-03T21:55:32Z

I'd like to get some Java based tooling, but that will be in a separate PR.
Edit: ran into some toolchain snags, so java tooling not ready yet. I'd like to land this current PR first anyways.

docs/ldml/tr35-keyboards.md

miloush · 2025-01-04T15:55:53Z

docs/ldml/tr35-keyboards.md

+           | DIGIT
+           | '_'
+ASCII-CTRLS
+         ::= [#x1-#x8#xB-#xC#xE-#x1F]


these are not allowed in XML 1.0

docs/ldml/tr35-keyboards.md

srl295 · 2025-01-04T23:00:19Z

Yes, my comment about surrogates is redundant because they are already covered in the BNF. As to errors, what I was trying to express is that an implementation would reject a keyboard layout with an error… But I can refer to that as a constraint for consistency

macchiati

I requested changes for the constraints.

- per review

- remove illegal ctrls - reorder members - cleanup

macchiati · 2025-01-08T23:17:00Z

docs/ldml/tr35-keyboards.md

+```ebnf
+[ wfc: No more than 9 capture groups may be present. ]
+[ vc: all variables referenced must be defined in the <variables> element ]
+[ vc: The CLDR repository may define additional constraints on the repertoire, such as requiring all characters to be in a published Unicode version and disallowing private-use characters. ]


Suggested change

[ vc: The CLDR repository may define additional constraints on the repertoire, such as requiring all characters to be in a published Unicode version and disallowing private-use characters. ]

[ vc: If a keyboard definitions is submitted to the the CLDR repository, it must satisfy additional constraints on the character repertoire. For more information, see [CLDR keyboard repertoire constraints](#repertoire-constraints). ]

We have to point to where those constraints are documented. So add a little section header for that and point to it. Also make a change to a similar line below.

I suggest that the contents of that section be:

No characters can have any of the following General_Category values in the latest version of the Unicode Standard:

Private Use (Co)

Surrogate (Cs)

Unassigned (Cn)

Not sure we can link from ebnf to spec.

Also the repository requirements are a separate activity - I don't see a reason to specify them here in detail (though I'll take the list back to kbd-wg). Perhaps it's better to just say- see a future version of the spec.

What about this

[ vc: If a keyboard definitions is submitted to the the CLDR repository, it must satisfy additional constraints on the character repertoire. ]

The problem with that is there is no way for the reader to find out what those requirements are — we can't have undefined vc requirements.

Then I'd propose to drop it from vc and just note. Or not even note. We don't have the requirements yet.

Yes there's no way to find out the requirements because they are future.

docs/ldml/tr35-keyboards.md

srl295 · 2025-01-09T14:09:29Z

@miloush how does it look?

srl295 self-assigned this Jan 1, 2025

srl295 force-pushed the srl295/kbd/cldr-18197/bnf branch from 121aab0 to afa4af5 Compare January 2, 2025 20:04

srl295 changed the title ~~CLDR-18197 kbd: update spec to mention abnf~~ CLDR-18197 kbd: update spec with ABNF Jan 2, 2025

srl295 force-pushed the srl295/kbd/cldr-18197/bnf branch from a461f03 to 308448c Compare January 3, 2025 18:14

srl295 changed the title ~~CLDR-18197 kbd: update spec with ABNF~~ CLDR-18197 kbd: add EBNF to spec for transforms Jan 3, 2025

CLDR-16836 kbd: add EBNF to spec, ABNF to data for transform grammar

d38f212

- add keyboard abnf and sample files and automated tests Temporarily skip non-BMP chars, see hildjj/node-abnf#25 which is being fixed.

srl295 force-pushed the srl295/kbd/cldr-18197/bnf branch from 308448c to d38f212 Compare January 3, 2025 21:31

srl295 changed the title ~~CLDR-18197 kbd: add EBNF to spec for transforms~~ CLDR-16836 kbd: add EBNF to spec for transforms Jan 3, 2025

macchiati requested changes Jan 3, 2025

View reviewed changes

CLDR-16836 kbd: per review, link to LDML ebnf

c0e32b4

srl295 force-pushed the srl295/kbd/cldr-18197/bnf branch from 199f93e to c0e32b4 Compare January 3, 2025 21:51

srl295 requested a review from macchiati January 3, 2025 21:53

srl295 marked this pull request as ready for review January 3, 2025 21:54

srl295 requested review from mcdurdin, xadxura, miloush and sven-oly January 3, 2025 21:54

miloush reviewed Jan 4, 2025

View reviewed changes

docs/ldml/tr35-keyboards.md Outdated Show resolved Hide resolved

miloush reviewed Jan 4, 2025

View reviewed changes

docs/ldml/tr35-keyboards.md Outdated Show resolved Hide resolved

miloush reviewed Jan 4, 2025

View reviewed changes

docs/ldml/tr35-keyboards.md Show resolved Hide resolved

miloush reviewed Jan 4, 2025

View reviewed changes

docs/ldml/tr35-keyboards.md Outdated Show resolved Hide resolved

macchiati requested changes Jan 8, 2025

View reviewed changes

srl295 added 2 commits January 8, 2025 16:22

CLDR-16836 kbd: ebnf: move spec body content into wfc / vc

5dfac4f

- per review

CLDR-16836 kbd: ebnf: update BNF per review

98d36cb

- remove illegal ctrls - reorder members - cleanup

srl295 requested review from macchiati and miloush January 8, 2025 22:46

CLDR-16836 merge from main

a7438e2

macchiati requested changes Jan 8, 2025

View reviewed changes

srl295 commented Jan 8, 2025

View reviewed changes

docs/ldml/tr35-keyboards.md Outdated Show resolved Hide resolved

docs/ldml/tr35-keyboards.md Outdated Show resolved Hide resolved

srl295 added 2 commits January 8, 2025 18:00

CLDR-16836 kbd ebnf: remove future vc

f2177e1

CLDR-16836 kbd ebnf: remove future vc

2054511

macchiati approved these changes Jan 9, 2025

View reviewed changes

srl295 merged commit 2b78422 into unicode-org:main Jan 9, 2025
10 checks passed

srl295 deleted the srl295/kbd/cldr-18197/bnf branch January 9, 2025 17:45

srl295 mentioned this pull request Jan 9, 2025

CLDR-16836 kbd: ebnf: move tests to js out of shell, check XML files #4270

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLDR-16836 kbd: add EBNF to spec for transforms #4261

CLDR-16836 kbd: add EBNF to spec for transforms #4261

srl295 commented Jan 1, 2025 •

edited

Loading

jira-pull-request-webhook bot commented Jan 2, 2025

srl295 commented Jan 2, 2025

srl295 commented Jan 2, 2025 •

edited

Loading

jira-pull-request-webhook bot commented Jan 3, 2025

jira-pull-request-webhook bot commented Jan 3, 2025

macchiati left a comment

srl295 commented Jan 3, 2025 •

edited

Loading

jira-pull-request-webhook bot commented Jan 3, 2025

srl295 commented Jan 3, 2025 •

edited

Loading

miloush Jan 4, 2025

srl295 commented Jan 4, 2025 via email

macchiati left a comment

macchiati Jan 8, 2025

srl295 Jan 8, 2025

srl295 Jan 8, 2025

macchiati Jan 8, 2025

srl295 Jan 8, 2025 •

edited

Loading

srl295 commented Jan 9, 2025

	[ vc: The CLDR repository may define additional constraints on the repertoire, such as requiring all characters to be in a published Unicode version and disallowing private-use characters. ]
	[ vc: If a keyboard definitions is submitted to the the CLDR repository, it must satisfy additional constraints on the character repertoire. For more information, see [CLDR keyboard repertoire constraints](#repertoire-constraints). ]

CLDR-16836 kbd: add EBNF to spec for transforms #4261

CLDR-16836 kbd: add EBNF to spec for transforms #4261

Conversation

srl295 commented Jan 1, 2025 • edited Loading

jira-pull-request-webhook bot commented Jan 2, 2025

srl295 commented Jan 2, 2025

srl295 commented Jan 2, 2025 • edited Loading

jira-pull-request-webhook bot commented Jan 3, 2025

jira-pull-request-webhook bot commented Jan 3, 2025

macchiati left a comment

Choose a reason for hiding this comment

srl295 commented Jan 3, 2025 • edited Loading

jira-pull-request-webhook bot commented Jan 3, 2025

srl295 commented Jan 3, 2025 • edited Loading

miloush Jan 4, 2025

Choose a reason for hiding this comment

srl295 commented Jan 4, 2025 via email

macchiati left a comment

Choose a reason for hiding this comment

macchiati Jan 8, 2025

Choose a reason for hiding this comment

srl295 Jan 8, 2025

Choose a reason for hiding this comment

srl295 Jan 8, 2025

Choose a reason for hiding this comment

macchiati Jan 8, 2025

Choose a reason for hiding this comment

srl295 Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

srl295 commented Jan 9, 2025

srl295 commented Jan 1, 2025 •

edited

Loading

srl295 commented Jan 2, 2025 •

edited

Loading

srl295 commented Jan 3, 2025 •

edited

Loading

srl295 commented Jan 3, 2025 •

edited

Loading

srl295 Jan 8, 2025 •

edited

Loading