Fix token declaration and verification in the model lexer #25137
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This appears to not have too much of a detrimental effect, but it doesn't seem to be what is intended either. antlr doesn't mind that `PLUS` isn't declared in `tokens` and happily uses the `PLUS` that appears later in the file, but the generated RustLexer.tokens had PLUS at the end rather than where it was intended: NOT=10 TILDE=11 PLUT=12 MINUS=13 ... PLUS=56
There were some tokens used in the grammar but not declared. Antlr doesn't really seem to care and happily uses them, but they appear in RustLexer.tokens in a potentially-unexpected order.
To prevent the reference grammar from getting out of sync with the real grammar, panic if RustLexer.tokens contains an unknown token in a similar way that verify.rs panics if it encounters an unknown binary operation token.
|
r? @pcwalton (rust_highfive has picked a reviewer for you, use r? to override) |
steveklabnik
added a commit
to steveklabnik/rust
that referenced
this pull request
May 7, 2015
…, r=alexcrichton I know this is the most trivial thing since it's *just* the model lexer, not the real lexer, but [it is simpler to read](rust-lang#15883 (comment)) and it'd be great if it was up to date but [it's been rotting](rust-lang#22379) and this is a tiny bit of that. Thanks!!!!!!
steveklabnik
added a commit
to steveklabnik/rust
that referenced
this pull request
May 7, 2015
…, r=alexcrichton I know this is the most trivial thing since it's *just* the model lexer, not the real lexer, but [it is simpler to read](rust-lang#15883 (comment)) and it'd be great if it was up to date but [it's been rotting](rust-lang#22379) and this is a tiny bit of that. Thanks!!!!!!
Manishearth
added a commit
to Manishearth/rust
that referenced
this pull request
May 18, 2015
…richton Hiiii soooo I'm trying to get the reference grammar and associated tests running again, and I swear I tested before but I must have had multiple things going on when I did, because the change I made in rust-lang#25137 to verify.rs is totally wrong. The RustLexer.tokens file that antlr generates has two sections: ``` EQ=1 LT=2 LE=3 EQEQ=4 NE=5 ... COMMENT=56 SHEBANG=57 UTF8_BOM=58 '='=1 '<'=2 '<='=3 '=='=4 ... ``` and verify.rs is only interested in the first half-- the `continue` is to ignore the second half. In 9c7d5ae, I made it panic instead. I was trying to make sure verify.rs handled everything that might happen in the first half and complain if it didn't. That would mean the reference grammar was out of sync with at least verify.rs, if not the real grammar. But it's totally ok for verify.rs to not handle the entire second half of the file. I'm sorry for breaking this :( Good thing these tests aren't being run regularly yet...? 😳
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I know this is the most trivial thing since it's just the model lexer, not the real lexer, but it is simpler to read and it'd be great if it was up to date but it's been rotting and this is a tiny bit of that.
Thanks!!!!!!