From fe549f161de91e73998493905318061f81265731 Mon Sep 17 00:00:00 2001 From: Nicholas Nethercote Date: Fri, 22 Apr 2022 14:55:10 +1000 Subject: [PATCH] Fix macro matching for `NoDelim` delimited sequences. PR #95159 unintentionally changed the behaviour of declarative macro matching for `NoDelim` delimited sequences. - Before #95159, delimiters were implicit in `mbe::Delimited`. When doing macro matching, normal delimiters were generated out of thin air as necessary, but `NoDelim` delimiters were not. This was done within `TokenTree::{get_tt,len}`. - After #95159, the delimiters were explicit. There was an unintentional change whereby `NoDelim` delimiters were represented explicitly just like normal delimeters. - #95555 introduced a new matcher representation (`MatcherLoc`) and the `NoDelim` delimiters were made explicit within it, just like `mbe::Delimited`. - #95797 then changed `mbe::Delimited` back to having implicit delimiters, but because matching is now being done on `MatcherLoc`, the behavioural change persisted. The fix is simple: remove the explicit `NoDelim` delimiters in the `MatcherLoc` representation. This gives back the original behaviour. As for why this took so long to be found: it seems that `NoDelim` sequences are unusual. It took a macro that generates another macro (from the `yarte_lexer` crate, found via a crater run) to uncover this. Fixes #96305. --- compiler/rustc_expand/src/mbe/macro_parser.rs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/compiler/rustc_expand/src/mbe/macro_parser.rs b/compiler/rustc_expand/src/mbe/macro_parser.rs index 8f260e1cdb5c8..4eb69fda36514 100644 --- a/compiler/rustc_expand/src/mbe/macro_parser.rs +++ b/compiler/rustc_expand/src/mbe/macro_parser.rs @@ -144,9 +144,13 @@ pub(super) fn compute_locs(matcher: &[TokenTree]) -> Vec { let close_token = Token::new(token::CloseDelim(delimited.delim), span.close); locs.push(MatcherLoc::Delimited); - locs.push(MatcherLoc::Token { token: open_token }); + if delimited.delim != token::DelimToken::NoDelim { + locs.push(MatcherLoc::Token { token: open_token }); + } inner(&delimited.tts, locs, next_metavar, seq_depth); - locs.push(MatcherLoc::Token { token: close_token }); + if delimited.delim != token::DelimToken::NoDelim { + locs.push(MatcherLoc::Token { token: close_token }); + } } TokenTree::Sequence(_, seq) => { // We can't determine `idx_first_after` and construct the final