-
Notifications
You must be signed in to change notification settings - Fork 13k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pretty-printing adds additional parentheses, breaking the proc-macro reparse check #75734
Comments
Also causes dtolnay/async-trait#118. Rustc inserted unnecessary parentheses around the closure before calling async_trait, then lost all the location information on the source tokens and blamed the user for handwriting those unnecessary parentheses. |
It would break expressions with "virtual parentheses", like |
Cases like this could be caught when comparing the original and re-parsed token streams though, because the "virtual parentheses" ( So we can print without parentheses -> reparse -> compare, and then re-print with parentheses if the comparison fails. |
…etrochenkov Attach tokens to all AST types used in `Nonterminal` We perform token capturing when we have outer attributes (for nonterminals that support attributes - e.g. `Stmt`), or when we parse a `Nonterminal` for a `macro_rules!` argument. The full list of `Nonterminals` affected by this PR is: * `NtBlock` * `NtStmt` * `NtTy` * `NtMeta` * `NtPath` * `NtVis` * `NtLiteral` Of these nonterminals, only `NtStmt` and `NtLiteral` (which is actually just an `Expr`), support outer attributes - the rest only ever have token capturing perform when they match a `macro_rules!` argument. This makes progress towards solving rust-lang#43081 - we now collect tokens for everything that might need them. However, we still need to handle `#[cfg]`, inner attributes, and misc pretty-printing issues (e.g. rust-lang#75734) I've separated the changes into (mostly) independent commits, which could be split into individual PRs for each `Nonterminal` variant. The purpose of having them all in one PR is to do a single Crater run for all of them. Most of the changes in this PR are trivial (adding `tokens: None` everywhere we construct the various AST structs). The significant changes are: * `ast::Visibility` is changed from `type Visibility = Spanned<VisibilityKind>` to a `struct Visibility { kind, span, tokens }`. * `maybe_collect_tokens` is made generic, and used for both `ast::Expr` and `ast::Stmt`. * Some of the statement-parsing functions are refactored so that we can capture the trailing semicolon. * `Nonterminal` and `Expr` both grew by 8 bytes, as some of the structs which are stored inline (rather than behind a `P`) now have an `Option<TokenStream>` field. Hopefully the performance impact of doing this is negligible.
…etrochenkov Attach tokens to all AST types used in `Nonterminal` We perform token capturing when we have outer attributes (for nonterminals that support attributes - e.g. `Stmt`), or when we parse a `Nonterminal` for a `macro_rules!` argument. The full list of `Nonterminals` affected by this PR is: * `NtBlock` * `NtStmt` * `NtTy` * `NtMeta` * `NtPath` * `NtVis` * `NtLiteral` Of these nonterminals, only `NtStmt` and `NtLiteral` (which is actually just an `Expr`), support outer attributes - the rest only ever have token capturing perform when they match a `macro_rules!` argument. This makes progress towards solving rust-lang#43081 - we now collect tokens for everything that might need them. However, we still need to handle `#[cfg]`, inner attributes, and misc pretty-printing issues (e.g. rust-lang#75734) I've separated the changes into (mostly) independent commits, which could be split into individual PRs for each `Nonterminal` variant. The purpose of having them all in one PR is to do a single Crater run for all of them. Most of the changes in this PR are trivial (adding `tokens: None` everywhere we construct the various AST structs). The significant changes are: * `ast::Visibility` is changed from `type Visibility = Spanned<VisibilityKind>` to a `struct Visibility { kind, span, tokens }`. * `maybe_collect_tokens` is made generic, and used for both `ast::Expr` and `ast::Stmt`. * Some of the statement-parsing functions are refactored so that we can capture the trailing semicolon. * `Nonterminal` and `Expr` both grew by 8 bytes, as some of the structs which are stored inline (rather than behind a `P`) now have an `Option<TokenStream>` field. Hopefully the performance impact of doing this is negligible.
The following code:
is pretty-printed as:
The extra parenthesis completely change the structure of the re-parsed
TokenStream
(we now have tokens wrapped in aTokenTree::Delimited
), breaking the pretty-print and reparse check done during proc-macro expansion. This results in an instance of #43081.This behavior was introduced by #43742. The PR description states that is intended to "make pprust easier to use with programmatically constructed ASTs." - however, I can't find a concrete example of code that is affected by this.
I see a few possible ways of fixing this:
TokenStream
.ExprKind::Paren
when pretty-printing the AST. I'm not exactly certain what the consequences of doing this are. Do we ever end up pretty-printing code-generated AST nodes (e.g. from builtin macros), other than when explicitly requested via-Z unpretty=expanded
?If we could get it to work without breaking anything, solution 3 seems like the cleanest way of solving this.
cc @petrochenkov
The text was updated successfully, but these errors were encountered: