Skip to content

Commit

Permalink
Handle parentheses when formatting slice expressions
Browse files Browse the repository at this point in the history
**Summary** Fix the formatter crash with `x[(1) :: ]` and related code.

**Problem** For assigning comments in slices in subscripts, we need to find the positions of the colons to assign comments before and after the colon to the respective lower/upper/step node (or dangling in that section). Formatting `x[(1) :: ]` was broken because we were looking for a `:` after the `1` but didn't consider that there could be a `)` outside the range of the lower node, which contains just the `1` and no optional parentheses.

**Solution** Use the simple tokenizer directly and skip all closing parentheses.

**Test Plan** I added regression tests.

Closes #5733
  • Loading branch information
konstin committed Jul 19, 2023
1 parent a1b1e5a commit 0634adc
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 16 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,8 @@ def a():

# Regression test for https://github.com/astral-sh/ruff/issues/5605
f = "f"[:,]

# Regression test for https://github.com/astral-sh/ruff/issues/5733
g1 = "g"[(1):(2)]
g2 = "g"[(1):(2):(3)]

33 changes: 17 additions & 16 deletions crates/ruff_python_formatter/src/expression/expr_slice.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use rustpython_parser::ast::{Expr, Ranged};
use ruff_formatter::prelude::{hard_line_break, line_suffix_boundary, space, text};
use ruff_formatter::{write, Buffer, Format, FormatError, FormatResult};
use ruff_python_ast::node::{AnyNodeRef, AstNode};
use ruff_python_whitespace::{first_non_trivia_token, Token, TokenKind};
use ruff_python_whitespace::{SimpleTokenizer, Token, TokenKind};

use crate::comments::{dangling_comments, SourceComment};
use crate::context::PyFormatContext;
Expand Down Expand Up @@ -162,10 +162,12 @@ pub(crate) fn find_colons(
let after_lower = lower
.as_ref()
.map_or(range.start(), |lower| lower.range().end());
let first_colon =
first_non_trivia_token(after_lower, contents).ok_or(FormatError::SyntaxError {
message: "Din't find any token for slice first colon",
})?;
let mut tokens = SimpleTokenizer::new(contents, TextRange::new(after_lower, range.end()))
.skip_trivia()
.skip_while(|token| token.kind == TokenKind::RParen);
let first_colon = tokens.next().ok_or(FormatError::SyntaxError {
message: "Din't find any token for slice first colon",
})?;
if first_colon.kind != TokenKind::Colon {
return Err(FormatError::SyntaxError {
message: "slice first colon token was not a colon",
Expand All @@ -175,17 +177,16 @@ pub(crate) fn find_colons(
let after_upper = upper
.as_ref()
.map_or(first_colon.end(), |upper| upper.range().end());
// At least the closing bracket must exist, so there must be a token there
let next_token =
first_non_trivia_token(after_upper, contents).ok_or(FormatError::SyntaxError {
message: "Din't find any token for slice second colon",
})?;
let second_colon = if next_token.kind == TokenKind::Colon {
debug_assert!(
next_token.range.start() < range.end(),
"The next token in a slice must either be a colon or the closing bracket"
);
Some(next_token)
let mut tokens = SimpleTokenizer::new(contents, TextRange::new(after_upper, range.end()))
.skip_trivia()
.skip_while(|token| token.kind == TokenKind::RParen);
let second_colon = if let Some(token) = tokens.next() {
if token.kind != TokenKind::Colon {
return Err(FormatError::SyntaxError {
message: "Expected a colon for the second colon token",
});
}
Some(token)
} else {
None
};
Expand Down

0 comments on commit 0634adc

Please sign in to comment.