python: Refine highlighting

[JaagupAverin]

Fixes:

• Types in binary unions as per PEP 604 not highlighted;
• except* keyword not highlighted;
• Classes beginning with _ not recognized as such, however _ is a valid first character for private classes; additionally the regex for parsing constant/class names appeared inconsistent and incomplete so was adjusted;
• Builtin types such as float, dict, etc not recognized as types;
• Update: decorators with arguments not recognized as decorators;
• Update: docstrings after type alias assignments not recognized as docstrings;
• Update: and/in/is/not/or/is not/not in not capturable as keywords;
• Update: decorators with "nesting" (@x.y.z) not recognized as decorators;

Before:

After:

Release Notes:

• N/A

[arv-anshul]

Hi @JaagupAverin, I have some issues around Syntax Highlighting too. Review and change it, if you find it right.

1. Mark these not, in, is operator as operator.special.

   📷 Screenshot

   

2. Identify # TODO:, # FIXME: as special comment in comments (comments.special).

3. (I don't know this); Can we identify function parameter as function.parameter.

Thank You!

[JaagupAverin]

Hi @JaagupAverin, I have some issues around Syntax Highlighting too. Review and change it, if you find it right.

1. Mark these not, in, is operator as operator.special.

📷 Screenshot

2. Identify # TODO:, # FIXME: as special comment in comments (comments.special).
3. (I don't know this); Can we identify function parameter as function.parameter.

Thank You!

1. Personally I'm for extra configurability but I'm not sure if Zed team might be worried about polluting the grammar with very specific non standard rules? @osiewicz ? I think the current list of captures is already out of date and I've thought if and where I should document the all the new captures for python?
2. I really like this personally, would add it too. Again I'd have to defer to Zed team if there is planned a more standard/documented way of approaching TODOs in code?
3. This is doable in the function signature but not in the body without LSP highlighting support so I'm not sure if it would be very helpful as a half baked solution yet.

[uncenter]

I really like this personally, would add it too. Again I'd have to defer to Zed team if there is planned a more standard/documented way of approaching TODOs in code?

The way this is done elsewhere (nvim-treesitter) is by adding the following to injections.scm:

((comment) @injection.content
  (#set! injection.language "comment"))

And then https://github.com/stsewd/tree-sitter-comment (which syntax highlights those kind of TODO comments) is added under the "comment" language name, so injected for comments as such.

This currently isn't standard practice in Zed though. See #9082 (comment) and #11895.

[uncenter]

1. Mark these not, in, is operator as operator.special.

I believe keyword.operator is what is normally used for these?

3. Can we identify function parameter as function.parameter.

I believe variable.parameter is what is standard for that.

[JaagupAverin]

@uncenter
Nice. Regarding documentation of all this, do I understand that this table shows only the captures that the official themes use, but not all the captures actually available? Could this table then be updated to show ALL of the captures, with new columns for whether these captures are supported by the official themes and also (optinally) if the capture is language-specific (e.g python's string.doc)

[uncenter]

Regarding documentation of all this, do I understand that this table shows only the captures that the official themes use, but not all the captures actually available?

This is correct. Themes support capturing any arbitrary capture. The table in the documentation is a list of common captures. See Helix's list of scopes: https://docs.helix-editor.com/themes.html?highlight=inherits#syntax-highlighting. And Neovim's, which is probably the most comprehensive: https://neovim.io/doc/user/treesitter.html#treesitter-highlight-groups.

Could this table then be updated to show ALL of the captures, with new columns for whether these captures are supported by the official themes

Definitely. I'll try to tackle updating it at some point soon :)

also (optinally) if the capture is language-specific

Captures should not be language specific really, they are supposed to be general (though it should be possible to scope a general capture for a theme to match for just one language, see #9461 (comment)).

(e.g python's string.doc)

I believe string.documentation is more commonly used (https://github.com/nvim-treesitter/nvim-treesitter/blob/0c26a5c4bc407fdbca545ab25cf6084fd069444b/queries/python/highlights.scm#L201-L222).

[arv-anshul]

Ok, Thanks for responding and correcting my comment @JaagupAverin and @uncenter.

Above @JaagupAverin mentioned about semantic tokens highlighting through LSP, specifically basedpyright (which has support for now). Is this feature is available now to use?

[JaagupAverin]

I added one more commit for classifying keyword operators as keyword.operator.
For #TODOs, I suppose #11895 would be a much better solution than a temporary hack.
Also I had a look at the nvim implementation for implementing @variable.parameter and to my mind that is quite a hardcore solution for something of minimal value (the parameters in declaration and body will be highlighted differently). Nvim has a few hacks that won't work correctly in all cases it seems.

[uncenter]

Nvim has a few hacks that won't work correctly in all cases it seems.

Can you elaborate? That list of queries for variable parameters seems fine to me fwiw.

[JaagupAverin]

Can you elaborate? That list of queries for variable parameters seems fine to me fwiw.

There's one where any 2nd argument to isinstance is always treated as type. Its somewhat giving special treatment to a function and might cause an identifier to switch colors back and forth (if it not properly identified as a type overall). There's a few other cases where best attempts are made to highlight types using their relations to keywords like TypeAlias, but the identifier will again just pop up with different colours throughout the file which imo is more confusing.

((call
  function: (identifier) @_isinstance
  arguments: (argument_list
    (_)
    (identifier) @type))
  (#eq? @_isinstance "isinstance"))

There's another that tries to fix what I believe is a bug in the python tree-sitter. This workaround won't always work since it could have several kinds of parents, not just type.

(type
  (subscript
    (identifier) @type)) ; type subscript: Tuple[int]

btw the only reason I picked these out was because I have been trying to solve the same issues here, but came to the conclusion that they're not worth the special handling.
Also I thought there was another hack regarding "dotted" decorators, that I had had issues with, but after trying it again it does seem to work :)

[osiewicz]

Thanks!

[osiewicz]

I've just noticed that there's a bunch of unresolved questions in this thread; I feel like these can be tackled in a separate PR.