feat(gazelle): Update to latest version of go-tree-sitter #2413
Conversation
There was a problem hiding this comment.
Hmm... still seems to not work on our pyle repo. I'll have more time to investigate early next week.
Please add a test case for python 3.12 syntax to gazelle/python/testdata. Specifically, we want to make sure that imports, gazelle annotations, and if __name__ that come after py3.12-only syntax are still parsed. For example:
# my_baz.py
import foo
# gazelle:include_dep //:bar
def baz[T]() -> T: # 3.12-only syntax
pass
# these still get parsed
import foobar
# gazelle:include_dep //:hello
if __name__ == "__main__":
passShould generate (assuming per-file target):
py_binary(
name = "my_baz",
srcs = ["my_baz.py"],
deps = [
"//:bar",
"//:foo",
"//:foobar",
"//:hello",
],
)If you only get 2 deps or a py_library, then the parsing still failed 😢
CHANGELOG.md
Outdated
| @@ -52,7 +52,7 @@ Unreleased changes template. | |||
|
|
|||
| {#v0-0-0-changed} | |||
| ### Changed | |||
| * Nothing yet. | |||
| * (gazelle): Update to latest version of go-tree-sitter. | |||
There was a problem hiding this comment.
Please include that this adds support for python 3.12 grammar (and 3.13?)
gazelle/go-tree-sitter.patch
Outdated
There was a problem hiding this comment.
nit: please move to a "patches" directory
|
maybe we can publish go-tree-sitter with patches to bazel registry? like https://registry.bazel.build/modules/circl |
While investigating #2396 and why #2413 doesn't appear to be working for us, I realized that one of the things I was making heavy use of was additional parser logging that I had added. This adds some of that logging. I also throw in some documentation because I found it helpful. Users can (attempt to) get additional parse failure information by setting the `GAZELLE_VERBOSE` environment variable to `1`. Eg: ```console $ GAZELLE_VERBOSE=1 bazel run //:gazelle ``` Here are some example logs: ```console $ GAZELLE_VERBOSE=1 bazel run //:gazelle INFO: Invocation ID: a4e026d8-17df-426c-b1cc-d3980690dd53 ... INFO: Running command line: bazel-bin/gazelle INFO: Streaming build results to: https://btx.cloud.google.com/invocations/a4e026d8-17df-426c-b1cc-d3980690dd53 gazelle: WARNING: failed to parse "hello/get_deps.py". The resulting BUILD target may be incorrect. gazelle: Parse error at {Row:1 Column:0}: def search_one_more_level[T](): gazelle: The above was parsed as: (ERROR (identifier) (call function: (list (identifier)) arguments: (argument_list))) gazelle: ERROR: failed to generate target "//hello:get_deps" of kind "py_library": a target of kind "pyle_py_binary" with the same name already exists. Use the '# gazelle:python_library_naming_convention' directive to change the naming convention. $ $ bazel run //:gazelle INFO: Invocation ID: 726c9fd6-f566-4c30-95ef-c4781ad155de ... INFO: Running command line: bazel-bin/gazelle INFO: Streaming build results to: https://btx.cloud.google.com/invocations/726c9fd6-f566-4c30-95ef-c4781ad155de gazelle: WARNING: failed to parse "hello/get_deps.py". The resulting BUILD target may be incorrect. gazelle: ERROR: failed to generate target "//hello:get_deps" of kind "py_library": a target of kind "pyle_py_binary" with the same name already exists. Use the '# gazelle:python_library_naming_convention' directive to change the naming convention. ``` --------- Co-authored-by: Richard Levasseur <richardlev@gmail.com> Co-authored-by: Richard Levasseur <rlevasseur@google.com>
Include a patch to the build files generated by the go_repository to support building with uplevel files references.
de98fae to
f915bd1
Compare
|
I think I got things working now. Had to remove I also moved the patch file into a |
|
Just relaying info that @maffoo and I had at work: This still doesn't work for
It's looking like we might have to do that. We can apply patches like @maffoo has done, but then those patches won't be used when some other project uses @hunshcn How do you recommend we publish to BCR? Fork go-tree-sitter, add MODULE.bazel and rules, and go from there? Or is there another option? |
|
Just like bazelbuild/bazel-central-registry#674 |
|
Ah thanks, that process looks pretty easy. I'll start it now. |
|
I filed bazel-contrib/bazel-gazelle#1984 to propose allowing default patches on libraries included by go_deps and put up bazel-contrib/bazel-gazelle#1985 to actually implement this. We'll see what the bazel-gazelle maintainers say... |
|
Will the BCR option not be limited to the bzlmod implementation? I have a fork of the gazelle plugin and made this upgrade using http_archive approach, which is compatible with MODULE.bazel and WORKSPACE. I can contribute the change here. |
|
Superseded by #2496. |
… Parameter Syntax) by using dougthor42's fork of go-tree-sitter (#2496) Replaces #2413. Fixes #2396. This updates the `go-tree-sitter` dependency to use my fork that includes `BUILD.bazel` files. Specifically, the `BUILD.bazel` files in the fork include references to top-level code like `array.h` which the original Gazelle-generated files for `go-tree-sitter` were not able to handle. I also include the test cases that @maffoo created in #2413 and verified that they (a) fail before the fix and (b) pass after the fix. The fork is: https://github.com/dougthor42/go-tree-sitter The branch that includes all changes is: https://github.com/dougthor42/go-tree-sitter/tree/for-rules-python-gazelle-plugin A couple notes: + I have a PR open to get `go-tree-sitter` into BCR [here](bazelbuild/bazel-central-registry#3366). However: 1. I'm having trouble getting tests to pass and to get things running locally to validate it 2. Using BCR would not fix things for people who still use WORKSPACE (right?) + The fork is _mostly_ [autogenerated BUILD.bazel files from gazelle](smacker/go-tree-sitter@cfa9bdf) but also contains: + [manual updates so that build files reference the toplevel `array.h` and other files](smacker/go-tree-sitter@63f89cd) + [replace all `smacker` with `dougthor42` so that `go build` works](smacker/go-tree-sitter@8a73cbd) + various other more minor things. + I was unable to get `go mod edit -replace` to work, so I've just manually updated `go.mod` and whatnot everywhere. If someone with more go knowledge has a suggestion I'm happy to hear it. --------- Co-authored-by: Matthew Neeley <maffoo@google.com> Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
The latest version of go-tree-sitter includes grammer changes for python 3.12. To use it, we need to patch the build files generated by the go_repository to support building with uplevel files references. This should fix #2396.