Existing protoc output causes issues with multiple rules

[bweston92]

So I have imported my protoc output into my project directly.

However when running Gazelle to generate build files I get the output displayed underneath. Probably because they already have BUILD files here: https://github.com/grpc-ecosystem/grpc-gateway/tree/master/protoc-gen-swagger/options
How would I get around this?

gazelle: found packages date () and money () in /workspace/src/github.com/legalweb/{prjectName}/idl/{prjectName}/protobuf
gazelle: /workspace/src/github.com/legalweb/{prjectName}/pkg/dep/sources/https---github.com-grpc--ecosystem-grpc--gateway/codegenerator/BUILD.bazel: go_default_xtest is no longer necessary. Run 'gazelle fix' to rename to go_default_test.
gazelle: /workspace/src/github.com/legalweb/{prjectName}/pkg/dep/sources/https---github.com-grpc--ecosystem-grpc--gateway/examples/integration/BUILD.bazel: go_default_xtest is no longer necessary. Run 'gazelle fix' to rename to go_default_test.
gazelle: /workspace/src/github.com/legalweb/{prjectName}/pkg/dep/sources/https---github.com-grpc--ecosystem-grpc--gateway/runtime/BUILD.bazel: go_default_xtest is no longer necessary. Run 'gazelle fix' to squash with go_default_test.
gazelle: /workspace/src/github.com/legalweb/{prjectName}/pkg/dep/sources/https---github.com-grpc--ecosystem-grpc--gateway/utilities/BUILD.bazel: go_default_xtest is no longer necessary. Run 'gazelle fix' to rename to go_default_test.
gazelle: /workspace/src/github.com/legalweb/{prjectName}/src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/BUILD.bazel: go_default_xtest is no longer necessary. Run 'gazelle fix' to squash with go_default_test.
gazelle: /workspace/src/github.com/legalweb/{prjectName}/src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/utilities/BUILD.bazel: go_default_xtest is no longer necessary. Run 'gazelle fix' to rename to go_default_test.
gazelle: multiple rules (//src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and //src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from //src/{prjectName}/pbgo/{prjectName}/alie/v1:go_default_library
gazelle: multiple rules (//src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and //src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from //src/{prjectName}/pbgo/{prjectName}/brokers/v1:go_default_library
gazelle: multiple rules (//src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and //src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from //src/{prjectName}/pbgo/{prjectName}/clients/v1:go_default_library
gazelle: multiple rules (//src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and //src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from //src/{prjectName}/pbgo/{prjectName}/identities/v1:go_default_library
gazelle: multiple rules (//src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and //src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from //src/{prjectName}/pbgo/{prjectName}/introducers/v1:go_default_library
gazelle: multiple rules (//src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and //src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from //src/{prjectName}/pbgo/{prjectName}/lenders/v1:go_default_library
gazelle: multiple rules (//src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/internal:internal_go_proto and //src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/internal:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/runtime/internal" from //src/{prjectName}/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime:go_default_library

[jayconrod]

In general, I would not recommend vendoring projects that already have Bazel build files. Build files have absolute paths within the repository, so when the contents of a repository are vendored, they don't make sense anymore.

If the project is buildable with go build anyway, you may be able to delete the build files and re-generate them with Gazelle. You'll lose any customizations from that project though. You might also find //cmd/move_labels useful. It was written for this purpose, but there are a lot of corner cases, so it doesn't end up being very helpful.

If you can, I'd recommend accessing proto-related dependencies through external repository rules (go_repository, git_repository, http_archive in WORKSPACE). You may need to remove the external = "vendored" attribute from your //:gazelle rule. This will switch Gazelle's default dependency resolution (when no known go_library or go_proto_library satisfies a dependency) to use external repos. All the libraries you've already vendored will still be preferred over external repos, so this should work pretty seamlessly.

You may find gazelle update-repos useful for adding go_repository rules to WORKSPACE or importing from dep's Gopkg.lock file.

[bweston92]

Thanks for getting back to me, I managed to offload the vendor to use the go_repository import in the WORKSPACE however I now get the following which I believe is because of it not understanding some of it is vendored and some of it isn't.

cannot use g (type *"cosmicapis/src/vendor/google.gol
ang.org/grpc".Server) as type *"google.golang.org/grpc".Server in argument to identities.RegisterIdentitiesServer

Any ideas?

[jayconrod]

This error comes from a conflict between two libraries that are linked into the build: one vendored, one not.

This is a problem especially with gRPC and proto dependencies because go_proto_library has some implicit dependencies on the external libraries. So you may get conflicts by using go_proto_library and importing some of the same dependencies from regular Go code. google.golang.org/grpc is one of these dependencies.

Gazelle currently tries to avoid conflicts by hard coding dependency resolution for the Well Known Type libraries (e.g., github.com/golang/protobuf/ptypes/any always resolves to @com_github_golang_protobuf//ptypes/any:go_default_library, even if you have it vendored). We should probably extend this to all other implicit proto dependencies.

For now, try a similar workaround to this comment: in your vendor directory, replace go_proto_library and go_library rules for conflicting libraries with alias rules. For example:

# in vendor/google.golang.org/grpc
alias(
    name = "go_default_library",
    actual = "@org_golang_google_grpc//:go_default_library",
)

Other implicit dependencies are:

github.com/golang/protobuf/proto => @com_github_golang_protobuf//proto:go_default_library
golang.org/x/net/context => @org_golang_x_net//context:go_default_library

[bweston92]

Sorry I don't fully understand the issue it self or the solution.

I added a BUILD file in vendor/google.golang.org/grpc with

alias(
    name = "go_default_library",
    actual = "@org_golang_google_grpc//:go_default_library",
)

Re-ran gazelle and same error gets outputted. Would this be fixed by just not vendoring but using go_repositories instead? :/

[jayconrod]

Where exactly is the package cosmicapis/src/vendor/google.golang.org/grpc coming from? I assumed that was in the vendor directory of your repository. Is it in an external repo?

Would this be fixed by just not vendoring but using go_repositories instead?

That is the most likely thing to work, though vendor directories in external repositories may still interfere. That would be true whether you're using go build or Bazel though.

[jayconrod]

The fixes for bazel-contrib/rules_go#1548 have landed at tip of master and will be in the 0.13.0 release in a couple days. We now basically support two modes of use:

• If you want to use pre-generated .pb.go files and don't want to depend on protoc for generating code at build time, add # gazelle:proto disable_global to your root build file, and add build_file_proto_mode = "disable_global" to any go_repository rules that may contain protos.
• If you want to generate code from protos at build time, run Gazelle normally. You won't be able to vendor protoc, protoc-gen-go, gRPC, Well Known Types, or Google APIs, or anything those things depend on. If you run into conflicts because of this, you can prevent Gazelle from indexing specific directories using # gazelle:exclude comments. For example, you could add # gazelle:exclude vendor/google.golang.org/grpc to your root build file.

Please re-open if you're still running into issues after this.

[bweston92]

I have run into some trouble, I don't know if it is my fault or the packages fault any help would be appreciated.

So the steps I had taken is:

• I moved back to dep for dependency management.
• I added ignored for my Golang code generated from protobuf, and put in the repo.
• I added the snippet(1) in the BUILD file in the root.
• I ran dep ensure -update
• I ran bazel run //:gazelle
• I then had to run bazel run //:gazelle -- fix because of some warnings in aboves output(2) but this had the following warnings(3).
• I then tried to build a Go binary and got (4)

(1)

# gazelle:proto disable_global

(2)

gazelle: /workspace/src/github.com/legalweb/cosmic/src/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/BUILD.bazel: go_default_xtest is no longer necessary. Run 'gazelle fix' to squash with go_default_test.
gazelle: /workspace/src/github.com/legalweb/cosmic/src/vendor/github.com/grpc-ecosystem/grpc-gateway/utilities/BUILD.bazel: go_default_xtest is no longer necessary. Run 'gazelle fix' to rename to go_default_test.

(3)

gazelle: multiple rules (@cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and @cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from @cosmicapis//src/lwebco.de/cosmicapis/pb-go/cosmicapis/alie/v1:go_default_library
gazelle: multiple rules (@cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and @cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from @cosmicapis//src/lwebco.de/cosmicapis/pb-go/cosmicapis/brokers/v1:go_default_library
gazelle: multiple rules (@cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and @cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from @cosmicapis//src/lwebco.de/cosmicapis/pb-go/cosmicapis/clients/v1:go_default_library
gazelle: multiple rules (@cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and @cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from @cosmicapis//src/lwebco.de/cosmicapis/pb-go/cosmicapis/identities/v1:go_default_library
gazelle: multiple rules (@cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and @cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from @cosmicapis//src/lwebco.de/cosmicapis/pb-go/cosmicapis/introducers/v1:go_default_library
gazelle: multiple rules (@cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:options_go_proto and @cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options" from @cosmicapis//src/lwebco.de/cosmicapis/pb-go/cosmicapis/lenders/v1:go_default_library
gazelle: multiple rules (@cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/internal:internal_go_proto and @cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/internal:go_default_library) may be imported with "github.com/grpc-ecosystem/grpc-gateway/runtime/internal" from @cosmicapis//src/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime:go_default_library

(4)

GoCompile: missing strict dependencies:
        /home/bweston/.cache/bazel/_bazel_bweston/48208206adaa8cfcc39a42c2d001dfe6/sandbox/linux-sandbox/10/execroot/cosmicapis/src/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/handler.go: import of "github.com/grpc-ecosystem/grpc-gateway/runtime/internal"
Known dependencies are:
        github.com/golang/protobuf/jsonpb
        github.com/golang/protobuf/proto
        github.com/golang/protobuf/ptypes
        github.com/golang/protobuf/ptypes/any
        github.com/golang/protobuf/ptypes/duration
        github.com/golang/protobuf/ptypes/timestamp
        github.com/grpc-ecosystem/grpc-gateway/utilities
        google.golang.org/grpc/codes
        google.golang.org/grpc/grpclog
        google.golang.org/grpc/metadata
        google.golang.org/grpc/status
        github.com/golang/protobuf/jsonpb
        github.com/golang/protobuf/proto
        github.com/golang/protobuf/ptypes
        github.com/golang/protobuf/ptypes/any
        github.com/golang/protobuf/ptypes/duration
        github.com/golang/protobuf/ptypes/timestamp
        github.com/grpc-ecosystem/grpc-gateway/utilities
        google.golang.org/grpc/codes
        google.golang.org/grpc/grpclog
        google.golang.org/grpc/metadata
        google.golang.org/grpc/status
Check that imports in Go sources match importpath attributes in deps.

[jayconrod]

From the warnings in (2) and (3), it sounds like the build files in src/vendor were stale. The warnings in (3) come up because there are multiple libraries with the same importpath.

Normally stale build files are fine, because Gazelle can fix most things, but the old go_proto_library rules may cause problems. In disable and disable_global proto modes, Gazelle will ignore any existing proto_library and go_proto_library rules, assuming they were manually written. So you may need to get rid of those yourself. Something like buildozer delete $(bazel query 'kind(go_proto_library, //src/vendor/...)') might work (buildozer is here). Or if you don't have any manual modifications, just find src/vendor -name BUILD.bazel -delete then re-run Gazelle.

The error in (4) is related. When Gazelle fails to resolve an import because of a conflict, it won't write out a label in deps. So you'll get this missing strict dependencies error when you build.

[bweston92]

Stale how? I only just switched to using dep/vendored again to try this, I was using the gazelle update-repos -from_file={} that put them in the WORKSPACE.

[jayconrod]

By stale, I meant that the build files in vendor still have rules that were generated before you added # gazelle:proto disable_global. Those rules won't be removed automatically.

But wait: you said you're adding go_repository rules to WORKSPACE with gazelle update-repos. Does that mean you're not depending on the vendored libraries? If that's the case, you'll need to make sure Gazelle doesn't look in vendor for dependency resolution (# gazelle:exclude src/vendor). You'll also need to add build_file_proto_mode = "disable_global" to all go_repository rules in WORKSPACE that might contain protos.

[vvrpradeep]

@jayconrod Just curious to know the solution for the issue (4) and how can we identify the conflict?
I am using bazel 25.1 and rules_go 0.18.1

I am also facing the same issue while compilation of a .proto file with "grpc-gateway" annotations.

WORKSPACE changes: Below http_archive added to the workspace.

http_archive(
    name = "googleapi",
    sha256 = "51849d3ef693c88eb7692875eb444ef7131502e3fa200f25fc0a37b1e7e55ab5",
    strip_prefix = "googleapis-a111a53c0c6722afcd793b64724ceef7862db5b9",
    url = "https://github.com/googleapis/googleapis/archive/a111a53c0c6722afcd793b64724ceef7862db5b9.zip",
)

http_archive(
    name = "grpc-gateway",
    sha256 = "d3a8565e63fe65d45270bcafb804f0c3778b90218528efc22de2fa273c724bfc",
    strip_prefix = "grpc-gateway-1.9.4",
    url = "https://github.com/grpc-ecosystem/grpc-gateway/archive/v1.9.4.zip",
)

error is:
job-reporting-service.pb.go: import of "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options"
job-reporting-service.pb.go: import of "google.golang.org/genproto/googleapis/api/annotations"
Known dependencies are:
github.com/golang/protobuf/proto
google.golang.org/grpc
golang.org/x/net/context
github.com/golang/protobuf/ptypes/any
google.golang.org/genproto/protobuf/api
github.com/golang/protobuf/protoc-gen-go/plugin
github.com/golang/protobuf/protoc-gen-go/descriptor
github.com/golang/protobuf/ptypes/duration
github.com/golang/protobuf/ptypes/empty
google.golang.org/genproto/protobuf/field_mask
google.golang.org/genproto/protobuf/source_context
github.com/golang/protobuf/ptypes/struct
github.com/golang/protobuf/ptypes/timestamp
google.golang.org/genproto/protobuf/ptype
github.com/golang/protobuf/ptypes/wrappers
Check that imports in Go sources match importpath attributes in deps.

Build.bazel

load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")

package(default_visibility = ["//visibility:public"])

#gazelle:ignore
proto_library(
    name = "job_proto",
    srcs = ["job-reporting-service.proto"],
    deps = ["@com_google_protobuf//:any_proto",
    "@googleapi//google/api:annotations_proto",
    "@grpc-gateway//protoc-gen-swagger/options:options_proto",
    ],
)

#gazelle:ignore
go_proto_library(
    name = "job_go_proto",
    compiler = "@io_bazel_rules_go//proto:go_grpc",
    importpath = "test/test/job-reporting-service/proto/v1/job",
    proto = ":job_proto",
    visibility = ["//visibility:public"],
)

Any pointers to debug or any known solution(s) are highly appreciated.

[jayconrod]

@vvrpradeep Have you opened an issue with grpc-gateway? I can't provide support for that project, but the people over there are pretty knowledgable. I imagine there's a recommended way to import it into a Bazel project, but I don't know what that is. This doesn't seem like a bug with Gazelle or rules_go, but if you do find a bug, please open a new issue.

[vvrpradeep]

@jayconrod
Thanks for the prompt response.

I will approach the grpc-gateway team as well.

Just to bring to your notice that

• It is (compilation of the proto file) working manually issue seen only with the bazel build.
• It's exactly the same error what I am seeing was reported in this thread.

The error in (4) is related. When Gazelle fails to resolve an import because of a conflict, it won't write out a label in deps. So you'll get this missing strict dependencies error when you build.

• Wanted to know how to identify the conflicts. If you can provide some details that will be helpful.

Thanks in advance for your support.