Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unwanted namespacing warning/ or error #15122

Closed
mamaart opened this issue Dec 16, 2023 · 4 comments
Closed

Unwanted namespacing warning/ or error #15122

mamaart opened this issue Dec 16, 2023 · 4 comments
Labels

Comments

@mamaart
Copy link

mamaart commented Dec 16, 2023

I originally posted this issue in grpc-go - they told me to post it here instead:

What version of gRPC are you using?

1.59.0

What version of protoc are you using?

libprotoc 25.1

What version of Go are you using (go version)?

1.21.4

What operating system (Linux, Windows, …) and version?

Linux 6.6.4-arch1-1

What did you do?

I have multiple protos defined in a git repo and the generated code is in this repo as well and I use it to import the functionality into the server and clients.

├── events_service
│   ├── admin.proto
│   ├── golang
│   │   ├── admin
│   │   │   ├── admin_grpc.pb.go
│   │   │   └── admin.pb.go
│   │   ├── generate.sh
│   │   ├── go.mod
│   │   ├── go.sum
│   │   ├── shared
│   │   │   └── shared.pb.go
│   │   └── user
│   │   ├── user_grpc.pb.go
│   │   └── user.pb.go
│   ├── shared.proto
│   └── user.proto
├── mailbox_service
│   ├── admin.proto
│   ├── golang
│   │   ├── admin
│   │   │   ├── admin_grpc.pb.go
│   │   │   └── admin.pb.go
│   │   ├── generate.sh
│   │   ├── go.mod
│   │   └── user
│   │   ├── user_grpc.pb.go
│   │   └── user.pb.go
│   └── user.proto
├── member_service
│   ├── admin.proto
│   ├── golang
│   │   ├── admin
│   │   │   ├── admin_grpc.pb.go
│   │   │   └── admin.pb.go
│   │   ├── generate.sh
│   │   ├── go.mod
│   │   ├── registration
│   │   │   ├── registration_grpc.pb.go
│   │   │   └── registration.pb.go
│   │   ├── shared
│   │   │   └── shared.pb.go
│   │   └── statistics
│   │   ├── statistics_grpc.pb.go
│   │   └── statistics.pb.go
│   ├── registration.proto
│   ├── shared.proto
│   └── statistics.proto
└── README.md

  • events service
  • mailbox service
  • member service

They are not related to eachother.

Inside some of them have a protofile called shared (its only used inside the domain and not shared with the others.)

When I have a client application that use all of the protos I get a panic, but I managed to reduce it to a warning after reading the FAQ in the error message.

I have even namespaced the package name in the proto - for instance:

syntax = "proto3";

package eventsservice.shared;

option go_package = "github.com/dk-slack/protos/events_service/golang/shared";

What did you expect to see?

Nothing, no warning and no errors.

What did you see instead?

After running this:

❯ GOLANG_PROTOBUF_REGISTRATION_CONFLICT=warn go run ./cmd
WARNING: proto: file "shared.proto" is already registered
	previously from: "github.com/dk-slack/protos/events_service/golang/shared"
	currently from:  "github.com/dk-slack/protos/member_service/golang/shared"
See https://protobuf.dev/reference/go/faq#namespace-conflict

WARNING: proto: file "admin.proto" is already registered
	previously from: "github.com/dk-slack/protos/events_service/golang/admin"
	currently from:  "github.com/dk-slack/protos/member_service/golang/admin"
See https://protobuf.dev/reference/go/faq#namespace-conflict
@mamaart mamaart added the untriaged auto added to all issues by default when created. label Dec 16, 2023
@mamaart
Copy link
Author

mamaart commented Dec 28, 2023

I've successfully updated all package names for both the Golang files and the general package names to ensure uniqueness. Despite these changes, I've noticed that warnings persist, specifically related to the names of the proto files.

I find this situation a bit puzzling. The reason is that, despite the presence of warnings, the files have been successfully compiled, and the resulting code is fully functional. The warnings appear to be more like artifacts—remnants from an earlier stage of development—rather than genuine indications of a current problem or issue.

@zhangskz
Copy link
Member

I believe you're looking for https://github.com/golang/protobuf which is for the protobuf go implementation, but please reopen if I'm mistaken here.

@neild
Copy link
Contributor

neild commented Feb 23, 2024

The behavior here is not specific to the Go implementation.

The protobuf descriptors which describe the structure of a set of protobuf declarations (messages, enums, etc.) include the names of the files in which each declaration appears. These filenames are determined by the protobuf compiler (protoc) and provided to the language-specific code generator.

The Go protobuf runtime enforces a requirement that there be no namespace conflicts among the descriptors in a program. (See https://protobuf.dev/reference/go/faq/#namespace-conflict for some more details.) This includes conflicts among filenames: There may not be multiple files with the same name.

This requirement is enforced to avoid ambiguity when, for example, attempting to look up the descriptors contained in the file with a given name (using protoreflect.Files.FindFileByPath, for example).

Not every protobuf runtime enforces this requirement. Some ignore conflicts, some print a warning at startup. The Go runtime's current behavior was mandated by the protobuf-team and the Go maintainers don't have discretion to change it, so we have historically deferred questions about it to this issue tracker.

In the specific case described here, it is not possible to say with certainty how to resolve the problem since the report doesn't include details on how protoc is being invoked, but I can infer a probably cause and solution from the report:

You have, for example, files named:

events_service/admin.proto
mailbox_service/admin.proto

You are probably invoking protoc with a command like:

protoc --go_out=... -Ievents_service events_service/admin.proto
protoc --go_out=... -Imailbox_service mailbox_service/admin.proto

or

(cd events_service && protoc --go_out=... admin.proto)
(cd mailbox_service && protoc --go_out=... admin.proto)

In both cases, protoc associate the filename admin.proto with each of the generated files.

What you want is for the files to have unique names: events_service/admin.proto and mailbox_service/admin.proto. You can achieve this with something like (note the lack of -I):

protoc --go_out=... events_service/admin.proto
protoc --go_out=... mailbox_service/admin.proto

This may require adjusting the --go_out option or other aspects of your configuration.

@mamaart
Copy link
Author

mamaart commented Feb 26, 2024

In each service I have a script to generate the go code for all the files in that service

├── eventservice
│   ├── generate.sh
│   ├── go.mod
│   ├── go.sum
│   └── protos
│       ├── admin.proto
│       ├── eventtype.proto
│       ├── location.proto
│       ├── locationtype.proto
│       ├── models.proto
│       ├── public.proto
│       └── services.proto
├── mailboxservice
│   ├── generate.sh
│   ├── go.mod
│   ├── go.sum
│   └── protos
│       ├── admin.proto
│       ├── models.proto
│       ├── services.proto
│       └── user.proto
└── memberservice
    ├── generate.sh
    ├── go.mod
    ├── go.sum
    └── protos
        ├── activist.proto
        ├── admin.proto
        ├── models.proto
        ├── public.proto
        ├── registration.proto
        ├── services.proto
        ├── slacktype.proto
        └── statistics.proto

My development flow for the services are autonomous and they could as well have been developed by 3rd party developers in each separate codebase.

The service eventservice and mailboxservice each have their own generation script, so I am basically doing what you said and calling protoc on a file ./protos/services,proto in each project. which is then giving me the namespace issue.

for instance in mailboxservice

protoc -I./protos \
  --go_out=paths=source_relative:./mailboxadmin \
  --go-grpc_out=paths=source_relative:./mailboxadmin \
  protos/admin.proto \

protoc -I./protos \
  --go_out=paths=source_relative:./mailboxuser \
  --go-grpc_out=paths=source_relative:./mailboxuser \
  protos/user.proto \

protoc -I./protos \
  --go_out=paths=source_relative:./mailboxmodels \
  --go-grpc_out=paths=source_relative:./mailboxmodels \
  protos/models.proto \

protoc -I./protos \
  --go_out=paths=source_relative:. \
  --go-grpc_out=paths=source_relative:. \
  protos/services.proto \

And then in event service

protoc -I./protos \
  --go_out=paths=source_relative:./eventadmin \
  --go-grpc_out=paths=source_relative:./eventadmin \
  protos/admin.proto \

protoc -I./protos \
  --go_out=paths=source_relative:./eventpublic \
  --go-grpc_out=paths=source_relative:./eventpublic \
  protos/public.proto \

protoc -I./protos \
  --go_out=paths=source_relative:./eventmodels \
  --go-grpc_out=paths=source_relative:./eventmodels \
  protos/models.proto \

protoc -I./protos \
  --go_out=paths=source_relative:./locationadmin \
  --go-grpc_out=paths=source_relative:./locationadmin \
  protos/location.proto \

protoc -I./protos \
  --go_out=paths=source_relative:./locationtypeadmin \
  --go-grpc_out=paths=source_relative:./locationtypeadmin \
  protos/locationtype.proto \

protoc -I./protos \
  --go_out=paths=source_relative:./eventtypeadmin \
  --go-grpc_out=paths=source_relative:./eventtypeadmin \
  protos/eventtype.proto \

protoc -I./protos \
  --go_out=paths=source_relative:. \
  --go-grpc_out=paths=source_relative:. \
  protos/services.proto \

Conclusion

I guess I was assuming some kind of encapsulation when I was developing the services separately like this, but I see now that grpc is looking at everything like its on the same level, and I can solve it by naming my 'protos' folder in each service project, '{service_name}protos' instead and then I just have to make sure not to give 2 services the same name.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants