reflect/protoregistry: conflicts with same filename #1122
Comments
|
I don't see any good answers here, so we're probably stuck picking from bad ones. There is a naming conflict which will prevent the two different The options I see are to continue to mandate that filenames must be globally unique (do any other implementations require this?), or to permit a conflict. If we permit a conflict, we need to figure out what |
|
Is there any reason why |
|
It seems plausible that someone might want to link packages outside their control that both contain a |
Sure, but that doesn't seem to be problem in this issue though. In my opinion, the protobuf developer site is lacking guidance in general in this area, and the |
We can do that and yes it will avoid the problem. To be more accurate, A and B are two micro services and we can see I think we can compare this issue to a project (whatever the language) doing |
@dsnet may I ask how would one compile |
|
I also had similar problem and what I don't understand is how it does not take "package" into account. In my case I have two events.proto files in two different folders, they generate events.pb.go , also in two different folders AND both of them have different "package" and go_package. Why do I get sounds like I should not have this problem, yet warning is printed |
|
The problem is a conflict in the source file name. The This can also be a problem when running import "events.proto";and there are two different I suspect that looking up |
|
I have solved the problem including all the path (relative to gopath) in the "import" and compiling with protoc using the absolute path to the proto file. After this, in the pb files generated, in the grpc.ServiceDesc structure, Metadata field contains the full path (relative to gopath) and works without warnings. |
|
So we have the situation @dsnet describes. We have Go program that depends on two external libraries both of which have a I agree with @strowk, at the very least there should be some second level logic based on package name to disambiguate source file name collisions. The workaround of changing how the protobuf is compiled does not work when an external library is managing how the protobuf is compiled. Along the current logical arc, the implication is that all protobufs in all repositories globally, need to have unique filenames - which is not reasonable. I really hope the threat of |
|
I receive these warnings now on and older project I just came back to after a year. This project had been having no issues but, since I've come back I've made the following updates:
This is happening in a layout model I was using that had components setup such as: componentA/ So now I'm getting the: WARNING: proto: file "io.proto" is already registered, A future release will panic on registration conflicts I also moved to using go_package as well with dsymonds overloading setup, for example: package marmot.example.scripts.juniper_password_change.ssh;
option go_package = "github.com/bearlytools/bearmetal/framework/examples/scripts/juniper_password_change/cogs/ssh/io;io";I'm sure I haven't thought of a million different ways this could break or is undesirable, but could we avoid this conflict by registering using the go_package full path with a flag passed, like --register_using_go_package? I'm sure this has some undesirable behavior, but I'm 100% sure neild is right, that there is probably just no great solution. |
|
Has the protobuf team made any further decisions on this issue? Much like @rcgoodfellow, I have a Go program that depends on two third-party libraries, each of which defines a This can't be that uncommon a situation. At this point, I'm not really clear how the protobuf team expect the file-by-path registry to work outside of a monorepo setting. |
|
I have the same issue too.
It seems unacceptable to got this panic threat from protobuf.
Is there any solution? |
It seems like a proto library problem that we were getting these warnings, but we were. See golang/protobuf#1122 for more details. At the time of this commit that issue is still open, but since I'm renaming the files it should fix our problem.
|
finally, I got the conflict panic! as the doc states: The protobuf language assumes that all declarations are universally unique. |
|
This is really becoming an issue. My protos structure: different I know I can solve this with absolute paths but it's definitely not ideal. |
|
Agree with previous comments. Not everyone uses a monorepo, and enforcing unique file path across different services is unnecessary and troublesome. |
|
We're coming up this issues' first birthday 🍰 |
|
I tried to update protobuf and golang yesterday, and I met this issue, on its birthday... Just an option for namespace in proto file? |
|
Ran into this problem as well, tried adding a unique namespace to the two, but that didn't help either How can you generate the go files with a unique inlined path? |
|
Different I don't understand why the file name is a problem if content is scoped to a package... Does any of you have a reasonable solution? Should I modify all files (but it's not a proper solution), or cross fingers so their panic warnings will never be implemented? Thank you, |
|
I met this issue.
|
|
I've hit hit issue recently. First I followed what documentation I could find about namespacing: https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict I followed the guide, specifically the bit We're running mutliple services each with a If this becomes a panic later I can see this breaking a huge number of projects around the open source community due to assumption (based on official guides) that a unique package name is sufficient to namespace a service. |
Warning is already a panic with latest release 😕 It has been introduced in release v1.26.0 : https://github.com/protocolbuffers/protobuf-go/releases/tag/v1.26.0 (Fatal namespace conflicts part) |
|
@dsnet @neild It seems that the latest release went ahead and added enforcement as the default behavior. This of course broke code and we will have to go back and mitigate via env flags or build flags. There could be a million reasons Go team has decided to do this, so this isn't a criticism and like everything else, people aren't going to be happy with the decision. I'm hoping you could address this issue here to let us know that this is just the way it is and close the issue or that there is some other way you plan to fix this in the future (by adding enforcement, this seems like the issue really should just be closed). I would just like to move this issue towards resolution so we are all on the same page. For anyone reading, your mitigations are: go build -ldflags "-X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn" Env var: |
|
Instead of using only the file path, use the concatenation of the package name with the file path can partially resolve this issue? |
This remains just as bizarre and unreasonable today as it was two years ago. |
It seems like a proto library problem that we were getting these warnings, but we were. See golang/protobuf#1122 for more details. At the time of this commit that issue is still open, but since I'm renaming the files it should fix our problem.
|
Hi, I've experienced this problem too. I have two Go modules with two *.proto files in different locations inside the modules. The only common they have is the basename. Generate commands are the same and looks like And when I run the command which imports both of this packages it panics. So it knows the Go path of both packages, it knows they are different, but when it registers files it checks only relative path from protoc command executions. I've made a little change and it seems solved the problem. Is this change reasonable? |
In your protos, two files should declare in two namespace or package, like this Hope it might help |
What's you pwd when you generate proto.go files? |
package directives are already different and that doesn't help.
pwd is local directory. But this is not the point. I found a way to avoid the panic, but it's not right. The same relative file names should not be a problem. Shouldn't we find the reason and fix it once for ever and for everybody? |
|
The right place to bring this up is either the general protobuf issue tracker (https://github.com/protocolbuffers/protobuf/issues) or mailing list (https://groups.google.com/g/protobuf). The question of conflicting |
protocolbuffers/protobuf#15122
|
|
I added a comment on protocolbuffers/protobuf#15122. The rejection of naming conflicts is a general requirement for protobuf implementations. Not every runtime rejects conflicts, but I believe the protobuf owners consider the cases which do not to be buggy. |
|
Prost (the Rust protobuf implementation) handles this just fine. As I mentioned upthread a year ago, the requirement that all source filenames are globally unique across all software libraries is simply not a reasonable assumption. Perhaps this works for Google, but out in the real world, not everything is inside one monorepo under one company's control. |
|
As mentioned by @neild, this is an issue that needs to be addressed at the https://github.com/protocolbuffers/protobuf level. Repeatedly bringing up this issue in this repo won't do much good. |
|
As noted on that comment, there are in fact other options available:
Nothing other than the choices of this library's maintainers prevents them from fixing this problem. They should be honest about that choice, rather than punting responsibility off to another team. |
|
Keep in mind that the Rust implementation is not maintained by Google, so it doesn't adhere to the level of expectations that the protobuf project has for languages with first-class support. It also doesn't help that protobuf doesn't have a formal specification for implementations (whether formally maintained by Google or not) to agree on what is "correct semantics". I can say this transparently as an outsider (since I don't work for Google anymore) that the Go protobuf maintainers are ultimately under the authority of the wider protobuf project in terms of what semantics and features they an provide. Complaining about the problem here is not going to change anything as they don't have the authority to make this change. Protobuf has gone through some leadership changes since I was involved with the Go protobuf implementation, so they may feel differently today, but the stance several years ago is that the rejection of duplicates is correct behavior. Again, this needs to be brought up at the https://github.com/protocolbuffers/protobuf level. |
This comment was marked as abuse.
This comment was marked as abuse.
|
Joe has done great work for Go both in and out of the protocol buffer
space, so let's be nice to him.
It is clear you disagree, we get it.
…On Fri, Feb 23, 2024 at 3:14 PM Eliza Weisman ***@***.***> wrote:
Keep in mind that the Rust implementation is not maintained by Google, so
it doesn't adhere to the level of expectations that the protobuf project
has for languages with first-class support.
So, you mean that — unlike software from Google — it isn't broken?
—
Reply to this email directly, view it on GitHub
<#1122 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJQZLSTGFQTYKDXTC442WLYVEPHDAVCNFSM4M5H462KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJWGIYTEMRSGA2A>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
John Doak
www.obscuredworld.com
|
|
Any resolution after 4 years? In real world, many companies may has some projects, like this project_a/protos/error.proto has why we must ensure these proto must has uniq name since these protos has complete different package and go mod ? |
|
So, recently, I’ve come into some work with these specifics. It can often happen that because of I don’t have answers, and I don’t have solutions here either, but in some cases we’re losing specificities of filepath in how we run the protoc commandline. |
|
I will attempt to summarize. Every The The It is possible to look up a file descriptor by file name. In Go, this is While The Go protobuf runtime will panic at init time if there are multiple file descriptors in the global registry with the same file name. Why does the Go runtime panic on filename conflicts? A I disagree with that reasoning. How do I get you to change your mind? This decision is from the overall protobuf maintainers, not the maintainers of the Go implementation. If you would like to argue for a change, the place to do so is with https://github.com/protocolbuffers/protobuf. Nobody working on the Go implementation has any ability to change this. I have You're probably invoking When you pass If you instead invoke I have two completely unrelated packages that each contain a The requirement that file names in the metadata be unique means that you should generally aim to give files in public packages names that are unlikely to be used by other packages. For example, all the I don't know of any canonical guidance on how to choose a non-conflicting name. You could use your project name if it is sufficiently unique, or use a Java-style URL-based naming scheme such as Does this mean I need to put my That's probably simplest. (This is what is done for the However, you could also copy your files into an appropriate hierarchy in a temporary directory at code generation time. Is there anything else I can do to avoid this panic at init time? You can set the environment variable You can also set the conflict policy at build time with a linker-initialized variable: |
|
It seems putting proto files under Yet it would be nice to be able to keep your proto files in arbitrary directory and still have them registered under uniq names. Here is an example project structure and protoc command usage. And here are few changes to protoc-gen-go to append path prefix to registry filenames. The key part is change in |
|
Just hit the same issue. And this is a 4 year old bug, and it seems so simple to fix in Making the error to a warning is also in many cases (my in particular) is not acceptable: I'm doing command line programs that shouldn't be spitting useless (from the end-user point of view) warnings. Creating a cumbersome unique directory structure ( Now, after reading this very long issue I see that GOLANG_PROTOBUF_REGISTRATION_CONFLICT can be set to But finally, for anyone else out there who may be interested, I adopted a strategy of creating a temporary soft link named after the Go import path. It is presumably unique, but one could add a random id if needed. It soft links the current directory, and feeding this self-link to the #!/bin/bash
set -e
#!/bin/bash
set -e
# Find Go import path: presumably unique.
import_path="$(go list -f '{{.ImportPath}}')"
# Extract the domain and the rest of the path
domain=$(echo "$import_path" | cut -d '/' -f 1)
rest_of_path=$(echo "$import_path" | cut -d '/' -f 2-)
# Reverse the domain part (split by '.')
reversed_domain=$(echo "$domain" | awk -F '.' '{ for (i=NF; i>1; i--) printf "%s.", $i; print $1 }' | sed 's/\.$//')
# Combine the reversed domain with the rest of the path
tmp_link="$reversed_domain/$rest_of_path"
tmp_link=$(echo "$tmp_link" | tr '/.' '__')
# Create temporary self-link, run protoc and clean link.
rm -f "${tmp_link}"
ln -s . "${tmp_link}"
protoc --go_out=. --go_opt=paths=source_relative "./${tmp_link}/_your_proto_file_.proto"
rm -f "${tmp_link}" |
Hello,
Sharing a same
.protofilename with different proto package names results in these errors:2020/05/10 15:03:12 WARNING: proto: file "common.proto" is already registered previously from: "git.example.com/proto/common" currently from: "git.example.com/proto/a" A future release will panic on registration conflicts. See: https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict2020/05/10 15:03:12 WARNING: proto: file "common.proto" is already registered previously from: "git.example.com/proto/common" currently from: "git.example.com/proto/b" A future release will panic on registration conflicts. See: https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflictHere is my layout:
I don't think my case is similar to both quoted in documentation https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict. I don't generate multiple times a same proto file (here I generate one time
common/common.protoand I import from others files) and I'm using different package names. It would be better if I rename package common to something else, but I don't think it's my issue here.What's the best practice to solve this issue ? Is it normal to get this warning in my case ? I mean, it doesn't look uncommon to have the same filename in different projects.
Thanks
The text was updated successfully, but these errors were encountered: