Include transitive protos, 'compile' works but 'protobuf' doesn't work

[shqyking]

Description

I have 3 projects, one is a web app, called shop. two are backend services, called shop and iam. Here is the directory layout.

• services
  • iam
    • build.gradle
    • src/main/proto/iam.proto
  • shop
    • build.gradle
    • src/main/proto/shop.proto
• webapps
  • shop
    • build.gradle

shop.proto uses iam.proto. webapps uses shop.proto. Here is what build.gradle looks like

services/iam/build.gradle

protobuf {
    protoc {
        artifact = "com.google.protobuf:protoc:$protobufVersion"
    }
    plugins {
        grpc {
            artifact = "io.grpc:protoc-gen-grpc-java:$protocGenGrpcVersion"
        }
    }

    generateProtoTasks {
        all().each { task ->
            task.plugins {
                grpc {}
            }
            task.generateDescriptorSet = true
            task.descriptorSetOptions.includeImports = true
        }
    }
}
dependencies {
    implementation "com.google.protobuf:protobuf-java:$protobufVersion"
    implementation "com.google.protobuf:protobuf-java-util:$protobufVersion"

    // gRPC
    implementation group: 'io.grpc', name: 'grpc-services'
    implementation group: 'com.google.api.grpc', name: 'proto-google-common-protos', version: "$googleCommonProtosVersion"
}

services/shop/build.gradle

protobuf {
   ......   // Exactly same with protobuf in services/iam/build.gradle
}
dependencies {
  ......  // Exactly same with dependencies in services/iam/build.gradle
  protobuf project(":services:iam")  // shop.proto depends on iam.proto
}

webapps/shop/build.gradle

protobuf {
    protoc {
        artifact = 'com.google.protobuf:protoc:3.6.1'
    }
    plugins {
        "grpc-web" {
            path = "$rootProject.projectDir/tools/bin/protoc-gen-grpc-web-1.0.4-${osdetector.classifier}"
        }
    }

    generateProtoTasks {
        all().each { task ->
            task.builtins {
                js {
                    option 'import_style=commonjs'
                }
                remove java
            }
            task.plugins {
                "grpc-web" {
                    option 'import_style=commonjs+dts'
                    option 'mode=grpcweb'
                    outputSubDir = 'js'
                }
            }
        }
    }
}

dependencies {
  protobuf project(':services:shop') //  only depends on direct proto target.
}

What I expect:

Executing :webapps:shop:generateProto should succeed.

What actually happens:

However, it reports error that it can't find iam.proto used by shop.proto.

Interestingly, if I change "protobuf" to "compile", it works. Seems "protobuf" does not transitively include dependency protos? What is the recommended way to include transitive protos? I would like to learn more about how it works so I can have better judgement on when to use "protobuf".

Thanks!

[voidzcy]

It should be transitive given that your services/shop/build.gradle also uses protobuf configuration for project(":services:iam") dependency. The project(':services:shop') dependency jar should contain an archived resources package, which contains both iam.proto and shop.proto. Can you turn on debug log to see what files are extracted and fed to protoc for your webapps/shop's build?

More specifically, can you provide log output for

protobuf-gradle-plugin/src/main/groovy/com/google/protobuf/gradle/ProtobufExtract.groovy

Line 61 in 3fd3c64

logger.debug "Extracting protos from ${file} to ${destDir}"

and

protobuf-gradle-plugin/src/main/groovy/com/google/protobuf/gradle/GenerateProtoTask.groovy

Line 523 in 3fd3c64

logger.log(LogLevel.INFO, cmd.toString())

for webapps/shop's build?

---

However, for your use case compile (or implementation) configuration should be the correct usage instead of protobuf as you do not need to recompile proto files from dependent projects, in which they are already compiled. For more information, please see here.

[voidzcy]

Can you try with 0.8.11 release and see if the issue persists?

[jd3nn1s]

I am also experiencing issues with using projects with protobuf project('name'). These issues are intermittent and show up more frequently when I use parallel workers. They have persisted even with 0.8.11.

For me, when the jar is created for the equivalent of :services:shop, it does not have iam's proto files in it, even though those protos exist in the build/extracted-protos directory.

I reviewed the logs for the build when it fails and I see that :services:shop:processResources is called before :services:shop:extractProtos is called.

I added logging to output the source for the :services:shop:processResources task and found that the source does not include iam's protos if I run:
./gradlew :services:shop:clean :services:shop:processResources
while if I run:
./gradlew :services:shop:clean :services:shop:generateProto && ./gradlew :services:shop:processResources it does.