not all outputs were created or valid (Remote caching, all platforms)

[excitoon]

0.23.2

	ERROR: /some/path/BUILD:25:1: output 'some/outpu.t' was not created
	ERROR: /some/path/BUILD:25:1: not all outputs were created or valid

@meteorcloudy
This issue happens every time when cache server (https://github.com/neitanod/toxy) terminates connection without an answer:

[toxy]$ ./main -p 9999 -t leave:asap
2019/04/24 07:18:31 Starting reverse proxy for leave:asap
2019/04/24 07:19:02 unsupported protocol scheme "leave"

[excitoon]

Basically the problem is zero answer for action result is perfectly fine (??) and that leads to 0 outputs of an action, which is downloading without an error.

ef7fe3d fixes that, but the problem is more general.

I can guess Bazel shall have some kind of tolerance to network errors. Maybe some checksums for cache entries?

[meteorcloudy]

/cc @buchgr Our remote caching/execution expert.

[buchgr]

@excitoon

Basically the problem is zero answer for action result is perfectly fine (??) and that leads to 0 outputs of an action, which is downloading without an error.

Yes it's fine for an action to not produce an output file, but then it must also not declare an output file or else you'll see this error.

I can guess Bazel shall have some kind of tolerance to network errors. Maybe some checksums for cache entries?

We do have that, but it looks like that this is not the problem here. Would have a reproducer for me to take a look at?

[excitoon]

@buchgr

git clone https://github.com/neitanod/toxy
go run toxy/main.go -p 9999 -t bad &
bazel build //some:thing --spawn_strategy=standalone --remote_http_cache=http://localhost:9999

[buchgr]

@excitoon how does the //some:thing target look like?

[excitoon]

@buchgr Any cacheable target with outputs. E.g. for bazel itself:

[chebotarev@some_machine bazel]$ bazel build //src:bazel --spawn_strategy=standalone --remote_http_cache=http://localhost:9999
INFO: Invocation ID: 42a78889-b7bf-4617-b9d7-e65ec085bc84
INFO: Analysed target //src:bazel (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 7] [-----] BazelWorkspaceStatusAction stable-status.txt
URI (GET): localhost:9999/ac/6953a6c58d4dc26c7160d766f3293f0d78621719e78c2cba85e7c006ec9257a1
ERROR: /home/chebotarev/bazel/src/tools/android/java/com/google/devtools/build/android/BUILD:21:1: output 'src/tools/android/java/com/google/devtools/build/android/all_android_tools_deploy.jar' was not created
ERROR: /home/chebotarev/bazel/src/tools/android/java/com/google/devtools/build/android/BUILD:21:1: not all outputs were created or valid
[1 / 10] 4 actions running
    Building deploy jar .../devtools/build/lib/bazel/BazelServer_deploy.jar; 0s remote-cache
    Building deploy jar .../devtools/build/android/all_android_tools_deploy.jar; 0s remote-cache
    .../java/com/google/devtools/coverageoutputgenerator:all_lcov_merger_tools; 0s remote-cache
    Building deploy jar .../build/importdeps/ImportDepsChecker_deploy.jar; 0s remote-cache
URI (GET): localhost:9999/ac/9d51dc88f1684479abed944587c2ce01605e75ab215a49e4c068645f9e7bb527
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.320s, Critical Path: 0.07s
INFO: 1 process: 1 remote cache hit.
FAILED: Build did NOT complete successfully

[excitoon]

@buchgr about a target without outputs and without output file, what is the caching strategy for it? Shall Bazel ask cache for that absence of files?

[buchgr]

@excitoon aww I see what you mean - that's indeed a bug. thanks!

This issue happens every time when cache server (https://github.com/neitanod/toxy) terminates connection without an answer:

This statement is not fully correctly. what toxy seems to do is send a 200 OK response with an empty body. So that's a badly behaving server. That's different from the server just terminating the connection.

I think the proper fix would be to add a check that the generated outputs are a super set of the declared outputs.

[excitoon]

@buchgr why not exact set?

…

On Wed, Apr 24, 2019, 14:34 Jakob Buchgraber ***@***.***> wrote: @excitoon <https://github.com/excitoon> aww I see what you mean - that's indeed a bug. This issue happens every time when cache server ( https://github.com/neitanod/toxy) terminates connection without an answer: This statement is not fully correctly. what toxy seems to do is send a 200 OK response with an empty body. So that's a badly behaving server. That's different from the server just terminating the connection. I think the proper fix would be to add a check that the generated outputs are a super set of the declared outputs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8125 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARGSB2HQVQPI4XM6OJYK4DPSBAVDANCNFSM4HIAHNZQ> .

[buchgr]

@excitoon it's totally legal for a command to create more output files than declared in Bazel (i.e. debug output).

[excitoon]

@buchgr Content-Length is not a mandatory header, so theoretically, Bazel (with server which do not set this header) can confuse accidental disconnect from graceful shutdown and fall to this bug, as I can guess.

…

On Wed, Apr 24, 2019, 14:34 Jakob Buchgraber ***@***.***> wrote: @excitoon <https://github.com/excitoon> aww I see what you mean - that's indeed a bug. This issue happens every time when cache server ( https://github.com/neitanod/toxy) terminates connection without an answer: This statement is not fully correctly. what toxy seems to do is send a 200 OK response with an empty body. So that's a badly behaving server. That's different from the server just terminating the connection. I think the proper fix would be to add a check that the generated outputs are a super set of the declared outputs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8125 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARGSB2HQVQPI4XM6OJYK4DPSBAVDANCNFSM4HIAHNZQ> .

[excitoon]

That thing that 0 bytes protobuf is a valid entity is indeed interesting.

[buchgr]

@excitoon it's a bit unfortunate. 0 bytes is a valid protobuf in that all fields in the protobuf will have its default values. We check the int exit_code field and the default value of an int is 0 which also happens to be the exit code signaling success...

[excitoon]

@buchgr Is e530f68 fine enough?

[buchgr]

Mostly yes. Open a PR (with a test) and we can discuss it further? I added a comment on the commit. Thanks!

[EricBurnett]

Changing the API to require a positively set field (so an empty proto isn't valid) is also bazelbuild/remote-apis#6 . I don't think it can happen until V3, since it'd be a breaking change, but would be a long-term defense against this.

[buchgr]

I believe Bazel should be able to handle it client side independent of the API. To the best of my knowledge if the output is of type Artifact it's a declared and thus required output by Bazel.

[ob]

Any status on this, we are hitting this problem when using bazel-remote with Bazel 0.25.1. I still don't know how an empty file got into the ac, but:

% find ac -size 0
ac/05/0520dc913c0d5fcf9791bb16affdeef7ea13ee5f8d6b42924b89c84a65d40fc6

When bazel asks for that action, bazel-remote happily hands it with size 0 which causes the build to fail because the output wasn't created.

[mostynb]

@ob: you might want to try upgrading bazel-remote, I added some ActionResult validation when implementing the gRPC interface, and then refactored the code so that the HTTP interface also has this validation.

[mostynb]

Duh, this part of the validation I added to bazel-remote is obviously out of spec (followup in buchgr/bazel-remote#121).

[MikhailTymchukFT]

Starting with 4.0.0 release, bazel hangs when using multiplex-worker strategy. It fails immediately with the message described in #0 post when using other strategies like sandboxed or local.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 2+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!