Bazel CI: Downstream projects broken

[meteorcloudy]

(02:07:00) ERROR: /var/lib/buildkite-agent/builds/bk-docker-v5fw/bazel-downstream-projects/rules_jvm_external/examples/scala_akka/BUILD.bazel:68:1: every rule of type scala_test implicitly depends upon the target '@bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main', but this target could not be found because of: no such package '@bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator': BUILD file not found in directory 'tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator' of external repository @bazel_tools. Add a BUILD file to a directory to mark it as a package.

rules_jvm_external, rules_scala, rules_webtesting, IntelliJ Plugin Aspect are all broken with above error.

Suspected culprit is 934c4fd

Culprit finder result:
https://buildkite.com/bazel/culprit-finder/builds/246#d3d639cf-fd0c-4bf8-bb57-a1e497a7673b
https://buildkite.com/bazel/culprit-finder/builds/247#6b6bab51-60a3-43ff-858c-a9cc1b4663fa

/cc @ulfjack

[ulfjack]

Sorry about that. I have a partial rollback of 934c4fd that should fix this.

[ulfjack]

I confirmed that this fixes at least the rules_jvm_external test examples/scala_akka.

[ulfjack]

It would help if Buildkite generated repro instructions as part of the culprit finder. It took me a while to figure out what to check out and how to run the tests. I tried building rules_scala and rules_jvm_external directly, and I get a ton of errors.

[meteorcloudy]

@ulfjack Thanks for the fix! I agree we should generate easy commands for reproducing failures.

Currently, we do print all the bazel command we execute in a job. As well as the git clone command to get a repo sources at a certain commit.
Checkout https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1318#ebfa0656-9a7a-40d4-9e35-251ddb02649d
You can find

Fetching rules_scala sources at 886bc9cf6d299545510b39b4872bbb5dc7526cb3
git clone -v https://github.com/bazelbuild/rules_scala.git /var/lib/buildkite-agent/builds/bk-docker-7tdb/bazel-downstream-projects/rules_scala
Cloning into '/var/lib/buildkite-agent/builds/bk-docker-7tdb/bazel-downstream-projects/rules_scala'...
...

and

bazel build --show_progress_rate_limit=5 --curses=yes --color=yes --terminal_columns=143 --show_timestamps --verbose_failures --keep_going --jobs=32 --announce_rc --experimental_multi_threaded_digest --experimental_repository_cache_hardlinks --disk_cache= --sandbox_tmpfs_path=/tmp --google_default_credentials --remote_cache=remotebuildexecution.googleapis.com --remote_instance_name=projects/bazel-untrusted/instances/default_instance --tls_enabled=true --remote_timeout=60 --remote_max_connections=200 --remote_default_platform_properties=properties:{name:"cache-silo-key" value:"0442b0ae1c41d62408465ed8eea86e24b61186a9c396a535d2f9252ae1202b7a"} -- //test/...

But we should simplify this and make reproducing CI failure only a copy&paste step.
Filed bazelbuild/continuous-integration#886