RBE platform uses outdated options

[davido]

Trying to activate Gerrit pipeline on RBE platform is failing with outdated options:

https://buildkite.com/bazel/gerrit/builds/12730#018c16a9-892d-4449-b76e-08709e838102

bazel build --show_progress_rate_limit=5 --curses=yes --color=yes --terminal_columns=143 --show_timestamps --verbose_failures --jobs=75 --announce_rc --experimental_repository_cache_hardlinks --disk_cache= --sandbox_tmpfs_path=/tmp --remote_executor=remotebuildexecution.googleapis.com --remote_instance_name=projects/bazel-untrusted/instances/default_instance --remote_timeout=3600 --incompatible_strict_action_env --google_default_credentials --toolchain_resolution_debug --remote_download_toplevel --bes_backend=buildeventservice.googleapis.com --bes_timeout=360s --project_id=bazel-untrusted --host_javabase=@buildkite_config//java:jdk --javabase=@buildkite_config//java:jdk --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8 --java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8 --crosstool_top=@buildkite_config//cc:toolchain --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 --extra_toolchains=@buildkite_config//config:cc-toolchain --extra_execution_platforms=@buildkite_config//config:platform --host_platform=@buildkite_config//config:platform --platforms=@buildkite_config//config:platform --test_env=HOME --test_env=BAZELISK_USER_AGENT -- //:release //:api-skip-javadoc
[...]
WARNING: Option 'project_id' is deprecated: Use --bes_instance_name instead
WARNING: Option 'host_javabase' is deprecated
WARNING: </span>Option 'java_toolchain' is deprecated

ERROR: /workdir/BUILD:38:8: While resolving toolchains for target //:release: com.google.devtools.build.lib.packages.BuildFileNotFoundException: no such package '@buildkite_config//config': The repository '@buildkite_config' could not be resolved: Repository '@buildkite_config' is not defined

Note, that we are building on RBE@GCP downstream for years now and all looks good. So that this must be a buildkite related issue.

As pointed out by @meteorcloudy in another comment, the logic is here:

continuous-integration/buildkite/bazelci.py

Lines 2027 to 2034 in 387688a

	for platform_flag, value in list(platform_flags.items()):
	found = False
	for original_flag in original_flags:
	if original_flag.startswith(platform_flag):
	found = True
	break
	if not found:
	flags += [platform_flag + "=" + value]

[meteorcloudy]

Updating the flags won't fix the actual failure. You'll have to specify relevant toolchain flags in your repo, check how Bazel does it: https://github.com/bazelbuild/bazel/blob/78f6b9494762674895e8d17f884ecc0c22d7cbf1/.bazelrc#L17-L29

[davido]

Thanks for the pointer.

TODO(bazel-team): Remove this toolchain when .bazelversion is 7.0.0rc2 or later.

^^^ Why it's saying this? Bazel@HEAD is newer then 7.0.0rc2 isn't it?

You'll have to specify relevant toolchain flags in your repo, check how Bazel does it [...]

It's already done in Gerrit, nevertheless, the build on RBE platform on Buildkite was failing:

https://gerrit.googlesource.com/gerrit/+/refs/heads/master/tools/remote-bazelrc#33

# Set several flags related to specifying the platform, toolchain and java
# properties.
build:remote_shared --crosstool_top=@rbe_jdk11//cc:toolchain
build:remote_shared --extra_toolchains=@rbe_jdk11//config:cc-toolchain
build:remote_shared --extra_execution_platforms=@rbe_jdk11//config:platform
build:remote_shared --host_platform=@rbe_jdk11//config:platform
build:remote_shared --platforms=@rbe_jdk11//config:platform
build:remote_shared --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1

On Gerrit@CI we are calling something like this for RBE@GCP

    $ bazel test --config=remote --remote_instance_name=projects/$PROJECT/instances/default_instance javatests/...

and it works like a charm, e.g. on Gerrit@HEAD: 1, 2.

[meteorcloudy]

How is the rbe_jdk11 repo defined?

[davido]

How is the rbe_jdk11 repo defined?

It's loaded in WORKSPACE file.

I maintain this repository to populate rbe_autoconfig:
https://github.com/davido/rbe_autoconfig

See the instruction there. Then I upload the tarball to Google Bucket, conduct GitHub release and consume it in the WORKSPACE like this:

http_archive(
    name = "ubuntu2204_jdk17",
    sha256 = "8ea82b81c9707e535ff93ef5349d11e55b2a23c62bcc3b0faaec052144aed87d",
    strip_prefix = "rbe_autoconfig-5.1.0",
    urls = [
        "https://gerrit-bazel.storage.googleapis.com/rbe_autoconfig/v5.1.0.tar.gz",
        "https://github.com/davido/rbe_autoconfig/releases/download/v5.1.0/v5.1.0.tar.gz",
    ],
)

See this CL, where I updated the RBE image to use Ubuntu2204, you've recently updated.
Please note, that this change wasn't merged yet on gerrit@HEAD.

[meteorcloudy]

OK, I found the problem of @buildkite_config:

continuous-integration/buildkite/bazelci.py

Lines 2050 to 2056 in 387688a

	if not remote_enabled(flags):
	if platform.startswith("rbe_"):
	aggregated_flags += rbe_flags(flags, accept_cached=enable_remote_cache)
	else:
	aggregated_flags += remote_caching_flags(
	platform, accept_cached=enable_remote_cache
	)

The ci script check if you have certain flags specified to decide whether to append rbe flags. You actually have RBE flags already defined in your .bazelrc file, but the ci script won't expand all the configs, to fix that, you have to put
--remote_executor=grpcs://remotebuildexecution.googleapis.com in the gerrit.yml file.

Just like Bazel: https://github.com/bazelbuild/bazel/blob/d8ed53897c1f0e13918a31a68b4013730a351670/.bazelci/postsubmit.yml#L353C72-L353C72

[davido]

@meteorcloudy Thanks for the analysis and for the pointer.

I tried once again to re-add RBE platform to gerrit pipeline in this PR.

PTAL. Thank you.