Release 0.23 - February 2019

[laurentlb]

Target RC date: February 1st

[philwo]

@katre Would you mind to swap with me? I'm assigned to the 0.24 release in March, but I'm on vacation the whole month.

[katre]

Sure, I'm fine to make that change.

[laurentlb]

Any update @philwo?

[philwo]

Marcel and me spent today with fixing breakages on our downstream pipelines and getting important last minute fixes in, without which we can’t judge whether a given commit is a good baseline / release.

I think I’ll cut 0.23.0rc1 either this evening or tomorrow, as we sorted most things out now.

[thundergolfer]

@philwo Apologies if this is a stupid question, but how do you discover what will get included in 0.23? I'm particularly interested in a92347e

[philwo]

@thundergolfer Not stupid at all. Our official release playbook is here: https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md

Basically, the release manager takes a good look at the last nightly downstream pipeline and decides on a good commit. I'll pick one today, probably just the current HEAD as the pipeline looks pretty good. The last run only had three failures:

• Bazel (fixed by 82a397b)
• Cloud Robotics Core (fixed by Fix .bazelrc to be compatible with Bazel 0.23.0 googlecloudrobotics/core#1)
• rules_swift (is just a pipeline configuration issue, which I'll fix when I'm in the office)

So your linked commit will definitely be included in the next release. :)

[thundergolfer]

@philwo Thanks for the detailed explanation 🙇

[philwo]

Looks like we're good to go. I picked this commit as the baseline: 441fd75

Log:

scripts/release/release.sh create 0.23.0 441fd75d0047f8a998d784c557736ab9075db893
scripts/release/release.sh push

[philwo]

Bazel 0.23.0rc1 was pushed to https://releases.bazel.build/0.23.0/rc1/index.html.

[philwo]

Bazel 0.23.0rc1 passed all of its own tests: https://buildkite.com/bazel/bazel-bazel/builds/6638

I will run the full downstream pipeline later, but don't expect any issues, as the last run was completely green apart from known issues that have been addressed.

[ulfjack]

We should make sure that 0.23.0 contains a fix for #7410. I suspect it already does, so that may not require any additional work.

[philwo]

Creating Bazel 0.23.0rc2 with a fix for #7397:

scripts/release/release.sh create --force_rc=2 0.23.0 441fd75d0047f8a998d784c557736ab9075db893 6ca7763669728253606578a56a205bca3ea883e9
scripts/release/release.sh push

[philwo]

@ulfjack I think this release is fine (I deliberately cut it after your flag flip went in, in order to reap its benefits for users and our CI ^^).

Everyone: I'm not aware of any release blocking issues for Bazel 0.23.0. If that doesn't change, I'll release Bazel 0.23.0 tomorrow (14 days since rc1 will have passed then).

[ulfjack]

There is one corner case - if someone's disabling the flag and using bash 5, then they'll see the error. There's a pending CL (in process of being submitted) to fix that as well, maybe we want to cherrypick that? (It's a two-line CL.)

[philwo]

I don't think it's worth it to do a cherry-pick (or even a patch release for 0.22.0) for that due to the following reasons:

• The cherry-pick would delay the release until Thursday thanks to our "2 business days passed since you pushed the last RC" policy, which is also the day I'm leaving for vacation.
• Bash 5 is an unsupported / untested setup. We don't test it on CI and none of our officially supported OS uses Bash 5 yet, so if it breaks, that's not great, but it's not like we promised that it works. (I'm fine with adding a more modern Linux that uses Bash 5 to our CI to test this in the future.)
• Running Bazel with a disabled feature flag also isn't really a supported configuration. If that then breaks (and only in combination with using an unsupported Bash version), I'd say, that's just bad luck.
• It's also not a regression compared to earlier Bazel releases.

[meteorcloudy]

This one looks like a release blocker for 0.23.0
#7459
Also happens on our CI, reported at #7464

[ulfjack]

For the record, commit is 2310b1c.

[philwo]

Thanks, Ulf. If we need another RC anyway due to that other breakage, let's also cherry pick that one. 👍

[meisterT]

f9eb1b5 needs to be cherry picked
cc @laurentlb

[laurentlb]

I wanted to help with the release, but the number of problems is too big for a Friday evening.

Commands I've run (the previous cherry-pick + two new cherry-picks):

• scripts/release/release.sh create --force_rc=3 0.23.0 441fd75 6ca7763 2310b1c f9eb1b5
• scripts/release/release.sh push

Issues:

• Playbook mentions the url https://buildkite.com/bazel-trusted/bazel-release. It doesn't work for me.
• Tests on Windows are failing (https://buildkite.com/bazel/bazel-bazel/builds/6901).
• I don't see any "Unblock step" anymore.
• I'm supposed to "Trigger a new pipeline in BuildKite to test the release candidate with all the downstream projects". But that button has been removed from https://buildkite.com/bazel/bazel-with-downstream-projects-bazel.

Edit. Philipp gave me the rights. I've triggered the downstream projects pipeline and rerun the Windows tests. I've deployed the artifacts.

[laurentlb]

rc3 is available: https://releases.bazel.build/0.23.0/rc3/index.html

Issues:

• A Windows test is still failing (https://buildkite.com/bazel/bazel-bazel/builds/6912#70da5be9-e880-4f03-b69e-42631a7c6c1d)
• Tulsi test is failing (https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/821#671667ba-87cc-437c-9a10-b843c034bdcb) with the error 2019/02/23 01:19:06 could not create directory /Users/buildkite/Library/Caches/bazelisk: mkdir /Users/buildkite/Library/Caches/bazelisk: operation not permitted.

I've deployed the artifacts as both issues seem unrelated to Bazel itself; it looks like infrastructure bugs.

[philwo]

• A Windows test is still failing (https://buildkite.com/bazel/bazel-bazel/builds/6912#70da5be9-e880-4f03-b69e-42631a7c6c1d)

This is a known issue (reported by some Googlers internally as b/125831768, was supposedly but not really fixed by bazelbuild/continuous-integration#496). The test is broken in that it doesn't run the bazel from its runfiles, instead it accidentally runs the bazel installed on the system, so doesn't test the Bazel from your workspace at all.

We can ignore this for now. It only started failing now due to infra changes on CI (installed Bazelisk instead of Bazel, which crashes when $HOME or %LocalAppDir% are not set in the environment), but it probably has always been broken.

[philwo]

• could not create directory /Users/buildkite/Library/Caches/bazelisk: mkdir /Users/buildkite/Library/Caches/bazelisk: operation not permitted

@fweikert This looks like Tulsi is running integration tests (?) that run Bazel (= Bazelisk) in a sandbox, which means it doesn't have access to $HOME/Library/Caches/bazelisk.

One way to fix this is to add --sandbox_writable_path=/Users/buildkite/Library/Caches/bazelisk to the test_flags of their bazelci.yml config for macOS (or maybe just add this in general to all tests running on macOS via bazelci.py here: https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/bazelci.py#L1106).

Another way would be to add no-sandbox to the tags of the failing tests.

[philwo]

@laurentlb Would you be OK with formally taking over this release?

I think there isn't much left to do except cherry-picking fixes for whatever regression still come up and then eventually releasing it.

I prettified the release notes manually for rc1 already and they are LGTM'd by Serge.

[laurentlb]

I think I can release later today. Let me know if I missed anything.

There's still some cleaning to do in the release notes, in case someone wants to help (https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit#heading=h.gpn2s9t6bosr).

[laurentlb]

Bazel 0.23.0 released: https://github.com/bazelbuild/bazel/releases/tag/0.23.0

cc @vbatts @petemounce @excitoon

[vbatts]

On 26/02/19 09:56 -0800, Laurent Le Brun wrote: Bazel 0.23.0 released: [1]https://github.com/bazelbuild/bazel/releases/tag/0.23.0 cc ***@***.*** ***@***.*** ***@***.***

https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/862699/

[thundergolfer]

Nice!

I would have thought java_binary.deploy_env would have made it under the "Important changes:" section.

Edit: Originally had java_library which was wrong

[laurentlb]

I'm not familiar with this specific change, but here's our process:

For the release notes, we first gather all commits with a RELNOTES tag in the description. Then we ask developers to review the announcement. If we missed something, you can still send a PR on https://github.com/bazelbuild/bazel-blog

[thundergolfer]

It's a change that is relevant to people looking to get Bazel support for Maven's provided or Gradle's compileOnly features.

Thanks for that info I'll send a PR if I get some time.

[meisterT]

Laurent: this needs to be cherry picked into a patch release f0a1597 to fix #7555. Thanks!