remote cache "SSLEngine Closed already" after multiple downloads/uploads

[em84]

Description of the problem / feature request:

When building a target with a lot of dependencies, which get all hit by the remote cache, while the final target doesn't we can systematically reproduce the error:

[898 / 899] PackageTar webconsole/webconsole.tar; 89s remote-cache
ERROR: /home/developer/project/webconsole/BUILD:170:1: PackageTar webconsole/webconsole.tar failed due to unexpected I/O exception: SSLEngine closed already
javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)
Target //webconsole:webconsole failed to build
INFO: Elapsed time: 206.327s, Critical Path: 105.56s
INFO: 883 processes: 870 remote cache hit, 13 processwrapper-sandbox.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. Turn on remote cache with an SSL remote (we use jfrog Artifactory).
2. Configure a rule (e.g. pkg_tar) with a "big" (we have > 800 remote cache hits) set of dependencies
3. build it
4. after successful build only remove the final target result from the remote cache, to force bazel to download all the dependencies before trying to upload the result in the next build.
5. clean local cache by bazel clean
6. rerun the build with the rule above

What operating system are you running Bazel on?

We can reproduce this on different debian and ubuntu linux systems.

What's the output of bazel info release?

INFO: Invocation ID: f7f9edff-e280-4e44-8123-2afe534e803c
release 2.2.0

Have you found anything relevant by searching the web?

Most likely related to:
#7041

Any other information, logs, or outputs that you want to share?

• In wireshark we see multiple uploads and downloads to the remote cache succeeding, so we can eliminate certificate issues.
• we reproduce this with --remote_max_connections=1, so we can eliminate any sort of parallel download/upload issues or to many open connections.

[ulfjack]

Is this reproducible?

[em84]

Yes it is reproducible (like I already wrote in the initial report of the issue). So far we only reproduced it with jfrog artifactory as remote. We are currently setting up another kind of remote to exclude any issues with our artifactory setup.

[ulfjack]

I fixed a bug like this a couple years ago. If it's reproducible on every run, then I'd be interested in looking at a repro to see if I can debug it. (Or you can try to debug if you like.)

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 2+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!