Bazel remote cache SSLEngine closed already

[sevki]

Description of the problem / feature request:

bazel fails reading/writing to remote-cache

WARNING: Error reading from the remote cache:
SSLEngine closed already
WARNING: Error writing to the remote cache:
SSLEngine closed already

the certificate is valid, and issued by DigiCert, it uses TLS 1.2 and ECDSA. The certificate is able to be verified with every up to date browser.

Feature requests: what underlying problem are you trying to solve with this feature?

Ship bazel with a more up to date trust store or provide a simpler way to use one

As the homogeny of operating systems becoming more and more prevalent, what would be great is if bazel just worked™. It would be a major pain if we'd have to go and package it for debian, archlinux, fedora, macos, windows and so on...

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

bazel build --remote_http_cache=https://cache.sevki.net/ ....
I've deployed this as an example, I'm not using this in production or anything, just wanted to share this as a quick repro step.

What operating system are you running Bazel on?

Linux 4.18.16-arch1-1-ARCH

What's the output of bazel info release?

release 0.19.0- (@non-git)

If bazel info release returns "development version" or "(@Non-Git)", tell us how you built Bazel.

pacman -S bazel

Have you found anything relevant by searching the web?

• Embedded JDK ignores system cacerts #5741 this points to the next issue as a solution
• https://groups.google.com/forum/#!topic/bazel-discuss/13uPDObyfQg this solution doesn't work for me

[buchgr]

Hey Sevik,

you seem to be using Bazel from an unofficial package. Can you please try with an official Bazel release? Bazel doesn't ship with a truststore but uses what's provided by it's JVM. Presumably arch linux ships its own JVM and truststore?

Thanks!

[sevki]

I tried using it with the oficial deb too.
Arch is also using an official release. https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/bazel#n15

[buchgr]

Thanks @sevki.

Apparently, the OpenJDK did a major overhaul of trusted CAs with the release of JDK10 and this list seems to also include DigiCert: http://openjdk.java.net/jeps/319. So once we upgrade Bazel's JDK to JDK10 this should solve your issues. You can do that already by specifying the --server_javabase and pointing it to a JDK10 folder.

[buchgr]

@sevki I just talked to people responsible for updating Bazel's JDK to 10. The plan is to do that in Bazel 0.22. There are a few changes that need to happen before we can safely update to JDK10, most importantly the decoupling of Bazel's JDK and javac's JDK. After 0.22 we will be able to update to new JDKs as their are released.

[sevki]

thanks @buchgr for the response. openjdk-10 seems to break bazel. it gets stuck somewhere in analysis stage

[sevki]

sorry doesn't break it, it's just less verbose with git

[sevki]

also openjdk-10 doesn't seem to fix the issue. still getting

WARNING: Error reading from the remote cache:
SSLEngine closed already
WARNING: Error writing to the remote cache:
SSLEngine closed already

[sevki]

Here is what I've tried and this problem still persists;

• changed server javabase to use openjdk10
• installed the latest version of ca-certificates-java ran with --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit after that
• combined both javabase and the host jvm args methods

nothing helped

[sevki]

I've tried this, with the both openjdk-8 and openjdk-10 and both worked no problem. bazel is still failing when I change the --server_javabase openjdk-10 or anything that is not embedded and I'm still getting SSLEngine already closed. i'm at my wits end, don't know how else to debug this, any suggestions @buchgr?

[sevki]

@buchgr so this does not seem to be a problem with the truststore but rather a problem with the http client that bazel uses, which does not seem to support SNI

[sevki]

attached is the pcap

https://sevki.io/bazel-cache.pcap

[buchgr]

Thanks for debugging this @sevki! So we'll need to add SNI support to it. This should actually be a one liner:

bazel/src/main/java/com/google/devtools/build/lib/remote/blobstore/http/HttpBlobStore.java

Line 211 in 612b1ed

SSLEngine engine = sslCtx.newEngine(ch.alloc());

Instead of

SSLEngine engine = sslCtx.newEngine(ch.alloc());

call

SSLEngine engine = sslCtx.newEngine(ch.alloc(), hostname, port);

Do you want to take a stab at it? :-)

[sevki]

Yeah sure 😄