Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remote: Occasional "SSLEngine closed already" #7041

Closed
nicolov opened this issue Jan 4, 2019 · 6 comments
Closed

remote: Occasional "SSLEngine closed already" #7041

nicolov opened this issue Jan 4, 2019 · 6 comments
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@nicolov
Copy link
Contributor

nicolov commented Jan 4, 2019

When using the HTTP remote cache, we get (very) occasional errors like:

(20:36:32) WARNING: Error reading from the remote cache:
SSLEngine closed already

Unlike #6551, which failed 100% of the time because of a certificate issue, we only get this once or twice over 10k actions.

This is 0.19.2 on ubuntu 14.
@jin jin added team-Remote-Exec Issues and PRs for the Execution (Remote) team and removed team-Execution labels Jan 14, 2019
@buchgr buchgr self-assigned this Jan 16, 2019
@buchgr buchgr added the P2 We'll consider working on this in future. (Assignee optional) label Jan 16, 2019
@buchgr
Copy link
Contributor

buchgr commented Jan 16, 2019

Does this error completely disable the remote cache for the build/test invocation or is it isolated to a single action?

@buchgr buchgr removed the untriaged label Jan 16, 2019
@uri-canva
Copy link
Contributor

In our case it failed the build, the fact the exception gets printed twice makes me think it caused the bazel server to crash:

Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//webclient/app:files BuildConfigurationValue.Key[e39613c467e02c426b1a9ca9df01b56d] false, actionIndex=1}' (requested by nodes 'ActionLookupData{actionLookupKey=//export-harness:export_harness_resources BuildConfigurationValue.Key[e39613c467e02c426b1a9ca9df01b56d] false, actionIndex=0}')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:531)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalArgumentException: Self-suppression not permitted
	at java.base/java.lang.Throwable.addSuppressed(Throwable.java:1025)
	at com.google.devtools.build.lib.remote.AbstractRemoteActionCache.download(AbstractRemoteActionCache.java:358)
	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:169)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:118)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:88)
	at com.google.devtools.build.lib.actions.SpawnActionContext.beginExecution(SpawnActionContext.java:41)
	at com.google.devtools.build.lib.exec.ProxySpawnActionContext.beginExecution(ProxySpawnActionContext.java:56)
	at com.google.devtools.build.lib.actions.SpawnContinuation$1.execute(SpawnContinuation.java:80)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction$SpawnActionContinuation.execute(SpawnAction.java:1342)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.beginExecution(SpawnAction.java:314)
	at com.google.devtools.build.lib.actions.Action.execute(Action.java:123)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$4.execute(SkyframeActionExecutor.java:851)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:985)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:957)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:116)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:77)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:577)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:760)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:275)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:454)
	... 4 more
Caused by: javax.net.ssl.SSLException: SSLEngine closed already
	at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)
 
INFO: Elapsed time: 441.264s, Critical Path: 6.00s
INFO: 824 processes: 808 remote cache hit, 16 linux-sandbox.
FAILED: Build did NOT complete successfully
Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//webclient/app:files BuildConfigurationValue.Key[e39613c467e02c426b1a9ca9df01b56d] false, actionIndex=1}' (requested by nodes 'ActionLookupData{actionLookupKey=//export-harness:export_harness_resources BuildConfigurationValue.Key[e39613c467e02c426b1a9ca9df01b56d] false, actionIndex=0}')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:531)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalArgumentException: Self-suppression not permitted
	at java.base/java.lang.Throwable.addSuppressed(Throwable.java:1025)
	at com.google.devtools.build.lib.remote.AbstractRemoteActionCache.download(AbstractRemoteActionCache.java:358)
	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:169)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:118)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:88)
	at com.google.devtools.build.lib.actions.SpawnActionContext.beginExecution(SpawnActionContext.java:41)
	at com.google.devtools.build.lib.exec.ProxySpawnActionContext.beginExecution(ProxySpawnActionContext.java:56)
	at com.google.devtools.build.lib.actions.SpawnContinuation$1.execute(SpawnContinuation.java:80)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction$SpawnActionContinuation.execute(SpawnAction.java:1342)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.beginExecution(SpawnAction.java:314)
	at com.google.devtools.build.lib.actions.Action.execute(Action.java:123)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$4.execute(SkyframeActionExecutor.java:851)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:985)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:957)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:116)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:77)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:577)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:760)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:275)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:454)
	... 4 more
Caused by: javax.net.ssl.SSLException: SSLEngine closed already
	at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//webclient/app:files BuildConfigurationValue.Key[e39613c467e02c426b1a9ca9df01b56d] false, actionIndex=1}' (requested by nodes 'ActionLookupData{actionLookupKey=//export-harness:export_harness_resources BuildConfigurationValue.Key[e39613c467e02c426b1a9ca9df01b56d] false, actionIndex=0}')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:531)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalArgumentException: Self-suppression not permitted
	at java.base/java.lang.Throwable.addSuppressed(Throwable.java:1025)
	at com.google.devtools.build.lib.remote.AbstractRemoteActionCache.download(AbstractRemoteActionCache.java:358)
	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:169)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:118)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:88)
	at com.google.devtools.build.lib.actions.SpawnActionContext.beginExecution(SpawnActionContext.java:41)
	at com.google.devtools.build.lib.exec.ProxySpawnActionContext.beginExecution(ProxySpawnActionContext.java:56)
	at com.google.devtools.build.lib.actions.SpawnContinuation$1.execute(SpawnContinuation.java:80)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction$SpawnActionContinuation.execute(SpawnAction.java:1342)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.beginExecution(SpawnAction.java:314)
	at com.google.devtools.build.lib.actions.Action.execute(Action.java:123)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$4.execute(SkyframeActionExecutor.java:851)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:985)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:957)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:116)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:77)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:577)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:760)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:275)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:454)
	... 4 more
Caused by: javax.net.ssl.SSLException: SSLEngine closed already
FAILED: Build did NOT complete successfully

@buchgr
Copy link
Contributor

buchgr commented Oct 18, 2019

@uri-canva what version of bazel is this?

@uri-canva
Copy link
Contributor

1.0.0, might be related to #9362 ?

@buchgr
Copy link
Contributor

buchgr commented Oct 18, 2019

yep. this is fixed in 1.0.1. Sorry about this!

@scele scele mentioned this issue Nov 4, 2019
Closed
@buchgr buchgr removed their assignment Jan 9, 2020
@em84 em84 mentioned this issue Apr 21, 2020
Closed
@coeuvre
Copy link
Member

coeuvre commented Dec 9, 2020

Closing since the issue seems to be fixed. Feel free to re-open if it still happens.

@coeuvre coeuvre closed this as completed Dec 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jin @buchgr @aiuto @coeuvre @nicolov @uri-canva