From 3d9a5f6ae3a5c67fb60520664e33a1c904dd7a90 Mon Sep 17 00:00:00 2001 From: Jason Gustafson Date: Thu, 11 Aug 2022 14:00:06 -0700 Subject: [PATCH] KAFKA-14163; Retry compilation after zinc compile cache error (#12507) We have been seeing builds fail due to errors such as the following: ``` Timeout waiting to lock zinc-1.6.1_2.13.8_17 compiler cache (/home/jenkins/.gradle/caches/7.5.1/zinc-1.6.1_2.13.8_17). It is currently in use by another Gradle instance. ``` This patch includes a workaround: if the compilation fails due to this zinc compile cache error, then we retry it. Reviewers: Ismael Juma Co-authored-by: Lucas Bradstreet --- Jenkinsfile | 2 +- retry_zinc | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+), 1 deletion(-) create mode 100755 retry_zinc diff --git a/Jenkinsfile b/Jenkinsfile index cce625b8d8ea3..e1b4c5b3a1d67 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -19,7 +19,7 @@ def doValidation() { sh """ - ./gradlew -PscalaVersion=$SCALA_VERSION clean compileJava compileScala compileTestJava compileTestScala \ + ./retry_zinc ./gradlew -PscalaVersion=$SCALA_VERSION clean compileJava compileScala compileTestJava compileTestScala \ spotlessScalaCheck checkstyleMain checkstyleTest spotbugsMain rat \ --profile --no-daemon --continue -PxmlSpotBugsReport=true """ diff --git a/retry_zinc b/retry_zinc new file mode 100755 index 0000000000000..9d7b5553fa997 --- /dev/null +++ b/retry_zinc @@ -0,0 +1,49 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Hacky workaround for https://github.com/gradle/gradle/issues/3777 +# There is currently no configurable timeout, so we retry builds jenkins when we can't get a lock on the zinc compiler cache +# Hopefully we can remove this in the future, but this will save us from having to manually rebuild for the time being. +# Example: +# [2021-10-19T17:25:07.234Z] * What went wrong: +# [2021-10-19T17:25:07.234Z] Execution failed for task ':streams:streams-scala:compileScala'. +# [2021-10-19T17:25:07.234Z] > Timeout waiting to lock zinc-1.3.5_2.13.6_8 compiler cache (/home/jenkins/.gradle/caches/7.0.2/zinc-1.3.5_2.13.6_8). It is currently in use by another Gradle instance. +# [2021-10-19T17:25:07.234Z] Owner PID: 3999 +# [2021-10-19T17:25:07.234Z] Our PID: 3973 +# [2021-10-19T17:25:07.234Z] Owner Operation: +# [2021-10-19T17:25:07.234Z] Our operation: +# [2021-10-19T17:25:07.234Z] Lock file: /home/jenkins/.gradle/caches/7.0.2/zinc-1.3.5_2.13.6_8/zinc-1.3.5_2.13.6_8.lock + +set -uf -o pipefail + +retryable=1 +while [[ "$retryable" != 0 ]]; do + retryable=0 + rm -f buildoutput.log + + "$@" 2>&1 | tee buildoutput.log + commandReturnCode=$? + + if [ $commandReturnCode -ne 0 ]; then + if grep "Timeout waiting to lock zinc" buildoutput.log; then + retryable=1 + echo 'Retrying due to zinc lock timeout' + continue + else + exit $commandReturnCode + fi + fi +done