community_build_a fails intermittently with corrupt scalatest artifacts

[griggt]

For some weeks or months now, community_build_a fails intermittently in CI with corrupt scalatest artifacts. The issue seems to arise more often when the CI is under heavy load.

Examples of error messages observed in the CI logs:

[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:///root/.ivy2/local/org.scalatest/scalatest-core_3.0.0-RC2/3.2.3/bundles/scalatest-core_3.0.0-RC2.jar: wrong checksum: /root/.ivy2/local/org.scalatest/scalatest-core_3.0.0-RC2/3.2.3/bundles/scalatest-core_3.0.0-RC2.jar (expected SHA-1 7e3c896002ef95528f8205305904a22b8e1c1f3f in /root/.ivy2/local/org.scalatest/scalatest-core_3.0.0-RC2/3.2.3/bundles/scalatest-core_3.0.0-RC2.jar.sha1, got 55de3b3142a807c18d64786b52a8a06fa49b7443)

[error] error while loading <root>,
[error] Error accessing /root/.ivy2/local/org.scalatest/scalatest-flatspec_3.0.0-RC1/3.2.3/bundles/scalatest-flatspec_3.0.0-RC1.jar
[error] Could not find package scala from compiler core libraries.
[error] Make sure the compiler core libraries are on the classpath.

[error] (scalatestCoreDotty / publishLocal) java.io.IOException: size of source file /__w/dotty/dotty/community-build/community-projects/scalatest/dotty/core/target/scala-3.0.0-M3/scalatest-core_3.0.0-M3-3.2.3.jar(0) differs from size of dest file /root/.ivy2/local/org.scalatest/scalatest-core_3.0.0-M3/3.2.3/bundles/scalatest-core_3.0.0-M3.jar(5270082) - please retry

[error] (scalatestCoreDotty / publishLocal) Missing files for publishing:
[error] 	/__w/dotty/dotty/community-build/community-projects/scalatest/dotty/core/target/scala-3.0.0-M3/scalatest-core_3.0.0-M3-3.2.3.jar

Links to some failing CI runs:

https://github.com/lampepfl/dotty/runs/2034290606?check_suite_focus=true#step:7:3717
https://github.com/lampepfl/dotty/runs/1814806014?check_suite_focus=true#step:7:2520
https://github.com/lampepfl/dotty/runs/1806534005?check_suite_focus=true#step:7:2488
https://github.com/lampepfl/dotty/runs/1701458513?check_suite_focus=true#step:7:2530
https://github.com/lampepfl/dotty/runs/1720784830?check_suite_focus=true#step:7:2533
https://github.com/lampepfl/dotty/runs/1715832346?check_suite_focus=true#step:7:2532
https://github.com/lampepfl/dotty/runs/1588578361?check_suite_focus=true#step:7:2480

[griggt]

The issue appears to be caused by a race condition in the sbt-osgi plugin in the particular manner in which it is used by the scalatest build.

The race condition seems to have been introduced by sbt/sbt-osgi#64 in sbt-osgi 0.9.5.

The scalatest build upgraded sbt-osgi from 0.9.4 to 0.9.6 for JDK 15 compatability (scalatest/scalatest#1899).
This change was incorporated into the Dotty community build by #10503.

The Build

The problem arises when an sbt project has a build structure similar to:

lazy val parent = project
  .enablePlugins(SbtOsgi)
  .settings(osgiSettings)
  .dependsOn(child)
  .aggregate(child)

lazy val child = project
  .enablePlugins(SbtOsgi)
  .settings(osgiSettings)

In particular, there must be a project which both aggregates and depends on another project in a multi-project build, and both projects must enable the SbtOsgi plugin and use osgiSettings.

In the scalatest build, the role of parent is played by scalatestDotty, which depends on and aggregates scalatestCoreDotty, scalatestFeatureSpecDotty, scalatestFlatSpecDotty, scalatestFreeSpecDotty, scalatestFunSuiteDotty, scalatestFunSpecDotty, scalatestPropSpecDotty, scalatestRefSpecDotty, scalatestWordSpecDotty, scalatestDiagramsDotty, scalatestMatchersCoreDotty, scalatestShouldMatchersDotty and scalatestMustMatchersDotty.

The Trigger

sbt> parent/publishLocal

The Race

1. The use of osgiSettings overrides the default publishing behavior to create the jar using the osgiBundle task rather than the default packageBin task:

def osgiSettings: Seq[Setting[_]] = Seq(
  packagedArtifact in (Compile, packageBin) := Scoped.mkTuple2(
    (artifact in (Compile, packageBin)).value,    // the jar artifact
    OsgiKeys.bundle.value                         // built with a custom task `osgiBundle`
  )
  ...
)

https://github.com/sbt/sbt-osgi/blob/4d8c4be70862a9f46990c46ae0c533239fde6883/src/main/scala/com/typesafe/sbt/osgi/SbtOsgi.scala#L37

2. Since parent aggregates child, child/publishLocal will be executed

3. child/publishLocal runs child/osgiBundle, which builds the child jar in the build target directory

4. child/publishLocal runs IvyActions.publish, which copies the child jar from the target directory to the local Ivy repository

5. parent/publishLocal runs parent/osgiBundle, to build the parent jar in the build target directory

6. Since osgiBundle depends on the value of the dependencyClasspathAsJars task, and parent dependsOn child, as a side effect of parent/osgiBundle the child jar is rebuilt (!) in the build target directory using the default packageBin task

The issue is that items 4 and 6 may happen concurrently, and the jar that is copied from the target directory to the local Ivy repository in (4) may be a partially written jar file from (6).

The Culprit

sbt/sbt-osgi#64 included the following change:

      bundle := Osgi.bundleTask(
        manifestHeaders.value,
        additionalHeaders.value,
-       (fullClasspath in Compile).value,
+       (dependencyClasspathAsJars in Compile).value.map(_.data) ++ (products in Compile).value,
        (artifactPath in (Compile, packageBin)).value,
        (resourceDirectories in Compile).value,
        embeddedJars.value,
        explodedJars.value,
        failOnUndecidedPackage.value,
        (sourceDirectories in Compile).value,
        (packageOptions in (Compile, packageBin)).value,
        streams.value)

Without this change, (6) does not occur, as the fullClasspath task does not cause the target jar to be built.

[smarter]

Amazing detective work!

• Would you mind opening an issue in https://github.com/sbt/sbt-osgi to track this?
• Having to keep around a commit that modifies the scalatest build makes upgrades a bit more complicated. I wonder if we could instead disable the osgi logic from the community-build, for example by resetting packagedArtifact to its original value. If I'm reading https://github.com/sbt/sbt/blob/ff6cbf2e1fb745b4b46218be9adf6d7c080c0693/main/src/main/scala/sbt/Defaults.scala#L1802 correctly this should be:

packagedArtifact in (Compile, packageBin) := ((artifact in (Compile, packageBin)).value -> (packageBin in Compile).value)