Support mill init from an existing SBT project (2000USD Bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 2000USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

---

We should be able to run ./mill init to do a best effort conversion of an existing SBT project to Mill:

• Code-generate a build.mill containing a MavenModulefor the root build.sbt, the package.mill for any subfolder build.sbts or SBT subprojects in the root build.sbt
• Convert the main and test dependencies on third-party libraries to ivyDeps
• Convert inter-subproject dependencies to moduleDeps
• Set any necessary javacOptions
• Implement PublishModule with relevant metadata

Such a conversion will never be 100%, but if we can automate 80% of the conversion that will already be a huge help for people migrating to Mill or even just trying it out

[Dylanb-dev]

hello, I am having a look at this. I haven't used java much (just a bit of clojure and kotlin) so might take a couple days for PR. If I don't have anything in 48 hours happy for anyone else to take this.

[steinybot]

I'll take a crack at this.

[lihaoyi]

@steinybot go for it

[Dylanb-dev]

Hi, I have been looking at this.

I am using scala-meta to parse the SBT files, can I confirm some test examples?

Test 1 - basic sbt from sbt documentation

  ThisBuild / scalaVersion := "2.13.12"
  ThisBuild / organization := "com.example"

  lazy val hello = project
    .in(file("."))
    .settings(
      name := "Hello",
      libraryDependencies ++= Seq(
        "org.scala-lang" %% "toolkit" % "0.1.7",
        "org.scala-lang" %% "toolkit-test" % "0.1.7" % Test
      )
    )

produces ./build.sc with

import mill._, scalalib._

object Hello extends MavenModule {
  def scalaVersion = "2.13.12"
  def organization = "com.example"

  def ivyDeps = Agg(
    ivy"org.scala-lang::toolkit:0.1.7"
  )

  object test extends ScalaTests with TestModule.Munit {
    def ivyDeps = Agg(
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )
  }
} 

test 2 - more complex realworld sbt with publishing

import Dependencies._

ThisBuild / scalaVersion := "2.12.19"
ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / organization := "com.example"
ThisBuild / organizationName := "example"

lazy val root = (project in file("."))
  .settings(
    name := "Hello",
    libraryDependencies += scalaTest % Test
  )
libraryDependencies += "org.scalameta" %% "scalameta" % "4.9.9"
// libraryDependencies += "com.typesafe.play" %% "play-json" % "2.9.2"

produces this mill build.sc

import mill._, scalalib._, publish._
import Dependencies._

object Hello extends MavenModule with PublishModule {
  def scalaVersion = "2.12.19"
  def publishVersion = "0.1.0-SNAPSHOT"
  
  def ivyDeps = Agg(
    ivy"org.scala-lang::toolkit:0.1.7"
  )

  def pomSettings = PomSettings(
    name = "example",
    organization = "com.example",
  )

  object test extends ScalaTests with TestModule.Munit {
    def ivyDeps = Agg(
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )
  }
}

test 3 - build.sbt with subprojects

ThisBuild / version      := "0.1.0"
ThisBuild / scalaVersion := "2.13.6"
ThisBuild / organization := "com.example"

val scalaTest = "org.scalatest" %% "scalatest" % "3.2.7"
val gigahorse = "com.eed3si9n" %% "gigahorse-okhttp" % "0.5.0"
val playJson  = "com.typesafe.play" %% "play-json" % "2.9.2"

lazy val hello = (project in file("."))
  .aggregate(helloCore)
  .dependsOn(helloCore)
  .enablePlugins(JavaAppPackaging)
  .settings(
    name := "Hello",
    libraryDependencies += scalaTest % Test,
  )

lazy val helloCore = (project in file("core"))
  .settings(
    name := "Hello Core",
    libraryDependencies ++= Seq(gigahorse, playJson),
    libraryDependencies += scalaTest % Test,
  )

Produces the following build.sc

import mill._, scalalib._

object Hello extends MavenModule with PublishModule {
  def scalaVersion = "2.13.6"
  def publishVersion = "0.1.0"
  
  def ivyDeps = Agg(
      ivy"com.eed3si9n::gigahorse-okhttp:0.5.0",
      ivy"com.typesafe.play::play-json:2.9.2",
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )

  def pomSettings = PomSettings(
    name = "Hello",
    organization = "com.example",
  )

  object test extends ScalaTests with TestModule.Munit {
    def ivyDeps = Agg(
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )
  }
}

with the following in ./core/package.mill

package build.core

import mill._, scalalib._

object `package` extends MavenModule with ScalaModule {
  def name = "Hello Core"
  def ivyDeps = Agg(
      ivy"com.eed3si9n::gigahorse-okhttp:0.5.0",
      ivy"com.typesafe.play::play-json:2.9.2",
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )
  object test extends ScalaTests with TestModule.Munit {
      def ivyDeps = Agg(
        ivy"org.scala-lang::toolkit-test:0.1.7"
      )
}

[lihaoyi]

Those examples look reasonable. One issue is that Scala-Meta parsing has a pretty low ceiling of how sophisticated builds it can handle. e.g. once people start having variables or helper methods, which is very common, it no longer works.

I'd say that to do this well, we want to actually run SBT to evaluate the SBT build files, and export the metadata that results from this evaluation. That way you don't care how much indirection exists in the SBT build: SBT has to be able to handle it, and in the end convert it to a "dumb metadata" format that will be much easier for Mill to process

[Dylanb-dev]

Sounds reasonable, I checked sbt for meta data output and https://github.com/sbt/sbt-buildinfo but it seemed too limited. I will take a closer look.

[lihaoyi]

There's an SBT -> Bazel converter that I think goes through SBT to dump its metadata https://github.com/stripe-archive/sbt-bazel

[lihaoyi]

This may be relevant as well https://stackoverflow.com/a/62767456/871202

[sideeffffect]

@Dylanb-dev https://github.com/oyvindberg/bleep can import sbt builds too. AFAIK it works by exporting sbt build to BSP (the .bsp/ directory) and then importing information to Bleep from it.
Maybe you could use this trick too.

[llvee]

@Dylanb-dev Did you complete this?

If not please let me know in the next 24 hours as I am interested in potentially attempting this.

[Dylanb-dev]

@llvee

Go ahead, although it might be better to wait for #3449 to be finished which will help make this issue much easier to complete.

[llvee]

@Dylanb-dev Thank you for mentioning that, I will review #3449.

[lihaoyi]

@ajaychandran's PR #3449 has landed, in case anyone wants to take a crack at this. We should be able to re-use all the rendering logic, and the only issue them becomes how to extract the relevant metadata from an SBT build to feed into the renderer

[lihaoyi]

Bumping the bounty on this from 1500 -> 2000USD

[Friendseeker]

I am currently working on this bounty. I have a concrete plan (use sbt-structure to parse the sbt settings & use #3449 to transpile the parsed settings), however I have other commitments in my life so my progress is slow.

I am down teaming up with others. I do not aim to take a monetary cut (I plan to ask Haoyi for information as reward, instead of monetary reward). If you would like to team up with me, feel free to join Mill's discord channel and ping me.

[lihaoyi]

Haha if you want information just email me and ask, no need to wait for completing the ticket. Can answer whatever questions you have today to the best of my abilities

[sake92]

Had a look at sbt-structure, it works well but it doesn't give us declared dependencies, but all (including transitive ones).
It isn't possible (to my understanding) to figure out which ones are transitive.. :/
Same thing with Bloop export, full classpath only.

Maybe we can steal the code from bloop:
https://github.com/scalacenter/bloop/blob/main/integrations/sbt-bloop/src/main/scala/bloop/integrations/sbt/SbtBloop.scala#L1146
or sbt-structure https://github.com/JetBrains/sbt-structure/tree/master/extractor/src/main/scala/org/jetbrains/sbt/extractors
but it's a looot of code there. 😄

But it's almost certainly easier than parsing sbt's output.
It doesn't print dependencies properly, e.g. you cant disambiguate between a java (%) and scala dep (%%)

[sideeffffect]

doesn't give us declared dependencies, but all (including transitive ones)

Is sbt-structure or Bloop open to adding this feature?

[lihaoyi]

I'd suggest anyone who wants to tackle this should just copy-paste/fork the implementations from sbt-structure or bloop. Long term maintenance of the fork and whether or not we can upstream it can be discussed later, first priority is just to get something working at all

[lihaoyi]

#4363 has landed, so mill init now has support for both Maven and Gradle projects. That PR also substantially refactors the auto-migration code to clean it up which should make it substantially easier to plug in support for SBT: you no longer need to worry about build-file generation at all! Just extract the subproject metadata from an SBT build, put it into the case classes defined in ir.scala, and we're done.

Seems like it should be an easy 2000USD for anyone who wants to pick this up and bring it over the finish line

[ShreckYe]

Hi I see it's been some days since #4363 landed. Is anyone working on this now? I helped review #4363, got free recently, and would like to give this a try, but I don't want to compete if there's someone already working on it.