Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for 'mill init' from existing Gradle project #4363

Merged
merged 44 commits into from
Jan 23, 2025
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
a71cecb
Added support for importing a Gradle project
ajaychandran Jan 17, 2025
49386ba
Escaped path to fix Windows error and fixed format
ajaychandran Jan 17, 2025
4cf060c
Fix format
ajaychandran Jan 17, 2025
73c31e2
Fix initscript
ajaychandran Jan 17, 2025
8e448da
Cleanup
ajaychandran Jan 17, 2025
c4a0549
Cleanup
ajaychandran Jan 17, 2025
a01f852
Cleanup
ajaychandran Jan 17, 2025
7b08e2e
inline type aliases
lihaoyi Jan 22, 2025
96ffc8a
rename reps/interpIvy
lihaoyi Jan 22, 2025
723653e
make all string rendering methods start with the word render
lihaoyi Jan 22, 2025
8133658
move maven/gradle/buildgen into init
lihaoyi Jan 22, 2025
6539731
extract common config fields
lihaoyi Jan 22, 2025
491ef5b
merge
lihaoyi Jan 22, 2025
688f27a
remove unnecessary scalaconversions
lihaoyi Jan 22, 2025
c151767
autofix
lihaoyi Jan 22, 2025
c277e71
tidy up main/gradle/BuildGen
lihaoyi Jan 22, 2025
df20659
.
lihaoyi Jan 22, 2025
dbddfbd
more standardization between maven and gradle
lihaoyi Jan 22, 2025
8d6aa36
extract common module rendering logic into BuildGenUtil
lihaoyi Jan 22, 2025
0519f16
extract renderTrait helper
lihaoyi Jan 22, 2025
b03ae2d
extract Ir* case classes
lihaoyi Jan 22, 2025
c136f18
more refactoring of BuildGen
lihaoyi Jan 22, 2025
ce4a916
extract renderTrait helper
lihaoyi Jan 22, 2025
3285006
more refactoring
lihaoyi Jan 22, 2025
d5c68ee
.
lihaoyi Jan 22, 2025
f1b72a9
wip
lihaoyi Jan 22, 2025
00017ed
wip
lihaoyi Jan 22, 2025
2ad6263
wip
lihaoyi Jan 22, 2025
0cd3a69
wip
lihaoyi Jan 22, 2025
3def3e4
fix unit tests
lihaoyi Jan 22, 2025
5942db5
fix missing repos in gradle
lihaoyi Jan 22, 2025
46654ce
fix
lihaoyi Jan 22, 2025
35bcc35
cleanup
lihaoyi Jan 22, 2025
da1a56c
[autofix.ci] apply automated fixes
autofix-ci[bot] Jan 22, 2025
c61fce9
Fix warnings
ajaychandran Jan 22, 2025
2cf079c
Added/updated code docs
ajaychandran Jan 22, 2025
632be2a
Fix golden files
ajaychandran Jan 22, 2025
ff8747b
[autofix.ci] apply automated fixes
autofix-ci[bot] Jan 22, 2025
466a0bb
Use variable to improve maintainability
ajaychandran Jan 22, 2025
d8e7be4
Fixed failing test for Ehcache3 and set jvmId for Moshi test
ajaychandran Jan 22, 2025
afca05e
Document quirk
ajaychandran Jan 22, 2025
428310e
Resolved some review comments
ajaychandran Jan 22, 2025
3b863dd
Added support for custom repos in Maven conversion
ajaychandran Jan 23, 2025
25a427e
Added suppot for publishing protperties in Gradle Conversion.
ajaychandran Jan 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions build.mill
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ object Deps {
val mavenResolverTransportWagon =
ivy"org.apache.maven.resolver:maven-resolver-transport-wagon:$mavenResolverVersion"
val coursierJvmIndexVersion = "0.0.4-70-51469f"
val gradleApi = ivy"dev.gradleplugins:gradle-api:8.11.1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest is 8.12 but this is not important anyway.

I get it now that you are using "dev.gradleplugins:gradle-api" whose latest is 8.11.1.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, the version is conventionally extracted into a constant:

Suggested change
val gradleApi = ivy"dev.gradleplugins:gradle-api:8.11.1"
val gradleApiVersion = "8.11.1"
val gradleApi = ivy"dev.gradleplugins:gradle-api:$gradleApiVersion"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, the version is conventionally extracted into a constant:

Looks premature to me.


object RuntimeDeps {
val dokkaVersion = "2.0.0"
Expand Down
1 change: 1 addition & 0 deletions docs/modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
** xref:cli/query-syntax.adoc[]
* xref:migrating/migrating.adoc[]
** xref:migrating/maven.adoc[]
** xref:migrating/gradle.adoc[]
// This section gives a tour of the various user-facing features of Mill:
// library deps, out folder, queries, tasks, etc.. These are things that
// every Mill user will likely encounter, and are touched upon in the various
Expand Down
121 changes: 121 additions & 0 deletions docs/modules/ROOT/pages/migrating/gradle.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
= Migrating From Gradle to Mill
:page-aliases: Migrating_A_Gradle_Build_to_Mill.adoc
:icons: font

include::partial$gtag-config.adoc[]

The Mill `init` command can be used to convert a Gradle build to Mill. This has
xref:#limitations[limitations] and is not intended to reliably migrate 100% of
Gradle builds out there in the wild, but is instead meant to provide the basic
scaffolding of a Mill build for you to further refine and update manually.

Each Gradle project in a build tree is converted to a Mill module.
A nested `test` module is defined, if `src/test` exists, and is configured with a supported xref:javalib/testing.adoc[test framework], if found.

Again, note that `mill init` imports a Gradle build on a best-effort basis.
This means that while simple projects can be expected to complete without issue:

include::partial$example/javalib/migrating/3-gradle-complete.adoc[]

Projects with a complex build often require some manual tweaking in order to work:

include::partial$example/javalib/migrating/4-gradle-incomplete.adoc[]

== Capabilities

The conversion

* handles deeply nested modules
* captures publish settings
* configures dependencies for https://docs.gradle.org/current/userguide/dependency_configurations.html#sub:what-are-dependency-configurations[configurations]
** implementation / api
** compileOnly / compileOnlyApi
** runtimeOnly
** testImplementation
** testCompileOnly
* configures testing frameworks
** JUnit 4
** JUnit 5
** TestNG

[#arguments]
=== Command line arguments

The conversion and it's output (the generated Mill build files) can be customized using
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The conversion and it's output (the generated Mill build files) can be customized using
The conversion and its output (the generated Mill build files) can be customized using


* `--base-module` (`-b`): name of generated base module trait defining shared settings
[source,sh]
----
./mill init --base-module MyModule
----
Copy link
Member

@lefou lefou Jan 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example how to keep the source block in the list item.

https://docs.asciidoctor.org/asciidoc/latest/lists/continuation/#list-continuation

Suggested change
* `--base-module` (`-b`): name of generated base module trait defining shared settings
[source,sh]
----
./mill init --base-module MyModule
----
* `--base-module` (`-b`): name of generated base module trait defining shared settings
+
[source,sh]
----
./mill init --base-module MyModule
----


ajaychandran marked this conversation as resolved.
Show resolved Hide resolved
* `--base-project` (`-g`): name of Gradle project to extract settings for `--base-module`
[source,sh]
----
./mill init --base-module MyModule --base-project lib
----
NOTE: If not specified, a project will be selected arbitrarily.

* `--jvm-id` (`-j`): version of xref:fundamentals/configuring-jvm-versions.adoc[custom JVM] to configure in `--base-module`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"JVM ID" is not a widely know term. From the code I get it that it can include the distribution and the version, so you might've forgotten to update the docs here:

Suggested change
* `--jvm-id` (`-j`): version of xref:fundamentals/configuring-jvm-versions.adoc[custom JVM] to configure in `--base-module`
* `--jvm-id` (`-j`): distribution and version of xref:fundamentals/configuring-jvm-versions.adoc[custom JVM] to configure in `--base-module`, in the form of "($DISTRIBUTION:)$VERSION"

[source,sh]
----
./mill init --base-module MyModule --jvm-id 17
----

* `--test-module` (`-t`): name of generated nested test module (defaults to `test`)
[source,sh]
----
./mill init --test-module test
----

* `--deps-object` (`-d`): name of generated companion object defining dependency constants
[source,sh]
----
./mill init --deps-object Deps
----

* `--merge` (`-m`): merge build files generated for a multi-module build
[source,sh]
----
./mill init --merge
----

TIP: You can run `mill init` multiple times. It is recommended to run it first without any options.

[#limitations]
== Limitations

The conversion does not support

* custom dependency configurations
* custom tasks
* non-Java sources

Gradle plugin support is limited to

* https://docs.gradle.org/current/userguide/java_plugin.html[java]
* https://docs.gradle.org/current/userguide/publishing_maven.html[maven-publish]

[TIP]
====
These limitations can be overcome by:

* configuring equivalent Mill xref:extending/contrib-plugins.adoc[contrib]
or xref:extending/thirdparty-plugins.adoc[third party] plugins
* defining custom xref:extending/writing-plugins.adoc[plugins]
* defining custom xref:fundamentals/tasks.adoc[tasks]
* defining custom xref:fundamentals/cross-builds.adoc[cross modules]
====

== FAQ

How to fix `java.lang.UnsupportedClassVersionError` error thrown by `mill init`?

Select a https://docs.gradle.org/current/userguide/compatibility.html#java_runtime[Java Runtime] compatible with the version of Gradle used and pass it using the `--jvm-id` xref:arguments[argument].

How to fix test compilation errors?

* The test framework configured may be for an unsupported version; try upgrading the
corresponding dependencies.
* Mill does not add `compileIvyDeps` dependencies to the transitive dependencies of the nested
test module; specify the dependencies again, in `ivyDeps` or `runIvyDeps`.
70 changes: 39 additions & 31 deletions docs/modules/ROOT/pages/migrating/maven.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,10 @@ Maven builds out there in the wild, but is instead meant to provide the basic
scaffolding of a Mill build for you to further refine and update manually.

Each Maven module with a `pom.xml` is converted to a Mill `build.mill`/`package.mill`
file containing a top-level `MavenModule`. A nested `test` module is defined if both:
file containing a top-level `MavenModule`.
A nested `test` module is defined, if `src/test` exists, and is configured with a supported xref:javalib/testing.adoc[test framework], if found.

* `src/test` exists
* a supported xref:javalib/testing.adoc[test framework] is detected (for a _tests only_
module with test sources in `src/main/java`)


Again, note that `mill init` importing Maven builds is best effort.
Again, note that `mill init` importing Maven builds is best-effort.
This means that while small projects can be expected to complete without issue:

include::partial$example/javalib/migrating/1-maven-complete.adoc[]
Expand All @@ -36,56 +32,72 @@ to get a working Mill build for any existing Maven project.
The conversion

* handles deeply nested modules
* captures project metadata
* configures dependencies for scopes:
* captures publish settings
* configures dependencies for scopes
** compile
** provided
** runtime
** test
* configures testing frameworks:
* configures testing frameworks
** JUnit 4
** JUnit 5
** TestNG
* configures multiple, compile and test, resource directories

=== Command line arguments
.name of generated base module trait defining project metadata settings

The conversion and it's output (the generated Mill build files) can be customized using
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The conversion and it's output (the generated Mill build files) can be customized using
The conversion and its output (the generated Mill build files) can be customized using


* `--base-module` (`-b`): name of generated base module trait defining shared settings
[source,sh]
----
./mill init --base-module MyModule
----
.name of generated nested test module (defaults to `test`)

* `--jvm-id` (`-j`): version of xref:fundamentals/configuring-jvm-versions.adoc[custom JVM] to configure in `--base-module`
[source,sh]
----
./mill init --base-module MyModule --jvm-id 17
----

* `--test-module` (`-t`): name of generated nested test module (defaults to `test`)
[source,sh]
----
./mill init --test-module test
----
.name of generated companion object defining constants for dependencies

* `--deps-object` (`-d`): name of generated companion object defining dependency constants
[source,sh]
----
./mill init --deps-object Deps
----
.capture properties defined in `pom.xml` for publishing

* `--merge` (`-m`): merge build files generated for a multi-module build
[source,sh]
----
./mill init --publish-properties
./mill init --merge
----
.merge build files generated for a multi-module build

* `--publish-properties` (`-p`): capture properties defined in `pom.xml` for publishing
[source,sh]
----
./mill init --merge
./mill init --publish-properties
----

.use cache for Maven repository system
* `--cache-repository`: use cache for Maven repository system
[source,sh]
----
./mill init --cache-repository
----
.process Maven plugin executions and configurations

* `--process-plugins`: process Maven plugin executions and configurations
[source,sh]
----
./mill init --process-plugins
----

TIP: You can run `mill init` multiple times. It is recommended to run it first without any options.

=== Verified projects

The conversion has been tested with the following projects:
Expand Down Expand Up @@ -115,7 +127,7 @@ The conversion does not support

* build extensions
* build profiles
* non-Java (native) sources
* non-Java sources

Maven plugin support is limited to

Expand All @@ -134,22 +146,18 @@ These limitations can be overcome by:

== FAQ

.How to fix compilation errors in generated build files?
How to fix compilation errors in generated build files?

This could happen if a module and task name collision occurs. Either rename the module or enclose the name in backticks.

How to fix JPMS `module not found` compilation errors?

.How to fix JPMS `module not found` compilation errors?

Set https://github.com/tfesenko/Java-Modules-JPMS-CheatSheet#how-to-export-or-open-a-package[additional command line options]
for dependencies.

Add https://github.com/tfesenko/Java-Modules-JPMS-CheatSheet#how-to-export-or-open-a-package[options]
for Java modules to `javacOptions`.

.How to fix test compilation errors?
How to fix test compilation errors?

* The test framework configured may be for an unsupported version; try upgrading the
corresponding dependencies.
* Mill does not add `provided` dependencies to the transitive dependencies of the nested
test module; specify the dependencies again, in one of `ivyDeps`, `compileIvyDeps`, `runIvyDeps`, in the test module.


* Mill does not add `compileIvyDeps` dependencies to the transitive dependencies of the nested
test module; specify the dependencies again, in `ivyDeps` or `runIvyDeps`.
17 changes: 13 additions & 4 deletions docs/modules/ROOT/pages/migrating/migrating.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,19 @@
include::partial$gtag-config.adoc[]

This page documents a playbook for migrating existing projects
from Maven, Gradle, or SBT to Mill. Some build tools also have additional semi-automated
tooling (e.g. see xref:migrating/maven.adoc[]), other automation is work-in-process
(e.g. migrating from https://github.com/com-lihaoyi/mill/issues/3962[Gradle] or
https://github.com/com-lihaoyi/mill/issues/3450[SBT]) but while automation helps with
from Maven, Gradle, or SBT to Mill.

Some build tools also have additional semi-automated
tooling.

* xref:migrating/maven.adoc[]
* xref:migrating/gradle.adoc[]

Other automation is work-in-process.

* https://github.com/com-lihaoyi/mill/issues/3450[SBT]

While automation helps with
some of the scaffolding the general principles laid out on this page still apply.

== How Long Does Migration Take?
Expand Down
32 changes: 32 additions & 0 deletions example/javalib/migrating/3-gradle-complete/build.mill
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
/** Usage

> rm build.mill # remove any existing build file

> git init .
> git remote add -f origin https://github.com/komamitsu/fluency.git
> git checkout 2.7.3 # multi-module Java project that requires Java 16+

> ./mill init --base-module FluencyModule --jvm-id 16
converting Gradle build
writing Mill build file to fluency-aws-s3/package.mill
writing Mill build file to fluency-core/package.mill
writing Mill build file to fluency-fluentd-ext/package.mill
writing Mill build file to fluency-fluentd/package.mill
writing Mill build file to fluency-treasuredata/package.mill
writing Mill build file to build.mill
init completed, run "mill resolve _" to list available tasks

> ./mill __.compile
compiling 9 Java sources to ...out/fluency-aws-s3/compile.dest/classes ...
compiling 6 Java sources to ...out/fluency-aws-s3/test/compile.dest/classes ...
compiling 27 Java sources to ...out/fluency-core/compile.dest/classes ...
compiling 8 Java sources to ...out/fluency-core/test/compile.dest/classes ...

> ./mill fluency-core.test # running all tests takes too long
Test org.komamitsu.fluency.FluencyTest finished, ...
Test org.komamitsu.fluency.validation.ValidatableTest finished, ...
Test org.komamitsu.fluency.buffer.BufferTest finished, ...
Test org.komamitsu.fluency.buffer.BufferPoolTest finished, ...
Test org.komamitsu.fluency.flusher.FlusherTest finished, ...
Test org.komamitsu.fluency.recordformat.MessagePackRecordFormatterTest finished, ...
*/
Comment on lines +1 to +32
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these files be converted into READMEs since they are just commands and results in comments?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These commands are automatically used as integration tests and the (expected) output will be asserted.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Thanks for pointing out.

16 changes: 16 additions & 0 deletions example/javalib/migrating/4-gradle-incomplete/build.mill
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
/** Usage

> rm build.mill # remove any existing build file

> git init .
> git remote add -f origin https://github.com/mockito/mockito.git
> git checkout v5.15.2 # multi-module Java project that requires Java 17+

> ./mill init --base-module MockitoModule --jvm-id 17 # init ignores custom dependency configurations
converting Gradle build
ignoring errorprone dependency (com.google.errorprone,error_prone_core,2.23.0)
init completed, run "mill resolve _" to list available tasks

> ./mill __.compile # compilation error can be by fixed by using the Mill plugin for ErrorProne
error: ...Unexpected javac output: error: plug-in not found: ErrorProne
*/
22 changes: 22 additions & 0 deletions integration/feature/init/src/BuildGenTestSuite.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package mill.integration

import mill.testkit.{IntegrationTester, UtestIntegrationTestSuite}

abstract class BuildGenTestSuite extends UtestIntegrationTestSuite {

def initMessage(buildFileCount: Int): String =
s"generated $buildFileCount Mill build file(s)"

def integrationTest[T](githubSrcZipUrl: String)(f: IntegrationTester => T): T =
super.integrationTest { tester =>
println(s"downloading $githubSrcZipUrl")
val zipFile = os.temp(requests.get(githubSrcZipUrl))
val unzipDir = os.unzip(zipFile, os.temp.dir())
val sourceDir = os.list(unzipDir).head
// move fails on Windows, so copy
for (p <- os.list(sourceDir)) {
os.copy.into(p, tester.workspacePath, replaceExisting = true, createFolders = true)
}
f(tester)
}
}
Loading
Loading