WIP: [Java] Require Java 22 to build Arrow #43050

laurentgo · 2024-06-25T19:01:34Z

** This is work in progress **

Rationale for this change

This change demonstrates how to modernize Java toolchain to build Arrow and open opportunities to introduce support for some of the recent Java new features while maintaining compatibility with Java 11.

Java tools used by our build system (like spotless, google-java-format, error-prone, etc.) have dropped support for Java 8 and may do so as well with Java 11 causing the build system to use unsupported versions.

The lack of recent java versions in our toolchains also means that adopting recent Java features (through mechanisms like multi-release jars for example) is currently not possible.

What changes are included in this PR?

This change makes Java 22 the minimum version required to build Arrow while Java 11 remains the minimum Java version required to use Arrow libraries and tools.

For the purpose of demonstrating the end result, this change actually captures multiple changes:

~~Compile arrow code base with --release flag, allowing true Java 11 compilation independently of the JDK version used by the build~~
Add support for cross version testing: Introduce a property arrow.test.jdk-version which accepts a JDK version to run unit and integration tests with (the JDK needs to be declared in Maven toolchains first), which is different from the JDK version used to build Arrow
Finally make Java 22 the minimum version required to build Arrow and update CI builds accordingly (there are several image changes needed to support Java 22. This change also introduces a new multi-JDK docker image)

Are these changes tested?

Those changes have been tested via github actions as much as possible. I haven't tested with crossbow yet/

Are there any user-facing changes?

No direct user-facing changes but developers will be required to update their Java toolchain.

github-actions · 2024-06-25T19:02:00Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

danepitkin

Thanks, Laurent! This looks like a nice bit of cleanup. It'll make it easier to drop Java 8 in the future as well.

danepitkin · 2024-06-26T20:27:12Z

ci/docker/java.dockerfile

+# specific language governing permissions and limitations
+# under the License.
+
+FROM maven:3.9.6-eclipse-temurin-22


Is it possible to parameterize the version numbers in a central location?

Of course, it should be possible :)

danepitkin · 2024-06-26T20:28:43Z

ci/docker/java.dockerfile

+env JAVA11_HOME /opt/java/openjdk11
+env JAVA17_HOME /opt/java/openjdk17
+env JAVA21_HOME /opt/java/openjdk21
+env JAVA22_HOME /opt/java/openjdk


Maybe we call this JAVA_LATEST?

There is a convention about using JAVAX_HOME where X represents the Java version. This convention is used by Maven JDK toolchains discovery mechanism. At the same time, JAVA_HOME already points to the latest JDK version.

For those reasons, I'm not sure we need JAVA_LATEST?

Ah, I agree if there is already a convention to follow.

danepitkin · 2024-06-26T20:35:35Z

@github-actions crossbow submit -g java

github-actions · 2024-06-26T20:37:57Z

Revision: e31e655

Submitted crossbow builds: ursacomputing/crossbow @ actions-7d97f6c645

Task	Status
java-jars
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.11-spark-master
test-conda-python-3.8-spark-v3.5.0
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

laurentgo · 2024-06-26T21:02:43Z

Looks like crossbow tests are failing (which is expected since I didn't try to run them and modified all of the actions).

I will try and see if I can modify, but I am also trying to gauge the interest for this work, and if yes, if I should decomposing this work into sub issues

lidavidm · 2024-06-26T23:47:21Z

So we'd use Java 21, but we'd still be limited to Java 8 features and dependencies, other than maybe things like being able to upgrade ErrorProne now that we don't build with Java 8 - is that right?

laurentgo · 2024-06-26T23:58:21Z

So we'd use Java 21, but we'd still be limited to Java 8 features and dependencies, other than maybe things like being able to upgrade ErrorProne now that we don't build with Java 8 - is that right?

Not necessarily. By leveraging MRJAR feature, you could create src/main/java21 directory and instruct the maven-compiler-plugin to use Java21 to build that code and get it packaged inside the jar under META-INF/versions/M/.... Using this mechanism you can now have two implementations of the same class, one for Java 8 up to Java 20, and one for java 21 and higher (the right one being picked up by the JVM automatically)

We could start with module-info.java which are Java9+ files for example...

lidavidm · 2024-06-27T00:10:07Z

Ah, I see.

@danepitkin the plan was to drop Java 8 for the 18.0.0 release, right? Or was it 19.0.0?

danepitkin · 2024-06-27T05:29:06Z

The current plan is to drop in v18. It would be nice to get some help on this though.

lidavidm · 2024-06-27T07:07:08Z

I will try and see if I can modify, but I am also trying to gauge the interest for this work, and if yes, if I should decomposing this work into sub issues

I think this work is valuable and I am at least OK with requiring Java 21 for building. But if we are going to drop JDK8 support entirely soon-ish then it sounds like there's less to be gained from this work.

laurentgo · 2024-06-27T15:53:21Z

There are still some long term gains:

Some plugins are already considering dropping Java 11 support as the official OpenJDK end of life date is October 2024 (while Java 8 is supported until November 2026...)
I believe a couple of recent Java features could be of great interest to the Arrow project:

Foreign Function and Memory API: FFI could provide a better solution than JNI for some of adapters and Gandiva. The new memory API could provide a better memory allocator than the current Netty one
Vector API: Could possibly help with vectorized operations

But to be able to experiment/introduce support for those new API, a prerequisite is still to have a modern toolchain

lidavidm · 2024-07-01T00:38:52Z

Yes, I think this is still helpful - but dropping Java 8 may simplify this?

laurentgo · 2024-07-02T15:24:02Z

Yes. it would not be necessary to refactor sun.misc.Unsafe access although it would be still worth it to remove it as a public field from MemoryUtil IMHO

Change miminum java build version to 22. This doesn't change the minimum version to use Arrow which is still Java 11 Update docker images for java jni/conda integration tests to use recent python image in order to install java 22

laurentgo · 2024-07-29T22:38:33Z

I just rebased on top of current master and remove already merged code and the changes to MemoryUtil. Will update the description as well

lidavidm · 2024-07-30T00:00:51Z

docs/source/developers/java/building.rst

@@ -32,8 +32,12 @@ Arrow Java uses the `Maven <https://maven.apache.org/>`_ build system.

 Building requires:

+* JDK 22+


is it 21 or 22? (preferably we would at least build with the latest LTS?)

Would like to test with 22, one of the latest Java features I'd like see Arrow adopt is JEP 454 - Foreign Function and Memory API which was released as part of Java 22

Testing with 22 is fine, but I'd like the minimum version to at least be an LTS

Is it possible to have an optionally-enabled submodule for FFM experiments?

Yes, it is possible but what about the release process? wouldn't that require Java 22?

delaying any (optional) use of FFM/FFI to Java 25 so not before september 2025.

hm, I see your point. The main issue I see is when we need to re-build the binaries either for a patch release or other reasons, if we originally release with a non lts version and this version e.g. 22 is no longer available additional changes will be required that potentially have other issues that need to be fixed (e.g. dependencies). vs. just using the lts that will continue to be available for much longer.

Though this is extrapolated from previous patch/backport experience and not actually based on java, so maybe my fears are unfounded?

As far as docker images or adoptium/temurin availability of non-LTS version. it seems older versions are still available. It is to be the same for the Github setup-java action. The issue may be homebrew though

I'm OK packaging with 22, I just want to leave it open for people to build with 21 (or ideally, even 17) if needed, minus any FFM code.

Ok. So for now I think I will pause on this change and will wait until I'm ready to propose a change requiring the use of a newer Java version

@laurentgo what would be the expected changes with this PR, if we break it down to the list of proposed changes?
(Removing UNSAFE, etc)

danepitkin · 2024-08-13T14:48:49Z

@github-actions crossbow submit test-debian-12-docs

github-actions · 2024-08-13T14:50:04Z

Unable to match any tasks for `test-debian-12-docs`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/10372038877

lidavidm · 2024-10-08T01:07:14Z

It seems error-prone is requiring Java 17 for building (can still target Java 11) so we may want to consider this again if other tools are going to do the same thing

laurentgo requested review from assignUser, kou, raulcd and lidavidm as code owners June 25, 2024 19:01

github-actions bot added Component: Java Component: Documentation awaiting review Awaiting review labels Jun 25, 2024

danepitkin reviewed Jun 26, 2024

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jun 26, 2024

laurentgo mentioned this pull request Jul 16, 2024

GH-38051: [Java] Remove Java 8 support #43139

Merged

Make Java 22 minimum version to build Arrow

9bfdd59

Change miminum java build version to 22. This doesn't change the minimum version to use Arrow which is still Java 11 Update docker images for java jni/conda integration tests to use recent python image in order to install java 22

laurentgo force-pushed the laurentgo/wip-update-java-version branch from e31e655 to 9bfdd59 Compare July 29, 2024 22:37

laurentgo requested a review from jonkeane as a code owner July 29, 2024 22:37

lidavidm reviewed Jul 30, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jul 30, 2024

assignUser marked this pull request as draft July 30, 2024 01:31

laurentgo changed the title ~~WIP: [Java] Require Java 21 to build Arrow~~ WIP: [Java] Require Java 22 to build Arrow Jul 30, 2024

lidavidm mentioned this pull request Oct 8, 2024

MINOR: [Java] Bump error_prone_core.version from 2.31.0 to 2.33.0 in /java #44327

Closed

vibhatha mentioned this pull request Jan 15, 2025

Memory-Access Methods in sun.misc.Unsafe are deprecated for removal apache/arrow-java#511

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: [Java] Require Java 22 to build Arrow #43050

WIP: [Java] Require Java 22 to build Arrow #43050

laurentgo commented Jun 25, 2024 •

edited

Loading

github-actions bot commented Jun 25, 2024

danepitkin left a comment

danepitkin Jun 26, 2024

laurentgo Jun 26, 2024

danepitkin Jun 26, 2024

laurentgo Jun 26, 2024

danepitkin Jul 2, 2024

danepitkin commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

laurentgo commented Jun 26, 2024

lidavidm commented Jun 26, 2024

laurentgo commented Jun 26, 2024 •

edited

Loading

lidavidm commented Jun 27, 2024

danepitkin commented Jun 27, 2024

lidavidm commented Jun 27, 2024

laurentgo commented Jun 27, 2024

lidavidm commented Jul 1, 2024

laurentgo commented Jul 2, 2024

laurentgo commented Jul 29, 2024

lidavidm Jul 30, 2024

laurentgo Jul 30, 2024

lidavidm Jul 30, 2024

lidavidm Jul 30, 2024

laurentgo Jul 30, 2024

assignUser Jul 30, 2024 •

edited

Loading

laurentgo Jul 30, 2024

lidavidm Jul 31, 2024

laurentgo Jul 31, 2024

vibhatha Aug 26, 2024

danepitkin commented Aug 13, 2024

github-actions bot commented Aug 13, 2024

lidavidm commented Oct 8, 2024

		@@ -32,8 +32,12 @@ Arrow Java uses the `Maven <https://maven.apache.org/>`_ build system.

		Building requires:

		* JDK 22+

WIP: [Java] Require Java 22 to build Arrow #43050

Are you sure you want to change the base?

WIP: [Java] Require Java 22 to build Arrow #43050

Conversation

laurentgo commented Jun 25, 2024 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Jun 25, 2024

danepitkin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danepitkin commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

laurentgo commented Jun 26, 2024

lidavidm commented Jun 26, 2024

laurentgo commented Jun 26, 2024 • edited Loading

lidavidm commented Jun 27, 2024

danepitkin commented Jun 27, 2024

lidavidm commented Jun 27, 2024

laurentgo commented Jun 27, 2024

lidavidm commented Jul 1, 2024

laurentgo commented Jul 2, 2024

laurentgo commented Jul 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

assignUser Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danepitkin commented Aug 13, 2024

github-actions bot commented Aug 13, 2024

lidavidm commented Oct 8, 2024

laurentgo commented Jun 25, 2024 •

edited

Loading

laurentgo commented Jun 26, 2024 •

edited

Loading

assignUser Jul 30, 2024 •

edited

Loading