Improve GraalVM/Mandrel version detection #34161

gsmet · 2023-06-20T12:36:18Z

At the moment, we are parsing the output of native-image --version to determine the GraalVM/Mandrel version. This is brittle and needs to be adjusted.

~~Note that we need to adjust things before the next minor version of GraalVM because the new format introduced in 23.0 forces the 23.0 version, which won't be true when 23.1 is released.~~ @jerboaa corrected me on this, my mistake.

In the GraalVM distribution, there is a release file at the root containing some properties with versions:

IMPLEMENTOR="Oracle Corporation"
JAVA_VERSION="17.0.7"
JAVA_VERSION_DATE="2023-04-18"
LIBC="gnu"
MODULES="java.base java.logging java.management jdk.internal.vm.ci java.compiler java.instrument jdk.jfr jdk.unsupported org.graalvm.sdk java.datatransfer java.xml java.prefs java.desktop java.transaction.xa java.sql jdk.management org.graalvm.truffle jdk.internal.vm.compiler com.oracle.graal.graal_enterprise com.oracle.svm.extraimage_enterprise java.security.sasl java.naming java.rmi java.management.rmi java.net.http java.scripting java.security.jgss java.sql.rowset java.xml.crypto java.se java.smartcardio jdk.accessibility jdk.internal.jvmstat jdk.attach jdk.charsets jdk.compiler jdk.crypto.ec jdk.crypto.cryptoki jdk.dynalink jdk.internal.ed jdk.editpad jdk.hotspot.agent jdk.httpserver jdk.incubator.foreign jdk.incubator.vector jdk.internal.le jdk.internal.opt jdk.internal.vm.compiler.management jdk.internal.vm.compiler.truffle.jfr jdk.jartool jdk.javadoc jdk.jcmd jdk.management.agent jdk.jconsole jdk.jdeps jdk.jdwp.agent jdk.jdi jdk.jlink jdk.jpackage jdk.jshell jdk.jsobject jdk.jstatd jdk.localedata jdk.management.jfr jdk.naming.dns jdk.naming.rmi jdk.net jdk.nio.mapmode jdk.random jdk.sctp jdk.security.auth jdk.security.jgss jdk.unsupported.desktop jdk.xml.dom jdk.zipfs org.graalvm.extraimage.builder org.graalvm.js.scriptengine org.graalvm.locator"
OS_ARCH="x86_64"
OS_NAME="Linux"
SOURCE=".:git:8ed58a0062e6 open:git:64e1e8a15e62 labsjdk-builder:6fdba7be9f6310a40fce95618f0a2195d0cbf183 compiler:f9ef31f0c872872f7008ae771540f3020223414c graal-enterprise:4d30240e899ca78e72314883702155c0ffd28030 graal-js:940f9a35184ed498c66f156bd9f7adc99a3cec97 graal-microservices:4d30240e899ca78e72314883702155c0ffd28030 java-benchmarks:f9ef31f0c872872f7008ae771540f3020223414c regex:f9ef31f0c872872f7008ae771540f3020223414c sdk:f9ef31f0c872872f7008ae771540f3020223414c substratevm:f9ef31f0c872872f7008ae771540f3020223414c substratevm-enterprise:4d30240e899ca78e72314883702155c0ffd28030 substratevm-enterprise-gcs:c55803e351323c2e16a4689700cb152afbe114eb tools:f9ef31f0c872872f7008ae771540f3020223414c tools-enterprise:4d30240e899ca78e72314883702155c0ffd28030 truffle:f9ef31f0c872872f7008ae771540f3020223414c vm:f9ef31f0c872872f7008ae771540f3020223414c vm-enterprise:4d30240e899ca78e72314883702155c0ffd28030"
GRAALVM_VERSION="23.0.0"
COMMIT_INFO={"compiler": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090907, "commit.rev": "f9ef31f0c872872f7008ae771540f3020223414c"}, "graal-enterprise": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090953, "commit.rev": "4d30240e899ca78e72314883702155c0ffd28030"}, "graal-js": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090643, "commit.rev": "940f9a35184ed498c66f156bd9f7adc99a3cec97"}, "graal-microservices": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090953, "commit.rev": "4d30240e899ca78e72314883702155c0ffd28030"}, "java-benchmarks": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090907, "commit.rev": "f9ef31f0c872872f7008ae771540f3020223414c"}, "regex": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090907, "commit.rev": "f9ef31f0c872872f7008ae771540f3020223414c"}, "sdk": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090907, "commit.rev": "f9ef31f0c872872f7008ae771540f3020223414c"}, "substratevm": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090907, "commit.rev": "f9ef31f0c872872f7008ae771540f3020223414c"}, "substratevm-enterprise": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090953, "commit.rev": "4d30240e899ca78e72314883702155c0ffd28030"}, "substratevm-enterprise-gcs": {"commit.committer": "Christian Haeubl <christian.haeubl@oracle.com>", "commit.committer-ts": 1674227214, "commit.rev": "c55803e351323c2e16a4689700cb152afbe114eb"}, "tools": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090907, "commit.rev": "f9ef31f0c872872f7008ae771540f3020223414c"}, "tools-enterprise": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090953, "commit.rev": "4d30240e899ca78e72314883702155c0ffd28030"}, "truffle": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090907, "commit.rev": "f9ef31f0c872872f7008ae771540f3020223414c"}, "vm": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090907, "commit.rev": "f9ef31f0c872872f7008ae771540f3020223414c"}, "vm-enterprise": {"commit.committer": "Gilles Duboscq <gilles.m.duboscq@oracle.com>", "commit.committer-ts": 1686090953, "commit.rev": "4d30240e899ca78e72314883702155c0ffd28030"}}
component_catalog="rest://gds.oracle.com/api/20220101/|uln://linux-update.oracle.com/rpc/api/?linux=generic_linux_x86_64_graalvm_jdk17&macos=macos_64_graalvm_jdk17&windows=windows_64_graalvm_jdk17|https://www.graalvm.org/component-catalog/v2/graal-updater-ee-component-catalog-java17.properties"

What is unsure is if they will continue to populate the GRAALVM_VERSION property now that they switched to using the JDK version scheme but I suppose we won't know that before the next GraalVM release.

@zakkak @Karm @jerboaa @galderz could any of you add a comment with the Mandrel output for it? Also do you have any idea how it will evolve in the future and if it's a good idea to use this file?

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2023-06-20T12:36:21Z

/cc @Karm (mandrel), @galderz (mandrel), @zakkak (mandrel)

jerboaa · 2023-06-20T12:48:21Z

@gsmet GraalVM is a jlinked JDK. It adds modules to the standard set of OpenJDK. We've been pointed at looking at the release file for GraalVM as you note, but that doesn't work for Mandrel. Mandrel's release file is the one from OpenJDK verbatim. Mandrel is not a jlinked JDK distro. It merely adds jars to the OpenJDK install. Thus, looking at the release file of Mandrel tells you nothing about the GraalVM code level.

I believe it's true that the GraalVM team plans to continue to populate the GRAALVM_VERSION env var in release.

jerboaa · 2023-06-20T12:51:26Z

Note that we need to adjust things before the next minor version of GraalVM because the new format introduced in 23.0 forces the 23.0 version, which won't be true when 23.1 is released.

@gsmet Could you clarify this statement? How is the 23.0 introduced version format forcing us to the 23.0 release?

gsmet · 2023-06-20T13:01:05Z

Actually, I misread one test and missed you were extracting 23.0 from build 20+34-jvmci-23.0-b10. So it works but it looks relatively brittle given they don't really advertise this version.

As for Mandrel not populating release, could you add information there or have a specific mandrel.release file that we could use?

Using the native-image --version is very brittle and the parsing has become significantly more complex and fragile so I think we should start the process of trying to simplify it. And having a file with the version looks like an interesting solution.

jerboaa · 2023-06-20T13:58:04Z

Actually, I misread one test and missed you were extracting 23.0 from build 20+34-jvmci-23.0-b10. So it works but it looks relatively brittle given they don't really advertise this version.

Yes, agreed. We had a long discussion and that was the only token available to us at this point (other than release).

As for Mandrel not populating release, could you add information there or have a specific mandrel.release file that we could use?

mandrel.release is acceptable even though it would complicate our release workflow a little. We cannot touch release as that file is owned by OpenJDK.

Using the native-image --version is very brittle and the parsing has become significantly more complex and fragile so I think we should start the process of trying to simplify it. And having a file with the version looks like an interesting solution.

Keep in mind that by using the release file or some such, you'd have to know where it is. Especially in the container context that's more finicky. I'd really like to hear input from @zakkak and @Karm as well.

Note that we've discussed this at the time here:
graalvm/mandrel#491

tqvarnst · 2023-06-20T16:25:41Z

We could put a hard requirement that for custom build images, they should provide either a release or a *.release inside $GRAALVM_HOME and that GRAALVM_HOME env needs to be set to the install location of GraalVM, Mandrel, or other native compile distribution.

zakkak · 2023-06-23T08:20:27Z

I'd really like to hear input from @zakkak and @Karm as well.

I am OK with introducing a mandrel.release file to mandrel and requiring it to be under GRAALVM_HOME as @tqvarnst suggests.

Note that AFAIU GraalVM's end goal is to completely drop the YY.X (e.g. 23.0) versioning at some point, meaning that even if we get it from the release file it's not guaranteed it will stay there in future releases and at some point we might end up needing to look at the JDK version only (reading the release file should still work in that case).

jerboaa · 2023-06-23T08:44:48Z

Note that AFAIU GraalVM's end goal is to completely drop the YY.X (e.g. 23.0) versioning at some point, meaning that even if we get it from the release file it's not guaranteed it will stay there in future releases

This seems a long way out, though.

Karm · 2023-09-06T09:25:05Z

My 2c: I would like us to keep parsing what native-image gives you as that is the executable (script) used to run the build. Having something in a text file in GRAALVM_HOME is nice, but it might not belong to the native-image executed.
e.g. you can have some native-image on your PATH and that would be driving the build:

https://github.com/quarkusio/quarkus/blob/main/core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildStep.java#L502

Looking for a text file in new File(pathDir.getParent(), "mandrel.release") seems rather adventurous to me.

Temporarily works around quarkusio#36246, till we have a consensus on how to move forward in quarkusio#34161 Closes quarkusio#36246

gsmet · 2023-10-04T09:47:04Z

I thought a bit more about it and my proposal would be to completely drop the old versioning for new versions.

E.g. in our Version object with a short versioningScheme. The old one would be 0, the new one would be 1 (and we could have another one later if they change the versioning again).

When comparing the versions, we would first compare versioningScheme.

And we would switch the the new GraalVM advertised version as the version instead of relying on the old scheme that is legacy and will go away at some point.

Now I don't know what's your plan for Mandrel and if this would work too :).

jerboaa · 2023-10-04T09:59:23Z

For GraalVM for JDK 21 (internal version 23.1) and Mandrel 23.1 we can explore going with release/mandrel.release files. Mandrel 23.1 release should have a mandrel.release file. That said, it'll be another bet that the internal version GRAALVM_VERSION in that file will stay there. If it does go away later, we are back to square one.

zakkak · 2023-10-04T10:32:12Z

I like @gsmet's proposal. This way we avoid depending on files (which has the issue @Karm mentions), and we align with upstream GraalVM.

Now I don't know what's your plan for Mandrel and if this would work too :).

The main issue I see with this approach is for Mandrel releases based on the same source level but on different JDKs (something that upstream GraalVM won't do but we have not crossed out it yet for Mandrel). E.g. Mandrel 24.0.x could potentially have two builds, one "for JDK 21" and another "for JDK 22", in that case we would need to resolve to the internal version (24.0) to distinguish it from 23.1 which is also "for JDK 21" and will be using the same 21 version since it will also get a CPU update.

The --version output of the releases described above could look like:

native-image 21.0.2 2024-01-23
OpenJDK Runtime Environment Mandrel-23.1.1.0-Final (build 21.0.2+5)
OpenJDK 64-Bit Server VM Mandrel-23.1.1.0-Final (build 21.0.2+5, mixed mode)

from

native-image 21.0.2 2024-01-23
OpenJDK Runtime Environment Mandrel-24.0.0.0-Final (build 21.0.2+5)
OpenJDK 64-Bit Server VM Mandrel-24.0.0.0-Final (build 21.0.2+5, mixed mode)

jerboaa · 2023-10-04T12:04:14Z

I like @gsmet's proposal. This way we avoid depending on files (which has the issue @Karm mentions), and we align with upstream GraalVM.

Sorry, I must be missing something :( Why does the classification of versioning schemes free us from knowing what the version actually is? My understanding was that the proposal would treat the single line versioning as schema 0 (or some such) and the 3-line output as schema 1. Yet, we need to figure out what the version is and what it means irrespective of the schema in place. I'm confused... Why would we not depend on files? How would we determine whether or not it's version scheme X if the native-image --version output is only telling us the (base) JDK version?

zakkak · 2023-10-04T12:21:01Z

The suggestion is to keep doing what we do for schema 0, but if schema is 1 we should use the new GraalVM versioning scheme and not try to figure out the internal/old version of GraalVM/Mandrel.

IIRC the three line version scheme appeared with the introduction of the "GraalVM for JDK XX" versioning scheme, thus if we detect the three line output the suggestion is to just compare the JDK versions.

Now to compare across schemas, it's as simple as schemaA < schemaB.

You would essentially have a comparesTo starting with the schema comparison, and only if the schema is the same, proceed to compare the version (which for one scheme will be the JDK version and for the other the Graal version)

So the version object in quarkus would be something like:

{schema = 0, version = 22.3.0.1} for

native-image 22.3.0.1-Final Mandrel Distribution (Java Version 17.0.5+8)

{schema = 1, version = 17.0.8+7.1} for

native-image 17.0.8 2023-07-18
GraalVM Runtime Environment GraalVM CE 17.0.8+7.1 (build 17.0.8+7-jvmci-23.0-b15)
Substrate VM GraalVM CE 17.0.8+7.1 (build 17.0.8+7, serial gc)

{schema = 1, version = 20.0.2+9.1} for

native-image 20.0.2 2023-07-18
GraalVM Runtime Environment GraalVM CE 20.0.2+9.1 (build 20.0.2+9-jvmci-23.0-b15)
Substrate VM GraalVM CE 20.0.2+9.1 (build 20.0.2+9, serial gc)

{schema = 1, version = 21+35} for

native-image 21 2023-09-19
GraalVM Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
Substrate VM GraalVM CE 21+35.1 (build 21+35, serial gc)

Now the issue with that is that {schema = 1, version = 20.0.2+9.1} and {schema = 1, version = 17.0.8+7.1} are actually the same GraalVM version, which however is supposed to be an one time thing for upstream GraalVM and it won't happen again. I am not sure we can say the same for Mandrel though.

jerboaa · 2023-10-04T12:25:48Z

I see. Thanks for clarifying.

Temporarily works around quarkusio#36246, till we have a consensus on how to move forward in quarkusio#34161 Closes quarkusio#36246

zakkak · 2024-10-04T08:02:04Z

It has been a while and we haven't made any progress on that, mainly because there was no clear winner (even the schema approach has its drawback). It turns out the work around in #36267 is doing its job just fine. This in conjunction with the fact that with the 25 release the internal and JDK version will get aligned I am considering closing this as "not planned".

To give a glimpse into the future, the expected versions to come are:

GraalVM version	JDK version	Internal version	Mandrel version
24	24	24.2	24.2.0.0
25	25	25	25.0.0.0
26	26	26	26.0.0.0
27	27	27	27.0.0.0

Note that there is a (small) chance to release non-LTS Mandrel releases (e.g. 26) built on the LTS JDK, so we might end up getting 26 and 27 built with JDK 25 but this should be easier to handle given we already parse the Java version as well.

WDYT?

jerboaa · 2024-10-04T08:33:07Z

Sounds good to me. FWIW, the last row should read 27 everywhere. Not 26 in the first column.

quarkus-bot bot added area/mandrel env/windows Impacts Windows machines labels Jun 20, 2023

gsmet mentioned this issue Jun 21, 2023

Locally built GraalVM (with local built JDK) fails to build Quarkus Native with NPE #34191

Closed

zakkak mentioned this issue Jul 17, 2023

Generate a mandrel.release file in Mandrel's distributions graalvm/mandrel-packaging#353

Closed

This was referenced Sep 7, 2023

[GR-48470] Remove extra version component. oracle/graal#7366

Open

[23.1] GraalVM depends on org.graalvm.home.Version which is polyglot API #35714

Closed

zakkak mentioned this issue Oct 2, 2023

Quarkus main fails to parse GraalVM version starting with GraalVM for JDK 22 #36246

Closed

zakkak added a commit to zakkak/quarkus that referenced this issue Oct 3, 2023

Work around missing graalvm internal version starting with JDK 22

13d5d5b

Temporarily works around quarkusio#36246, till we have a consensus on how to move forward in quarkusio#34161 Closes quarkusio#36246

zakkak added a commit to zakkak/quarkus that referenced this issue Oct 3, 2023

Work around missing graalvm internal version starting with JDK 22

092b6f5

Temporarily works around quarkusio#36246, till we have a consensus on how to move forward in quarkusio#34161 Closes quarkusio#36246

zakkak mentioned this issue Oct 3, 2023

Work around missing GraalVM internal version starting with JDK 22 #36267

Merged

zakkak added a commit to zakkak/quarkus that referenced this issue Oct 4, 2023

Work around missing graalvm internal version starting with JDK 22

dbc1962

Temporarily works around quarkusio#36246, till we have a consensus on how to move forward in quarkusio#34161 Closes quarkusio#36246

zakkak mentioned this issue Nov 23, 2023

[error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to get GraalVM version #37272

Closed

zakkak closed this as not planned Won't fix, can't repro, duplicate, stale Jan 27, 2025

zakkak mentioned this issue Jan 27, 2025

Quarkus fails to properly detect the patch part of internal versions of GraalVM CE builds #45873

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve GraalVM/Mandrel version detection #34161

Improve GraalVM/Mandrel version detection #34161

gsmet commented Jun 20, 2023 •

edited

Loading

quarkus-bot bot commented Jun 20, 2023

jerboaa commented Jun 20, 2023

jerboaa commented Jun 20, 2023

gsmet commented Jun 20, 2023

jerboaa commented Jun 20, 2023

tqvarnst commented Jun 20, 2023

zakkak commented Jun 23, 2023

jerboaa commented Jun 23, 2023

Karm commented Sep 6, 2023

gsmet commented Oct 4, 2023

jerboaa commented Oct 4, 2023

zakkak commented Oct 4, 2023

jerboaa commented Oct 4, 2023

zakkak commented Oct 4, 2023 •

edited

Loading

jerboaa commented Oct 4, 2023

zakkak commented Oct 4, 2024 •

edited

Loading

jerboaa commented Oct 4, 2024

Improve GraalVM/Mandrel version detection #34161

Improve GraalVM/Mandrel version detection #34161

Comments

gsmet commented Jun 20, 2023 • edited Loading

quarkus-bot bot commented Jun 20, 2023

jerboaa commented Jun 20, 2023

jerboaa commented Jun 20, 2023

gsmet commented Jun 20, 2023

jerboaa commented Jun 20, 2023

tqvarnst commented Jun 20, 2023

zakkak commented Jun 23, 2023

jerboaa commented Jun 23, 2023

Karm commented Sep 6, 2023

gsmet commented Oct 4, 2023

jerboaa commented Oct 4, 2023

zakkak commented Oct 4, 2023

jerboaa commented Oct 4, 2023

zakkak commented Oct 4, 2023 • edited Loading

jerboaa commented Oct 4, 2023

zakkak commented Oct 4, 2024 • edited Loading

jerboaa commented Oct 4, 2024

gsmet commented Jun 20, 2023 •

edited

Loading

zakkak commented Oct 4, 2023 •

edited

Loading

zakkak commented Oct 4, 2024 •

edited

Loading