Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JDK17 compatibility #3168

Closed
nmandrescu opened this issue May 31, 2021 · 33 comments
Closed

JDK17 compatibility #3168

nmandrescu opened this issue May 31, 2021 · 33 comments

Comments

@nmandrescu
Copy link

nmandrescu commented May 31, 2021

Problem

JEP 403 targeted for JDK 17 (release planned for Sept 2021) will remove --illegal-access flag. That will be equivalent to --illegal-access=deny.

Running with --illegal-access=warn apps using jackson-databind and other jackson modules reported here #2856 (comment) can discover issues like
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil

Workaround

For illegal reflective access pointing to jackson use --add-open for the package containing the restricted field.

Update
Not related to jackson library, since JDK 9 a number of standard replacements are provided for the most commonly used internal elements of the JDK.

@cowtowncoder
Copy link
Member

cowtowncoder commented May 31, 2021

Unfortunately most if not all remaining issues that surface with that rather non-descriptive warning are due to dynamic introspection for various JDK types and depend on actual usage. Jackson core itself does not force access to JDK, but to access fields, getters, setters, constructors, introspection is used.
ClassUtil specifically gets called to try to force access when it is deemed necessary based on types user tries to serialize/deserialize; it has no "agenda of its known".

So unfortunately this is not as easy as (for example) replacing use of sun.misc.Unsafe with alternatives.

So what tends to be necessary are user reports for specific usage that triggers these issues: this helps figure out how to prevent specific problems. I am not confident there is a general simple solution; although more and more general improvements can be developed when finding patterns common to reported problems.

@nmandrescu
Copy link
Author

@cowtowncoder, thank you for clarification, fully agree! I was about to update the description that this is not solvable by jackson itself.

A way to deal with it is to expose such illegally accessed packages with --add-opens.

@cowtowncoder
Copy link
Member

@nmandrescu Agreed! And this is one area that I really hope we can get improvements on, from better diagnostics to tooling perhaps. One thing that has been unfortunate is that from what I have seen, stack traces do not always seem to identify context in which problem occurs: while it is ClassUtil that tries to apply operation, it is the target class and (parts of) stack trace that really matter. I realize that it may not be something JDK can easily provide, but maybe there are settings that would let developers get more information.

@GedMarc
Copy link

GedMarc commented Jun 1, 2021

This is something that I am specifically looking at -

The problem is closer to usage, than implementation though, through devious attempts of resolution -

  1. Field Accessors to JDK Internals is banned across the board, this is correct according to the principle of encapsulation. This issue is only specific to referencing JDK Internal Value Objects, you must use Property fetch and Set. As value objects though, these don't bring back the final result, only whats necessary to calculate the final result -> LocalDateTime/OffsetDateTime best example here
    1,a) Our final result ended up in us simply registering a serializer and deserializer for each of the value objects in the JDK, and handling it properly by property, getters and setters
    1.b) The list of value objects in JDK is not exhaustive, setting all objects found in package java. to find by property only removes all invalid encapsulation requests

  2. Kotlin so-far does not seem to be adopting JDK 9 and up, and are following the JDK 8 pathway, going non-modular (even on higher jdk's making a move to JDK 11 very hard, JDK 17 is completely impossible for them with the block). This library would have the largest impact from forcing modularizaiton

  3. Version 3.0 is under way with Jakarta specific releases for anything I've found on that front, and once finalized will port across. Jakarta EE 9.1 is still not even JDK 9 compatible which is a huge hammer in a lot of people plans, and are so far away from being 11 compatible , that 16 or 17 right now for them seems out of reach for at least 2 years. (VERY ARRRRGGGGGHHHHH)

For now I think Jackson caters perfectly for modularization, and adheres to all encapsulation limitations placed - The user however must be aware of the new encapsulation rules (which are correct) and bend accordingly to them.

We can perhaps improve on the time-jsr a bit more to assist users move across properly, but right now, I am looking at the jakarta module split, and seeing what's happening in Version 3.

Outcome Analysis -
User Issue

@nmandrescu nmandrescu changed the title JDK17 compatibility for strongly encapsulate JDK internals JDK17 compatibility Jun 1, 2021
@cowtowncoder
Copy link
Member

Good overview @GedMarc. One specific area where we can probably make progress is to start preventing what must be illegal accesses into java.* types (and probably javax., if any remain).
And then as you pointed out trying to cover more of JDK types that formerly "happened to work", f.ex by using non-public single-String constructors, based on user reports (or active searching if feasible).

@nmandrescu
Copy link
Author

If it helps and possible to do anything about, here are some java.* samples:

WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field java.lang.reflect.Proxy.h
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field java.lang.ref.Reference.referent
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field java.lang.ref.Reference.queue
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field java.lang.ref.Reference.next

For javax.*

WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field javax.naming.InitialContext.myProps
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field javax.naming.InitialContext.defaultInitCtx
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:~/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.12.3/jackson-databind-2.12.3.jar) to field javax.naming.InitialContext.gotDefault

@cowtowncoder
Copy link
Member

Thank you @nmandrescu! It'd be great to know the path through which these came about (for reproduction), but this is helpful as-is.

@amCap1712
Copy link

amCap1712 commented Sep 2, 2021

Not sure if this is helpful but you can pass --illegal-access=debug on the command line to print a stack trace along with the message for each illegal access.

@JJBRT
Copy link

JJBRT commented Sep 20, 2021

Please take a look at my new article that explain how to export all modules to all modules at runtime in Java 16 and later without using any JVM parameter

@GedMarc
Copy link

GedMarc commented Sep 21, 2021

Look at the new frameworks for how write modular and encapsulated applications, something like guicedee.com, that works completely on proper encapsulation.
What is the point - best that you'll do is force oracle to lock it down even more, load up proxy classes in anonymous modules and force 'exports' for all packages because of this? you can't modify a module chain once in JRT!! it is a read-only vfs!

You don't "Solve" this problem with --add-exports and add-modules, or any other temporary work around in order to give people time to adapt, you Adapt to the shift in language by just doing it properly, so you can get all the benefits from it, which far out way and out perform any classpath-based application, by miles.

I apologize for the below tone but..... in order to get my facial expression across xD (in jest please)
No. Hard No,
Definitely. No.
I'm sorry - No.
insert meme here NO!

@AISCo
Copy link

AISCo commented Oct 25, 2021

Could these details help me in my specific case:
[ERROR] WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:/home/user/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.10.2/jackson-databind-2.10.2.jar) to constructor java.util.Optional() [ERROR] at com.fasterxml.jackson.databind.util.ClassUtil.checkAndFixAccess(ClassUtil.java:939) [ERROR] at com.fasterxml.jackson.databind.deser.impl.CreatorCollector._fixAccess(CreatorCollector.java:263) [ERROR] at com.fasterxml.jackson.databind.deser.impl.CreatorCollector.setDefaultCreator(CreatorCollector.java:123) [ERROR] at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addDeserializerConstructors(BasicDeserializerFactory.java:383) [ERROR] at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:283) [ERROR] at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:224) [ERROR] at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:220) [ERROR] at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) [ERROR] at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:443) [ERROR] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.createContextual(MapDeserializer.java:262) [ERROR] at com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:650) [ERROR] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:484) [ERROR] at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.resolve(SuperSonicBeanDeserializer.java:79) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) [ERROR] at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) [ERROR] at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:476) [ERROR] at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4389) [ERROR] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4198) [ERROR] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3205) [ERROR] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3173) [ERROR] at com.application.commun.echange.utils.JsonUtils.deserializerJSON(JsonUtils.java:53)

with this sample code:
public static <T> T deserializerJSON(final String jsonMessage, final Class<T> classObject) { try { final var mapper = new ObjectMapper(); mapper.registerModule(new AfterburnerModule()); mapper.configure(FAIL_ON_UNKNOWN_PROPERTIES, false); return mapper.readValue(jsonMessage, classObject); } catch (final IOException e) { throw new TechnicalException(e); } }

the error is on this line : return mapper.readValue(jsonMessage, classObject);

@sruffatti
Copy link

So do we support JDK 17 now?

@GedMarc
Copy link

GedMarc commented Nov 3, 2022

Yep - including full support for jpms

@nmandrescu
Copy link
Author

Thank you! We've used successfully several jackson-databind newer releases with JDK17. No more issues reported. Closing.

@Aksh125
Copy link

Aksh125 commented Nov 15, 2022

Can Anyone Help Me to resolved thw following error
I have migrated Spring Boot application from jdk 11 to jdk 17. I got the error java.lang.illegalAccessError for jackson Module afterburner

@JJBRT
Copy link

JJBRT commented Nov 15, 2022

Can Anyone Help Me to resolved thw following error I have migrated Spring Boot application from jdk 11 to jdk 17. I got the error java.lang.illegalAccessError for jackson Module afterburner

The fastest solution is to export all modules to all modules at runtime

@GedMarc
Copy link

GedMarc commented Nov 15, 2022

@Aksh125 please provide a little more detail

@Roberto-Gentili
Copy link

Roberto-Gentili commented Nov 15, 2022

@Aksh125 please provide a little more detail

Please stop down-voting working solutions even if they use some hack because they might be interpreted as broken solutions. And very often people don't have time to fix problems caused by new implementations of the jdk

@GedMarc
Copy link

GedMarc commented Nov 15, 2022

@Roberto-Gentilli As the developer for the modularization of the Jackson library and assisting with the usage of modules, unfortunately no - Giving the impression to say, new developers, that that process is acceptable is giving the wrong guiding and direction entirely -
To override all module links and dependencies in a JDK above JDK9, the solution is not valid - but I appreciate your constructive feedback :)

Reference : https://stackoverflow.com/questions/68867895/in-java-17-how-do-i-avoid-resorting-to-add-opens

@Roberto-Gentili
Copy link

Roberto-Gentili commented Nov 15, 2022

@Roberto-Gentilli As the developer for the modularization of the Jackson library and assisting with the usage of modules, unfortunately no - To override all module links and dependencies in a JDK above JDK9, the solution is not valid - but I appreciate your constructive feedback :)

You can tell people that there are better solutions, but if you vote against a solution it might appear that the solution doesn't work. I can assure that the solution proposed by @JJBRT works perfectly (except on Android) and I can also assure you that people's frustration when they see that their software no longer works with the new jdk implementation is often very high

@Aksh125
Copy link

Aksh125 commented Nov 15, 2022

@GedMarc @Roberto-Gentili when my spring boot application running on JDK 11 its show the warning
Warning: illegal reflective access by com.fasterxml.jackson.module.afterburner.util.myclassloader
And When I Updated to JDK 17 it get the error
java.lang.illegalAccessError
My Springboot version is 2.6.5

@GedMarc
Copy link

GedMarc commented Nov 15, 2022

@Aksh125 A stack trace please, and which version of Jackson are you using? :)

@GedMarc
Copy link

GedMarc commented Nov 15, 2022

@Aksh125 can you please log it against the afterburner project?

@Roberto-Gentili
Copy link

Roberto-Gentili commented Nov 15, 2022

@GedMarc @Roberto-Gentili when my spring boot application running on JDK 11 its show the warning Warning: illegal reflective access by com.fasterxml.jackson.module.afterburner.util.myclassloader And When I Updated to JDK 17 it get the error java.lang.illegalAccessError My Springboot version is 2.6.3

If you need a fast solution and you are not developing on Android you need only to add this to your pom:

<dependency>
    <groupId>org.burningwave</groupId>
    <artifactId>core</artifactId>
    <version>12.62.0</version>
</dependency>

... And then you need only to call the method org.burningwave.core.assembler.StaticComponentContainer.Modules.exportAllToAll() in a static block of your main class e.g.:

public class MainClass {

	static {
		org.burningwave.core.assembler.StaticComponentContainer.Modules.exportAllToAll();
	}
	
	public static void main(String[] args) {
		... Do the stuff
	}
}

if you see a lot of logs, they can be disabled

@GedMarc
Copy link

GedMarc commented Nov 15, 2022

and only if it's not enterprise, and if it is enterprise, only if the burningwave is on the approved dependencies list for that organisation...
and you are still circumventing standard Java JPMS so, from my perspective, not a solution

@Roberto-Gentili
Copy link

Roberto-Gentili commented Nov 15, 2022

and only if it's not enterprise, and if it is enterprise, only if the burningwave is on the approved dependencies list for that organisation... and you are still circumventing standard Java JPMS so, from my perspective, not a solution

if this is a solution to @Aksh125's problem, I'd let him decide
And if the jackson library is throwing these errors, it's doing non-jdk compliant operations, so some moralizing should be avoided

@GedMarc
Copy link

GedMarc commented Nov 15, 2022

Rather just learn how JPMS works and just fix it, Classpath mode is fading away - all of these -- hacks --- at one point will stop working, each Java major release adds another layer of modular security, and the benefit's of modularization far outweighs the push-back against it.
When looking for support, you want solutions, not hack-patches that will only work for x amount of time.

@Roberto-Gentili
Copy link

When looking for support, you want solutions, not hack-patches that will only work for x amount of time.

Ok, you made it clear, but let him decide which solution to adopt especially when your library is the first to use illegal reflection operations

@GedMarc
Copy link

GedMarc commented Nov 15, 2022

Only on private field-access properties - which is against strict-encapsulation, but we want to keep that option available for backwards compatibility......
But everything can be resolved, in a proper, viable and operational way, without breaking or abusing the internals of Java to get "JDK8"-like functionality back in JDK's that are modular by definition,

It could just be as simple as an opens clause in their module-info file, or proxying the classes referenced from afterburner....

@Vishal-0319
Copy link

Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @1d8b0500

Getting the above error while upgrading from jdk 11 to 17. Could you please help here?

@pjfanning
Copy link
Member

When calling Java command, you can add an add-opens option. See https://docs.oracle.com/javase/9/migrate/toc.htm#JSMIG-GUID-7BB28E4D-99B3-4078-BDC4-FC24180CE82B

@cowtowncoder
Copy link
Member

@Vishal-0319 that error message would not be enough to help, fwtw, as it does not show which class is being problematic (or reproduction to figure it out). You also did not include Jackson version. Going forward it'd make sense to add more information as well as file a separate issue.

@Vishal-0319
Copy link

I will try out and get back with the info if it doesnt workout, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests