ClassNotFoundException: com.jcraft.jzlib.Deflater when generating native image

[dsyer]

JZlib is not used or required by my app. Looks like maybe something Graal is trying to accesss?

$ native-image -H:Name=micro -H:ReflectionConfigurationFiles=netty_reflection_config.json --report-unsupported-elements-at-runtime -cp $CP com.example.micro.MicroApplication
Build on Server(pid: 24101, port: 33515)
   classlist:   1,877.39 ms
       (cap):   2,166.71 ms
       setup:   2,645.74 ms
    analysis:  20,473.22 ms
fatal error: java.lang.NoClassDefFoundError
	at jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.lookupType(HotSpotJVMCIRuntime.java:389)
	at jdk.vm.ci.hotspot.HotSpotUnresolvedJavaType.resolve(HotSpotUnresolvedJavaType.java:91)
	at com.oracle.graal.pointsto.meta.AnalysisField.<init>(AnalysisField.java:104)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.createField(AnalysisUniverse.java:357)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookupAllowUnresolved(AnalysisUniverse.java:345)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookup(AnalysisUniverse.java:312)
	at com.oracle.graal.pointsto.meta.AnalysisType.convertFields(AnalysisType.java:857)
	at com.oracle.graal.pointsto.meta.AnalysisType.convertInstanceFields(AnalysisType.java:850)
	at com.oracle.graal.pointsto.meta.AnalysisType.getInstanceFields(AnalysisType.java:841)
	at com.oracle.graal.pointsto.flow.context.object.AnalysisObject.getInstanceFieldTypeStore(AnalysisObject.java:209)
	at com.oracle.graal.pointsto.flow.context.object.AnalysisObject.getInstanceFieldFlow(AnalysisObject.java:199)
	at com.oracle.graal.pointsto.flow.StoreFieldTypeFlow$StoreInstanceFieldTypeFlow.onObservedUpdate(StoreFieldTypeFlow.java:160)
	at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:347)
	at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:389)
	at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:500)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:174)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.ClassNotFoundException: com.jcraft.jzlib.Deflater
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at jdk.vm.ci.hotspot.CompilerToVM.lookupType(Native Method)
	at jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.lookupType(HotSpotJVMCIRuntime.java:381)
	... 20 more
Error: Processing image build request failed

[tarsa]

AFAIK JSch has implicit dependency on JZlib, but since Java's class loading is lazy you won't get ClassNotFoundException on JVM if you don't trigger JZlib usage. Native-image tool in Graal doesn't load classes lazily, but eagerly and that's why it fails, I guess. Some stubbing is probably needed.

[cstancu]

@tarsa is right, however, when using --report-unsupported-elements-at-runtime the error should be pushed to runtime and only reported when that execution path is taken. This is a bug.

[dsyer]

I found where it came from: jzlib is an optional dependency of netty-codec, but not on my classpath. So it's surprising that other people seem to have been successful with native images and netty. That's probably enough to be able to reproduce the issue?

[cstancu]

Thank you for investigating further. I was able to reproduce this with a minimal test case, however it's a somewhat complicated corner case, a proper solution for it will take some work.

I guess your code reaches into some parts of netty that makes jzlib reachable. You probably use more of netty than we have tried before.

[dsyer]

Adding jzlib to the classpath fixes it. But that's a workaround really.

[tsachev]

@dsyer I am using this substitution to resolve this

@TargetClass(io.netty.handler.codec.compression.ZlibCodecFactory.class)
@SuppressWarnings({"unused"})
final class Target_io_netty_handler_codec_compression_ZlibCodecFactory {

    @Substitute
    public static ZlibEncoder newZlibEncoder(ZlibWrapper wrapper, int compressionLevel, int windowBits, int memLevel) {
        return new JdkZlibEncoder(wrapper, compressionLevel);
    }
}

Looking at original method here I think SVM thinks there are paths that will call this method with memLevel != 8. In my case with only netty and reactory-netty on classpath I think there is no such code?

[dsyer]

I believe that is correct. The app runs fine for me with no jzlib anyway.

[cstancu]

@tsachev your substitution looks correct. You might want to add a runtime check for the value of memLevel to make sure it is never the wrong/unexpected value. This issue is encountered during bytecode parsing and we don't do any flow sensitive analysis, i.e., we parse all control flow paths. In the later phases we do constant folding and can eliminate some paths.

[tonit]

Problem exists in release 1.0 RC5 too.
--report-unsupported-elements-at-runtime has no effect and reports errors at build time about (accepted because never used) missing classes in classpath.

[sdeleuze]

I guess we need to see if the condition can be updated on Netty side to be more strict or if Graal flow analysis could be improved to avoid this false positive. Any thoughts @normanmaurer?

[normanmaurer]

honestly I have no idea what you are asking for on the netty side... @sdeleuze @cstancu @dsyer can you clarify ?

[timboudreau]

I think the problem here is that jzlib is an optional dependency of Netty's codec, codec-http and codec-http2. You only need it on the classpath if you choose to use the jzlib encoders and decoders at runtime - but Graal is trying to compile classes that reference types from it.

In theory, if Graal does some "tree shaking" unused code elimination before attempting a compile, the JZlib classes should get shaken out, if they are not referenced whatsoever in the closure of the application entry point. How doable that is in practice is an open question - flow analysis has its limits.

The only Netty-side fix I can imagine would be to move the JZlib-based encoders and decoders to a separate project.

Workaround is simple: Include JZlib on the classpath. Not nice, since in most cases it will not be used, but does solve the problem.

[cstancu]

@timboudreau our analysis already shakes off a lot of unused code, however, as you point out, flow analysis has its limits and it sees the code that uses JZlib as reachable.

[cstancu]

This change set ddbf8fe should allow --report-unsupported-elements-at-runtime to correctly push the ClassNotFoundException exception to runtime for this specific situation. It eagerly triggers resolution of the declared type of fields during bytecode parsing so type resolution errors don't pop up later when it is to late for --report-unsupported-elements-at-runtime to intercept them.

[sdeleuze]

Thanks, I can confirm that Spring WebFlux + Reactor Netty compiles and run correctly with --report-unsupported-elements-at-runtime with master. While I was really puzzled to ask our users to add artificially an unused dependency, I am ok putting in our documentation the --report-unsupported-elements-at-runtime trick for now. So I am ok to close this issue on Graal side.

@normanmaurer Even if @cstancu workaround makes it less critical, if you want to improve Netty compatibilty with Graal, GraalVM now offers an official way to detect its presence via the org.graalvm.nativeimage.imagecode system property. We use that in Spring to adapt to Graal (see GraalDetector). You could maybe add similar capabilities to Netty PlatformDependent and leverage that to use JdkZlibEncoder, avoid using unsafe, etc. That's seems similar to what you are doing now depending on JDK versions and other parameters, and I think Netty + Graal native images are a pretty exciting combination. I can create the related Netty issue if you think that's an idea worth.

[normanmaurer]

@sdeleuze yes please and link this issue.

[timboudreau]

@cstancu is it possible that some small change in the way Netty uses the jzlib code would allow that code to be shaken out at image build time? Do we know the path that makes it determined to be reachable?

[cstancu]

The decision over which zlib encoder/decoder to use seems to be made in ZlibCodecFactory. The decision is made based on the Java version and the value of io.netty.noJdkZlibDecoder or io.netty.noJdkZlibEncoder properties. Look for example at ZlibCodecFactory.newZlibEncoder(int):

    public static ZlibEncoder newZlibEncoder(int compressionLevel) {
        if (PlatformDependent.javaVersion() < 7 || noJdkZlibEncoder) {
            return new JZlibEncoder(compressionLevel);
        } else {
            return new JdkZlibEncoder(compressionLevel);
        }
    }

With sufficient inlining by the end of the compilation we should be able to constant fold PlatformDependent.javaVersion() < 7 to false and, assuming noJdkZlibEncoder is also false, eliminate the if branch. However we currently determine during bytecode parsing that some JZLib classes cannot be found and we report that immeditaly (unless --report-unsupported-elements-at-runtime is set). We could delay that decision until later, after Graal has a chance to optimize the code and determine that the if branch is unreachable.

With --report-unsupported-elements-at-runtime we essentially replace the body of the first branch with a DeoptimizeNode that pushes the java.lang.NoClassDefFoundError to runtime, if that branch is ever taken. We could instead always add the DeoptimizeNode during bytecode parsing and if the DeoptimizeNode is still reachable after inlining and constant folding report the error, unless --report-unsupported-elements-at-runtime is set, in which case we push the error reporting to runtime like before.

I think this approach would fix this Netty use case. I cannot think of a small change in Netty that would currently work.

[cstancu]

I will close this issue since the original reported error is fixed with the current approach, i.e., --report-unsupported-elements-at-runtime does what it should.

[dsyer]

I'm not sure when this was "fixed". I'm using rc5 and still having the same problem. If you mark an issue "fixed" could you please consider adding a milestone or label to it?

[cstancu]

This fix will be available in rc6. We usually close issues once the fix is merged into master. We will look into tagging issues with the release version.

[ilopmar]

This is happening again using rc9. Steps to reproduce it:

• Clone this repo: https://github.com/micronaut-guides/micronaut-creating-first-graal-app
• cd micronaut-creating-first-graal-app/complete
• git checkout graal-rc9
• Use Java Graal-VM RC9
• Execute: ./build-native-image.sh
• I've tried adding the substitution suggested in this same issue ClassNotFoundException: com.jcraft.jzlib.Deflater when generating native image #494 (comment) and also adding the flag --report-unsupported-elements-at-runtime, but it still fails.
• Please note that using rc8 I didn't need neither the substitution nor the flag.

[complete:23479]    classlist:   4,409.65 ms
[complete:23479]        (cap):     852.78 ms
[complete:23479]        setup:   3,291.91 ms
Warning: class initialization of class io.netty.handler.ssl.util.BouncyCastleSelfSignedCertGenerator failed with exception java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider. This class will be initialized at runtime because option --report-unsupported-elements-at-runtime is used for image building. Use the option --delay-class-initialization-to-runtime=io.netty.handler.ssl.util.BouncyCastleSelfSignedCertGenerator to explicitly request delayed initialization of this class.
Warning: class initialization of class io.netty.handler.ssl.JdkNpnApplicationProtocolNegotiator failed with exception java.lang.ExceptionInInitializerError. This class will be initialized at runtime because option --report-unsupported-elements-at-runtime is used for image building. Use the option --delay-class-initialization-to-runtime=io.netty.handler.ssl.JdkNpnApplicationProtocolNegotiator to explicitly request delayed initialization of this class.
Warning: class initialization of class io.netty.handler.ssl.ReferenceCountedOpenSslEngine failed with exception java.lang.NoClassDefFoundError: io/netty/internal/tcnative/SSL. This class will be initialized at runtime because option --report-unsupported-elements-at-runtime is used for image building. Use the option --delay-class-initialization-to-runtime=io.netty.handler.ssl.ReferenceCountedOpenSslEngine to explicitly request delayed initialization of this class.
[complete:23479]     analysis:  48,994.96 ms
fatal error: java.lang.NoClassDefFoundError: Lcom/jcraft/jzlib/Inflater;
	at java.lang.Class.getDeclaredFields0(Native Method)
	at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
	at java.lang.Class.getDeclaredField(Class.java:2068)
	at jdk.vm.ci.hotspot.HotSpotResolvedJavaFieldImpl.toJava(HotSpotResolvedJavaFieldImpl.java:191)
	at jdk.vm.ci.hotspot.HotSpotResolvedJavaFieldImpl.getAnnotation(HotSpotResolvedJavaFieldImpl.java:179)
	at com.oracle.svm.hosted.SVMHost.platformSupported(SVMHost.java:150)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.createField(AnalysisUniverse.java:347)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookupAllowUnresolved(AnalysisUniverse.java:341)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookup(AnalysisUniverse.java:308)
	at com.oracle.graal.pointsto.meta.AnalysisType.convertFields(AnalysisType.java:875)
	at com.oracle.graal.pointsto.meta.AnalysisType.convertInstanceFields(AnalysisType.java:868)
	at com.oracle.graal.pointsto.meta.AnalysisType.getInstanceFields(AnalysisType.java:859)
	at com.oracle.graal.pointsto.flow.context.object.AnalysisObject.getInstanceFieldTypeStore(AnalysisObject.java:209)
	at com.oracle.graal.pointsto.flow.context.object.AnalysisObject.getInstanceFieldFlow(AnalysisObject.java:199)
	at com.oracle.graal.pointsto.flow.StoreFieldTypeFlow$StoreInstanceFieldTypeFlow.onObservedUpdate(StoreFieldTypeFlow.java:160)
	at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:347)
	at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:389)
	at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:508)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:174)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.ClassNotFoundException: com.jcraft.jzlib.Inflater
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 24 more
Error: Image building with exit status 1

[satb]

Running into the same issue with rc9 even with --report-unsupported-elements-at-runtime