Native libraries for multiple Operating Systems

[maziyarpanahi]

Hi,

I would like to see the best practice to have native libraries for PyTorch engine on the same device (CPU) for multiple operating systems in one project offline. For instance, if I wish to offer a PyTorch engine to all osx, linux, and windows users in one SBT project (Fat JAR, I am not worried about the size), what would be the best way to do this since there is no Maven package by DJL for this specific purpose.

As an example, tensorflow-java has option for each specific operating system/device, however it also has an option to automatically include all versions of the native library for a given configuration. For example, the tensorflow-core-platform for all supported OSes on CPU, tensorflow-core-platform-mkl for all supported OSes on CPU with MKL, tensorflow-core-platform-gpu for all supported OSes on GPU, etc.

Since there is no option for this, I have to include all supported pytorch-native-cpu with multiple classifiers in the SBT project. DJL PyTorch doesn't work properly in this situation when you have all the 3 osx, win, and linux CPU dependencies in the same project: No deep learning engine found.

What would be the best way to have a package that supports all 3 operating systems on the CPU without using auto for offline usage given these are included inside the Fat JAR (and not in some default cache folder)

Seq(
    "ai.djl.pytorch" % "pytorch-engine" % "0.14.0",
    "ai.djl.pytorch" % "pytorch-native-cpu" % "1.9.1" classifier "osx-x86_64" classifier "linux-x86_64" classifier "win-x86_64"
  )

/jnilib
├── linux-x86_64
│   ├── cpu
│   │   └── libdjl_torch.so
│   ├── cpu-precxx11
│   │   └── libdjl_torch.so
│   ├── cu102
│   │   └── libdjl_torch.so
│   └── cu111
│       └── libdjl_torch.so
├── osx-x86_64
│   └── cpu
│       └── libdjl_torch.dylib
└── win-x86_64
    ├── cpu
    │   └── djl_torch.dll
    ├── cu102
    │   └── djl_torch.dll
    └── cu111
        └── djl_torch.dll

PS: I have read the discussions/issues regarding offline usage of native libraries, but none of them asked about multiple targets. They all knew what was the actual target (like windows on CPU).

[frankfliu]

@maziyarpanahi
You can include all classifiers in your fat jar. DJL will pickup correct pytorch native library based on the operating system. But you have to handle the duplicate files when you creating the fat jar. Each pytorch-native jar file contains a pytorch.properties files which has its metadata in it. You need make sure all the pytorch.properties files are included in your fatjar. See this issue for more information: #940

Here is an example how gradle handle it:

djl/engines/mxnet/jnarator/build.gradle

Line 28 in 5ffc373

duplicatesStrategy = DuplicatesStrategy.INCLUDE

[maziyarpanahi]

Hi @frankfliu

Many thanks for your quick response and sorry for the delay. I was making sure I try everything before reporting back here. I have successfully made it work both as a maven package and a Fat JAR, however, it worked only on my own operating system (osx). When I use the same Fat JAR on Linux I get the following error:

Caused by: java.lang.IllegalStateException: Your PyTorch native library jar does not match your operating system. Make sure the Maven Dependency Classifier matches your system type.

Is this because of the pytorch.properties inside native/lib? After setting the merge strategy to .contact to merge them all together it looks like this now:

MergeStrategy:

case x if x.contains("pytorch.properties") => MergeStrategy.concat

pytorch.properties:

version=1.9.1-20211025
classifier=cpu-win-x86_64
libraries=asmjit.dll,c10.dll,fbgemm.dll,libiomp5md.dll,torch.dll,torch_cpu.dll
version=1.9.1-20211025
classifier=cpu-linux-x86_64
libraries=libc10.so,libgomp-52f2fd74.so.1,libtorch.so,libtorch_cpu.so
version=1.9.1-20211025
classifier=cpu-osx-x86_64
libraries=libc10.dylib,libiomp5.dylib,libtorch.dylib,libtorch_cpu.dylib

• Does this error come from this file? (pytorch.properties)
• Is it because of the format after being concatenated in the wrong format? (what should be the right format to have all the classifier and libraries correctly listed in the same file?)

Many thanks again @frankfliu

[frankfliu]

@maziyarpanahi
MergeStrategy.concat won't work for DJL,

We expect multiple pytorch.properties files in the jar. jar file allows duplicated files in the same jar, you probably need to use
MergeStrategy.deduplicate

[maziyarpanahi]

Thanks @frankfliu, appreciate the help.

Unfortunately, the deduplicate in sbt-assembly will verify that all candidates have the same contents and error out otherwise. Which in this case will error out with contents are different.

Just to be clear so we can start finding ways to do this by other means, are we looking for a situation where there are 3 pytorch.properties files inside native/lib where all the libraries are? Or are we looking for having 3 duplicated native/lib/ directories each with their own files? (native directory is currently in the root of the Fat JAR)

native
└── lib
    ├── asmjit.dll
    ├── c10.dll
    ├── fbgemm.dll
    ├── libc10.dylib
    ├── libc10.so
    ├── libgomp-52f2fd74.so.1
    ├── libiomp5.dylib
    ├── libiomp5md.dll
    ├── libtorch.dylib
    ├── libtorch.so
    ├── libtorch_cpu.dylib
    ├── libtorch_cpu.so
    ├── pytorch.properties
    ├── torch.dll
    └── torch_cpu.dll

[frankfliu]

@maziyarpanahi

Now i see the problem. We are scanning all pytorch.properties files in the jar, so duplicated pytorch.properties is expected.
However, if you put all native jars into single jar file, we won't be able to figure out which native/lib/libtorch.so file is for linux and which is for mac.

It looks like we won't be able to support fatjar for linux and mac together.
What we can do is add classifier folder for each native jar file in future: native/lib/linux-x86_64/libtorch.so

[maziyarpanahi]

That'd be great! Any way that makes it possible to have multiple OSes for 1 device in one JAR would be a great help. Unfortunately, Apache Spark (spark-shell and spark-submit) doesn't support classifier in SBT packages. It will entirely skip it, so we have to improvise and make another package that already dealt with classifiers and have everything in 1 place.

Many thanks again @frankfliu for the help and looking forward to a solution for 1 JAR and multiple OSes.

[saudet]

TensorFlow for Java uses the same mechanism as used by the JavaCPP Presets, so it is possible to do this wih the JavaCPP Presets for PyTorch. I have offered DJL my help to package their support for PyTorch in the same way, but they are not interested, and I am not sure why. @frankfliu Could you please elaborate on the reasons why DJL isn't interested in reusing functionality from JavaCPP that is known to work well? I would be happy to fix whatever is wrong with JavaCPP to get cooperation going.

[maziyarpanahi]

Hi @frankfliu

Is there any ETA on adding classifier folders for each native jar file based on OS like: native/lib/linux-x86_64/libtorch.so? Is there anything I can do to help?

[saudet]

We've been talking about using JavaCPP for that. If you have some time to look into this, it would help if you could take a look at that to have a better understanding of any hurdles that need to be overcome.

[maziyarpanahi]

Hi @saudet
Is this similar to how tensorflow-core for instance is using JavaCPP? (I am not familiar with JavaCPP but I can look at how it is used somewhere else)

[saudet]

Yes, it could be exactly like that, but from what I understand DJL needs a few things that JavaCPP's loader doesn't already do. I think the way to go about it would be to add whatever is required to JavaCPP, but we first need someone to look into this to identify the missing pieces.