Java packages starting with io can conflict with Python's io module #29
Comments
|
Also I just tested the jep console, where the import statements work without any problems... |
|
I also just tested to set the LD_PRELOAD variable using (Again I am able to import and find Yet this would be wrong syntax in python. If I use the correct syntax: EDIT: |
|
Hi, you've made good progress on this. Have you checked the LD_LIBRARY_PATH environment variable? Besides LD_PRELOAD, the jep console script sets LD_LIBRARY_PATH. It looks like maybe you have more than one version of python on your machine? |
|
Which path goes into the LD_LIBRARY_PATH? |
|
Look at the jep script that the build produced, it should be in there. Typically it needs to point at the python/lib dir, wherever libpython2.7.so is. |
|
Alright, thanks for the quick reply. I tried it by setting: However I still get the same error. (Also used -Djava.library.path with no result) |
|
Just to make sure I'm following this correctly.
The jep console should just about be identical to a java application. Maybe try explicitly setting PATH, it still seems like it might be picking up a different python. With LD_PRELOAD, LD_LIBRARY_PATH, and PATH explicitly set there's not much chance it can get the wrong python. If that doesn't work, do you know if there's another python installed on that system? And are you using virtualenv or not? This is quite perplexing. |
|
That's correct. However I checked. Here are the environment vars: However I still get the same error. It works in the python console as well as the jep console, however fails in code. The actual code is (where io.BufferedIOBase is the object from the import error thrown by nltk): jep.eval("from io import BufferedIOBase")However the following code evaluates correctly: jep.eval("import io") |
|
If you look at the jep script, the jep console is a simple java program that is emulating the python interpreter. So it should behave very similarly to running any java application. I'm down to two theories:
For theory 1, have you tried importing nltk through the jep console? I'd take it a step further and use the jep console and import just like your script, from gensim.models import doc2vec
import nltk
import string
import osI suspect the import of nltk will fail and the BufferedIOBase is a misleading message. Glancing at nltk's For theory 2, I would recommend printing out various things on the |
|
Here is the output form the jep console: Here is the output from the java application: The only difference I see is the Also: |
|
I don't know what the |
|
Alright, after reading some documentation it seems like the _ Variable is the path to the execution context (therefore the jep file for the jep console and java for my application). |
|
It looks like the io module is significantly different between 2.6 and 2.7. Do you have a python 2.6 on the system that the Java application could be erroneously picking up? |
|
I printed out sys.version in the Java application, which says 2.7.3. Also in both cases sys.executable relates to /usr/bin/python |
|
Since you are able to import io but not BufferedIOBase, you may be getting the wrong io module. It would be interesting to see what is in the io module that you are importing with the output of something like this(from the java application): import io
print(dir(io))Im guessing that BufferedIOBase will not be included in the output and perhaps whatever else is in the module or not will give us a clue as to where it is coming from. |
|
Alright. The dir(io) from the jep console is: While the one from the java application is: |
|
That looks promising, can you print(io.__file__)in both. |
|
Okay. That seems to be pretty close to the issue. More interestingly the java applications gets the correct path for the string package: |
|
hook.py could potentially create an io module. For example, from java.io import FileThat will create a module java and a module io with the module io attached as an attribute to module java. But in sys.modules it will be java.io, not just io.... Are you passing a |
|
Hmm. My creation code for JEP looks like this: Try(new Jep()) match {
case Success(jep) => Right(new PythonInterop(jep))
case Failure(e) => Left(e)
}Where PythonInterop is just a class for error handling. |
|
You are getting into the import mechanism which is an area I am less familiar with, so I'm making some assumptions reading through code here. It looks like the ClassList inspects your classpath and allows imports of any class in any jar in your class path. If any jar has a package in it that starts with 'io' then jep will let you import it.(I think just 'io', doesn't even need to be 'io.'). Can you look at your classpath and see if anything starts with io. From java the list of jars that jep is using should be in System.getProperty("java.class.path") |
|
I explicitly set the classpath at the start of the application, which results in: |
|
For the current version of Jep I recommend creating a custom ClassEnquirer. For this particular case you could do something as simple as extending ClassList and overriding the contains method to return false when the package is 'io'. I think we should leave this issue open and in a future version of jep we should look into making it easier to avoid conflicts with python and java(scala?) naming. Combining the 2 namespaces is a difficult problem, and I'm not sure we will be able to get all the kinks worked out with the default Enquirer, especially when you throw scala names in the mix. Can you explain where your 'io' package comes from so we can understand the scope of the problem. I don't know anything about scala, but a quick google reveals that scala has an 'io' package. At the java level it looks like the package is 'scala.io' which shouldn't have problems but Im not sure if different scala implementations might be handling this differently. |
|
First of all thanks for the great support! |
|
You will need to call a more detailed constructor. The default arguments are (false, null, null, null), so you can use that and just change the last argument to a custom enquirer. |
|
I traced down why the error message was so unhelpful. Is there a way to e.printStacktrace() with scala? And if so can you verify it has all that information? |
|
Scala can use the same function as Java (it just has some neat additions, like real functional programming ;) ), so here you go: So looks like the same ClassNotFound Exception. |
|
I'm going to look into improving the quality of the error message a bit for jep 3.4. For 3.5 I was planning on reworking ClassList and ClassEnquirer somewhat, so I'll try and make the defaults smart enough to handle this scenario for that release. |
provided ClassEnquirer implementations will not identify io and re as Java packages Change-Id: Ic5f95d16bfa5def27e0fd0095a26b3928406829f
|
Fixed on dev_3.5 branch. Java packages starting with "io" or "re" will be removed by the two provided ClassEnquirer implementations, |
|
I did not test the fix well enough. Working on a more complete fix and unit test. |
Change-Id: I60e83e95933e2f4f8623229d1f0108c8ca4f9e7a
|
Fixed on dev_3.6. |
Hey,
I am using Jep 3.4.1 (installed with
python setup.py install) on a server running python 2.7. I am calling Jep from Scala code. I am able to instanciate python, however I get the following error when I try to load a script:jep.JepException: <type 'exceptions.ImportError'>: /usr/lib/python2.7/lib-dynload/_multiprocessing.so: undefined symbol: _Py_ZeroStructThe script itself includes the following code:
The script works if I load it in python in the console. I made some experiments and the script runs fine, if I delete the import statements. Any ideas, why that might be?
Greetings
The text was updated successfully, but these errors were encountered: