-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
joblib muliprocessing not working on > spyder 5? #16415
Comments
Hi @dkent-trimac, Can you please post here an example code that I can use to try to reproduce this issue? |
The code below reproduces the problem. The code runs fine on 5.0.3, but not on 5.1.4. For 5.1.4, it seems that passing in the sklearn object is causing the problem - if you uncomment ' # helper[0] = 'other string', then the code also works fine in 5.1.4
|
Works for me on Linux. |
Could it be just a Windows 10 problem then? |
@dalthviz, could you test @dkent-trimac example on Windows? |
These are some resources to deal with this problem: https://joblib.readthedocs.io/en/latest/auto_examples/serialization_and_wrappers.html |
Thanks - but am interested to see if you can replicate the problem on your side in Windows 10. |
@ccordoba12 I was able to run the code provided by @dkent-trimac . This is the output that I'm seeing: However for the test I used the Windows installer and an environment with
@dkent-trimac is there any difference between the packages I have in my testing env and your setup related with |
joblib and scikit-learn are the same. Are there any other I should check? Does your test run ok if you run the script within Spyder (5.1.4) as opposed to running a file from the command line? |
I don't think so but could you share with us the list of packages your using when the code fails? Also, maybe doing a diff of the packages list between the install with Spyder 5.0.3 and the one with Spyder 5.1.4 could give us some clues.
I'm running the script within Spyder with a file open and the @dkent-trimac let us know! |
I am running the code the same way as you - but I get the output below. (Happy to close this if we have done the doable here - thanks for having a look.)
|
Sending my list of packages in case that might be helpful.
|
Here is the diff (i.e. diff list1 list2)
|
You omitted this crucial part, which shows that this problem has nothing to do with Spyder. Please recreate your environment and install everything from conda-forge. The problem is conda-forge and defaults packages are binary incompatible, so you can't mix them (as you're doing right now). |
That worked - works find now. |
Issue Report Checklist
conda update spyder
(orpip
, if not using Anaconda)jupyter qtconsole
(if console-related)spyder --reset
Problem Description
I am using joblib for multiprocessing.
I get the following error with 5.1.3 and later: BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
I seem not to get the problem with 5.0.3.
What steps reproduce the problem?
What is the expected output? What do you see instead?
Both of the above should work. I obsrve today that only step #1 works.
Paste Traceback/Error Below (if applicable)
Versions
Dependencies
The text was updated successfully, but these errors were encountered: