-
-
Notifications
You must be signed in to change notification settings - Fork 523
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Simplistic multiprocessing.Pool replacement for parallel docbuild on older Cygwin #27490
Comments
Branch: u/embray/ticket-27490 |
Author: Erik Bray |
Commit: |
comment:1
Jeroen, you will likely have thoughts about this. Keep in mind, it's not meant to be at all robust--just a quick workaround so I don't have to spend too much more time on it. But if you have any thoughts on straightforward improvements to this I'm all for it. Obviously a better workaround, if it were possible, would be to use the much talked-about idea for generalizing the parallel processing loop from the Sage doctester. But since we don't have that yet this will do for now. New commits:
|
comment:2
I'm just wondering why you even bother with parallel docbuilding in the first place. The obvious solution is just using a single process. |
comment:3
Also, in general I don't like code of the form
If This is especially relevant when So I would like to know if there is a good reason to not use your "simplistic multiprocessing.Pool replacement" on all systems. |
comment:4
Some comments on the code:
In general, I like it. It's simple but the use case is sufficiently simple that we don't need anything more complicated. And it says a lot about |
comment:5
Thanks for having a look. Replying to @jdemeyer:
Yes, that should be much better.
+1
I thought it was pretty straightforward, but I guess the
I think there is: Or at least to try to SIGTERM first. Reason being this block can be reached if one process exits in error, while other processes are still working perfectly fine. You want to gracefully shut them down and ensure that their atexit handlers run, clean up temp files, etc.
I don't know that it says a lot. I don't think it actually works "better" on the whole, just in this one case. Keep in mind also that One downside to this approach is that there is no data returned from the child processes to the parent. So for example an exception raised in a worker cannot be re-raised from the parent. Instead I just raise a generic |
comment:6
Replying to @jdemeyer:
Partly for the reason I mentioned at the end of my previous comment, and partly just because I need this now and although I'm convinced it's robust-enough for my use it's still not well-tested. How about for now we special-case this, and then for the next release make it a priority to finally get at least an initial version of the doctest forker code released and replace it with that? |
comment:7
Replying to @jdemeyer:
At first I did just replace this with just plain The parallel version only took about 20 minutes to get working. |
comment:9
I addressed most of your comments, but I still have the big I'd still be open to just using it on all platforms, but I'm wary, given that this isn't battle-tested. |
comment:10
Replying to @embray:
I see your point. |
Reviewer: Jeroen Demeyer |
comment:11
I'm willing to give this the benefit of the doubt. You'll probably be the only user of this code anyway. I'm sure that there is room for improvement (I still don't like the |
comment:12
Thanks. However, I need to double back now since my last change broke something. It's not starting new processes up after previous ones finish. |
comment:21
Which "abomination"? I'm not necessarily going to do anything at your behest if you put it in such negative terms. |
comment:22
(Which is not not say I necessarily think this is pretty as-is but it's still not even clear exactly what you're asking to move). |
comment:23
Im asking you to move the parallel implementation of multiprocessing into a separate file, ideally with a small doctstring explaining what is going on here. |
comment:24
But both versions? Just the one I added? |
comment:25
At least for now just the one you added in the else branch. Thanks. |
comment:26
Okay, I'll do that. |
Branch pushed to git repo; I updated commit sha1. New commits:
|
Branch pushed to git repo; I updated commit sha1. New commits:
|
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
|
comment:30
(Just removed an unused import.) |
Changed branch from u/embray/ticket-27490 to |
comment:33
Followup at #27514 |
Changed commit from |
…regular-guess * u/dkrenn/sequences/rec-hash: (8211 commits) Updated SageMath version to 8.7 Updated SageMath version to 8.7.rc0 Trac sagemath#27490: Moved the alternate build_many implementation into a sage_setup.docbuild.utils module. Trac sagemath#27490: Further fixes in use of os.wait() Trac sagemath#27214: Patch GAP to allocate its memory pool using MAP_NORESERVE on Cygwin Trac sagemath#27490: Address some review comments and other cleanup: A little bit of import cleanup Trac sagemath#27490: Simplistic multiprocessing.Pool replacement for parallel docbuild on older Cygwin Fix alarm() test when cysignals was compiled with debugging Trac sagemath#27485: Use sdh_cmake in the spkg-install for primecount. Trac sagemath#27484: Add shd_cmake helper for running cmake with the correct flags for building Sage SPKGs. cysignals should be a normal dependency Upgrade to Cysignals 1.10.2 Upgrade to notebook-5.7.6 Trac sagemath#27461: Add abs tol on this test to account for minor numerical difference on Cygwin due to libm differences. Replacing None < infinity comparison with equivalent code. Some last little tidbits for uniformity. Removing some code duplication for __pth_root (changed to _pth_root_func). One more xderinv added. trac 27474: move some references to the master bibliography file. ...
…ular-warning * u/dkrenn/sequences/k-regular-guess: (8211 commits) Updated SageMath version to 8.7 Updated SageMath version to 8.7.rc0 Trac sagemath#27490: Moved the alternate build_many implementation into a sage_setup.docbuild.utils module. Trac sagemath#27490: Further fixes in use of os.wait() Trac sagemath#27214: Patch GAP to allocate its memory pool using MAP_NORESERVE on Cygwin Trac sagemath#27490: Address some review comments and other cleanup: A little bit of import cleanup Trac sagemath#27490: Simplistic multiprocessing.Pool replacement for parallel docbuild on older Cygwin Fix alarm() test when cysignals was compiled with debugging Trac sagemath#27485: Use sdh_cmake in the spkg-install for primecount. Trac sagemath#27484: Add shd_cmake helper for running cmake with the correct flags for building Sage SPKGs. cysignals should be a normal dependency Upgrade to Cysignals 1.10.2 Upgrade to notebook-5.7.6 Trac sagemath#27461: Add abs tol on this test to account for minor numerical difference on Cygwin due to libm differences. Replacing None < infinity comparison with equivalent code. Some last little tidbits for uniformity. Removing some code duplication for __pth_root (changed to _pth_root_func). One more xderinv added. trac 27474: move some references to the master bibliography file. ...
SageMath version 8.7, Release Date: 2019-03-23 * tag '8.7': (943 commits) Updated SageMath version to 8.7 Updated SageMath version to 8.7.rc0 Trac sagemath#27490: Moved the alternate build_many implementation into a sage_setup.docbuild.utils module. Trac sagemath#27490: Further fixes in use of os.wait() Trac sagemath#27214: Patch GAP to allocate its memory pool using MAP_NORESERVE on Cygwin Trac sagemath#27490: Address some review comments and other cleanup: A little bit of import cleanup Trac sagemath#27490: Simplistic multiprocessing.Pool replacement for parallel docbuild on older Cygwin Fix alarm() test when cysignals was compiled with debugging Trac sagemath#27485: Use sdh_cmake in the spkg-install for primecount. Trac sagemath#27484: Add shd_cmake helper for running cmake with the correct flags for building Sage SPKGs. cysignals should be a normal dependency Upgrade to Cysignals 1.10.2 Upgrade to notebook-5.7.6 Trac sagemath#27461: Add abs tol on this test to account for minor numerical difference on Cygwin due to libm differences. Replacing None < infinity comparison with equivalent code. Some last little tidbits for uniformity. Removing some code duplication for __pth_root (changed to _pth_root_func). One more xderinv added. trac 27474: move some references to the master bibliography file. ...
…ounded * u/dkrenn/k-regular-warning: (8211 commits) Updated SageMath version to 8.7 Updated SageMath version to 8.7.rc0 Trac sagemath#27490: Moved the alternate build_many implementation into a sage_setup.docbuild.utils module. Trac sagemath#27490: Further fixes in use of os.wait() Trac sagemath#27214: Patch GAP to allocate its memory pool using MAP_NORESERVE on Cygwin Trac sagemath#27490: Address some review comments and other cleanup: A little bit of import cleanup Trac sagemath#27490: Simplistic multiprocessing.Pool replacement for parallel docbuild on older Cygwin Fix alarm() test when cysignals was compiled with debugging Trac sagemath#27485: Use sdh_cmake in the spkg-install for primecount. Trac sagemath#27484: Add shd_cmake helper for running cmake with the correct flags for building Sage SPKGs. cysignals should be a normal dependency Upgrade to Cysignals 1.10.2 Upgrade to notebook-5.7.6 Trac sagemath#27461: Add abs tol on this test to account for minor numerical difference on Cygwin due to libm differences. Replacing None < infinity comparison with equivalent code. Some last little tidbits for uniformity. Removing some code duplication for __pth_root (changed to _pth_root_func). One more xderinv added. trac 27474: move some references to the master bibliography file. ...
For Cygwin versions less than 3.0.0 (and only Cygwin) this replaces the use of
multiprocessing.Pool
in thesage_setup.docbuild.build_many
function, with a naïve but "good enough" (it works in general) parallel process pool that does not rely on starting processes from threads.This is needed for #27214, because the specific combination of using
MAP_NORESERVE
mmap
s and forking processes from a thread can result in a bug in Cygwin (fixed in 3.0.0) which causes unhandled segfaults to occur in any code that is run during the docbuild which uses libgap.So this is really only needed so that the docs can continue to be built on systems (including my primary development environment, as well as the buildbot) that do not yet have Cygwin >= 3.0.0 once #27214 is applied.
CC: @jdemeyer
Component: porting: Cygwin
Author: Erik Bray
Branch:
fe0e3ea
Reviewer: Jeroen Demeyer
Issue created by migration from https://trac.sagemath.org/ticket/27490
The text was updated successfully, but these errors were encountered: