Skip to content

Commit

Permalink
pythongh-84559: Remove the new multiprocessing warning, too disruptiv…
Browse files Browse the repository at this point in the history
…e. (python#101551)

This reverts the core of python#100618 while leaving relevant documentation
improvements and minor refactorings in place.
  • Loading branch information
gpshead authored Feb 3, 2023
1 parent f6c53b8 commit d4c410f
Show file tree
Hide file tree
Showing 11 changed files with 27 additions and 189 deletions.
16 changes: 8 additions & 8 deletions Doc/library/concurrent.futures.rst
Original file line number Diff line number Diff line change
Expand Up @@ -281,18 +281,18 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.

Added the *initializer* and *initargs* arguments.

.. note::
The default :mod:`multiprocessing` start method
(see :ref:`multiprocessing-start-methods`) will change away from
*fork* in Python 3.14. Code that requires *fork* be used for their
:class:`ProcessPoolExecutor` should explicitly specify that by
passing a ``mp_context=multiprocessing.get_context("fork")``
parameter.

.. versionchanged:: 3.11
The *max_tasks_per_child* argument was added to allow users to
control the lifetime of workers in the pool.

.. versionchanged:: 3.12
The implicit use of the :mod:`multiprocessing` *fork* start method as a
platform default (see :ref:`multiprocessing-start-methods`) now raises a
:exc:`DeprecationWarning`. The default will change in Python 3.14.
Code that requires *fork* should explicitly specify that when creating
their :class:`ProcessPoolExecutor` by passing a
``mp_context=multiprocessing.get_context('fork')`` parameter.

.. _processpoolexecutor-example:

ProcessPoolExecutor Example
Expand Down
11 changes: 6 additions & 5 deletions Doc/library/multiprocessing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,11 @@ to start a process. These *start methods* are

Available on POSIX systems. Currently the default on POSIX except macOS.

.. note::
The default start method will change away from *fork* in Python 3.14.
Code that requires *fork* should explicitly specify that via
:func:`get_context` or :func:`set_start_method`.

*forkserver*
When the program starts and selects the *forkserver* start method,
a server process is spawned. From then on, whenever a new process
Expand All @@ -138,11 +143,6 @@ to start a process. These *start methods* are
Available on POSIX platforms which support passing file descriptors
over Unix pipes such as Linux.

.. versionchanged:: 3.12
Implicit use of the *fork* start method as the default now raises a
:exc:`DeprecationWarning`. Code that requires it should explicitly
specify *fork* via :func:`get_context` or :func:`set_start_method`.
The default will change away from *fork* in 3.14.

.. versionchanged:: 3.8

Expand Down Expand Up @@ -1107,6 +1107,7 @@ Miscellaneous
launched (before creating a :class:`Pool` or starting a :class:`Process`).

Only meaningful when using the ``'forkserver'`` start method.
See :ref:`multiprocessing-start-methods`.

.. versionadded:: 3.4

Expand Down
16 changes: 7 additions & 9 deletions Doc/whatsnew/3.12.rst
Original file line number Diff line number Diff line change
Expand Up @@ -440,12 +440,6 @@ Deprecated
warning at compile time. This field will be removed in Python 3.14.
(Contributed by Ramvikrams and Kumar Aditya in :gh:`101193`. PEP by Ken Jin.)

* Use of the implicit default ``'fork'`` start method for
:mod:`multiprocessing` and :class:`concurrent.futures.ProcessPoolExecutor`
now emits a :exc:`DeprecationWarning` on Linux and other non-macOS POSIX
systems. Avoid this by explicitly specifying a start method.
See :ref:`multiprocessing-start-methods`.

Pending Removal in Python 3.13
------------------------------

Expand Down Expand Up @@ -510,9 +504,13 @@ Pending Removal in Python 3.14
* Testing the truth value of an :class:`xml.etree.ElementTree.Element`
is deprecated and will raise an exception in Python 3.14.

* The default :mod:`multiprocessing` start method will change to one of either
``'forkserver'`` or ``'spawn'`` on all platforms for which ``'fork'`` remains
the default per :gh:`84559`.
* The default :mod:`multiprocessing` start method will change to a safer one on
Linux, BSDs, and other non-macOS POSIX platforms where ``'fork'`` is currently
the default (:gh:`84559`). Adding a runtime warning about this was deemed too
disruptive as the majority of code is not expected to care. Use the
:func:`~multiprocessing.get_context` or
:func:`~multiprocessing.set_start_method` APIs to explicitly specify when
your code *requires* ``'fork'``. See :ref:`multiprocessing-start-methods`.

Pending Removal in Future Versions
----------------------------------
Expand Down
17 changes: 0 additions & 17 deletions Lib/concurrent/futures/process.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@
import itertools
import sys
from traceback import format_exception
import warnings


_threads_wakeups = weakref.WeakKeyDictionary()
Expand Down Expand Up @@ -651,22 +650,6 @@ def __init__(self, max_workers=None, mp_context=None,
mp_context = mp.get_context("spawn")
else:
mp_context = mp.get_context()
if (mp_context.get_start_method() == "fork" and
mp_context == mp.context._default_context._default_context):
warnings.warn(
"The default multiprocessing start method will change "
"away from 'fork' in Python >= 3.14, per GH-84559. "
"ProcessPoolExecutor uses multiprocessing. "
"If your application requires the 'fork' multiprocessing "
"start method, explicitly specify that by passing a "
"mp_context= parameter. "
"The safest start method is 'spawn'.",
category=mp.context.DefaultForkDeprecationWarning,
stacklevel=2,
)
# Avoid the equivalent warning from multiprocessing itself via
# a non-default fork context.
mp_context = mp.get_context("fork")
self._mp_context = mp_context

# https://github.com/python/cpython/issues/90622
Expand Down
28 changes: 1 addition & 27 deletions Lib/multiprocessing/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,6 @@ class TimeoutError(ProcessError):
class AuthenticationError(ProcessError):
pass

class DefaultForkDeprecationWarning(DeprecationWarning):
pass

#
# Base type for contexts. Bound methods of an instance of this type are included in __all__ of __init__.py
#
Expand Down Expand Up @@ -284,23 +281,6 @@ def _Popen(process_obj):
from .popen_fork import Popen
return Popen(process_obj)

_warn_package_prefixes = (os.path.dirname(__file__),)

class _DeprecatedForkProcess(ForkProcess):
@classmethod
def _Popen(cls, process_obj):
import warnings
warnings.warn(
"The default multiprocessing start method will change "
"away from 'fork' in Python >= 3.14, per GH-84559. "
"Use multiprocessing.get_context(X) or .set_start_method(X) to "
"explicitly specify it when your application requires 'fork'. "
"The safest start method is 'spawn'.",
category=DefaultForkDeprecationWarning,
skip_file_prefixes=_warn_package_prefixes,
)
return super()._Popen(process_obj)

class SpawnProcess(process.BaseProcess):
_start_method = 'spawn'
@staticmethod
Expand All @@ -324,9 +304,6 @@ class ForkContext(BaseContext):
_name = 'fork'
Process = ForkProcess

class _DefaultForkContext(ForkContext):
Process = _DeprecatedForkProcess

class SpawnContext(BaseContext):
_name = 'spawn'
Process = SpawnProcess
Expand All @@ -342,16 +319,13 @@ def _check_available(self):
'fork': ForkContext(),
'spawn': SpawnContext(),
'forkserver': ForkServerContext(),
# Remove None and _DefaultForkContext() when changing the default
# in 3.14 for https://github.com/python/cpython/issues/84559.
None: _DefaultForkContext(),
}
if sys.platform == 'darwin':
# bpo-33725: running arbitrary code after fork() is no longer reliable
# on macOS since macOS 10.14 (Mojave). Use spawn by default instead.
_default_context = DefaultContext(_concrete_contexts['spawn'])
else:
_default_context = DefaultContext(_concrete_contexts[None])
_default_context = DefaultContext(_concrete_contexts['fork'])

else:

Expand Down
5 changes: 2 additions & 3 deletions Lib/test/_test_multiprocessing.py
Original file line number Diff line number Diff line change
Expand Up @@ -4098,10 +4098,9 @@ def test_shared_memory_SharedMemoryServer_ignores_sigint(self):
def test_shared_memory_SharedMemoryManager_reuses_resource_tracker(self):
# bpo-36867: test that a SharedMemoryManager uses the
# same resource_tracker process as its parent.
cmd = f'''if 1:
cmd = '''if 1:
from multiprocessing.managers import SharedMemoryManager
from multiprocessing import set_start_method
set_start_method({multiprocessing.get_start_method()!r})
smm = SharedMemoryManager()
smm.start()
Expand Down
19 changes: 0 additions & 19 deletions Lib/test/test_concurrent_futures.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
import threading
import time
import unittest
import warnings
import weakref
from pickle import PicklingError

Expand Down Expand Up @@ -572,24 +571,6 @@ def test_shutdown_no_wait(self):
assert all([r == abs(v) for r, v in zip(res, range(-5, 5))])


@unittest.skipIf(mp.get_all_start_methods()[0] != "fork", "non-fork default.")
class ProcessPoolExecutorDefaultForkWarning(unittest.TestCase):
def test_fork_default_warns(self):
with self.assertWarns(mp.context.DefaultForkDeprecationWarning):
with futures.ProcessPoolExecutor(2):
pass

def test_explicit_fork_does_not_warn(self):
with warnings.catch_warnings(record=True) as ws:
warnings.simplefilter("ignore")
warnings.filterwarnings(
'always', category=mp.context.DefaultForkDeprecationWarning)
ctx = mp.get_context("fork") # Non-default fork context.
with futures.ProcessPoolExecutor(2, mp_context=ctx):
pass
self.assertEqual(len(ws), 0, msg=[str(x) for x in ws])


create_executor_tests(ProcessPoolShutdownTest,
executor_mixins=(ProcessPoolForkMixin,
ProcessPoolForkserverMixin,
Expand Down
5 changes: 2 additions & 3 deletions Lib/test/test_logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -4759,9 +4759,8 @@ def test_multiprocessing(self):
# In other processes, processName is correct when multiprocessing in imported,
# but it is (incorrectly) defaulted to 'MainProcess' otherwise (bpo-38762).
import multiprocessing
mp = multiprocessing.get_context('spawn')
parent_conn, child_conn = mp.Pipe()
p = mp.Process(
parent_conn, child_conn = multiprocessing.Pipe()
p = multiprocessing.Process(
target=self._extract_logrecord_process_name,
args=(2, LOG_MULTI_PROCESSING, child_conn,)
)
Expand Down
85 changes: 0 additions & 85 deletions Lib/test/test_multiprocessing_defaults.py

This file was deleted.

3 changes: 1 addition & 2 deletions Lib/test/test_re.py
Original file line number Diff line number Diff line change
Expand Up @@ -2431,8 +2431,7 @@ def test_regression_gh94675(self):
input_js = '''a(function() {
///////////////////////////////////////////////////////////////////
});'''
mp = multiprocessing.get_context('spawn')
p = mp.Process(target=pattern.sub, args=('', input_js))
p = multiprocessing.Process(target=pattern.sub, args=('', input_js))
p.start()
p.join(SHORT_TIMEOUT)
try:
Expand Down

This file was deleted.

0 comments on commit d4c410f

Please sign in to comment.