Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-100228: Document the os.fork threads DeprecationWarning. #109767

Merged
merged 9 commits into from
Sep 23, 2023
8 changes: 8 additions & 0 deletions Doc/library/concurrent.futures.rst
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,14 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
The *max_tasks_per_child* argument was added to allow users to
control the lifetime of workers in the pool.

.. versionchanged:: 3.12
On POSIX systems if your application has multiple threads and the
:mod:`multiprocessing` context uses the ``"fork"`` start method:
gpshead marked this conversation as resolved.
Show resolved Hide resolved
The :func:`os.fork` called internally to spawn workers may raise a
gpshead marked this conversation as resolved.
Show resolved Hide resolved
:exc:`DeprecationWarning`. Pass a *mp_context* configured to use a
different start method. See the :func:`os.fork` documentation
further explanation.
gpshead marked this conversation as resolved.
Show resolved Hide resolved

.. _processpoolexecutor-example:

ProcessPoolExecutor Example
Expand Down
6 changes: 6 additions & 0 deletions Doc/library/multiprocessing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,12 @@ to start a process. These *start methods* are
Code that requires *fork* should explicitly specify that via
:func:`get_context` or :func:`set_start_method`.

.. versionchanged:: 3.12
If Python is able to detect that your process has multiple threads, the
:func:`os.fork` function that this start method calls internally will
raise a :exc:`DeprecationWarning`. Use a different start method.
See the :func:`os.fork` documentation further explanation.
gpshead marked this conversation as resolved.
Show resolved Hide resolved

*forkserver*
When the program starts and selects the *forkserver* start method,
a server process is spawned. From then on, whenever a new process
Expand Down
36 changes: 32 additions & 4 deletions Doc/library/os.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4157,15 +4157,38 @@ written in Python, such as a mail server's external command delivery program.

.. audit-event:: os.fork "" os.fork

.. warning::

If you use TLS sockets in an application calling ``fork()`` see
the warning in the :mod:`ssl` documentation.
gpshead marked this conversation as resolved.
Show resolved Hide resolved

.. versionchanged:: 3.8
Calling ``fork()`` in a subinterpreter is no longer supported
(:exc:`RuntimeError` is raised).

.. warning::

See :mod:`ssl` for applications that use the SSL module with fork().
.. versionchanged:: 3.12
If Python is able to detect that your process has multiple
threads, :func:`os.fork` now raises a :exc:`DeprecationWarning`.

We chose to surface this as a warning, when detectable, to better
inform developers of a design problem that the POSIX platform
specificly calls this out as not supported. Even in code that
*appears* to work, it has never been safe to mix threading with
:func:`os.fork` on POSIX platforms. The CPython runtime itself has
always made API calls that are not safe for use in the child
process when threads existed in the parent (such as ``malloc`` and
``free``).

Users of macOS or users of libc or malloc implementations other
than those typically found in glibc to date are among those
already more likely to experience deadlocks running such code.

See `this discussion on fork being incompatible with threads
<https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555>`_
for technical details of why we're surfacing this long standing
gpshead marked this conversation as resolved.
Show resolved Hide resolved
platform compatibility problem to developers.
gpshead marked this conversation as resolved.
Show resolved Hide resolved

.. availability:: Unix, not Emscripten, not WASI.
.. availability:: POSIX, not Emscripten, not WASI.


.. function:: forkpty()
Expand All @@ -4178,6 +4201,11 @@ written in Python, such as a mail server's external command delivery program.

.. audit-event:: os.forkpty "" os.forkpty

.. versionchanged:: 3.12
If Python is able to detect that your process has multiple
threads, this now raises a :exc:`DeprecationWarning`. See the
longer explanation on :func:`os.fork`.

.. versionchanged:: 3.8
Calling ``forkpty()`` in a subinterpreter is no longer supported
(:exc:`RuntimeError` is raised).
Expand Down
13 changes: 13 additions & 0 deletions Doc/whatsnew/3.12.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1225,6 +1225,19 @@ Pending Removal in Python 3.14
:func:`~multiprocessing.set_start_method` APIs to explicitly specify when
your code *requires* ``'fork'``. See :ref:`multiprocessing-start-methods`.

* :func:`os.fork` now raises a :exc:`DeprecationWarning` on POSIX when
it can detect that it is being called from a multithreaded process.
There has always been a fundamental incompatibility with the POSIX
platform when doing so. Even if such code *appeared* to work. We
added the warning to to raise awareness as issues encounted by code
doing this are becoming more frequent. See the :func:`os.fork`
documentation for more details.

When this warning appears to come from within :mod:`multiprocessing`
while using that or :mod:`concurrent.futures` the safe fix is to use a
different start method such as ``spawn`` (anywhere) or ``forkserver``
(not on macOS).
gpshead marked this conversation as resolved.
Show resolved Hide resolved

* :mod:`pkgutil`: :func:`pkgutil.find_loader` and :func:`pkgutil.get_loader`
now raise :exc:`DeprecationWarning`;
use :func:`importlib.util.find_spec` instead.
Expand Down