PEP 594: Update with discussion items (#1063)

* Deprecate parser module
* Keep fileinput module
* Elaborate why crypt and spwd are dangerous and bad
* Improve sections for cgitb, colorsys, nntplib, and smtpd modules
* The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable substitutions.
* Mention that socketserver is going to stay for http.server and xmlrpc.server

pep-0594.rst

@@ -5,7 +5,7 @@ Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 20-May-2019
-Post-History:
+Post-History: 21-May-2019
 
 
 Abstract
@@ -96,7 +96,8 @@ features should be added. Bug should still be fixed.
 ---
 
 Starting with Python 3.9, deprecated modules will start issuing
-`DeprecationWarning`.
+`DeprecationWarning`. The `parser`_ module is removed and potentially
+replaced with a new module.
 
 
 3.10
@@ -116,7 +117,9 @@ analogous Python's two release deprecation process.
 
 The first *provisionally accepted* phase targets Python 3.8.0b1. In the first
 phase no code is changes or removed. Modules are only documented as
-deprecated.
+deprecated. The only exception is the `parser`_ module. It has been
+documented as deprecated since Python 2.5 and is scheduled for removal for
+3.9 to make place for a more advanced parser.
 
 The final decision, which modules will be removed and how the removed code
 is preserved, can be delayed for another year.
@@ -133,6 +136,7 @@ audio processing.
 
 .. csv-table:: Table 1: Proposed modules deprecations
    :header: "Module", "Deprecated in", "To be removed", "Replacement"
+   :widths: 1, 1, 1, 2
 
     aifc,3.8,3.10,\-
     asynchat,3.8,3.10,asyncio
@@ -142,13 +146,13 @@ audio processing.
     cgi,3.8,3.10,\-
     cgitb,3.8,3.10,\-
     chunk,3.8,3.10,\-
-    colorsys,**3.8?**,**3.10?**,\-
-    crypt,3.8,3.10,\-
-    fileinput,3.8,3.10,argparse
+    colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow"
+    crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib"
+    fileinput,\-,**keep**,argparse
     formatter,3.4,3.10,\-
     fpectl,**3.7**,**3.7**,\-
     getopt,**3.2**,**keep**,"argparse, optparse"
-    imghdr,3.8,3.10,\-
+    imghdr,3.8,3.10,"filetype, puremagic, python-magic"
     imp,**3.4**,3.10,importlib
     lib2to3,\-,**keep**,
     macpath,**3.7**,**3.8**,\-
@@ -157,10 +161,11 @@ audio processing.
     nis,3.8,3.10,\-
     optparse,\-,**keep**,argparse
     ossaudiodev,3.8,3.10,\-
+    parser,**2.5**,**3.9**,"ast, lib2to3.pgen2"
     pipes,3.8,3.10,subprocess
-    smtpd,**3.7**,3.10,aiosmtpd
-    sndhdr,3.8,3.10,\-
-    spwd,3.8,3.10,\-
+    smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd
+    sndhdr,3.8,3.10,"filetype, puremagic, python-magic"
+    spwd,3.8,3.10,"python-pam, simplepam"
     sunau,3.8,3.10,\-
     uu,3.8,3.10,\-
     wave,\-,**keep**,
@@ -250,7 +255,8 @@ The `audioop <https://docs.python.org/3/library/audioop.html>`_ module
 contains helper functions to manipulate raw audio data and adaptive
 differential pulse-code modulated audio data. The module is implemented in
 C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_
-module depend on `audioop`_ for some operations.
+module depend on `audioop`_ for some operations. The byteswap operation in
+the `wave`_ module can be substituted with little work.
 
 Module type
   C extension
@@ -266,8 +272,11 @@ colorsys
 
 The `colorsys <https://docs.python.org/3/library/colorsys.html>`_ module
 defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate
-systems. The Pillow library provides much faster conversation between
-color systems.
+systems.
+
+The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and
+advanced features. The Pillow library is better suited to transform images
+between color systems.
 
 Module type
   pure Python
@@ -276,8 +285,10 @@ Deprecated in
 To be removed in
   3.10
 Substitute
-  `Pillow <https://pypi.org/project/Pillow/>`_,
-  `colorspacious <https://pypi.org/project/colorspacious/>`_
+  `colormath <https://pypi.org/project/colormath/>`_,
+  `colour <https://pypi.org/project/colour/>`_
+  `colorspacious <https://pypi.org/project/colorspacious/>`_,
+  `Pillow <https://pypi.org/project/Pillow/>`_
 
 chunk
 ~~~~~
@@ -311,7 +322,9 @@ Deprecated in
 To be removed in
   3.10
 Substitute
-  *n/a*
+  `puremagic <https://pypi.org/project/puremagic/>`_,
+  `filetype <https://pypi.org/project/filetype/>`_,
+  `python-magic <https://pypi.org/project/python-magic/>`_
 
 ossaudiodev
 ~~~~~~~~~~~
@@ -348,7 +361,9 @@ Deprecated in
 To be removed in
   3.10
 Substitute
-  *n/a*
+  `puremagic <https://pypi.org/project/puremagic/>`_,
+  `filetype <https://pypi.org/project/filetype/>`_,
+  `python-magic <https://pypi.org/project/python-magic/>`_
 
 sunau
 ~~~~~
@@ -439,6 +454,10 @@ cgitb
 The `cgitb <https://docs.python.org/3/library/cgitb.html>`_ module is a
 helper for the cgi module for configurable tracebacks.
 
+The ``cgitb`` module is not used by any major Python web framework (Django,
+Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an
+optional debugging middleware.
+
 Module type
   pure Python
 Deprecated in
@@ -453,7 +472,8 @@ smtpd
 
 The `smtpd <https://docs.python.org/3/library/smtpd.html>`_ module provides
 a simple implementation of a SMTP mail server. The module documentation
-recommends ``aiosmtpd``.
+marks the module as deprecated and recommends ``aiosmtpd`` instead. The
+deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1.
 
 Module type
   pure Python
@@ -471,11 +491,15 @@ The `nntplib <https://docs.python.org/3/library/nntplib.html>`_ module
 implements the client side of the Network News Transfer Protocol (nntp). News
 groups used to be a dominant platform for online discussions. Over the last
 two decades, news has been slowly but steadily replaced with mailing lists
-and web-based discussion platforms.
+and web-based discussion platforms. Twisted is also
+`planning <https://twistedmatrix.com/trac/ticket/9405>`_ to deprecate NNTP
+support.
 
 The ``nntplib`` tests have been the cause of additional work in the recent
-past. Python only contains client side of NNTP. The test cases depend on
-external news server. These servers were unstable in the past.
+past. Python only contains client side of NNTP. The tests connect to
+external news server. The servers are sometimes unavailble, too slow, or do
+not work correctly over IPv6. The situation causes flaky test runs on
+buildbots.
 
 Module type
   pure Python
@@ -508,6 +532,10 @@ quality and insecure. Users are discouraged to use them.
   commonly available on Linux.
 * Depending on the platform, the ``crypt`` module is not thread safe. Only
   implementations with ``crypt_r(3)`` are thread safe.
+* The module was never useful to interact with system user and password
+  databases. On BSD, macOS, and Linux, all user authentication and
+  password modification operations must go through PAM (pluggable
+  authentication module), see `spwd`_ deprecation.
 
 Module type
   C extension + Python module
@@ -561,9 +589,18 @@ spwd
 
 The `spwd <https://docs.python.org/3/library/spwd.html>`_ module provides
 direct access to Unix shadow password database using non-standard APIs.
+
 In general it's a bad idea to use the spwd. The spwd circumvents system
-security policies, it does not use the PAM stack, and is
-only compatible with local user accounts.
+security policies, it does not use the PAM stack, and is only compatible
+with local user accounts, because it ignores NSS. The use of the ``spwd``
+module for access control must be consider a *security bug*, as it bypasses
+PAM's access control.
+
+Further more the ``spwd`` module uses the
+`shadow(3) <http://man7.org/linux/man-pages/man3/shadow.3.html>`_ APIs.
+Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This
+is dangerous and even forbidden for confined services on systems with a
+security engine like SELinux or AppArmor.
 
 Module type
   C extension
@@ -572,28 +609,12 @@ Deprecated in
 To be removed in
   3.10
 Substitute
-  **none**
+  `python-pam <https://pypi.org/project/python-pam/>`_,
+  `simpleplam <https://pypi.org/project/simplepam/>`_
 
 Misc modules
 ------------
 
-fileinput
-~~~~~~~~~
-
-The `fileinput <https://docs.python.org/3/library/fileinput.html>`_ module
-implements a helpers to iterate over a list of files from ``sys.argv``. The
-module predates the optparser and argparser module. The same functionality
-can be implemented with the argparser module.
-
-Module type
-  pure Python
-Deprecated in
-  3.8
-To be removed in
-  3.10
-Substitute
-  argparse
-
 formatter
 ~~~~~~~~~
 
@@ -649,6 +670,35 @@ To be removed in
 Substitute
   **none**
 
+parser
+~~~~~~
+
+The `parser <https://docs.python.org/3/library/parser.html>`_ module provides
+an interface to Python’s internal parser and byte-code compiler. The stdlib
+has superior ways to interact with the parse tree. From Python 2.5 onward,
+it's much more convenient to cut in at the Abstract Syntax Tree (AST)
+generation and compilation stage.
+
+The ``parser`` module causes additional work. It's C code that must be
+kept in sync with any change to Python's grammar and internal parser.
+Pablo wants to remove the parser module and promote lib2to3's pgen2 instead
+[6]_.
+
+Most importantly the presence of the ``parser`` module makes it harder to
+switch to something more powerful than a LL(1) parser [7]_. Since the
+``parser`` module is documented as deprecated since Python 2.5 and a new
+parsing technology is planned for 3.9, the ``parser`` module is scheduled for
+removal in 3.9.
+
+Module type
+  C extension
+Deprecated in
+  3.8, documented as deprecated since **2.5**
+To be removed in
+  **3.9**
+Substitute
+  ast, lib2to3.pgen2
+
 pipes
 ~~~~~
 
@@ -693,6 +743,20 @@ Modules to keep
 
 Some modules were originally proposed for deprecation.
 
+fileinput
+---------
+
+The `fileinput <https://docs.python.org/3/library/fileinput.html>`_ module
+implements a helpers to iterate over a list of files from ``sys.argv``. The
+module predates the optparser and argparser module. The same functionality
+can be implemented with the argparser module.
+
+Several core developers expressed their interest to keep the module in the
+standard library, as it is handy for quick scripts.
+
+Module type
+  pure Python
+
 lib2to3
 -------
 
@@ -798,7 +862,25 @@ Discussions
 * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...)
   convinced me to keep the `wave`_ module. [4]_
 * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_
+* Andrew Svetlov mentioned the ``socketserver`` module is questionable.
+  However it's used to implement ``http.server`` and ``xmlrpc.server``. The
+  stdlib doesn't have a replacement for the servers, yet.
+
+
+Update history
+==============
+
+Update 1
+--------
 
+* Deprecate `parser`_ module
+* Keep `fileinput`_ module
+* Elaborate why `crypt`_ and `spwd`_ are dangerous and bad
+* Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules
+* The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now
+  list suitable substitutions.
+* Mention that ``socketserver`` is going to stay for ``http.server`` and
+  ``xmlrpc.server``
 
 References
 ==========
@@ -808,6 +890,8 @@ References
 .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/
 .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169
 .. [5] https://twitter.com/dabeaz/status/1130278844479545351
+.. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html
+.. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379
 
 
 Copyright