diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst index 5e07a4caeb9ebe..47b01da58ecb1c 100644 --- a/Doc/whatsnew/3.12.rst +++ b/Doc/whatsnew/3.12.rst @@ -66,6 +66,10 @@ Summary -- Release highlights .. PEP-sized items next. +New grammar features: + +* :pep:`701`: Syntactic formalization of f-strings + New typing features: * :pep:`688`: Making the buffer protocol accessible in Python @@ -136,22 +140,70 @@ Improved Error Messages New Features ============ -* Add :ref:`perf_profiling` through the new - environment variable :envvar:`PYTHONPERFSUPPORT`, - the new command-line option :option:`-X perf <-X>`, - as well as the new :func:`sys.activate_stack_trampoline`, - :func:`sys.deactivate_stack_trampoline`, - and :func:`sys.is_stack_trampoline_active` APIs. - (Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes - with contributions from Gregory P. Smith [Google] and Mark Shannon - in :gh:`96123`.) -* The extraction methods in :mod:`tarfile`, and :func:`shutil.unpack_archive`, - have a new a *filter* argument that allows limiting tar features than may be - surprising or dangerous, such as creating files outside the destination - directory. - See :ref:`tarfile-extraction-filter` for details. - In Python 3.14, the default will switch to ``'data'``. - (Contributed by Petr Viktorin in :pep:`706`.) +.. _whatsnew312-pep701: + +PEP 701: Syntactic formalization of f-strings +--------------------------------------------- + +:pep:`701` lifts some restrictions on the usage of f-strings. Expression components +inside f-strings can now be any valid Python expression including backslashes, +unicode escaped sequences, multi-line expressions, comments and strings reusing the +same quote as the containing f-string. Let's cover these in detail: + +* Quote reuse: in Python 3.11, reusing the same quotes as the containing f-string + raises a :exc:`SyntaxError`, forcing the user to either use other available + quotes (like using double quotes or triple quotes if the f-string uses single + quotes). In Python 3.12, you can now do things like this: + + >>> songs = ['Take me back to Eden', 'Alkaline', 'Ascensionism'] + >>> f"This is the playlist: {", ".join(songs)}" + 'This is the playlist: Take me back to Eden, Alkaline, Ascensionism + + Note that before this change there was no explicit limit in how f-strings can + be nested, but the fact that string quotes cannot be reused inside the + expression component of f-strings made it impossible to nest f-strings + arbitrarily. In fact, this is the most nested f-string that could be written: + + >>> f"""{f'''{f'{f"{1+1}"}'}'''}""" + '2' + + As now f-strings can contain any valid Python expression inside expression + components, it is now possible to nest f-strings arbitrarily: + + >>> f"{f"{f"{f"{f"{f"{1+1}"}"}"}"}"}" + '2' + +* Multi-line expressions and comments: In Python 3.11, f-strings expressions + must be defined in a single line even if outside f-strings expressions could + span multiple lines (like literal lists being defined over multiple lines), + making them harder to read. In Python 3.12 you can now define expressions + spaning multiple lines and include comments on them: + + >>> f"This is the playlist: {", ".join([ + ... 'Take me back to Eden', # My, my, those eyes like fire + ... 'Alkaline', # Not acid nor alkaline + ... 'Ascensionism' # Take to the broken skies at last + ... ])}" + 'This is the playlist: Take me back to Eden, Alkaline, Ascensionism' + +* Backslashes and unicode characters: before Python 3.12 f-string expressions + couldn't contain any ``\`` character. This also affected unicode escaped + sequences (such as ``\N{snowman}``) as these contain the ``\N`` part that + previously could not be part of expression components of f-strings. Now, you + can define expressions like this: + + >>> print(f"This is the playlist: {"\n".join(songs)}") + This is the playlist: Take me back to Eden + Alkaline + Ascensionism + >>> print(f"This is the playlist: {"\N{BLACK HEART SUIT}".join(songs)}") + This is the playlist: Take me back to Eden♥Alkaline♥Ascensionism + +See :pep:`701` for more details. + +(Contributed by Pablo Galindo, Batuhan Taskaya, Lysandros Nikolaou, Cristián +Maureira-Fredes and Marta Gómez in :gh:`102856`. PEP written by Pablo Galindo, +Batuhan Taskaya, Lysandros Nikolaou and Marta Gómez). .. _whatsnew312-pep709: @@ -223,6 +275,24 @@ See :pep:`692` for more details. Other Language Changes ====================== +* Add :ref:`perf_profiling` through the new + environment variable :envvar:`PYTHONPERFSUPPORT`, + the new command-line option :option:`-X perf <-X>`, + as well as the new :func:`sys.activate_stack_trampoline`, + :func:`sys.deactivate_stack_trampoline`, + and :func:`sys.is_stack_trampoline_active` APIs. + (Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes + with contributions from Gregory P. Smith [Google] and Mark Shannon + in :gh:`96123`.) + +* The extraction methods in :mod:`tarfile`, and :func:`shutil.unpack_archive`, + have a new a *filter* argument that allows limiting tar features than may be + surprising or dangerous, such as creating files outside the destination + directory. + See :ref:`tarfile-extraction-filter` for details. + In Python 3.14, the default will switch to ``'data'``. + (Contributed by Petr Viktorin in :pep:`706`.) + * :class:`types.MappingProxyType` instances are now hashable if the underlying mapping is hashable. (Contributed by Serhiy Storchaka in :gh:`87995`.) @@ -543,6 +613,14 @@ tkinter like ``create_*()`` methods. (Contributed by Serhiy Storchaka in :gh:`94473`.) +tokenize +-------- + +* The :mod:`tokenize` module includes the changes introduced in :pep:`701`. ( + Contributed by Marta Gómez Macías and Pablo Galindo in :gh:`102856`.) + See :ref:`whatsnew312-porting-to-python312` for more information on the + changes to the :mod:`tokenize` module. + types ----- @@ -687,6 +765,11 @@ Optimizations * Speed up :class:`asyncio.Task` creation by deferring expensive string formatting. (Contributed by Itamar O in :gh:`103793`.) +* The :func:`tokenize.tokenize` and :func:`tokenize.generate_tokens` functions are + up to 64% faster as a side effect of the changes required to cover :pep:`701` in + the :mod:`tokenize` module. (Contributed by Marta Gómez Macías and Pablo Galindo + in :gh:`102856`.) + CPython bytecode changes ======================== @@ -1130,6 +1213,8 @@ Removed Iceape, Firebird, and Firefox versions 35 and below (:gh:`102871`). +.. _whatsnew312-porting-to-python312: + Porting to Python 3.12 ====================== @@ -1201,6 +1286,40 @@ Changes in the Python API that may be surprising or dangerous. See :ref:`tarfile-extraction-filter` for details. +* The output of the :func:`tokenize.tokenize` and :func:`tokenize.generate_tokens` + functions is now changed due to the changes introduced in :pep:`701`. This + means that ``STRING`` tokens are not emitted any more for f-strings and the + tokens described in :pep:`701` are now produced instead: ``FSTRING_START``, + ``FSRING_MIDDLE`` and ``FSTRING_END`` are now emitted for f-string "string" + parts in addition to the appropriate tokens for the tokenization in the + expression components. For example for the f-string ``f"start {1+1} end"`` + the old version of the tokenizer emitted:: + + 1,0-1,18: STRING 'f"start {1+1} end"' + + while the new version emits:: + + 1,0-1,2: FSTRING_START 'f"' + 1,2-1,8: FSTRING_MIDDLE 'start ' + 1,8-1,9: OP '{' + 1,9-1,10: NUMBER '1' + 1,10-1,11: OP '+' + 1,11-1,12: NUMBER '1' + 1,12-1,13: OP '}' + 1,13-1,17: FSTRING_MIDDLE ' end' + 1,17-1,18: FSTRING_END '"' + + Aditionally, there may be some minor behavioral changes as a consecuence of the + changes required to support :pep:`701`. Some of these changes include: + + * Some final ``DEDENT`` tokens are now emitted within the bounds of the + input. This means that for a file containing 3 lines, the old version of the + tokenizer returned a ``DEDENT`` token in line 4 whilst the new version returns + the token in line 3. + + * The ``type`` attribute of the tokens emitted when tokenizing some invalid Python + characters such as ``!`` has changed from ``ERRORTOKEN`` to ``OP``. + Build Changes =============