'\040' instead of space in repl history on macOS with 3.13 or main branch

[aisk]

Bug report

Bug description:

When using the version of Python (3.9.6) that comes with Apple at /usr/bin/python, or when installing the latest version of Python (3.12.3) via Homebrew, the readline functionality is based on libedit. There is a file ~/.python_history that stores the REPL history, and its content is similar to:

_HiStOrY_V2_
import\040os

When using the latest version of Python built from the main branch, which includes the new REPL, pressing the up arrow key will display:

❯ ./python.exe
Python 3.14.0a0 experimental free-threading build (heads/main:73f4a58d36, May 21 2024, 21:12:51) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import\040os

The '\04' is not converted to space character.

CPython versions tested on:

3.13

Operating systems tested on:

macOS Sonoma 14.2

Linked PRs

• gh-119310: Fix encoding when reading old history file #121779
• [3.13] gh-119310: Fix encoding when reading old history file (GH-121779) #123784

[ambv]

That's because editline used by the basic REPL. We'll address this in the betas.

[Constantin1489]

See also, #102130 (comment) (may ignore some of my argument in the url. But take the examples or links or symptoms. The handling for encoding part in my example code is just wrong.)

it looks like the problem is deeper than thought...

# print_all_printable_unicode.py
# https://stackoverflow.com/a/77352299/20307768
for i in range(0x0000, 1 + 0xffff, 1):
    if str.isprintable(chr(i)):
        print(chr(i))

1. backup the ~/.python_history
   mv ~/.python_history ~/.python_history.bak
2. initialize editline history files with its header
   echo "_HiStOrY_V2_" > ~/.python_history > ~/.python_history.utf8
3. run the code and redirect output to files.
   python print_all_printable_unicode.py >> ~/.python_history >> ~/.python_history.utf8
4. run python repl with editline and exit (It will modify ~/.python_history)
   python
5. compare result using vim(diff)
   vim -d ~/.python_history ~/.python_history.utf8