Different output for internal and external console on Windows

[s-weigand]

Description of your problem

Depending on if i run my code, which uses subprocess to execute Windows commandline code, in the internal console ("Execute in new dedicated Python console") or the external console ("Execute in an external System terminal") i get different output and exception bevaior, if none ascii characters are contained.
This is due to different encodings of the internal console (retrived by locale.getdefaultlocale('LANG') in spyderlib/widgets/externalshell/sitecustomize.py lines 149-165) and the system console (run the chcp command in cmd).
Also since the values of ctypes.windll.kernel32.SetConsoleCP(_cp) and ctypes.windll.kernel32.SetConsoleOutputCP(_cp) are changed there is no way (or at least none i found), to determine the right encoding.

What steps will reproduce the problem?

To reproduce the the problem you can execute the following code.
Note that cp850 is the codepage of my windows and may differ for you same as the printability of the test_string.
To reproduce the error u may want to change test_string with non ascii character of your language aswell as cp1252 for the output of locale.getpreferredencoding() and cp850 for the codepage given by running the chcp command in cmd.

# -*- coding: utf-8 -*-
import subprocess, ctypes, locale


def get_cmd_encoding():
    return_string = ""
    test_string = r"äöüÄÖÜß"
    arglist = ("echo "+test_string).split(" ")
    output = subprocess.Popen(arglist, shell=True, stdout=subprocess.PIPE,
                              stdin=subprocess.PIPE).communicate()[0]
    endocing_list = ["utf-8", "cp1252", "cp850"]
    for encoding in endocing_list:
        print(encoding)
        try:
            return_string = output.decode(encoding).strip()
            print(return_string)
        except:
            print("exception happened")
            return_string = ""

ctypes_cdll = ctypes.cdll.kernel32
ctypes_windll = ctypes.windll.kernel32


get_cmd_encoding()
print("\nretrived ecodings:\n")
print("locale.getdefaultlocale() \t\t: ", locale.getdefaultlocale())
print("locale.getlocale() \t\t\t: ", locale.getlocale())
print("locale.getpreferredencoding() \t: ", locale.getpreferredencoding())
print("ctypes_cdll.GetConsoleCP() \t\t: ", ctypes_cdll.GetConsoleCP())
print("ctypes_cdll.GetConsoleOutputCP()\t: ", ctypes_cdll.GetConsoleOutputCP())
print("ctypes_windll.GetConsoleCP() \t\t: ", ctypes_windll.GetConsoleCP())
print("ctypes_windll.GetConsoleOutputCP()\t: ", ctypes_windll.GetConsoleOutputCP())
print("chcp command\t\t\t: ", subprocess.check_output("chcp", shell=True))
# just to keep the external System Terminal open
input()

Expected output : System terminal

utf-8
exception happened
cp1252
exception happened
cp850
äöüÄÖÜß

retrived ecodings:

locale.getdefaultlocale()       :  ('de_DE', 'cp1252')
locale.getlocale()          :  (None, None)
locale.getpreferredencoding()   :  cp1252
ctypes_cdll.GetConsoleCP()      :  850
ctypes_cdll.GetConsoleOutputCP()    :  850
ctypes_windll.GetConsoleCP()        :  850
ctypes_windll.GetConsoleOutputCP()  :  850
chcp command            :  b'Aktive Codepage: 850.\r\n'

False output : internal console

utf-8
exception happened
cp1252
äöüÄÖÜß
cp850
õ÷³─Í▄▀

retrived ecodings:

locale.getdefaultlocale()       :  ('de_DE', 'cp1252')
locale.getlocale()          :  (None, None)
locale.getpreferredencoding()   :  cp1252
ctypes_cdll.GetConsoleCP()      :  1252
ctypes_cdll.GetConsoleOutputCP()    :  1252
ctypes_windll.GetConsoleCP()        :  1252
ctypes_windll.GetConsoleOutputCP()  :  1252
chcp command            :  b'Aktive Codepage: 1252.\r\n'

Solution

Simply deleting lines 149-165 spyderlib/widgets/externalshell/sitecustomize.py solved that problem for me. But i have no idea which problems may arrise of this "fix". I hope the person who wrote that part still knows why and knows how to test if this produes a problem (none found here yet).

spyderlib/widgets/externalshell/sitecustomize.py lines 149-165

#==============================================================================
# Setting console encoding (otherwise Python does not recognize encoding)
# for Windows platforms
#==============================================================================
if os.name == 'nt':
    try:
        import locale, ctypes
        _t, _cp = locale.getdefaultlocale('LANG')
        try:
            _cp = int(_cp[2:])
            ctypes.windll.kernel32.SetConsoleCP(_cp)
            ctypes.windll.kernel32.SetConsoleOutputCP(_cp)
        except (ValueError, TypeError):
            # Code page number in locale is not valid
            pass
    except ImportError:
        pass

Guess the problem with the encoding is that even the python sourcecode says that locale.getdefaultlocale is guessing the encoding.

Versions and main components

• Spyder Version: 3.0.0b2
• Python Version: 3.5.1 64bit
• Operating system: Windows 7 64bit

Dependencies

IPython >=3.0 : 4.1.1 (OK)
jedi >=0.8.1;<0.9.0: 0.9.0 (NOK)
matplotlib >=1.0 : 1.5.1 (OK)
numpy >=1.7 : 1.10.4 (OK)
pandas >=0.13.1 : 0.18.0 (OK)
pep8 >=0.6 : 1.7.0 (OK)
pyflakes >=0.6.0 : 1.1.0 (OK)
pygments >=1.6 : 2.1.1 (OK)
pylint >=0.25 : 1.5.4 (OK)
qtconsole >=4.0 : 4.1.1 (OK)
rope >=0.9.4 : 0.9.4-1 (OK)
sphinx >=0.6.6 : 1.3.1 (OK)
sympy >=0.7.3 : 1.0 (OK)
zmq >=13.0.0 : 15.2.0 (OK)

[ccordoba12]

@PierreRaybaut, you were the one who added those lines. Could you give us some light here?

[PierreRaybaut]

I did some tests on both Python 2.7 and Python 3.4 and it seems that Windows console encoding is now well handled by Python 3 (at least since v3.4, may also in v3.0-3.3).

The original spirit of the code snippet you intend to remove was to fix Windows console so that utf-8 would work as well as on Linux platforms. Python was not supporting cp encoding in Windows console, so at least in Spyder the console output was consistent with the results obtained on Linux -- it was the best compromise I could find. Now that Python 3 works well with Windows console encoding, there is no need to keep this patch... but it's still necessary for Python 2.

I hence recommend keeping the "Setting console encoding [...] for Windows platforms" code block for Python 2, so the right fix for me would rather be the following:

#==============================================================================
# Setting console encoding (otherwise Python does not recognize encoding)
# for Windows platforms
#==============================================================================
+if os.name == 'nt' and PY2:
-if os.name == 'nt':
    try:
        import locale, ctypes
        _t, _cp = locale.getdefaultlocale('LANG')
        try:
            _cp = int(_cp[2:])
            ctypes.windll.kernel32.SetConsoleCP(_cp)
            ctypes.windll.kernel32.SetConsoleOutputCP(_cp)
        except (ValueError, TypeError):
            # Code page number in locale is not valid
            pass
    except ImportError:
        pass