Pip opening file with cp1252 encoding raises exception when temp folder path contains non-latin char

[jacekchalupka]

pip version

21.0.1

Python version

3.9

OS

Windows 10

Additional information

My home folder contains ł character (polish language)

Description

When doing pip install . exception is being raised (see output field).

During installation, the file sitecustomize.py is opened to writing in default system encoding code. On Windows it is being opened with cp-1252.

The content that is being written to a file is:

import os, site, sys

# First, drop system-sites related paths.
original_sys_path = sys.path[:]
known_paths = set()
for path in {'c:\\python\\python39\\lib\\site-packages'}:
    site.addsitedir(path, known_paths=known_paths)
system_paths = set(
    os.path.normcase(path)
    for path in sys.path[len(original_sys_path):]
)
original_sys_path = [
    path for path in original_sys_path
    if os.path.normcase(path) not in system_paths
]
sys.path = original_sys_path

# Second, add lib directories.
# ensuring .pth file are processed.
for path in ['C:\\Users\\jchałupka\\AppData\\Local\\Temp\\pip-build-env-94due9qe\\overlay\\Lib\\site-packages', 'C:\\Users\\jchałupka\\AppData\\Local\\Temp\\pip-build-env-94due9qe\\normal\\Lib\\site-packages']: 
    assert not path in sys.path
    site.addsitedir(path)

WORKAROUNDS

Change mentioned above line to:

        with open(os.path.join(self._site_dir, 'sitecustomize.py'), 'w', encoding='utf-8') as fp:

Set PYTHONUTF8=1 env var.

Expected behavior

File is being opened with utf-8 encoding, no error raised.

How to Reproduce

The easiest way to reproduce the issue I found was with using poetry.

1. Make sure your temp folder contains come non latin character (in my example ł)
2. Install poetry 1.1.4
3. Create new project poetry new some_name
4. Enter the project folder: cd some_name
5. Try to install project: pip install .

Output

PS C:\repos\temp> pip install .
Processing c:\repos\temp
ERROR: Exception:
Traceback (most recent call last):
  File "c:\python\python39\lib\site-packages\pip\_internal\cli\base_command.py", line 189, in _main
    status = self.run(options, args)
  File "c:\python\python39\lib\site-packages\pip\_internal\cli\req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "c:\python\python39\lib\site-packages\pip\_internal\commands\install.py", line 316, in run
    requirement_set = resolver.resolve(
  File "c:\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 100, in resolve
    r = self.factory.make_requirement_from_install_req(
  File "c:\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 301, in make_requirement_from_install_req
    cand = self._make_candidate_from_link(
    self._link_candidate_cache[link] = LinkCandidate(
  File "c:\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 300, in __init__
    super().__init__(
  File "c:\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 144, in __init__
    self.dist = self._prepare()
  File "c:\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 226, in _prepare
    dist = self._prepare_distribution()
  File "c:\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 311, in _prepare_distribution
    return self._factory.preparer.prepare_linked_requirement(
  File "c:\python\python39\lib\site-packages\pip\_internal\operations\prepare.py", line 457, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "c:\python\python39\lib\site-packages\pip\_internal\operations\prepare.py", line 500, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "c:\python\python39\lib\site-packages\pip\_internal\operations\prepare.py", line 66, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "c:\python\python39\lib\site-packages\pip\_internal\distributions\sdist.py", line 39, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "c:\python\python39\lib\site-packages\pip\_internal\distributions\sdist.py", line 66, in _setup_isolation
    self.req.build_env = BuildEnvironment()
  File "c:\python\python39\lib\site-packages\pip\_internal\build_env.py", line 83, in __init__
    fp.write(textwrap.dedent(
  File "c:\python\python39\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u0142' in position 584: character maps to <undefined>

Code of Conduct

• I agree to follow the PSF Code of Conduct

[pfmoore]

Thanks for the report. Now that we only support Python 3, writing the file in UTF-8 seems like the correct solution.

[uranusjr]

Duplicate to #9054.