[BUG] Can't Modify an INI file that contains unicode characters

[twangboy]

Description
Can't modify an ini file that contains unicode characters.

Setup
Create an ini file with the following content (C:\temp\test.ini):

[Locales]
Mapping = "en_US:English (US),zh:简体中文,en_GB:English (UK),fi:Suomi,fr:Français,de:Deutsch,hi:हिंदी,it:Italiano,ja:日本語,ko:한국어,nl:Dutch,pt:Portuguese,es:Espanõl,es_ES:Espanõl (ES),zh_TW:繁體中文,sv:Svenska,hu:Magyar"

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
Attempt to get the value from the ini file using salt:

(venv) PS C:\src\salt> salt-call --local ini.get_option C:\Temp\test.ini Locales Mapping

Expected behavior
The ini value should be displayed without a stacktrace

Screenshots

(venv) PS C:\src\salt> salt-call --local ini.get_option C:\Temp\test.ini Locales Mapping
[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 229: character maps to <undefined>
Traceback (most recent call last):
  File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\src\salt\venv\Scripts\salt-call.exe\__main__.py", line 7, in <module>
    sys.exit(salt_call())
  File "C:\src\salt\salt\scripts.py", line 443, in salt_call
    client.run()
  File "C:\src\salt\salt\cli\call.py", line 50, in run
    caller.run()
  File "C:\src\salt\salt\cli\caller.py", line 95, in run
    ret = self.call()
  File "C:\src\salt\salt\cli\caller.py", line 202, in call
    ret["return"] = self.minion.executors[fname](
  File "C:\src\salt\salt\loader\lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "C:\src\salt\salt\loader\lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "C:\src\salt\salt\loader\lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\src\salt\salt\executors\direct_call.py", line 10, in execute
    return func(*args, **kwargs)
  File "C:\src\salt\salt\loader\lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "C:\src\salt\salt\loader\lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "C:\src\salt\salt\loader\lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\src\salt\salt\modules\ini_manage.py", line 106, in get_option
    inifile = _Ini.get_ini_file(file_name, separator=separator)
  File "C:\src\salt\salt\modules\ini_manage.py", line 463, in get_ini_file
    inifile.refresh()
  File "C:\src\salt\salt\modules\ini_manage.py", line 420, in refresh
    inicontents = salt.utils.stringutils.to_unicode(rfh.read())
  File "C:\Python310\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 229: character maps to <undefined>

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: 3.1.37
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 25.0.2
        relenv: Not Installed
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist:
        locale: cp1252
       machine: AMD64
       release: 10
        system: Windows
       version: 10 10.0.22631 SP0 Multiprocessor Free

Additional context

I can fix the problem by modifying salt/modules.ini_manage.py at line 419. The current line is:

                with salt.utils.files.fopen(self.name) as rfh:

If I open the file with the encoding="utf-8" option, then everything works:

                with salt.utils.files.fopen(self.name, encoding="utf-8") as rfh:

I'm not sure this is a valid fix though. Feedback welcome.

[s0undt3ch]

The functions should probably all need a ini_encoding keyword argument(which defaults to utf-8) that gets passed to fopen, at least this way, you allow users to override the encoding.