PR: Use chardet as a fallback for encoding detection

[rlaverde]

Fixes #3731

---

I think It's better to leave the logic that already exist to detect text encoding, and just add chardet as a fallback

Using only chardet some utf-8 text will be detect as ascii (because utf-8 is a superset of ascii)

[ccordoba12]

This will only work for text that's not unicode, so why not remove from CODECS above all encodings that are not unicode and use chardet for non-unicode stuff?

[goanpeca]

Plus typo, Fallback (with and extra L)

[rlaverde]

@ccordoba12 I think that It's better to relay in the information of the file that could be more accurate, and use encoding detection as a fallback, chardet recommend that way (although it make reference to mime types)

@goanpeca yep, duck-typing I'll correct it :)

[goanpeca]

I agree with Rafa, I think what is in there is good enough (checking the line at the top of the file) in case it is not found then we can use chardet that is heavier to check.

[ccordoba12]

Ok, I agree with you guys, thanks for the clarification :-)

@rlaverde, please don't forget to fix the little typo pointed out by @goanpeca on the line above this line :-)

[ccordoba12]

Where are we computing determining the encoding for files in the Editor @rlaverde? Could you point to the exact point where we are doing it?

[rlaverde]

@ccordoba12 editor call two functions of encoding module read() and write() and these functions call encode() and decode() and these call get_coding() (the one I modified)

The encoding is guessed in encode() and decode() and get_coding()

example: https://github.com/spyder-ide/spyder/blob/master/spyder/widgets/editor.py#L1258

[ccordoba12]

@rlaverde, please add some files to test this new feature:

1. One with cp1252 (or ANSI) Windows encoding.
2. Another one with Big5 or another Chinese encoding.
3. And a final one with a Cyrillic encoding (KOI8-R, MacCyrillic, IBM855, IBM866, ISO-8859-5, or windows-1251).

Thanks!

[ccordoba12]

There are failures on Python 2, please fix them.

[ccordoba12]

Are we assuring this also in the Editor, i.e. that text read from files in Python 2 is converted to bytes before detecting its encoding?

[rlaverde]

No, I was wrong, this wasn't the problem with python2

The problem is trying to convert bytes to string without knowing the encoding, in python3 str(some_bytes_with_no_utf8_encoding) return an empty string but It fails in python2

[ccordoba12]

I think this one is ready, thanks @rlaverde!