Character encoding strikes back

[wingman-jr-addon]

User Drago got me some great feedback about the ongoing battle to make the character detection work flawlessly. See #70 for past history.

on some websites (e.g. https://winfuture.de/news,123262.html) special characters like "ä", "ö", "ü", "ß" and probably become broken and shown as �. The developer reduced problematic pages like these to a minimum already, so its not a big deal.

Having an actual site to check against helps so much! I can reproduce the issue.

[wingman-jr-addon]

I looked into this a bit and the issue seems to be related to the fact that we are getting raw bytes that may NOT be UTF-8 encoding and always dump them out as UTF-8 encoded using TextEncoder. This is for sure not the fully correct way to handle this; however, Firefox doesn't support other character sets on the TextEncoder. (See

wingman_jr/background.js

Line 641 in 03523f1

// 2) Ensures the output Content-Type is UTF-8 because that is what TextEncoder supports

, https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder) Playing around with this PR as a possible solution, but it may introduce other things as I haven't check the regression tests:
#187

[wingman-jr-addon]

All tests passed after some tweaks - keeping an eye on this for regressions

[Dragodraki]

Don't want to bother you. If you seeking for yet more optimization for special characters fixing, here is another site which seems to have problems (german website):

https://uniconverter.wondershare.de/ogg/aac-vs-ogg.html

There you can find the "�" character again instead of "ä", "ö", "ü", "ß"