restore lenient libmagic decoding

[mjg]

176cffc ("refactor alot.db.utils.remove_cte", 2018-12-04) created a few
problems with 8bit quoted-printable e-mails, see #1291 #1360.

This commit restores the old libmagic fallback which did not cause this
problem.

(Maybe just goood as a 0.8 hotfix.)

[pazz]

Sorry I don't have time to review this properly.
Can you just confirm that this indeed fixes the issues in #1291 and #1360 and that we have test cases witnessing this?

[josch]

I just backported the change to version 0.8 that we are about to upload to Debian and my results are:

• I can confirm that this is not a regression of the issue I reported as UnicodeDecodeError with Python3 #1291, so no error message shown with python3 in affected emails
• It fixes Missing characters in extracted body #1360 in that characters are not missing from messages anymore (as also evident by this change fixing that testcase)

[pazz]

Thanks for this josh and mjg!

@lucc: what do you think?
Would this justify a minor release?

[mjg]

The force-pushed change shows that with this commit, the expected failure is turned into success.

What I don't know is whether the behavour change introduced in commit #176cffc was intentional, based on a concrete example, or just on the rationale "if there are decode errors then ignore undecodable bytes", which seems reasonable. I'm wondering: Is the message from the test case RFC compliant? It certainly is in a very common format. It is exactly this type that decode() fails to decode but on which libmagic succeeds. Maybe we can trace what libmagic actually does/guesses to understand what is going wrong? Maybe just a different CTE that we need to pass to decode()?

[dcbaker]

Libmagic actually does some sort of analysis to figure out what the content encoding is, the remove_cte function trusts that a) the payload is truthful in it's encoding or b) that it does what the RFC requires and is 7bit if it doesn't declare an encoding.

FWIW, I think this is the right thing to do; try to trust the payload, if it lied/is wrong then fall back to libmagic.

Or, bump the version to 3.6 which IIRC can actually do this correctly.

Since this is blocking debian's adoption of 0.8, I think there should be a 0.8.1 release.

[pazz]

@dcbaker: by "bump the version" you mean have alot eepend on python \ge 3.6 ?

[jljusten]

@pazz, @dcbaker: I believe we are seeing this with python 3.7 on debian.

[lucc]

@pazz sorry I'm late. I also vote for 0.8.1

[pazz]

OK, I've just tagged this bugfix release (but won't announce it on the notmuch list)

[jljusten]

@pazz: I think maybe you didn't change alot/__init__.py for the version before tagging, so the tarball content doesn't match the v0.8.1 tag. ?

[pazz]

@jljusten: thanks and sorry: I only pushed the tags, not the last commit. It should be fixed now.