Ticket 35581: Updated django.core.mail to Python's modern email API [WIP, DO-NOT-MERGE]

[medmunds]

Trac ticket number

ticket-35581

Branch description

Switch django.core.mail from Python's "legacy email API" to its newer "modern email API."

DO NOT MERGE until these security issues in Python's modern email API have been fixed: python/cpython#80222 and python/cpython#121284.

Checklist

• This PR targets the main branch.
  [This PR currently targets Refs #35581 -- Improved django.core.mail tests. django/django#18502]
• The commit message is written in past tense, mentions the ticket number, and ends with a period.
• I have checked the "Has patch" ticket flag in the Trac system.
• I have added or updated relevant tests.
• I have added or updated relevant docs, including release notes if applicable.
• n/a I have attached screenshots in both light and dark modes for any UI changes.

Notes

This is the second of two PRs involved in updating django.core.mail from Python's legacy to modern email APIs. See django#18502 for a separate PR that expands existing email tests prior to these changes.

Some info that may be helpful for reviewers:

1. Forced trailing newlines in text parts: Python's modern email API forces trailing newlines on text/* content in message bodies and attachments (see email: set_content() always assumes trailing EOL python/cpython#121515). In practice, this probably doesn't matter much: most text/* content ends with a newline, and it generally doesn't hurt to add one.

   Unfortunately, a lot of our test cases used variations on body="Content" and assertEqual(generated_body, "Content") without trailing newlines. Modern email's behavior broke all of these tests. I've updated them to include trailing newlines (in both fixture and assertion).

   I'm open to suggestions on this. I did investigate some workarounds for handling the body encoding ourselves, but these either had other downsides (e.g., sending all text/* content with base64 content transfer encoding) or involved copying a big chunk of Python's raw_content_manager implementation into Django's code, which kind of goes against the spirit of this ticket.

2. MIMEBase attachments: I had originally proposed continuing support for MIMEBase attachments without deprecation. But it turns out modern email's email.message.MIMEPart is a good replacement, so this PR puts MIMEBase on the usual deprecation path. I've also added a docs example of using MIMEPart for inline images.

3. Deprecations and changes to undocumented behavior:

   • Implemented the deprecation plan from the original django-developers discussion.
   • Removed support for the undocumented EmailMessage.mixed_subtype and EmailMultiAlternatives.alternative_subtype attributes. (But it will raise an error on attempts to use them.)
   • Removed undocumented support for legacy email.charset.Charset objects in the undocumented EmailMessage.encoding property. (There was a single test that covered this; it's been removed.)
   • Otherwise preserved support for the undocumented EmailMessage.encoding property. I suggest renaming this to charset if we ever want to document it.

4. Changes to RFC 2047 encoded-words in non-ASCII headers:

   • Modern email uses a different RFC 2047 encoder than legacy. The results are semantically equivalent, but modern email tries to minimize the portion of the header using RFC 2047 encoding. This affected several test cases that were looking for a specific encoded-word in the generated message.
   • Modern email supports only the utf-8 charset for generating RFC 2047 encoded-words. With legacy email, Django used the DEFAULT_CHARSET setting (or the undocumented EmailMessage.encoding override) to set the encoded-word charset. The results are semantically equivalent, but this affected one test case. (In practice, it seems extremely unlikely there would be email software still in use that supports RFC 2047 encoding but doesn't understand utf-8.)
   • I dropped tests for applying RFC 2047 encoding to the localpart ("username") of a non-ASCII email address. RFC 2047 specifically prohibits this, and no known MTA or email client supports it. See ticket-25986#12 et seq. (Python will actually still incorrectly use an encoded-word here, but it's a Python bug that might get fixed someday, so I'm no longer testing for it.)

[adamchainz]

Is this meant to be against the django/django repo, rather than your fork? Or are you deliberately avoiding that until the security issue are fixed?

[medmunds]

@adamchainz this is meant to be against django#18502, which updates and adds missing tests to tests/mail prior to changing the mail implementation. (GitHub doesn't allow a PR based on another PR, unless you create a branch for it.)

[medmunds]

Incidentally, I could pretty easily be convinced to just drop the undocumented functions and classes (get rid of core/mail/_deprecated.py) rather than taking them through a deprecation period.

This might actually be a "more compatible" approach for SafeMIMEText and SafeMIMEMultipart, depending on how you read the documentation. Those are currently covered only incidentally as the return type of EmailMessage.message():

class EmailMessage
…

• message() constructs a django.core.mail.SafeMIMEText object (a subclass of Python’s MIMEText class) or a django.core.mail.SafeMIMEMultipart object holding the message to be sent. If you ever need to extend the EmailMessage class, you’ll probably want to override this method to put the content you want into the MIME object.

(There is no other mention of the SafeMIME classes in Django's documentation.)

With the changes in this PR, EmailMessage.message() now returns a Python email.message.EmailMessage, so it's not possible to be strictly compatible with both of these past claims:

1. "message() constructs a SafeMIMEText … or a SafeMIMEMultipart object" (and returns it).
2. "SafeMIMEText [is] a subclass of Python's MIMEText class." (Incidentally, SafeMIMEMultipart is not, and nothing is documented about its base class.)

The implementation currently in the PR preserves claim 2 for the SafeMIMEText class. (And keeps both SafeMIME class implementations available, deprecated, for code that used them without documentation.)

If we would prefer to preserve claim 1, we could instead define SafeMIMEText = SafeMIMEMultipart = email.message.EmailMessage and deprecate the SafeMIME names on import. (And get rid of the unnecessary SafeMIME implementations.)

[medmunds]

Moved to django#18966