Improve handling of non-ASCII characters in message headers

Improve Anymail's normalized EmailAddress object:
- Stop using Django's undocumented, deprecated sanitize_address() helper
- Add ANYMAIL["IDNA_ENCODER"] setting, defaulting to idna2008 [breaking]
  - Implement a few useful IDNA_ENCODER presets
  - Add idna package as direct dependency (already sub-dependency of 'requests')
  - Add uts46 package optional dependency
- Add base backend idna_encode method for subclass use
- Subclass EmailAddress from Python's modern email.headerregistry.Address object
- Add EmailAddress.as_dict() and formatting options useful for various ESPs
- Add utils to apply/reverse RFC 2047 encoded-word and RFC 5322 quoted-string
  formatting

Update each ESP's EmailBackend to use the appropriate EmailAddress and header
encodings, based on testing its API's behavior for Unicode characters.

Add some ESP-specific unsupported feature errors to prevent particularly
problematic Unicode handling:
- Brevo: Prevent non-ASCII values in custom headers to avoid raw, 8-bit
  utf-8 (also affects metadata, which uses a custom header)
- Mailgun: Prevent EAI in from_email (API accepts EAI, but generated
  message is undeliverable)
- Scaleway: Prevent EAI in any address field (API accepts EAI, but
  generated message is undeliverable)

Documentation:
- Add a new "International email" page in the advanced section, with
  general details on Unicode handling and the new IDNA_ENCODER setting
- Update each ESP's page to indicate whether it handles EAI
- Document some other ESP-specific Unicode quirks uncovered during testing

Closes #444

Author: medmunds

CHANGELOG.rst

@@ -25,6 +25,66 @@ Release history
 ^^^^^^^^^^^^^^^
     ..  This extra heading level keeps the ToC from becoming unmanageably long
 
+vNext
+-----
+
+*Unreleased changes*
+
+This release improves handling of non-ASCII characters everywhere email messages
+allow them, based on extensive testing of Unicode handling for all supported
+ESPs. There are several new workarounds for ESP bugs and a handful of new
+errors to help you avoid bugs that don't have workarounds. See
+`International email <https://anymail.dev/en/latest/tips/international_email/#idna>`_
+in the docs for more information.
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+* **International domain names:** When sending email to a non-ASCII domain name,
+  use IDNA 2008 with UTS-46 pre-processing rather than obsolete IDNA 2003
+  encoding. This ensures email can be sent to newer domains enabled by IDNA 2008.
+
+  This change should make no difference for virtually all real-world email
+  addresses that worked with earlier Anymail releases. But trying to send to
+  emoji domains or others no longer allowed by IDNA 2008 will now raise an
+  ``AnymailInvalidAddress`` error.
+
+  To restore the old behavior or select a different encoding, use the new
+  ``IDNA_ENCODER`` setting. See
+  `Domains (IDNA) <https://anymail.dev/en/latest/tips/international_email/#idna>`_
+  in the docs.
+
+  As part of this change, Anymail now has a direct dependency on the ``idna``
+  package. (It was already being installed as a sub-dependency of ``requests``.)
+
+* **Brevo:** Raise an error if metadata or custom header values include non-ASCII
+  characters. This avoids a Brevo API bug that sends unencoded 8-bit headers,
+  which can cause bounces or dropped messages.
+
+* **Mailgun:** Raise an error if the ``from_email`` uses EAI (has a non-ASCII
+  local part). This avoids a Mailgun API bug that generates undeliverable
+  messages.
+
+* **Scaleway TEM:** Raise an error if any address field uses EAI (has a non-ASCII
+  local part). This avoids a Scaleway API bug that generates undeliverable messages.
+
+Fixes
+~~~~~
+
+* **Brevo:** Work around a Brevo API bug which loses non-ASCII display names
+  that also contain a comma or certain other punctuation.
+
+Other
+~~~~~
+
+* **Mandrill:** Document a Mandrill API bug that can cause an address with a
+  non-ASCII display name to display incorrectly in some email clients.
+
+* **Unisender Go:** Document a Unisender Go API bug that can cause an Reply-To
+  address (only) with a non-ASCII display name to display incorrectly in some
+  email clients.
+
+
 v13.1
 -----
 

anymail/_idna.py

@@ -0,0 +1,53 @@
+# Pre-packaged IDNA_ENCODER options
+import idna
+
+from anymail.exceptions import AnymailImproperlyInstalled, _LazyError
+
+try:
+    from uts46 import encode as uts46_encode
+except ImportError:
+    uts46_encode = _LazyError(
+        AnymailImproperlyInstalled("uts46", "<your-esp>,uts46", 'IDNA_ENCODER="uts46"')
+    )
+
+__all__ = ["idna2003", "idna2008", "uts46", "none"]
+
+
+def idna2003(domain: str) -> str:
+    """
+    Encode domain (if necessary) using IDNA 2003 standard.
+    This matches Django's own behavior and requires no extra libraries.
+    But it will fail to encode some newer IDNs that require IDNA 2008, and
+    it will use obsolete encoding for IDNs that contain deviation characters.
+    """
+    return domain.encode("idna").decode("ascii")
+
+
+def idna2008(domain: str) -> str:
+    """
+    Encode domain (if necessary) using the IDNA 2008 standard
+    with UTS46 preprocessing. (Preprocessing is required to handle
+    case insensitivity most users would expect.)
+
+    Will reject some domains (e.g., emojis) that browsers allow.
+    Relies on the third-party 'idna' package (installed with django-anymail).
+    """
+    return idna.encode(domain, uts46=True).decode("ascii")
+
+
+def uts46(domain: str) -> str:
+    """
+    Encode domain (if necessary) using the UTS46 standard.
+    This is the encoding used by all modern browsers.
+
+    Requires the 'uts46' package (installable via 'django-anymail[uts46]' extra).
+    """
+    return uts46_encode(domain).decode("ascii")
+
+
+def none(domain: str) -> str:
+    """
+    Leaves domain as unencoded Unicode characters.
+    Can be used with an ESP whose API correctly handles IDNA encoding.
+    """
+    return domain

anymail/backends/amazon_ses.py

@@ -7,7 +7,7 @@
 from .. import __version__ as ANYMAIL_VERSION
 from ..exceptions import AnymailAPIError, AnymailImproperlyInstalled
 from ..message import AnymailRecipientStatus
-from ..utils import UNSET, get_anymail_setting
+from ..utils import UNSET, get_anymail_setting, parse_address_list
 from .base import AnymailBaseBackend, BasePayload
 
 try:
@@ -153,7 +153,30 @@ class AmazonSESV2SendEmailPayload(AmazonSESBasePayload):
     def init_payload(self):
         super().init_payload()
         self.all_recipients = []  # for parse_recipient_status
-        self.mime_message = self.message.message()
+
+        # Temporarily replace the address fields on self.message with
+        # pre-IDNA-encoded versions, only while we're converting it
+        # to a MIME message. (We don't own self.message, so should not
+        # permanently modify it.)
+        address_fields = {"from_email", "to", "cc", "bcc", "reply_to"}
+        original_values = {
+            field: getattr(self.message, field) for field in address_fields
+        }
+        try:
+            for field in address_fields:
+                addresses = getattr(self.message, field)
+                idna_encoded_addresses = [
+                    address.format(idna_encode=self.backend.idna_encode)
+                    for address in parse_address_list(addresses, field)
+                ]
+                if field == "from_email":
+                    # from_email is a single string; all others are lists
+                    idna_encoded_addresses = ", ".join(idna_encoded_addresses)
+                setattr(self.message, field, idna_encoded_addresses)
+            self.mime_message = self.message.message()
+        finally:
+            for field, original_value in original_values.items():
+                setattr(self.message, field, original_value)
 
     def finalize_payload(self):
         # (The boto3 SES client handles base64 encoding raw_message.)
@@ -232,7 +255,8 @@ def set_recipients(self, recipient_type, emails):
             assert recipient_type in ("to", "cc", "bcc")
             destination_key = f"{recipient_type.capitalize()}Addresses"
             self.params.setdefault("Destination", {})[destination_key] = [
-                email.address for email in emails
+                email.format_addr_spec(idna_encode=self.backend.idna_encode)
+                for email in emails
             ]
 
     def set_subject(self, subject):
@@ -355,18 +379,28 @@ def finalize_payload(self):
         cc_and_bcc_addresses = {}
         if self.recipients["cc"]:
             cc_and_bcc_addresses["CcAddresses"] = [
-                cc.address for cc in self.recipients["cc"]
+                cc.format(use_rfc2047=True, idna_encode=self.backend.idna_encode)
+                for cc in self.recipients["cc"]
             ]
         if self.recipients["bcc"]:
             cc_and_bcc_addresses["BccAddresses"] = [
-                bcc.address for bcc in self.recipients["bcc"]
+                # (display-name is not relevant for bcc recipients)
+                bcc.format_addr_spec(idna_encode=self.backend.idna_encode)
+                for bcc in self.recipients["bcc"]
             ]
 
         # Construct an entry with merge data for each "to" recipient:
         self.params["BulkEmailEntries"] = []
         for to in self.recipients["to"]:
             entry = {
-                "Destination": dict(ToAddresses=[to.address], **cc_and_bcc_addresses),
+                "Destination": dict(
+                    ToAddresses=[
+                        to.format(
+                            use_rfc2047=True, idna_encode=self.backend.idna_encode
+                        )
+                    ],
+                    **cc_and_bcc_addresses,
+                ),
                 "ReplacementEmailContent": {
                     "ReplacementTemplate": {
                         "ReplacementTemplateData": self.serialize_json(
@@ -447,8 +481,9 @@ def parse_recipient_status(self, response):
         return dict(zip(to_addrs, anymail_statuses))
 
     def set_from_email(self, email):
-        # this will RFC2047-encode display_name if needed:
-        self.params["FromEmailAddress"] = email.address
+        self.params["FromEmailAddress"] = email.format(
+            use_rfc2047=True, idna_encode=self.backend.idna_encode
+        )
 
     def set_recipients(self, recipient_type, emails):
         # late-bound in finalize_payload
@@ -462,7 +497,10 @@ def set_subject(self, subject):
 
     def set_reply_to(self, emails):
         if emails:
-            self.params["ReplyToAddresses"] = [email.address for email in emails]
+            self.params["ReplyToAddresses"] = [
+                email.format(use_rfc2047=True, idna_encode=self.backend.idna_encode)
+                for email in emails
+            ]
 
     def set_extra_headers(self, headers):
         self.headers = headers

anymail/backends/base.py

@@ -3,12 +3,15 @@
 
 from django.conf import settings
 from django.core.mail.backends.base import BaseEmailBackend
+from django.utils.module_loading import import_string
 from django.utils.timezone import get_current_timezone, is_naive, make_aware
 from requests.structures import CaseInsensitiveDict
 
 from ..exceptions import (
     AnymailCancelSend,
+    AnymailConfigurationError,
     AnymailError,
+    AnymailInvalidAddress,
     AnymailRecipientsRefused,
     AnymailSerializationError,
     AnymailUnsupportedFeature,
@@ -32,6 +35,8 @@
     parse_single_address,
 )
 
+DEFAULT_IDNA_ENCODER = "idna2008"
+
 
 class AnymailBaseBackend(BaseEmailBackend):
     """
@@ -63,6 +68,26 @@ def __init__(self, *args, **kwargs):
             send_defaults.update(esp_send_defaults)
         self.send_defaults = send_defaults
 
+        # Initialize self._idna_encoder from IDNA_ENCODER setting
+        # (optionally ESP-specific)
+        self.idna_encoder = get_anymail_setting(
+            "idna_encoder", esp_name=self.esp_name, kwargs=kwargs, default=None
+        ) or get_anymail_setting("idna_encoder", default=DEFAULT_IDNA_ENCODER)
+        if callable(self.idna_encoder):
+            self._idna_encoder = self.idna_encoder
+        else:
+            try:
+                dotted_path = (
+                    self.idna_encoder
+                    if "." in self.idna_encoder
+                    else f"anymail._idna.{self.idna_encoder}"  # built-ins
+                )
+                self._idna_encoder = import_string(dotted_path)
+            except (ImportError, TypeError) as error:
+                raise AnymailConfigurationError(
+                    f"cannot resolve IDNA_ENCODER={self.idna_encoder!r}"
+                ) from error
+
     def open(self):
         """
         Open and persist a connection to the ESP's API, and whether
@@ -157,6 +182,15 @@ def _send(self, message):
 
         return True
 
+    def idna_encode(self, domain):
+        try:
+            return self._idna_encoder(domain)
+        except ValueError as error:
+            # ValueError includes UnicodeError, idna.IDNAError, uts46.UTS46Error, etc.
+            raise AnymailInvalidAddress(
+                f"Cannot encode {domain!r} using IDNA_ENCODER={self.idna_encoder!r}"
+            ) from error
+
     def run_pre_send(self, message):
         """Send pre_send signal, and return True if message should still be sent"""
         try: