Repository navigation
Should decode() offer best-effort per-label decoding?Β #248
Description
idna.decode() accepts a full multi-label domain and decodes it (e.g. idna.decode("www.xn--galit-femmes-hommes-9ybf.gouv.fr") -> "www.Γ©galitΓ©-femmes-hommes.gouv.fr"), which is great. But it is all-or-nothing: if a single label is invalid, the whole call raises, even for the labels that are fine.
>>> idna.decode("a.b.c.xn--pokxncvks")
idna.core.InvalidCodepoint: Codepoint U+3253 at position 1 of 'γγγγγγ' not allowedThere seems to be no option to get a best-effort result (uts46=True, strict=False don't change this on 3.13).
This matters for "decode for display" use cases (HTTP clients exposing a Unicode host). The relevant standards specify per-label processing that keeps an invalid xn-- label unchanged rather than failing the whole domain:
- WHATWG URL "domain to Unicode" returns the result and only signals validation
errors, never fails outright: https://url.spec.whatwg.org/#concept-domain-to-unicode - UTS Overly restrictive rules for KATAKANA MIDDLE DOTΒ #46 Β§4 Processing (Convert/Validate): on Punycode decode failure, "leave the label unchanged": https://www.unicode.org/reports/tr46/#Processing
- Go's
x/net/idnaimplements exactly this β comment literally says "Spec says keep the old label":
https://github.com/golang/net/blob/8c4c965e028475082408749b50ed7a686df0d265/idna/idna.go#L393-L401
Because decode() doesn't offer this, downstream projects need to re-implement the label-splitting themselves. Concretely, I got a PR in httpx2 (pydantic/httpx2#979) that implements the following:
labels = []
for label in host.split("."):
if label.startswith("xn--"):
try:
label = idna.decode(label)
except idna.IDNAError:
pass
labels.append(label)
host = ".".join(labels)Since decode() already accepts a whole domain and owns label splitting internally, the "keep invalid label raw" behavior arguably belongs here (maybe a flag? e.g. ignore_invalid_punycode=True, mirroring UTS #46's IgnoreInvalidPunycode) rather than being re-implemented in every consumer.
Would the maintainers be open to such an option? Or is all-or-nothing the intended contract and per-label recovery deliberately left to callers?
AI Disclaimer
This issue was created with the assistance of Claude Code. I've edited accordingly.
Activity
[-]Should decode() offer best-effort per-label decoding (keep invalid xn-- labels raw)?[/-][+]Should decode() offer best-effort per-label decoding?[/+][-]Should decode() offer best-effort per-label decoding?[/-][+]Should `decode()` offer best-effort per-label decoding?[/+]kjd commentedon May 24, 2026
Seems reasonable to consider. I think the major consideration is whether overloading the
strictargument with this behavior is appropriate or whether another argument is warranted. There should be a protocol-conformant path available and should think about those relying on invoking the library to check for protocol conformance shouldn't be surprised.Kludex commentedon May 24, 2026
What's the path forward? Should I bring a more concrete proposal?
kjd commentedon May 25, 2026
I have put together a potential implementation in the lenient-decode branch. I am not enamored with the API but it was the best I could think of so far.
Kludex commentedon May 26, 2026
What's wrong about it? The parameter name?
kjd commentedon May 27, 2026
I think my unease is that
strictcurrently applies to one dimension of strictness, whereaslenientapplies to a different dimension. I thinkignore_invalid_punycodeyour proposed is too verbose, plus it is not really being ignored it is being passed through. Maybe something about the use case, i.e. for display purposes, is the angle to down here. While the functionality makes sense I don't have a clear view on how to present the interface.Add lenient option to decode() for best-effort label recovery (issue #β¦
Kludex commentedon Jun 1, 2026
It seems more maintainable to add a parameter, not sure what the name should be.
On my side I have this PR that I want to act on: pydantic/httpx2#979 - but I don't want to merge it as is. Would it be okay if I vendor your approach on 47b5cde while you think about the best appraoch here?
Rename decode() lenient= option to display= (issue #248)
kjd commentedon Jun 2, 2026
Renamed the argument to
displayand have merged it into master, will make a new release shortly with the functionality included.Kludex commentedon Jun 2, 2026
Thanks! π