Repository navigation
Make the IANA_NO_ALIASES encodings resolvable by name - #800
Merged
Merged
Conversation
IANA_SUPPORTED folds in IANA_NO_ALIASES, but _IANA_NAMES is built only from encodings.aliases. Those names are in IANA_NO_ALIASES precisely because they have no alias entry, so seven of the nine never reach _IANA_NAMES: cp720, cp737, cp856, cp875, cp1006, koi8_t, koi8_u. (cp874 and koi8_r resolve only because they also appear in aliases.values().) iana_name() therefore raises for an encoding the package lists as supported, any_specified_encoding() ignores a declared charset of one of them, and CharsetMatch.output() raises when the target is one of them.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
IANA_SUPPORTEDfolds inIANA_NO_ALIASES, but_IANA_NAMESis built only fromencodings.aliases. Those nine names are inIANA_NO_ALIASESbecause they have no alias entry, so seven of them never reach_IANA_NAMESβcp720,cp737,cp856,cp875,cp1006,koi8_t,koi8_u.cp874andkoi8_rresolve only because they happen to also appear inaliases.values(), which is why this has been quiet.koi8_rvskoi8_uis the whole bug in one pair:Three consequences, all measured on
master(e239bdc):any_specified_encodingon a page declaring itkoi8_rNonefrom_bytes(...).best().encodingfor that pagekoi8_rshift_jis_2004.output('utf_8')utf-8koi8-u.output('koi8_u')ValueErrorSo a declared charset is ignored, the detector then picks the wrong encoding, and the re-encoded payload keeps a
<meta charset="koi8-u">that is no longer true. #328 added these nine to make them supported and updateddocs/user/support.rstaccordingly; only the lookup table was left behind.What I ran. Same script on
masterand on the patch:iana_name()over all ofIANA_NO_ALIASESβ 2/9 resolvable before, 9/9 after; the declared-charset round trip above forkoi8_r/koi8_u/cp1251; and.output()to four Cyrillic encodings.nox -s test(as CONTRIBUTING asks): thetest-3.14session gives 197 passed onmasterand 297 passed here β the added test is parametrised overIANA_SUPPORTED, which is 100 entries. Reverting onlyconstant.pyand keeping the test fails it exactly 7 times, one per unresolvable name, so it isn't vacuous. Fullnox -s testis green on 3.9β3.14 and pypy (3.7/3.8/3.15 skipped, interpreters not installed here).nox -s lintpasses every hook β ruff, ruff format, mypy, cython-lint, codespell, zizmor.The alternative I rejected, and it also passes. Building the self-map from
IANA_SUPPORTEDinstead is a one-liner that passes every check above and the full suite. It also silently drops eight names_IANA_NAMESresolves today βrot_13,mbcs,base64_codec,bz2_codec,hex_codec,quopri_codec,uu_codec,zlib_codecβ becauseIANA_SUPPORTEDfilters those out. AddingIANA_NO_ALIASESkeeps the change additive.Not addressed:
CHARDET_CORRESPONDENCEkeys offiso8859_1, which is not inIANA_SUPPORTED(Python's canonical name islatin_1), so that entry looks unreachable too β separate from this and left alone.AI assistance: found and patched with Claude Code (Claude Opus 5,
claude-opus-5); I reviewed and ran everything above myself.