Skip to content

Make the IANA_NO_ALIASES encodings resolvable by name - #800

Merged
Ousret merged 1 commit into
jawah:masterfrom
dylanpulver:resolve-iana-no-aliases
Sep 7, 2026
Merged

Ousret merged 1 commit into
jawah:masterfrom
dylanpulver:resolve-iana-no-aliases

Conversation

@dylanpulver

@dylanpulver dylanpulver commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

IANA_SUPPORTED folds in IANA_NO_ALIASES, but _IANA_NAMES is built only from encodings.aliases. Those nine names are in IANA_NO_ALIASES because they have no alias entry, so seven of them never reach _IANA_NAMES β€” cp720, cp737, cp856, cp875, cp1006, koi8_t, koi8_u. cp874 and koi8_r resolve only because they happen to also appear in aliases.values(), which is why this has been quiet.

koi8_r vs koi8_u is the whole bug in one pair:

koi8_r  in IANA_SUPPORTED: True   iana_name('koi8_r') -> 'koi8_r'
koi8_u  in IANA_SUPPORTED: True   iana_name('koi8_u') -> ValueError: Unable to retrieve IANA for 'koi8_u'

Three consequences, all measured on master (e239bdc):

koi8_r koi8_u
any_specified_encoding on a page declaring it koi8_r None
from_bytes(...).best().encoding for that page koi8_r shift_jis_2004
header after .output('utf_8') rewritten to utf-8 left saying koi8-u
.output('koi8_u') β€” ValueError

So a declared charset is ignored, the detector then picks the wrong encoding, and the re-encoded payload keeps a <meta charset="koi8-u"> that is no longer true. #328 added these nine to make them supported and updated docs/user/support.rst accordingly; only the lookup table was left behind.

What I ran. Same script on master and on the patch: iana_name() over all of IANA_NO_ALIASES β€” 2/9 resolvable before, 9/9 after; the declared-charset round trip above for koi8_r/koi8_u/cp1251; and .output() to four Cyrillic encodings. nox -s test (as CONTRIBUTING asks): the test-3.14 session gives 197 passed on master and 297 passed here β€” the added test is parametrised over IANA_SUPPORTED, which is 100 entries. Reverting only constant.py and keeping the test fails it exactly 7 times, one per unresolvable name, so it isn't vacuous. Full nox -s test is green on 3.9–3.14 and pypy (3.7/3.8/3.15 skipped, interpreters not installed here). nox -s lint passes every hook β€” ruff, ruff format, mypy, cython-lint, codespell, zizmor.

The alternative I rejected, and it also passes. Building the self-map from IANA_SUPPORTED instead is a one-liner that passes every check above and the full suite. It also silently drops eight names _IANA_NAMES resolves today β€” rot_13, mbcs, base64_codec, bz2_codec, hex_codec, quopri_codec, uu_codec, zlib_codec β€” because IANA_SUPPORTED filters those out. Adding IANA_NO_ALIASES keeps the change additive.

Not addressed: CHARDET_CORRESPONDENCE keys off iso8859_1, which is not in IANA_SUPPORTED (Python's canonical name is latin_1), so that entry looks unreachable too β€” separate from this and left alone.

AI assistance: found and patched with Claude Code (Claude Opus 5, claude-opus-5); I reviewed and ran everything above myself.

IANA_SUPPORTED folds in IANA_NO_ALIASES, but _IANA_NAMES is built only
from encodings.aliases. Those names are in IANA_NO_ALIASES precisely
because they have no alias entry, so seven of the nine never reach
_IANA_NAMES: cp720, cp737, cp856, cp875, cp1006, koi8_t, koi8_u.
(cp874 and koi8_r resolve only because they also appear in
aliases.values().)

iana_name() therefore raises for an encoding the package lists as
supported, any_specified_encoding() ignores a declared charset of one of
them, and CharsetMatch.output() raises when the target is one of them.
@dylanpulver
dylanpulver requested a review from Ousret as a code owner September 2, 2026 11:47

@Ousret Ousret left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Ousret
Ousret merged commit f6afd31 into jawah:master Sep 7, 2026
49 checks passed
@Ousret Ousret mentioned this pull request Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants