[Shimada666]

Problem

Commit 7835ef7 increased the CJK uncommon ratio from /10 to /5. For valid, meaningful Chinese text containing characters outside the limited common-character set, this can push UTF-8 chaos above the 10% fast-path threshold and allow single-byte encodings such as PTCP154 to rank first.

Minimal reproduction:

{"title":"海岛旅居会员忠诚动因持续探索"}

• 3.4.9 detects utf-8.
• 3.5.0 and 3.5.1 detect ptcp154.

Fix

• Restore the CJK uncommon ratio divisor to /10 in both the pure Python and Cython implementations.
• Keep SuspiciousKatakanaPlugin and the other detection improvements from 7835ef7 unchanged.
• Add a sanitized regression test through the public detect(bytes) API.

Validation

• Pure Python test suite: 198 passed.
• Cython test suite: 198 passed.
• Lint: passed.
• Backward compatibility: 98.0% (467/477).
• Dataset coverage: 97.904% (467/477).

[Ousret]

I tested it and it does regress other cases.
The CI itself report a few loss in coverage.

After running a couple of representative cases, I landed the following compromise:

if self._character_count < 4:
    return 0.0

uncommon_form_usage: float = (
    (2 * self._uncommon_count - self._character_count)
    / (5 * max(self._character_count, 16))
)

return max(0.0, uncommon_form_usage)

regards,

[Shimada666]

Thanks for testing this and suggesting the compromise, @Ousret! I've applied your formula in 5adc843 to both the Python and Cython implementations, keeping the UTF-8 Chinese JSON regression test.

I rechecked the short-input guard, the 50% boundary, the minimum denominator of 16, and the zero clamp in both implementations. Local validation passed:

• Pure Python (3.11) and compiled Cython (3.12): 297 tests and 11 subtests passed each.
• All pre-commit checks passed.
• Detection coverage: 98.323% (469/477) with preemptive detection, and 96.855% (462/477) without it.

The original JSON example still detects as UTF-8. Thanks again for helping preserve the other detection cases — could you take another look when you have a chance?

[Ousret]

LGTM. The compromise seems to work as intended.

regards,