Skip to content

Fix weekday/month abbreviation conflicts in translation (#1061) - #1476

Merged
serhii73 merged 3 commits into
scrapinghub:masterfrom
yamilmaud:fix_DC-1311_Italian_date_with_format_is_not_working
Sep 30, 2026
Merged

serhii73 merged 3 commits into
scrapinghub:masterfrom
yamilmaud:fix_DC-1311_Italian_date_with_format_is_not_working

Conversation

@yamilmaud

Copy link
Copy Markdown
Contributor

Summary

  • Fixes Italian date with format "%a, %d %b %Y %H:%M:%S %z" is not working #1061: dateparser.parse('mar, 07 giu 2022 08:56:47 +0200') (Italian) returned None, with or without an explicit date_formats.
  • Root cause: several locales (Italian and 20+ others, e.g. es, gl, ro, sq) reuse the same abbreviation for a weekday and a month — Italian mar is both martedì (Tuesday) and marzo (March). Dictionary.__init__ always resolved this in favor of the month, since months are listed after weekdays in KNOWN_WORD_TOKENS, so the ambiguous token was silently mistranslated whenever it should have meant a weekday.
  • Fix: Dictionary now tracks these conflicts in weekday_month_conflicts. Locale.translate() uses this to recognize the common "Weekday, day Month Year" pattern (an ambiguous token immediately followed by a comma at the start of the string) and resolves it as the weekday in that case, leaving all other occurrences (e.g. "mar 2022") resolving as the month as before.

Test plan

  • Added a regression test (test_leading_weekday_month_abbreviation_conflict) covering both the date_formats-based and format-free reproduction from the issue.
  • pytest tests/ — 24269 passed, 6 skipped, 1 xfailed, no regressions.
  • ruff check / ruff format --check on changed files — clean.

🤖 Generated with Claude Code

Some locales (Italian and 20+ others) reuse the same abbreviation for
a weekday and a month, e.g. Italian "mar" is both "martedì" (Tuesday)
and "marzo" (March). Dictionary construction always resolved this in
favor of the month since months are listed after weekdays in
KNOWN_WORD_TOKENS, so strings like "mar, 07 giu 2022 08:56:47 +0200"
translated to "march, 07 june 2022..." and failed to parse entirely,
with or without an explicit date_formats.

Track these conflicts in Dictionary.weekday_month_conflicts and, in
Locale.translate(), resolve a leading ambiguous token followed by a
comma (the common "Weekday, day Month Year" pattern) as the weekday.

Fixes scrapinghub#1061

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@codspeed

codspeed Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 7 untouched benchmarks


Comparing yamilmaud:fix_DC-1311_Italian_date_with_format_is_not_working (f190940) with master (3ec5fe3)

Open in CodSpeed

@codecov

codecov Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.23%. Comparing base (ef140a8) to head (f190940).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1476      +/-   ##
==========================================
+ Coverage   97.21%   97.23%   +0.01%     
==========================================
  Files         236      236              
  Lines        3090     3110      +20     
==========================================
+ Hits         3004     3024      +20     
  Misses         86       86              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdrianAtZyte

Copy link
Copy Markdown
Member

Thanks! I think the heuristic is too broad, though: with this change, parse("mar, 2022", languages=["it"]) goes from March 2022 to 2022-09-29, and "mar, 10, 2022" goes from March 10 to September 10. Meanwhile "mar 07 giu 2022" (no comma) still returns None.

I wonder if a better signal would be the presence of another month in the same string: if the string already has an unambiguous month (giu), the ambiguous token can only be the weekday, comma or not. That should fix the issue cases without changing how mar, 2022 is read.

Please, also add tests for the cases that must keep reading mar as March (e.g. mar 2022, mar, 2022), and prefix the new names (weekday_month_conflicts, WEEKDAY_TOKENS, MONTH_TOKENS, RE_LEADING_WORD_BEFORE_COMMA) with _, since they are not meant to be public API.

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, mar, 07 giu 2022 ... parses correctly for me now. I have one concern about the rule being too broad, details inline.

Comment thread dateparser/languages/locale.py Outdated
forced_weekday_token = (
leading_weekday_match.group(1).lower() if leading_weekday_match else None
)
if forced_weekday_token not in dictionary.weekday_month_conflicts:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fires for any ambiguous token followed by a comma, even when nothing else in the string names a month, so it also flips real months to weekdays. For example parse("mar, 2022", languages=["it"]) was March 2022 before and now becomes the current month with the year kept, and mar, 7 2022 goes from 2022-03-07 to 2022-05-07 (I ran it with RELATIVE_BASE in May). The same shape changes in es, ro, gl, sq, uk (сер) and the other locales in weekday_month_conflicts; nothing errors, the month is just silently replaced.

Would it work to take the weekday reading only when another token in the string translates to a month (and keep the month reading otherwise)? That still fixes mar, 07 giu 2022 and leaves mar, 2022 as March. A test for the month-only case would be good to have next to the new one.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds worth trying.

…biguous month present

Address review feedback on the previous commit: resolving any ambiguous
weekday/month token followed by a comma as a weekday was too broad and
flipped genuine month-only usages (e.g. Italian "mar, 2022" from March
2022 to the current month, "mar, 7 2022" from March 7 to a different
month/day), and still failed to parse the same string without a comma
("mar 07 giu 2022 ...").

Instead, only resolve an ambiguous token as a weekday when another,
unambiguous month is present elsewhere in the same string - a date
string only needs one month, so any other month-shaped token in that
case must be something else. Rename the new Dictionary members with a
leading underscore since they are not public API.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@yamilmaud

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback, both — good catch! Pushed a new commit that fixes it.

Now an ambiguous token (like Italian mar) is only read as a weekday when there's already another, unambiguous month elsewhere in the string (e.g. giu). If not, it still means the month, just like before. So mar, 2022, mar, 7 2022, and mar, 10, 2022 all stay correct now, and as a bonus it also fixes the no-comma case (mar 07 giu 2022...) that was still broken before.

Also renamed the new Dictionary fields with a leading underscore since they're internal, and added tests for all of the above.

One thing I didn't try to fix: mar 15 mar 2022 (same ambiguous word meant two different ways) still returns None — but that was already true on master, so no new regression there, just a pre-existing edge case.

@AdrianAtZyte AdrianAtZyte left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks good to me now. Please, also prefix is_unambiguous_month with _, and drop the “as it was before Issue #1061 was fixed…” part of the test_weekday_month_abbreviation_conflict_is_read_as_month docstring.

Small follow-up nits from review: rename Dictionary.is_unambiguous_month
to _is_unambiguous_month, since it's internal API like the other new
members, and drop the "as it was before Issue scrapinghub#1061 was fixed" clause
from test_weekday_month_abbreviation_conflict_is_read_as_month's
docstring.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@yamilmaud

Copy link
Copy Markdown
Contributor Author

Thanks, done! Renamed is_unambiguous_month → _is_unambiguous_month, and trimmed the docstring on test_weekday_month_abbreviation_conflict_is_read_as_month. Pushing now.

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@serhii73
serhii73 merged commit 2ab3588 into scrapinghub:master Sep 30, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Italian date with format "%a, %d %b %Y %H:%M:%S %z" is not working

3 participants