Repository navigation
Conversation
Some locales (Italian and 20+ others) reuse the same abbreviation for a weekday and a month, e.g. Italian "mar" is both "martedì" (Tuesday) and "marzo" (March). Dictionary construction always resolved this in favor of the month since months are listed after weekdays in KNOWN_WORD_TOKENS, so strings like "mar, 07 giu 2022 08:56:47 +0200" translated to "march, 07 june 2022..." and failed to parse entirely, with or without an explicit date_formats. Track these conflicts in Dictionary.weekday_month_conflicts and, in Locale.translate(), resolve a leading ambiguous token followed by a comma (the common "Weekday, day Month Year" pattern) as the weekday. Fixes scrapinghub#1061 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1476 +/- ##
==========================================
+ Coverage 97.21% 97.23% +0.01%
==========================================
Files 236 236
Lines 3090 3110 +20
==========================================
+ Hits 3004 3024 +20
Misses 86 86 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
serhii73
left a comment
There was a problem hiding this comment.
Thanks for this, mar, 07 giu 2022 ... parses correctly for me now. I have one concern about the rule being too broad, details inline.
| forced_weekday_token = ( | ||
| leading_weekday_match.group(1).lower() if leading_weekday_match else None | ||
| ) | ||
| if forced_weekday_token not in dictionary.weekday_month_conflicts: |
There was a problem hiding this comment.
This fires for any ambiguous token followed by a comma, even when nothing else in the string names a month, so it also flips real months to weekdays. For example parse("mar, 2022", languages=["it"]) was March 2022 before and now becomes the current month with the year kept, and mar, 7 2022 goes from 2022-03-07 to 2022-05-07 (I ran it with RELATIVE_BASE in May). The same shape changes in es, ro, gl, sq, uk (сер) and the other locales in weekday_month_conflicts; nothing errors, the month is just silently replaced.
Would it work to take the weekday reading only when another token in the string translates to a month (and keep the month reading otherwise)? That still fixes mar, 07 giu 2022 and leaves mar, 2022 as March. A test for the month-only case would be good to have next to the new one.
…biguous month present
Address review feedback on the previous commit: resolving any ambiguous
weekday/month token followed by a comma as a weekday was too broad and
flipped genuine month-only usages (e.g. Italian "mar, 2022" from March
2022 to the current month, "mar, 7 2022" from March 7 to a different
month/day), and still failed to parse the same string without a comma
("mar 07 giu 2022 ...").
Instead, only resolve an ambiguous token as a weekday when another,
unambiguous month is present elsewhere in the same string - a date
string only needs one month, so any other month-shaped token in that
case must be something else. Rename the new Dictionary members with a
leading underscore since they are not public API.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Thanks for the feedback, both — good catch! Pushed a new commit that fixes it. Now an ambiguous token (like Italian Also renamed the new One thing I didn't try to fix: |
AdrianAtZyte
left a comment
There was a problem hiding this comment.
Thanks, this looks good to me now. Please, also prefix
is_unambiguous_monthwith_, and drop the “as it was before Issue #1061 was fixed…” part of thetest_weekday_month_abbreviation_conflict_is_read_as_monthdocstring.
Small follow-up nits from review: rename Dictionary.is_unambiguous_month to _is_unambiguous_month, since it's internal API like the other new members, and drop the "as it was before Issue scrapinghub#1061 was fixed" clause from test_weekday_month_abbreviation_conflict_is_read_as_month's docstring. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Thanks, done! Renamed |
Summary
dateparser.parse('mar, 07 giu 2022 08:56:47 +0200')(Italian) returnedNone, with or without an explicitdate_formats.maris bothmartedì(Tuesday) andmarzo(March).Dictionary.__init__always resolved this in favor of the month, since months are listed after weekdays inKNOWN_WORD_TOKENS, so the ambiguous token was silently mistranslated whenever it should have meant a weekday.Dictionarynow tracks these conflicts inweekday_month_conflicts.Locale.translate()uses this to recognize the common"Weekday, day Month Year"pattern (an ambiguous token immediately followed by a comma at the start of the string) and resolves it as the weekday in that case, leaving all other occurrences (e.g."mar 2022") resolving as the month as before.Test plan
test_leading_weekday_month_abbreviation_conflict) covering both thedate_formats-based and format-free reproduction from the issue.pytest tests/— 24269 passed, 6 skipped, 1 xfailed, no regressions.ruff check/ruff format --checkon changed files — clean.🤖 Generated with Claude Code