Skip to content

Add a STRICT_DATE_ORDER setting to enforce the year position or the whole DATE_ORDER (#868) - #1443

Merged
serhii73 merged 9 commits into
masterfrom
fix-868-invalid-day-reordered-as-year
Oct 2, 2026
Merged

serhii73 merged 9 commits into
masterfrom
fix-868-invalid-day-reordered-as-year

Conversation

@serhii73

@serhii73 serhii73 commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #701, part of #868 (the DATE_ORDER half, opt-in through a new setting. The date_formats half is left for a separate discussion in #868).

With DATE_ORDER set as shown:

parse(...) "none" (default) "year" "all"
"32 DEC 10", DMY 2032-12-10 None None
"10 DEC 32", MYD 2032-12-10 None None
"13/12/10", MDY 2010-12-13 2010-12-13 None
"10 DEC 95", DMY 1995-12-10 1995-12-10 1995-12-10

The setting

STRICT_DATE_ORDER is a string. "none" keeps DATE_ORDER a preference, as on master. "year" reads a two-digit year only from the position DATE_ORDER gives the year. "all" also keeps the day and month where the order puts them, so no swap (#1431).

"year" and "all" raise a SettingValidationError if DATE_ORDER is not passed: the default one is only a preference, so there is nothing to enforce. search_dates and a reused DateDataParser are not affected. With the default PARSERS, parse() can only turn a date into None. A rejection is final across locales (ru would read "20 09 29" with MYD as a 2029 date) and skips the no-spaces parser. Only parts read from numbers are checked: "DEC 32" with DMY is still December 2032.

Stricter results under the opt-in

With "year", besides the table: an invalid month ("12/13/10" with DMY), CJK dates under an order that does not start with the year ("95年12月10日" with MDY, documented), rare orders ("40 12" with DYM), and, under YMD, YDM, MYD and DYM, a two-digit year in a position the order does not give ("DEC 10 95" with YMD, #519's "4-99" with YMD).

Known gaps

  • date_formats, the other half of Given a month with 32 days, dateparser fails with some years but parses with others #868.
  • search_dates without RELATIVE_BASE checks only the first date (Settings.replace() drops the explicit settings, a separate bug). A rejected span is dropped or split differently, and the dates after it resolve against the previous one, so they can change too.
  • The no-spaces parser still returns a time of day for a time-only match, as on master. With a custom PARSERS that runs absolute-time first, a rejected reading can be replaced by the next parser's.
  • Under MYD, a month name is not counted, so a day before the year with the month name after it ("10 95 DEC") is rejected by "year" too, although the year is in place.
  • Add parse_many() to infer a shared date order #1438 also adds a STRICT_DATE_ORDER, as a bool: to agree before either lands.

Tests

127 tests; the full suite (24,606), pre-commit, mypy and the docs build pass. Differential run against master and this PR's previous head (7,683 strings × every DATE_ORDER and none × auto/en/ru/ja; 860,496 parse() and 268,800 search_dates() results):

  • Unset or "none": 0 differences from master (430,248 and 134,400 compared), nor in 5,440 day ranges such as "5-5 juin 2021" in every language.
  • "year", "all": no exception; every parse() change is a date becoming None (17,100 and 26,219). "all" is the "year" result or None, except "00月31日" with MYD and auto-detected languages (as before).
  • "year" against the previous head: 5,632 more None, all under YMD, YDM and MYD (DMY and MDY cannot change: nothing comes after the year there). "all" is unchanged.

HISTORY.rst is left to the release PR. Suggested bullet:

- Add the ``STRICT_DATE_ORDER`` setting. With ``"year"``, a two-digit year is
  read only from the position ``DATE_ORDER`` gives the year, so "32 DEC 10" with
  ``DATE_ORDER="DMY"`` is ``None`` instead of 2032-12-10; with ``"all"``, the
  day and month are not swapped either. Both need ``DATE_ORDER`` (#868).

🤖 Generated with Claude Code

@codspeed

codspeed Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 7 untouched benchmarks


Comparing fix-868-invalid-day-reordered-as-year (d2e6c6b) with master (9073f25)

Open in CodSpeed

@codecov

codecov Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.96%. Comparing base (9073f25) to head (d2e6c6b).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1443      +/-   ##
==========================================
+ Coverage   98.94%   98.96%   +0.02%     
==========================================
  Files         239      239              
  Lines        3521     3590      +69     
==========================================
+ Hits         3484     3553      +69     
  Misses         37       37              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdrianAtZyte

Copy link
Copy Markdown
Member

Any thoughts on this LLM feedback?

The analysis is great, but this is a lot of machinery for #868, and I would rather not take on the shape-matching cache and the callback plumbing into _parser for it. The DATE_ORDER check on its own looks good to me. Please, could you split that into its own PR with a few tests?

For date_formats, I am not sure the year-position approach is worth it, given the known gaps (e.g. "32 December 10" with %d %b %y still gives 2032). I wonder if we should instead discuss in #868 whether the fallback should be skipped when the string has the shape of a format.

Also, returning None for all locales on the first _MisplacedYearError feels like a separate behavior change, and 95年12月10日 with DATE_ORDER="MDY" going from a correct date to None looks like a regression to me, even if the DMY case was already wrong.

@serhii73 serhii73 changed the title Do not read an invalid day as the year under an explicit DATE_ORDER or date format (#868) Do not read an invalid day as the year under an explicit DATE_ORDER (#868) Sep 28, 2026
@serhii73

Copy link
Copy Markdown
Collaborator Author

Thanks, agreed on the split. I reworked this PR into the DATE_ORDER check alone (in place, to keep this thread), commit cc8afd4:

On the other points:

date_formats. Agreed, let's take it to #868. Two things worth having in that discussion: "skip the fallback when the string has the shape of a format" needs the same shape test (strptime on the string with every digit normalised) but not the year positions, the callback or the cache, so it is much smaller. It does change the day/month swap the fallback gives today: "12/31/2020" with ["%d/%m/%Y"] is 2020-12-31 on master and would become None. And "32 December 10" with %b stays a gap under either approach, since December does not match %b.

Cross-locale finality. I kept it, because without it the fix only holds with an explicit languages. I re-ran the differential sweep (95,152 parses; DMY, MDY, YMD, MYD; auto-detected languages and en, ru, ja) with the rejection made non-final: 617 of the 8,473 rejected results, all with auto-detected languages, come back from another locale, and 583 of them as a date master never gave. zh reads "09年20月10日(日)" under DMY as a relative date (2020-06-05 from a 2020-06-15 base) and ru reads "20 09 29" under MYD as 2029-06-29, because it translates it to "29 29". The "20 09 29" test case guards this. It is narrow: only a two-digit year read from a wrong slot under an explicit DATE_ORDER is final; every other ValueError falls through to the next locale as before.

95年12月10日 with MDY. I can't separate it from "32 DEC 10" inside the parser. The ja translation is "95-12-10", so the parser sees exactly what it sees in "32-12-10" with DMY, and master's MDY result is a coincidence: the same string with DMY gives 1995-10-12. The one rule that keeps it is to check only the day slot, literally the issue's title: reject when the number read as the year sits where the order wants a day. That keeps 95年12月10日/MDY at 1995-12-10 (95 sits in the month slot) and still fixes "32 DEC 10"/DMY and "DEC 32 10"/MDY, but leaves "32/12/10"/MDY at 2032-12-10 and "12/13/10"/DMY at 2013-10-12, which are the same bug. I would rather keep the symmetric rule and list the ja case as a behaviour change, as the PR body does, but the narrower one is a small change to the same check if you prefer it.

@AdrianAtZyte

Copy link
Copy Markdown
Member

Should we get #1419 merged first?

@AdrianAtZyte

Copy link
Copy Markdown
Member

I think we also need to merge #1431 first.

And I think we should then rethink this PR. The documented way for DATE_ORDER to work as as a preference. If we want to be able to enforce specific date orders, we should expose some way for users to opt into that alternative behavior.

@serhii73
serhii73 force-pushed the fix-868-invalid-day-reordered-as-year branch from cc8afd4 to 95ea50b Compare September 30, 2026 06:28
@serhii73 serhii73 changed the title Do not read an invalid day as the year under an explicit DATE_ORDER (#868) Add a STRICT_DATE_ORDER setting so that an invalid day is not read as the year (#868) Sep 30, 2026
@serhii73

Copy link
Copy Markdown
Collaborator Author

Thanks! With #1419 and #1431 merged, I've rebased this onto master and reworked it along your suggestion. The check is now opt-in through a new setting, STRICT_DATE_ORDER, which defaults to False. The default is exactly master's behaviour: with the setting off, none of 527,380 results change in a differential run. The name is open, what do you think? I chose it to sit next to DATE_ORDER and mirror STRICT_PARSING.

With the setting on, the only thing it does in parse() is turn dates into None. A reading it rejects is not retried with the day and month swapped (the #1431 retry), because that would give a date the default does not. The points from your earlier review are now opt-in results, documented in docs/settings.rst and the PR body:

  • 95年12月10日 with MDY is None with the setting, because its translation, 95-12-10, has nothing left to mark the year.
  • The rejection stays final across locales, so ru cannot turn "20 09 29" with MYD into a date in 2029.

One question on scope: the setting enforces where the year is, but a day and month that do not fit are still swapped ("13/12/10" with MDY, and "01/13/21" with DMY through #1431). Should STRICT_DATE_ORDER also turn those swaps off, so that it enforces the whole order? That would fit the name better, but it is a bigger change, so I've left it out for now.

@AdrianAtZyte

AdrianAtZyte commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

What if we make STRICT_DATE_ORDER an enum (string), to support all 3 approaches?

serhii73 and others added 2 commits October 1, 2026 09:35
… the year (#868)

DATE_ORDER is a preference: any number that is not a valid day or month
is read as a two-digit year, so "32 DEC 10" with DATE_ORDER="DMY" is
2032-12-10. With the new setting, which defaults to False, a two-digit
year is read only where DATE_ORDER puts the year, and such a date string
gives None instead. The rejection is final: no other locale may guess
another reading, and the no-spaces parser, which would read the same
numbers in a date order again, is skipped. The other parsers of the
locale still get their turn.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The setting is now "none" (the default), "year" or "all", so that it can
name all three behaviours: DATE_ORDER as a preference, the position of a
two-digit year enforced, or the whole order enforced. With "all", a day
and a month that do not fit are no longer swapped, neither in the token
loop nor through the retry with the swapped order: "13/12/10" with
DATE_ORDER="MDY" is None. A four-digit year is still read wherever it is,
and a date that starts with one is still read as year, month and day.

With "all", a date string that cannot be read in DATE_ORDER at all is
final too, so that no other locale reads it differently: with
auto-detected languages, "10年20月00日" with DMY would otherwise come
back from zh as a relative date, 10 years and 20 months ago.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@serhii73
serhii73 force-pushed the fix-868-invalid-day-reordered-as-year branch from 95ea50b to 05f2737 Compare October 1, 2026 09:48
@serhii73 serhii73 changed the title Add a STRICT_DATE_ORDER setting so that an invalid day is not read as the year (#868) Add a STRICT_DATE_ORDER setting to enforce the year position or the whole DATE_ORDER (#868) Oct 1, 2026
@AdrianAtZyte

Copy link
Copy Markdown
Member
  • I think "year" should check both directions, i.e. reject a two-digit year read from anywhere other than where DATE_ORDER puts it, so "10 DEC 32" with MYD becomes None too. parse error #519’s "4-99" with YMD becoming None under it is fine for an opt-in setting. That would also make it much easier to document.
  • Setting STRICT_DATE_ORDER without DATE_ORDER silently does nothing. I think it should either raise a SettingValidationError, or enforce the order actually in use (the locale’s, with PREFER_LOCALE_DATE_ORDER).
  • The last paragraph in docs/settings.rst could be much shorter, I think only the “only with DATE_ORDER” part and the 95年12月10日 caveat matter to users.

serhii73 and others added 3 commits October 2, 2026 09:01
A two-digit year is now read only where DATE_ORDER puts the year: the day and
month read before it must be the ones the order puts before it. Until now only
a day or month that the order puts before the year, but is read after it, was
rejected, so "32 DEC 10" with DMY was None but "10 DEC 32" with MYD was still
2032-12-10, and "year" did nothing at all for YMD and YDM, where no part comes
before the year.

This also rejects #519's last-resort "4-99" with YMD, which is acceptable for an
opt-in setting. DMY and MDY are not affected, since nothing comes after the
year in them.

The day and month swap is now also tested under DYM and MYD, where the year is
between them and the swap changes which of them is read before it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… to enforce

"year" and "all" only enforce a DATE_ORDER that the caller sets, since the
default one is a preference, so without one they silently did nothing. Raise a
SettingValidationError from check_settings instead, which only looks at the
settings the caller passed, so the rebuilt settings of search_dates, which
forget those, cannot trigger it. "none" does not need a DATE_ORDER.

The cases that pinned "no effect without DATE_ORDER" are replaced by tests of
the error, of "none", and of search_dates and a reused parser, which keep working.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
"year" now reads as one rule: a two-digit year is read only from the position
DATE_ORDER gives the year. Of the details after the examples, only two matter
to users, that "year" and "all" need DATE_ORDER, and that '95年12月10日' with
MDY is rejected. The rest, such as that a rejection is final across locales,
which parsers still run, and that a month name or a missing part is not
checked, is in the code comments.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
serhii73 and others added 3 commits October 2, 2026 12:26
05f2737 records in auto_order that the end of a day range is dropped when a
month name follows it, but it always deleted the entry after the month. In a
day-first order, as in "12-12 June 2021" with DMY or "5-5 juin 2021" in French,
the day comes before the month, so the deletion raised IndexError, parse_alpha
swallowed it and the month name was not read: the date was None even without
STRICT_DATE_ORDER, unlike master. Remove the day entry itself instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_ORDER

Moving "4-99" with YMD and "10 DEC 32" with MYD to the rejected cases left no
test of #519's two-digit year fallback under STRICT_DATE_ORDER, and left its
comment on "10年20月00日", which goes through the day and month swap instead.
Also say what the search_dates test checks: that it does not raise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without it, "read a two-digit year only from the position DATE_ORDER gives the
year" suggests that "32 DEC" with DMY is rejected too, but a month name and a
missing part are not checked, and a four-digit year is read wherever it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@AdrianAtZyte AdrianAtZyte left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Please, drop the issue numbers from the _StrictDateOrderError docstring and the #1419 comment.
  • In docs/settings.rst, quote the values ('none', 'year', 'all'), and the doctests should not show a None line, since that is not what the REPL prints.

…amples

Quote the values of the setting in the docs, since they are strings, and drop
the None lines from its examples, since the REPL prints nothing for None.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@serhii73
serhii73 merged commit b7d0e8b into master Oct 2, 2026
22 checks passed
@serhii73
serhii73 deleted the fix-868-invalid-day-reordered-as-year branch October 2, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parser result wrong for weird input string

2 participants