Skip to content

Apply an explicit DATE_ORDER to every date that search_dates finds (#1512) - #1535

Merged
AdrianAtZyte merged 2 commits into
masterfrom
fix/search-dates-explicit-date-order
Oct 5, 2026
Merged

AdrianAtZyte merged 2 commits into
masterfrom
fix/search-dates-explicit-date-order

Conversation

@serhii73

@serhii73 serhii73 commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1512.

>>> search_dates('op 30-05-2027. Deze op 01-11-2026', languages=['nl'],
...              settings={'DATE_ORDER': 'DMY', 'STRICT_PARSING': True})
[('30-05-2027', datetime(2027, 5, 30)), ('01-11-2026', datetime(2026, 1, 11))]   # 1.4.3, master
[('30-05-2027', datetime(2027, 5, 30)), ('01-11-2026', datetime(2026, 11, 1))]   # 1.4.2, this branch

Root cause

Unless the caller sets RELATIVE_BASE, search_dates parses each date after the first one again, with the date found before it as RELATIVE_BASE. Up to 1.4.2 it set that on the caller's settings object in place. #1346 made this thread-safe by working on a copy, parser._settings.replace(RELATIVE_BASE=...). But Settings.replace() only keeps _mod_settings, the record of which settings the caller set, when it is passed mod_settings=. The copy has none, so _try_parser takes the explicit DATE_ORDER for the default one and lets the order of the locale win. Search parses the English translation of every language but vi and hu, so that order is MDY. STRICT_DATE_ORDER (#1443) stopped applying to those dates for the same reason, since it only applies with an explicit DATE_ORDER.

The issue also suspects the date_order change in _try_parser. That change is not involved:

tree second date of the repro
1.4.2, and de7b986 right before #1346 2026-11-01
cd9bb13 (#1346), 1.4.3, master 2026-01-11
cd9bb13 with only its search.py hunk reverted 2026-11-01

The fix

Settings.replace() keeps the caller's _mod_settings unless it is given others, so a copy made to change some settings still knows which of the others the caller set. Nothing is mutated: the copy is a new object, as before, and shares the read-only mapping.

#1515 (closed unmerged) proposed passing mod_settings= at the call site instead. The two give identical results on every input below, but fixing replace() also covers the next copy someone makes. The only other copy, in FullTextLanguageDetector, is always made from the default settings, which have nothing to keep, so it does not change. Neither does settings.replace(...) on the default settings, which is how a Settings object is built by hand.

Tests

The diff of tests/ is additions only; no existing test was touched.

  • the repro from the issue;
  • three dates per text, each with and without STRICT_PARSING: DMY in nl, de, fr, ru and en; MDY in vi and hu, whose own orders are DMY and YMD, and in en; YMD in en, es and vi with two-digit years, since a four-digit year first is read as the year under any order;
  • no DATE_ORDER, in en, vi and hu, alone and next to other settings: every date is still read in the order of the locale;
  • the other settings across the rebuilt settings: PREFER_DATES_FROM for a date with no year, a relative date (in 3 days, Π§Π΅Ρ€Π΅Π· 3 дня) counted from the date before it, TIMEZONE with TO_TIMEZONE, and with RETURN_AS_TIMEZONE_AWARE;
  • STRICT_DATE_ORDER "all" and "year" on the second date;
  • Settings.replace() itself: it keeps an explicit DATE_ORDER, and the default one it copies does not become explicit.

On master 10 tests and 20 subtests fail, each because a date is read in the order of the locale instead of the explicit one or, for STRICT_DATE_ORDER, is not rejected. Only the controls pass there: no DATE_ORDER, MDY in English and the replace() guard.

Verification

  • pytest: 24,761 passed, 18 skipped, 1 xfailed; master has 24,736 passed with the same skips and xfail. Same result with TZ set to Pacific/Auckland and Pacific/Fiji, as in CI.
  • tox: all, py311, py313, py314, mypy, docs, scripts and twinecheck pass. py310, min and min-all cannot install atheris on my machine, so I ran their suites in venvs with the same pins minus atheris, which only fuzzing/ imports. All pass.
  • pre-commit run --all-files passes: ruff check, ruff format, zizmor. The pre-commit that tox -e pre-commit installs needs a newer git than mine for --all-files, so I ran pre-commit 3.8.0 with the pinned hooks, and tox -e pre-commit -- --files $(git ls-files) as well.

Differential run against master over 138,985 results. The texts: 800 generated with two to four complete dates in 11 languages, 195 that mix in relative and yearless dates, and 289 strings from tests/test_search.py. Each ran under 33 settings, with and without RELATIVE_BASE, plus the ngram strategy and parse() on each date alone:

results changed
no DATE_ORDER 6,420 0
RELATIVE_BASE set by the caller 42,372 0
ngram strategy 8,800 0
parse() 45,441 0
explicit DATE_ORDER, no RELATIVE_BASE 35,952 10,458
this branch vs the #1515 call-site fix 138,985 0

Two independent checks show the changes are the fix:

  • A RELATIVE_BASE set by the caller keeps master off the broken path. With complete dates, this branch without one agrees with master with one on every date that does not take a part from that base. The 1,808 results that differ hold pieces with no full date, like 10:30, or two-digit years under PREFER_DATES_FROM: "future", which are resolved against the base.
  • Where 1.4.2 splits a text the same way, 3,497 of the 3,525 changed dates are the ones 1.4.2 returns. The other 28, in 7 texts, are parser changes since 1.4.2: a YYYY-MM-DD first date that 1.4.2 read as year-day-month under a day-first order (a first date, so untouched by this bug), the later dates that take their month or day from it, and a dans 3 jours, that 1.4.2 read as three days ago (Mistake search_dates for ru languageΒ #706).

Some changed results look odd. With DATE_ORDER: "YMD", 20 ΠΌΠ°Ρ€Ρ‚Π° after 19 ΠΌΠ°Ρ€Ρ‚Π° 2001 becomes 2020-03-19, because YMD reads a lone number as the year. That is what parse() returns with the same settings and base, and what 1.4.2 returned in search.

Note

The docstring of test_search_dates_does_not_check_the_rebuilt_settings (#1443) says the rebuilt settings "forget which settings the caller set", which is no longer true. Its assertions still hold. I left it alone so that this PR only adds tests; it can be reworded here or in a follow-up.

πŸ€– Generated with Claude Code

…1512)

search_dates parses each date after the first one again, with the date
found before it as RELATIVE_BASE, unless the caller set one. Since the
thread-safety change in 1.4.3, it does that on a copy of the settings
made by Settings.replace() instead of changing the caller's settings in
place. The copy did not keep track of which settings the caller set, so
an explicit DATE_ORDER counted as the default one and the order of the
locale won for every date after the first: with DATE_ORDER "DMY",
"01-11-2026" became January 11 instead of November 1. STRICT_DATE_ORDER
stopped applying to those dates for the same reason.

Have Settings.replace() keep the settings that the caller set unless it
is given others. That fixes every copy, not only the one search_dates
makes. The only other one, made for language detection, is always made
from the default settings, so it does not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codspeed

codspeed Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

βœ… 7 untouched benchmarks


Comparing fix/search-dates-explicit-date-order (3173b31) with master (b7d0e8b)

Open in CodSpeed

@AdrianAtZyte AdrianAtZyte left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, reword the docstring of test_search_dates_does_not_check_the_rebuilt_settings, since the rebuilt settings no longer forget which settings the caller set, and drop the # Issue #1512 comment from the repro test.

The docstring of test_search_dates_does_not_check_the_rebuilt_settings
said that the settings search_dates rebuilds for the dates after the
first one "forget which settings the caller set". Settings.replace()
keeps track of them now, so say that the rebuilt settings remember
which settings the caller set.

Also drop the "# Issue #1512" comment from the test of the repro in
the issue.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@AdrianAtZyte
AdrianAtZyte merged commit fed9cf9 into master Oct 5, 2026
20 checks passed
@serhii73
serhii73 deleted the fix/search-dates-explicit-date-order branch October 5, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1.4.3 regression: search_dates ignores explicit DATE_ORDER for every date after the first

2 participants