Skip to content

Stop search_dates() slowing down on long runs of brackets - #1565

Merged
serhii73 merged 2 commits into
scrapinghub:masterfrom
serhii73:fix-search-dates-bracket-runs
Nov 4, 2026
Merged

serhii73 merged 2 commits into
scrapinghub:masterfrom
serhii73:fix-search-dates-bracket-runs

Conversation

@serhii73

@serhii73 serhii73 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Regression from #1401, which is not in any release yet.

What was wrong

Two loops added by "Parse last, this and next before a weekday" (#1401) take much longer than the text grows on a long text made of brackets, for example search_dates("(UTC " * 3200, languages=["en"]). They are only in unreleased master; no release contains them.

  1. Locale._attached_group_end() (called from translate_search() for every token that opens a bracket or a quote) scans to the end of the tokens to find where the group closes. With n opening tokens that is n^2/2 checks. For a chunk made only of weekdays, translate_search() also re-checked every word already in the chunk at each bracket (an all(...) over the chunk).
  2. _clean_substring() dropped one unbalanced closing bracket per loop iteration, with two str.count() calls over the whole text and a copy of it each time.

CPU seconds of one search_dates(text, languages=["en"]) call (time.process_time(), Python 3.12, one run each):

text chars master this PR
"(UTC " * k 2,000 / 4,000 / 8,000 / 16,000 0.74 / 2.59 / 9.80 / 37.0 0.13 / 0.27 / 0.55 / 1.19
"Jan 5" + ")" * n 32,005 / 64,005 / 128,005 / 256,005 0.39 / 1.20 / 4.46 / 17.0 0.10 / 0.23 / 0.40 / 0.83

On master each doubling of the size multiplies the time by 3.1 to 3.8; with this PR it is 1.7 to 2.3.

How it is fixed

  • _attached_group_end() gets a known_ends dict, shared by all the groups of one sentence. A scan from token s that stops at token e already knows the answer for every group that opens between s and e, so it stores it; every token is looked at a constant number of times.
  • translate_search() remembers how many leading tokens of the current chunk it already knows to be empty or weekdays (_first_non_weekday()), and only checks the tokens added since the last bracket.
  • _clean_substring() tracks the part of the text that is left as a span of indexes, with the number of () and [] in it, instead of cutting and counting the text again and again. The position of the first space and the result of looking up the first word are kept, so that a long first word is not looked at again for every word dropped at the end.

No result changes (see below).

Tests and checks

  • 37 new tests in a new file, tests/test_search_bracket_runs.py; no existing test file is edited. They count work instead of timing it, so they are deterministic: test_groups_are_scanned_once (token checks for 500 groups in five shapes, (UTC , [UTC , (12:00 , (1 and (Mon : 124,750 on master, 499 with the fix), test_weekdays_before_a_group_are_checked_once (249,500 on master, 998) and test_long_word_is_looked_up_once (20,002,000 on master, 12,000). The one exception, test_many_unbalanced_closing_brackets, is bound by CPU time (time.process_time()): 5 to 12 s on master, below 0.05 s with the fix, bound 0.5 s. The other tests hold results taken from master for nested, unclosed and attached groups and edge cases, and compare _clean_substring() with the previous implementation on every text of up to 6 characters over " ()[]/a".
  • On a pristine tree of master with only the new test file, 11 of the 37 tests fail and 26 pass. Ten hand-made mutants of the changed code are each caught by at least one test.
  • Differential run of master against this branch (RELATIVE_BASE fixed, TIMEZONE=UTC): 20,837 strings through parse, DateDataParser.get_date_data, search_dates (explicit languages, language detection, the ngram strategy) and Locale.translate_search; 158,405 results compared, 0 differences. _clean_substring() agrees with the old implementation on 14.4 million exhaustive and 3 million random strings.
  • Full test suite on Python 3.12: 25,083 passed, 23 skipped, 4 xfailed (including the 37 new tests). A local mirror of the CI matrix passes on all 16 legs: Python 3.10 to 3.15 (3.13 also with TZ=Pacific/Auckland and TZ=Pacific/Fiji), the minimum supported dependency versions (with and without the optional ones), all optional dependencies, mypy, the docs build, the scripts, the benchmarks, twine check and the pre-commit hooks.

🤖 Generated with Claude Code

serhii73 and others added 2 commits November 3, 2026 18:00
Regression from scrapinghub#1401. Locale.translate_search() asks _attached_group_end()
for every token that opens a bracket or a quote where the group closes, and
every call scanned to the end of the tokens. That is n^2/2 token checks for
n groups: 124 750 checks for "(UTC " * 500. For a chunk made only of
weekdays, translate_search() also looked again, with all(), at every word
already in the chunk at each bracket: 249 500 checks for "(Mon " * 500.

What changed:
* _attached_group_end() takes the dict known_ends, shared by all the groups
  of one sentence. A scan from token s that stops at token e already knows
  the answer for every group that opens between s and e, so it stores it
  (the closing token itself is left out: a lone quote or bracket closes a
  group only if it does not open it). Every token is looked at a constant
  number of times.
* translate_search() only checks the tokens added to the current chunk since
  the last bracket, through the new _first_non_weekday().

Results are unchanged.

How it was proved:
* test_groups_are_scanned_once counts the checks of the tokens of 500 groups
  for five shapes: 124 750 on master, 499 now, bound 2000.
* test_weekdays_before_a_group_are_checked_once counts the weekday checks:
  249 500 on master, 998 now, bound 2000. Both fail on master for that reason
  and not for a missing name: they only use names that exist there.
* test_translate_search_of_groups holds the results of master for nested,
  unclosed and attached groups, among them the three shortest inputs found
  by a fuzzer that tell apart a wrong end of the memo, a wrong stop token
  and a chunk that is not checked again from its start. They pass on master
  and on this change.
* Mutations of this change (no memo lookup, memo filled for the first group
  only, memo range or stop token off by one, weekday check not restarted or
  restarted at every bracket) each make one of these tests fail.
* Differential run, master against this change, 20 837 strings through
  parse, get_date_data, search_dates (explicit languages, language
  detection, ngram search) and translate_search: 158 405 results compared,
  0 differences; 30 000 token-fuzz strings: 430 000 results, 0 differences;
  the new function agrees with a verbatim copy of the old one on 46 167
  group scans.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Why: _clean_substring() dropped one unbalanced closing bracket per loop
iteration, with two str.count() calls over the whole text and a copy of the
text each time. "Jan 5" + ")" * 200 000 took about 10 s of CPU time.

What changed: the text that is left is tracked as a span [start, end) of the
input, with the number of "()" and "[]" in it. The text is only ever
shortened at its ends, so the counts are updated from the piece that is
removed, and the position of the first space is kept until the span moves
past it (without that, a long first word is looked at again for every word
dropped at the end: the count test below). Apart from that nothing changes:
the same steps run in the same order.

How it was proved:
* test_many_unbalanced_closing_brackets: 200 000 closing brackets in four
  shapes, 5 to 12 s of process time on master, below 0.05 s now; the bound is
  0.5 s, about 10 times away from both.
* test_long_word_is_looked_up_once counts the characters looked up in the
  words to skip: 20 million for a 10 000-character word followed by 2000
  skipped words on master, 12 000 now, bound 20 000.
* test_short_texts_like_cutting_the_text compares _clean_substring() with the
  previous implementation, copied into the test, on every text of up to 6
  characters over " ()[]/a" with three sets of words to skip (some that
  contain brackets): 0 differences. It fails when the brackets of a dropped
  word are not taken off the counts.
* Mutations (counts not updated on a drop or for an opening bracket, counts
  recomputed from the slice, first-word result not kept) each make one of the
  tests fail.
* The new function agrees with the old one on 14.4 million exhaustive and 3
  million random strings (up to 40 characters, six skip sets): 0
  differences; the differential runs of the previous commit are unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codspeed

codspeed Bot commented Nov 3, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 10 untouched benchmarks


Comparing serhii73:fix-search-dates-bracket-runs (c3fb3fe) with master (2d93cdc)

Open in CodSpeed

@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.22%. Comparing base (2d93cdc) to head (c3fb3fe).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1565      +/-   ##
==========================================
+ Coverage   99.19%   99.22%   +0.03%     
==========================================
  Files         239      239              
  Lines        4098     4149      +51     
==========================================
+ Hits         4065     4117      +52     
+ Misses         33       32       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@serhii73
serhii73 requested review from AdrianAtZyte and a lite review from Copilot November 4, 2026 07:47
@serhii73
serhii73 merged commit 4b1eab8 into scrapinghub:master Nov 4, 2026
20 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain, so the proposed performance and parsing-preservation changes are not currently blocked.

No open issues 🎉

What changed in this PR

This pull request improves search_dates() performance for long runs of brackets while preserving parsing results through cached group scans, incremental weekday checks, and span-based substring cleanup.

Changes:

  • Memoizes attached bracket-group scans across each sentence.
  • Avoids repeatedly checking previously processed weekday tokens.
  • Reworks _clean_substring() to track spans and bracket counts efficiently.
  • Adds regression, equivalence, and performance tests.
File Description
tests/​test_search_bracket_runs.py Adds regression, performance, and behavior-preservation tests for bracket-heavy searches.
dateparser/​search/​search.py Optimizes substring cleanup, but a no-space input can still cause repeated rescans and quadratic behavior.
dateparser/​languages/​locale.py Adds cached group scanning and incremental weekday detection.

This pull request requires changes.

Review effort: Lite

@serhii73
serhii73 deleted the fix-search-dates-bracket-runs branch November 4, 2026 07:55
felixonmars added a commit to felixonmars/archriscv-packages that referenced this pull request Nov 8, 2026
Allow ten times the duration in the four long-input test functions behind
nine timing failures on shinx. Keep the full suite, input sizes, result
assertions and subprocess deadline while accommodating slower riscv64 CPUs.

The upstream performance fixes are already in 1.5.0; no applicable upstream
timing adjustment was found.

scrapinghub/dateparser#1565
scrapinghub/dateparser@3ef4387
scrapinghub/dateparser@66d5783
https://archriscv.felixc.at/.status/logs/python-dateparser/python-dateparser-1.5.0-1.log
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants