Repository navigation
Stop search_dates() slowing down on long runs of brackets - #1565
Merged
serhii73 merged 2 commits intoNov 4, 2026
Merged
Conversation
Regression from scrapinghub#1401. Locale.translate_search() asks _attached_group_end() for every token that opens a bracket or a quote where the group closes, and every call scanned to the end of the tokens. That is n^2/2 token checks for n groups: 124 750 checks for "(UTC " * 500. For a chunk made only of weekdays, translate_search() also looked again, with all(), at every word already in the chunk at each bracket: 249 500 checks for "(Mon " * 500. What changed: * _attached_group_end() takes the dict known_ends, shared by all the groups of one sentence. A scan from token s that stops at token e already knows the answer for every group that opens between s and e, so it stores it (the closing token itself is left out: a lone quote or bracket closes a group only if it does not open it). Every token is looked at a constant number of times. * translate_search() only checks the tokens added to the current chunk since the last bracket, through the new _first_non_weekday(). Results are unchanged. How it was proved: * test_groups_are_scanned_once counts the checks of the tokens of 500 groups for five shapes: 124 750 on master, 499 now, bound 2000. * test_weekdays_before_a_group_are_checked_once counts the weekday checks: 249 500 on master, 998 now, bound 2000. Both fail on master for that reason and not for a missing name: they only use names that exist there. * test_translate_search_of_groups holds the results of master for nested, unclosed and attached groups, among them the three shortest inputs found by a fuzzer that tell apart a wrong end of the memo, a wrong stop token and a chunk that is not checked again from its start. They pass on master and on this change. * Mutations of this change (no memo lookup, memo filled for the first group only, memo range or stop token off by one, weekday check not restarted or restarted at every bracket) each make one of these tests fail. * Differential run, master against this change, 20 837 strings through parse, get_date_data, search_dates (explicit languages, language detection, ngram search) and translate_search: 158 405 results compared, 0 differences; 30 000 token-fuzz strings: 430 000 results, 0 differences; the new function agrees with a verbatim copy of the old one on 46 167 group scans. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Why: _clean_substring() dropped one unbalanced closing bracket per loop iteration, with two str.count() calls over the whole text and a copy of the text each time. "Jan 5" + ")" * 200 000 took about 10 s of CPU time. What changed: the text that is left is tracked as a span [start, end) of the input, with the number of "()" and "[]" in it. The text is only ever shortened at its ends, so the counts are updated from the piece that is removed, and the position of the first space is kept until the span moves past it (without that, a long first word is looked at again for every word dropped at the end: the count test below). Apart from that nothing changes: the same steps run in the same order. How it was proved: * test_many_unbalanced_closing_brackets: 200 000 closing brackets in four shapes, 5 to 12 s of process time on master, below 0.05 s now; the bound is 0.5 s, about 10 times away from both. * test_long_word_is_looked_up_once counts the characters looked up in the words to skip: 20 million for a 10 000-character word followed by 2000 skipped words on master, 12 000 now, bound 20 000. * test_short_texts_like_cutting_the_text compares _clean_substring() with the previous implementation, copied into the test, on every text of up to 6 characters over " ()[]/a" with three sets of words to skip (some that contain brackets): 0 differences. It fails when the brackets of a dropped word are not taken off the counts. * Mutations (counts not updated on a drop or for an opening bracket, counts recomputed from the slice, first-word result not kept) each make one of the tests fail. * The new function agrees with the old one on 14.4 million exhaustive and 3 million random strings (up to 40 characters, six skip sets): 0 differences; the differential runs of the previous commit are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1565 +/- ##
==========================================
+ Coverage 99.19% 99.22% +0.03%
==========================================
Files 239 239
Lines 4098 4149 +51
==========================================
+ Hits 4065 4117 +52
+ Misses 33 32 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain, so the proposed performance and parsing-preservation changes are not currently blocked.
No open issues 🎉
What changed in this PR
This pull request improves search_dates() performance for long runs of brackets while preserving parsing results through cached group scans, incremental weekday checks, and span-based substring cleanup.
Changes:
- Memoizes attached bracket-group scans across each sentence.
- Avoids repeatedly checking previously processed weekday tokens.
- Reworks
_clean_substring()to track spans and bracket counts efficiently. - Adds regression, equivalence, and performance tests.
| File | Description |
|---|---|
tests/test_search_bracket_runs.py |
Adds regression, performance, and behavior-preservation tests for bracket-heavy searches. |
dateparser/search/search.py |
Optimizes substring cleanup, but a no-space input can still cause repeated rescans and quadratic behavior. |
dateparser/languages/locale.py |
Adds cached group scanning and incremental weekday detection. |
This pull request requires changes.
Review effort: Lite
felixonmars
added a commit
to felixonmars/archriscv-packages
that referenced
this pull request
Nov 8, 2026
Allow ten times the duration in the four long-input test functions behind nine timing failures on shinx. Keep the full suite, input sizes, result assertions and subprocess deadline while accommodating slower riscv64 CPUs. The upstream performance fixes are already in 1.5.0; no applicable upstream timing adjustment was found. scrapinghub/dateparser#1565 scrapinghub/dateparser@3ef4387 scrapinghub/dateparser@66d5783 https://archriscv.felixc.at/.status/logs/python-dateparser/python-dateparser-1.5.0-1.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Regression from #1401, which is not in any release yet.
What was wrong
Two loops added by "Parse last, this and next before a weekday" (#1401) take much longer than the text grows on a long text made of brackets, for example
search_dates("(UTC " * 3200, languages=["en"]). They are only in unreleased master; no release contains them.Locale._attached_group_end()(called fromtranslate_search()for every token that opens a bracket or a quote) scans to the end of the tokens to find where the group closes. With n opening tokens that is n^2/2 checks. For a chunk made only of weekdays,translate_search()also re-checked every word already in the chunk at each bracket (anall(...)over the chunk)._clean_substring()dropped one unbalanced closing bracket per loop iteration, with twostr.count()calls over the whole text and a copy of it each time.CPU seconds of one
search_dates(text, languages=["en"])call (time.process_time(), Python 3.12, one run each):"(UTC " * k"Jan 5" + ")" * nOn master each doubling of the size multiplies the time by 3.1 to 3.8; with this PR it is 1.7 to 2.3.
How it is fixed
_attached_group_end()gets aknown_endsdict, shared by all the groups of one sentence. A scan from tokensthat stops at tokenealready knows the answer for every group that opens betweensande, so it stores it; every token is looked at a constant number of times.translate_search()remembers how many leading tokens of the current chunk it already knows to be empty or weekdays (_first_non_weekday()), and only checks the tokens added since the last bracket._clean_substring()tracks the part of the text that is left as a span of indexes, with the number of()and[]in it, instead of cutting and counting the text again and again. The position of the first space and the result of looking up the first word are kept, so that a long first word is not looked at again for every word dropped at the end.No result changes (see below).
Tests and checks
tests/test_search_bracket_runs.py; no existing test file is edited. They count work instead of timing it, so they are deterministic:test_groups_are_scanned_once(token checks for 500 groups in five shapes,(UTC,[UTC,(12:00,(1and(Mon: 124,750 on master, 499 with the fix),test_weekdays_before_a_group_are_checked_once(249,500 on master, 998) andtest_long_word_is_looked_up_once(20,002,000 on master, 12,000). The one exception,test_many_unbalanced_closing_brackets, is bound by CPU time (time.process_time()): 5 to 12 s on master, below 0.05 s with the fix, bound 0.5 s. The other tests hold results taken from master for nested, unclosed and attached groups and edge cases, and compare_clean_substring()with the previous implementation on every text of up to 6 characters over" ()[]/a".RELATIVE_BASEfixed,TIMEZONE=UTC): 20,837 strings throughparse,DateDataParser.get_date_data,search_dates(explicit languages, language detection, the ngram strategy) andLocale.translate_search; 158,405 results compared, 0 differences._clean_substring()agrees with the old implementation on 14.4 million exhaustive and 3 million random strings.TZ=Pacific/AucklandandTZ=Pacific/Fiji), the minimum supported dependency versions (with and without the optional ones), all optional dependencies, mypy, the docs build, the scripts, the benchmarks,twine checkand the pre-commit hooks.🤖 Generated with Claude Code