Skip to content

Stop search_dates slowing down on long runs of words that do not parse - #1564

Merged
serhii73 merged 1 commit into
scrapinghub:masterfrom
serhii73:fix-search-dates-unparsed-word-runs
Nov 4, 2026
Merged

serhii73 merged 1 commit into
scrapinghub:masterfrom
serhii73:fix-search-dates-unparsed-word-runs

Conversation

@serhii73

@serhii73 serhii73 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Regression from #1401 and #1440, which are not in any release yet.

What was wrong

On master, search_dates takes much longer than the text grows on a long text made of words that do not parse as a date on their own, for example "ngΓ y " * k with languages=["vi"] or "hour " * k with languages=["en"].

The code that finds the date a relative date is relative to, _ExactLanguageSearch.set_relative_base, looks back through the items parsed so far for the last one that is a date. Since #1401 the look-back goes over every item that has no date, such as "ngΓ y" or "day" (and, since #1440, "hour", which used to parse as today), so in a long run of such words none of them stops the look-back and every word looks back over all the words before it. For every part of a split the caller also copied the whole list of items parsed so far ([*parsed, *current_parsed]).

How many times a DateData is read for runs of 200 and 400 words (this is what the new tests count):

text master this PR
"hour " * k 28,600 -> 111,685 (3.91x) 1,906 -> 3,808 (2.00x)
"ngΓ y " * k 27,864 -> 110,215 (3.96x) 1,170 -> 2,338 (2.00x)

CPU seconds of one search_dates call for "ngΓ y " * k with languages=["vi"] (time.process_time(), Python 3.12, one run each):

chars master this PR
16,000 0.69 0.17
64,000 9.28 0.69

A run of "hour " in English still costs more per character than one of "ngΓ y " in Vietnamese: since #1440 each word of it is tried on its own, which this PR does not change.

How it is fixed

The date that the next date is relative to is now kept up to date as each item is parsed (_last_relative_base), and parse_item receives that date instead of the list of everything parsed so far, so the time spent on an item no longer depends on how many items came before it. parse_found_objects hands the date to the two places where it parses part of a chunk again, in place of the already_parsed list it took (it is only called from search_parse and from itself). set_relative_base, now unused, is removed. Nothing else changes: the same date is chosen as before, so results are identical.

Tests and checks

  • 7 new tests in a new file, tests/test_search_unparsed_word_runs.py; no existing test file is edited. Two count the DateData reads for runs of 200 and 400 words ("hour " in English, "ngΓ y " in Vietnamese) and require fewer than 2.5x as many for the longer run; counting instead of timing makes them deterministic. Five pin which date each relative date is relative to, including after a run of words that do not parse, after a trailing quantity ("Fifty years after") and after dates in the same sentence.
  • On a pristine tree of master the two count tests fail (AssertionError: 111685 not less than 71500.0) and the five pinning tests pass; on this branch all 7 pass. Five hand-made mutants of the change (a relative date taken as the base, a date that did not parse taken as the base, the base not updated after a date, after a trailing quantity, or never used) are each caught by at least one test.
  • Differential run of master against this branch, with a frozen clock and TIMEZONE=UTC, with and without a fixed RELATIVE_BASE: 5,156 strings (a subset of a corpus of strings captured from the existing tests, per-language vocabulary combinations for all languages and edge cases with relative dates and unparsed words), 26,929 comparisons of parse, get_date_data, search_dates (explicit languages, RELATIVE_BASE set, languages=None) and parse_found_objects: no difference.
  • Full test suite on Python 3.12: 25,053 passed, 23 skipped, 4 xfailed. A local mirror of the CI matrix passes on all 16 legs: Python 3.10 to 3.15 (3.13 also with TZ=Pacific/Auckland and TZ=Pacific/Fiji), the minimum supported dependency versions (with and without the optional ones), all optional dependencies, mypy, the docs build, the scripts, the benchmarks, twine check and the pre-commit hooks.

πŸ€– Generated with Claude Code

Regression from scrapinghub#1401 and scrapinghub#1440. search_dates found the date that a relative
date is relative to by looking back through the items parsed so far for the
last one that is a date. Since scrapinghub#1401 the look-back goes over every item that
has no date, for example a word like "day" or "ngΓ y" that does not parse as a
date on its own ("hour" is one too since scrapinghub#1440), so in a long run of such
words each word looked back over all the words before it, and each word also
copied the whole list of items parsed so far.

Keep that date up to date as each item is parsed instead, and hand it to
parse_item in place of the list. The date is the same, so the results do not
change, but the time spent on a word no longer depends on how many words came
before it.

The new tests are in their own file, tests/test_search_unparsed_word_runs.py.
Two of them count how many times a DateData is read for runs of 200 and 400
words: the count doubles with this change and almost quadruples without it
(they fail on master and pass here). Five pin which date each relative date is
relative to (they pass on both).

Results are unchanged: a differential run of master against this change, with
a frozen clock, over 5,156 strings (26,929 comparisons of parse,
get_date_data, search_dates and parse_found_objects) showed no difference.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codspeed

codspeed Bot commented Nov 3, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

βœ… 10 untouched benchmarks


Comparing serhii73:fix-search-dates-unparsed-word-runs (84f36ab) with master (2d93cdc)

Open in CodSpeed

@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

βœ… All modified and coverable lines are covered by tests.
βœ… Project coverage is 99.19%. Comparing base (2d93cdc) to head (84f36ab).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1564      +/-   ##
==========================================
- Coverage   99.19%   99.19%   -0.01%     
==========================================
  Files         239      239              
  Lines        4098     4095       -3     
==========================================
- Hits         4065     4062       -3     
  Misses         33       33              

β˜” View full report in Codecov by Harness.
πŸ“’ Have feedback on the report? Share it here.

πŸš€ New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@serhii73
serhii73 merged commit d463089 into scrapinghub:master Nov 4, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant