Repository navigation
Stop search_dates slowing down on long runs of words that do not parse - #1564
Merged
serhii73 merged 1 commit intoNov 4, 2026
Merged
Conversation
Regression from scrapinghub#1401 and scrapinghub#1440. search_dates found the date that a relative date is relative to by looking back through the items parsed so far for the last one that is a date. Since scrapinghub#1401 the look-back goes over every item that has no date, for example a word like "day" or "ngΓ y" that does not parse as a date on its own ("hour" is one too since scrapinghub#1440), so in a long run of such words each word looked back over all the words before it, and each word also copied the whole list of items parsed so far. Keep that date up to date as each item is parsed instead, and hand it to parse_item in place of the list. The date is the same, so the results do not change, but the time spent on a word no longer depends on how many words came before it. The new tests are in their own file, tests/test_search_unparsed_word_runs.py. Two of them count how many times a DateData is read for runs of 200 and 400 words: the count doubles with this change and almost quadruples without it (they fail on master and pass here). Five pin which date each relative date is relative to (they pass on both). Results are unchanged: a differential run of master against this change, with a frozen clock, over 5,156 strings (26,929 comparisons of parse, get_date_data, search_dates and parse_found_objects) showed no difference. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Codecov Reportβ
All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1564 +/- ##
==========================================
- Coverage 99.19% 99.19% -0.01%
==========================================
Files 239 239
Lines 4098 4095 -3
==========================================
- Hits 4065 4062 -3
Misses 33 33 β View full report in Codecov by Harness. π New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Regression from #1401 and #1440, which are not in any release yet.
What was wrong
On master,
search_datestakes much longer than the text grows on a long text made of words that do not parse as a date on their own, for example"ngΓ y " * kwithlanguages=["vi"]or"hour " * kwithlanguages=["en"].The code that finds the date a relative date is relative to,
_ExactLanguageSearch.set_relative_base, looks back through the items parsed so far for the last one that is a date. Since #1401 the look-back goes over every item that has no date, such as"ngΓ y"or"day"(and, since #1440,"hour", which used to parse as today), so in a long run of such words none of them stops the look-back and every word looks back over all the words before it. For every part of a split the caller also copied the whole list of items parsed so far ([*parsed, *current_parsed]).How many times a
DateDatais read for runs of 200 and 400 words (this is what the new tests count):"hour " * k"ngΓ y " * kCPU seconds of one
search_datescall for"ngΓ y " * kwithlanguages=["vi"](time.process_time(), Python 3.12, one run each):A run of
"hour "in English still costs more per character than one of"ngΓ y "in Vietnamese: since #1440 each word of it is tried on its own, which this PR does not change.How it is fixed
The date that the next date is relative to is now kept up to date as each item is parsed (
_last_relative_base), andparse_itemreceives that date instead of the list of everything parsed so far, so the time spent on an item no longer depends on how many items came before it.parse_found_objectshands the date to the two places where it parses part of a chunk again, in place of thealready_parsedlist it took (it is only called fromsearch_parseand from itself).set_relative_base, now unused, is removed. Nothing else changes: the same date is chosen as before, so results are identical.Tests and checks
tests/test_search_unparsed_word_runs.py; no existing test file is edited. Two count theDateDatareads for runs of 200 and 400 words ("hour "in English,"ngΓ y "in Vietnamese) and require fewer than 2.5x as many for the longer run; counting instead of timing makes them deterministic. Five pin which date each relative date is relative to, including after a run of words that do not parse, after a trailing quantity ("Fifty years after") and after dates in the same sentence.AssertionError: 111685 not less than 71500.0) and the five pinning tests pass; on this branch all 7 pass. Five hand-made mutants of the change (a relative date taken as the base, a date that did not parse taken as the base, the base not updated after a date, after a trailing quantity, or never used) are each caught by at least one test.TIMEZONE=UTC, with and without a fixedRELATIVE_BASE: 5,156 strings (a subset of a corpus of strings captured from the existing tests, per-language vocabulary combinations for all languages and edge cases with relative dates and unparsed words), 26,929 comparisons ofparse,get_date_data,search_dates(explicit languages,RELATIVE_BASEset,languages=None) andparse_found_objects: no difference.TZ=Pacific/AucklandandTZ=Pacific/Fiji), the minimum supported dependency versions (with and without the optional ones), all optional dependencies, mypy, the docs build, the scripts, the benchmarks,twine checkand the pre-commit hooks.π€ Generated with Claude Code