Skip to content

Stop ignoring numbers without a unit in relative dates - #1410

Merged
AdrianAtZyte merged 6 commits into
scrapinghub:masterfrom
AdrianAtZyte:freshness-stray-numbers
Oct 21, 2026
Merged

AdrianAtZyte merged 6 commits into
scrapinghub:masterfrom
AdrianAtZyte:freshness-stray-numbers

Conversation

@AdrianAtZyte

@AdrianAtZyte AdrianAtZyte commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Fixes #1034. Part of #1265.

@codspeed

codspeed Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by ×13

⚡ 2 improved benchmarks
✅ 8 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ test_freshness_pattern_findall_long_digit_run 12,942.5 µs 965.3 µs ×13
⚡ test_freshness_pattern_sub_long_digit_run 12,942.9 µs 965.4 µs ×13

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing AdrianAtZyte:freshness-stray-numbers (13daf0a) with master (a3d3b89)

Open in CodSpeed

@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (663348c) to head (13daf0a).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1410   +/-   ##
=======================================
  Coverage   99.03%   99.03%           
=======================================
  Files         239      239           
  Lines        3833     3841    +8     
=======================================
+ Hits         3796     3804    +8     
  Misses         37       37           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdrianAtZyte
AdrianAtZyte marked this pull request as ready for review September 23, 2026 09:37

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! This fixes #1034 (03/06/2020a gives 2020-06-03). The catch is that a relative date followed
by any unrelated number is now rejected, and in search_dates that is common prose:

from dateparser.search import search_dates

for s in ["Posted 3 days ago 15 comments", "John Doe 2 hours ago 3 replies",
          "Reviewed 3 weeks ago 12 people found this helpful", "Answered 2 years ago 7 upvotes"]:
    print(s, search_dates(s, languages=["en"]))

On master these give 3 days / 2 hours / 3 weeks / 2 years ago. On this branch (no RELATIVE_BASE)
they give 6 days / ~13 hours / 6 weeks / 4 years ago: the offset is computed from a wrong base. Once 3 day ago 15 is
rejected, search splits the chunk. parse_item parses 3 day in one candidate split and sets
RELATIVE_BASE on parser._settings without restoring it, so the 3 day ago of the next split is
computed from that. The leak is old, but this change is what sends ordinary text through it. With a
RELATIVE_BASE, some chunks are lost instead: "Posted 3 days ago | 15 comments" and
"Last seen 3 days ago at 5" go from 3 days ago to None.

The same rejection hits auto-detected French: search_dates("Il y a 5 min") is translated as
year 1 5 minute, and the stray 1 now makes it January 5th (master: 5 minutes ago). parse() also
loses the day for yesterday at 5, ayer a las 9, вчера в 12 or 3 days ago 10.30 (#1450 recovers
some of these).

Would it work to keep rejecting bare numbers before the quantity (the #1034 shape) and ignore the ones
after the last one? For example, in _are_all_words_units:

matches = list(PATTERN.finditer(date_string))
if matches:
    end = matches[-1].end()
    date_string = date_string[:end] + re.sub(r"\d+", "", date_string[end:])
date_string = PATTERN.sub("", date_string)

With that, the tests in this PR still pass, 03/06/2020a and the 1st of last month are still handled,
and the search_dates and parse() examples above go back to master's results. The
parser._settings leak in search.py parse_item is probably worth fixing on its own too: restoring
the settings after the re-parse also fixes the doubled offsets.

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review (claude code-review, at 822ca19). No data-loss or crash found; the full suite passes on this head (24479 passed). The rejection of a unit-less number before a relative phrase is the intended change.

Non-blocking notes:

  • The fix is partial and order-dependent: a leading number is now rejected (2020 3 days ago -> None), but digits after the last unit are still dropped silently (3 days ago 15, 5 days ago 2020 and tomorrow 2025 parse as the plain relative date). The new test 3 days ago 15 pins that behaviour, which is the opposite of the PR title.
  • When a time is detected, \d+ joins the skip list, so a stray number plus any time token disables the check (2020 3 days ago is None, but 2020 3 days ago 5pm still parses and drops 2020).
  • Inputs that parsed before now return None (2020 3 days ago, 12 3 days ago, 3 ago 2 days, 5 yesterday). Please add tests for the rejected cases, since the headline behaviour is currently untested.
  • has_time is a new required positional argument on the private _parse_date and _are_all_words_units; a default of False keeps subclasses working.
  • The skip list is matched by prefix with re.match, and PATTERN is run twice per candidate on the parse() hot path.

@AdrianAtZyte
AdrianAtZyte merged commit dda8b21 into scrapinghub:master Oct 21, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dateparser.parse() prefers current year also when 'STRICT_PARSING' is True

2 participants