Skip to content

Support date_formats in search_dates with the ngram strategy - #1436

Merged
AdrianAtZyte merged 7 commits into
scrapinghub:masterfrom
AdrianAtZyte:search-date-formats
Oct 26, 2026
Merged

AdrianAtZyte merged 7 commits into
scrapinghub:masterfrom
AdrianAtZyte:search-date-formats

Conversation

@AdrianAtZyte

@AdrianAtZyte AdrianAtZyte commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Resolves #846, resolves #1160

@codspeed

codspeed Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

βœ… 10 untouched benchmarks


Comparing AdrianAtZyte:search-date-formats (a4b334e) with master (535089d)

Open in CodSpeed

@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

βœ… All modified and coverable lines are covered by tests.
βœ… Project coverage is 99.18%. Comparing base (535089d) to head (a4b334e).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1436      +/-   ##
==========================================
+ Coverage   99.09%   99.18%   +0.08%     
==========================================
  Files         239      239              
  Lines        3997     4052      +55     
==========================================
+ Hits         3961     4019      +58     
+ Misses         36       33       -3     

β˜” View full report in Codecov by Harness.
πŸ“’ Have feedback on the report? Share it here.

πŸš€ New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdrianAtZyte
AdrianAtZyte marked this pull request as ready for review September 23, 2026 15:08

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review (claude code-review, at 107080f; probes run on this head). date_formats is a new opt-in parameter of search_dates (master has none), so nothing that worked before can regress and I'm approving. One limitation deserves attention before this is advertised:

  • Formats are not authoritative. Because the longest-first n-gram scan lets a fuzzy parse of an earlier or longer n-gram win, search_dates('A 15 March 2024 B', strategy='ngram', languages=['en'], date_formats=['%d %B %Y']) returns [('A 15', ...), ('March 2024', ...)], the same as without formats, and 'x 2024-03-15 14:30 y' with ['%Y-%m-%d %H:%M'] returns the span 2024-03-15 14:30 y with a garbage time. The format only helps when the text before it does not fuzzy-parse. Trying date_formats strictly at every n-gram size first, and falling back to fuzzy parsing afterwards, would fix it.

Non-blocking notes:

  • A bare str or a one-shot iterator for date_formats returns None with a logged warning per candidate (the TypeError is swallowed by the broad except). Validate once up front, turn it into a tuple, and raise on a str.
  • Candidates are matched against the single-space-joined tokens rather than the original span, so formats with (, ), @, | or multiple spaces never match, silently.
  • The guard uses truthiness, so date_formats=[] with the split strategy is accepted; a non-empty list raises ValueError.
  • _parse_candidate gains a required positional parameter (give it = None), the docs in search.py and __init__.py differ, and tests cover only one happy path (no leading words, iterators, multi-token formats, add_detected_language or RETURN_TIME_SPAN).

@AdrianAtZyte
AdrianAtZyte merged commit 3b43c90 into scrapinghub:master Oct 26, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add custom formats to search_dates search_dates doesn't support custom date formats

2 participants