Repository navigation
Parse English number words from thirteen to ninety-nine - #1416
Conversation
Codecov Reportβ
All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1416 +/- ##
=======================================
Coverage 99.03% 99.04%
=======================================
Files 239 239
Lines 3853 3885 +32
=======================================
+ Hits 3816 3848 +32
Misses 37 37 β View full report in Codecov by Harness. π New features to boost your workflow:
|
serhii73
left a comment
There was a problem hiding this comment.
Thanks, the relative expressions work well ("in twenty minutes", "twenty one days ago", "forty-five seconds ago", "twenty-five minutes later").
One problem before merging: the new words are turned into digits everywhere, not only before a time unit, so ordinary English text now
behaves as if it had bare numbers in it, and search_dates loses dates it used to find:
from datetime import datetime
import dateparser
from dateparser.search import search_dates
S = {"RELATIVE_BASE": datetime(2020, 6, 15, 12)}
search_dates("She turned thirty on 14 February 2018.", settings=S)
# master: [('on 14 February 2018', datetime(2018, 2, 14, 0, 0))]
# this PR: None
search_dates("She turned thirty on February 14, 2018.", settings=S)
# master: [('on February 14, 2018', datetime(2018, 2, 14, 0, 0))]
# this PR: [('2018', datetime(2018, 6, 15, 0, 0))]
search_dates("About fifty students arrived in 2019 and forty left in 2020.", settings=S)
# master: [('in 2019 and', datetime(2019, 6, 15, 0, 0)), ('in 2020', datetime(2020, 6, 15, 0, 0))]
# this PR: [('in 2020', datetime(2020, 6, 15, 0, 0))]It also adds false positives: dateparser.parse("twenty") now returns the 20th of the current month, parse("Nineteen Eighty-Four") returns
1984-06-19, and search_dates("Chapter Twenty: The return. It was 5 June 2019.") now also returns ('Twenty: The', <20th of this month>).
Could the new entries only apply when a time unit follows, as the old ... later rule did? For example:
- thirteen(?=\s+(?:decades?|years?|y|months?|mo|weeks?|w|days?|d|hours?|hrs?|h|minutes?|mins?|m|seconds?|secs?|s)\b): '13'
- twenty[ -]([1-9])(?=\s+(?:decades?|years?|y|months?|mo|weeks?|w|days?|d|hours?|hrs?|h|minutes?|mins?|m|seconds?|secs?|s)\b): 2\1
- twenty(?=\s+(?:decades?|years?|y|months?|mo|weeks?|w|days?|d|hours?|hrs?|h|minutes?|mins?|m|seconds?|secs?|s)\b): '20'With that lookahead on each new entry, all the tests added here still pass, and the examples above behave as they do on master.
serhii73
left a comment
There was a problem hiding this comment.
Code review (claude code-review, at de74daf). The full suite passes on this head and I found no regression for inputs that parsed before; the notes below are edge cases in the new feature.
Non-blocking notes:
one thirty minutes agoandone twenty days agobecome1 30 minutes agoand1 20 days ago, and the leading1is silently dropped, giving 30 minutes or 20 days ago.one thirtyis more likely a clock time, so this is wrong output. The unconditionalone..twelverules run before the tens rules; a guard would fix it.- The unit lookahead includes the single letters
y d h m s wand the wordsecond, soTwenty second street meeting on 5 June 2019can produce a bogusTwenty secondmatch insearch_dates, andtwenty-1 days agois accepted as 21 days. - Only 1-99 are handled:
one hundred days agostill returns None. Compound forms accept a single space or hyphen only (twenty one days agowith two spaces fails). - The same ~200-character unit alternation is copied into 25 rules in
en.pyanden.yaml, so adding a unit (likefortnightin #1539) means editing 25 places; one rule with a tens-and-units mapping would be shorter and cheaper to run on every parse. - The
laterrule now only accepts digits and relies on rule order. Tests cover only a fewlaterforms and have no negative cases for the unit-ambiguity examples above.
Resolves #1236