Repository navigation
Keep the %z offset when parsing with date_formats - #1424
Conversation
Codecov Reportβ
All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1424 +/- ##
==========================================
+ Coverage 99.05% 99.08% +0.03%
==========================================
Files 239 239
Lines 3900 3935 +35
==========================================
+ Hits 3863 3899 +36
+ Misses 37 36 -1 β View full report in Codecov by Harness. π New features to boost your workflow:
|
serhii73
left a comment
There was a problem hiding this comment.
Thanks, this fixes #956 and #376 nicely, and the TIMEZONE / TO_TIMEZONE / RETURN_AS_TIMEZONE_AWARE handling now matches
the free-text parser.
One regression before merging: a format that combines a two-digit year with %z now raises.
import dateparser
dateparser.parse('26 Jan 18 16:32:21 +0000', date_formats=['%d %b %y %H:%M:%S %z'])- master:
datetime.datetime(2018, 1, 26, 16, 32, 21) - this PR:
TypeError: can't compare offset-naive and offset-aware datetimes
date_obj is now aware, and _apply_century_preference() compares it with a naive now. It happens with and without
RELATIVE_BASE, and under every PREFER_DATES_FROM. Comparing naive values there fixes it:
date_naive = date_obj.replace(tzinfo=None)
if now_naive < date_naive and prefer_from == "past":
...
elif now_naive >= date_naive and prefer_from == "future":
...With that change, '26 Jan 99 16:32:21 +0100' gives 1999 (past/default) and 2099 (future), keeping the +01:00 offset.
Could you add a %y + %z case to the new test?
serhii73
left a comment
There was a problem hiding this comment.
Code review (claude code-review, at e15e85d). The full suite passes on this head (24353 passed) and keeping the %z offset is the stated goal, so I'm approving. One thing needs a note before release:
- User-visible change.
parse('2021-09-22 12:30-1030', date_formats=['%Y-%m-%d %H:%M%z'])used to return a naive datetime and now returns2021-09-22 12:30:00-10:30with default settings. Callers comparing the result with naive datetimes will get aTypeError, and results change whenTIMEZONEorTO_TIMEZONEis set. Please add a changelog line and a sentence in thedate_formatsdocs about%z.
Non-blocking notes:
- The
RETURN_AS_TIMEZONE_AWARElogic now exists in three places (date_parser.py,freshness_date_parser.py,utils), andapply_timezone_from_settingschanged fromis not Trueto truthiness; a shared helper would avoid drift. - With
TIMEZONE='local'(the default) the result stays in the parsed offset, while any otherTIMEZONEconverts to it. That matches the non-format path, but the tests only cover theAsia/Tokyocase, notlocalwithTO_TIMEZONE. - The
%ycentury preference compares naive wall-clock times and ignores the parsed offset, so a date within a day ofRELATIVE_BASEcan land on the wrong side ofPREFER_DATES_FROM(e.g.1/1/99 00:00 +0100with a UTC base of 1999-01-01 00:00 andfuturegives 2099). %Znames (UTC,EST) are still dropped, so%Zresults are naive while%zresults are aware; worth documenting.
Keep both sentences of the date_formats docstring paragraphs: the PARSERS fallback sentence of this PR and the %z/%Z sentence from scrapinghub#1424. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Fixes #376, resolves #956, closes #840