Repository navigation
Speed up import by compiling timezone regexes lazily and combined - #1405
Conversation
Merging this PR will improve performance by 35.95%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing |
Codecov Reportβ
All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1405 +/- ##
==========================================
+ Coverage 98.98% 99.03% +0.05%
==========================================
Files 239 239
Lines 3628 3833 +205
==========================================
+ Hits 3591 3796 +205
Misses 37 37 β View full report in Codecov by Harness. π New features to boost your workflow:
|
serhii73
left a comment
There was a problem hiding this comment.
Thanks, the import-time win is great (dateparser's cumulative import drops from ~0.32 s to ~0.05 s here, and the combined regexes compile in ~0.1 s on first use).
One behaviour change, though: for the abbreviation template ((\W|\d|_)%s($|\W)), the precedence moved from "first entry of timezone_info_list that occurs anywhere in the string" to "leftmost occurrence in the string". So strings that carry two zone labels now resolve differently:
import dateparser
from dateparser.timezone_parser import pop_tz_offset_from_string as pop
dateparser.parse("10 Jan 2020 10:00 CET/CEST")
# master: 2020-01-10 10:00:00+02:00 this PR: None
# (same for "10 Jan 2020 10:00 CET / CEST" and "10.01.2020 10:00 (CET/CEST)")
pop("Jan 10, 2020 10:00 PST/PDT", as_offset=False)
# master: ('Jan 10, 2020 10:00 PST/', 'PDT') this PR: ('Jan 10, 2020 10:00 PDT', 'PST')
pop("Let's get on 10 Jan 2020 at 10:00 EST", as_offset=False)
# master: ("Let's get on 10 Jan 2020 at 10:00 ", 'EST') this PR: ("Let's on 10 Jan 2020 at 10:00 EST", 'GET')Since this is meant to be a pure speed-up (and first-entry-wins is what tz_abbreviation_conflicts.py and the table corrections assume), could _find_tz keep the old precedence? Scanning all matches of the non-anchored templates and keeping the lowest index does it. Running it against master over the test-suite strings plus ~20k synthetic one- and two-abbreviation strings and an offset grid gave identical output:
def _find_tz(string):
for pattern, timezones in _tz_regexes()[0]:
if pattern.flags & regex.REVERSE:
match = pattern.search(string)
matches = [match] if match else []
else:
# overlapped: in "EST/EDT" both matches share the "/"
matches = pattern.finditer(string, overlapped=True)
best = None
for match in matches:
group = next(k for k, v in match.groupdict().items() if v is not None)
index = int(group[2:])
if best is None or index < best[0]:
best = (index, match.span())
if best:
name, offset = timezones[best[0]]
return best[1], name, offset
return NoneA test such as pop_tz_offset_from_string("10 Jan 2020 10:00 CET/CEST", as_offset=False) == ("10 Jan 2020 10:00 CET/", "CEST") would pin it.
serhii73
left a comment
There was a problem hiding this comment.
Code review (claude code-review, at 6daf009). No wrong-output bug found: old and new pop_tz_offset_from_string, word_is_tz and is_timezone_token gave identical results on 60,000 random strings built from table names, case variants, UTC+5:30-style offsets, trailing text and newlines. The full suite passes on this head (24481 passed).
Non-blocking notes:
build_tz_offsetshas no leading underscore, and_tz_offsets,_search_regexand_search_regex_ignorecaseare removed. If external code imports them it will fail withImportError, so a small lazy shim (module__getattr__) would keep this non-breaking.@cacheon_tz_regexes()has no lock, so threads hitting the first call at the same time each compile the whole table (the result is the same, but the first request is slow). The repo treats thread safety as a requirement (#1346), so a lock around the build is worth considering.- The old quick gate on the combined regex is gone, so short strings without a timezone are slower (about 39 us against 21 us for a 17-character string; longer strings are faster).
- Picking the reverse search via
template.endswith("$")ties behaviour to the template text; an explicit field intimezone_info_listwould be safer._tz_regexes()returns a positional 3-tuple that is indexed as[1]and[2]; a NamedTuple would be safer. - The benchmark calls
re.purge()andregex.purge(), which clears the global caches for the other benchmarks in the same run. utilsnow relies on_find_tztie-breaking and the leading-space contract with no direct test.
Fixes #533