Skip to content

Speed up import by compiling timezone regexes lazily and combined - #1405

Merged
AdrianAtZyte merged 8 commits into
scrapinghub:masterfrom
AdrianAtZyte:faster-tz-import
Oct 21, 2026
Merged

AdrianAtZyte merged 8 commits into
scrapinghub:masterfrom
AdrianAtZyte:faster-tz-import

Conversation

@AdrianAtZyte

Copy link
Copy Markdown
Member

Fixes #533

@codspeed

codspeed Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 35.95%

1 warning

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

⚑ 2 improved benchmarks
βœ… 7 untouched benchmarks
πŸ†• 1 new benchmark

Performance Changes

Benchmark BASE HEAD Efficiency
⚑ test_parse_relative_date_strings 31.5 ms 21.2 ms +48.7%
⚑ test_parse_autodetected_date_strings 96.5 ms 77.6 ms +24.3%
πŸ†• test_timezone_regexes_first_use N/A 611.4 ms N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing AdrianAtZyte:faster-tz-import (2a3f20e) with master (f43ba92)

Open in CodSpeed

@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

βœ… All modified and coverable lines are covered by tests.
βœ… Project coverage is 99.03%. Comparing base (2f87862) to head (2a3f20e).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1405      +/-   ##
==========================================
+ Coverage   98.98%   99.03%   +0.05%     
==========================================
  Files         239      239              
  Lines        3628     3833     +205     
==========================================
+ Hits         3591     3796     +205     
  Misses         37       37              

β˜” View full report in Codecov by Harness.
πŸ“’ Have feedback on the report? Share it here.

πŸš€ New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdrianAtZyte
AdrianAtZyte marked this pull request as ready for review September 23, 2026 08:25

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the import-time win is great (dateparser's cumulative import drops from ~0.32 s to ~0.05 s here, and the combined regexes compile in ~0.1 s on first use).

One behaviour change, though: for the abbreviation template ((\W|\d|_)%s($|\W)), the precedence moved from "first entry of timezone_info_list that occurs anywhere in the string" to "leftmost occurrence in the string". So strings that carry two zone labels now resolve differently:

import dateparser
from dateparser.timezone_parser import pop_tz_offset_from_string as pop

dateparser.parse("10 Jan 2020 10:00 CET/CEST")
# master: 2020-01-10 10:00:00+02:00      this PR: None
# (same for "10 Jan 2020 10:00 CET / CEST" and "10.01.2020 10:00 (CET/CEST)")

pop("Jan 10, 2020 10:00 PST/PDT", as_offset=False)
# master: ('Jan 10, 2020 10:00 PST/', 'PDT')      this PR: ('Jan 10, 2020 10:00 PDT', 'PST')

pop("Let's get on 10 Jan 2020 at 10:00 EST", as_offset=False)
# master: ("Let's get on 10 Jan 2020 at 10:00 ", 'EST')      this PR: ("Let's on 10 Jan 2020 at 10:00 EST", 'GET')

Since this is meant to be a pure speed-up (and first-entry-wins is what tz_abbreviation_conflicts.py and the table corrections assume), could _find_tz keep the old precedence? Scanning all matches of the non-anchored templates and keeping the lowest index does it. Running it against master over the test-suite strings plus ~20k synthetic one- and two-abbreviation strings and an offset grid gave identical output:

def _find_tz(string):
    for pattern, timezones in _tz_regexes()[0]:
        if pattern.flags & regex.REVERSE:
            match = pattern.search(string)
            matches = [match] if match else []
        else:
            # overlapped: in "EST/EDT" both matches share the "/"
            matches = pattern.finditer(string, overlapped=True)
        best = None
        for match in matches:
            group = next(k for k, v in match.groupdict().items() if v is not None)
            index = int(group[2:])
            if best is None or index < best[0]:
                best = (index, match.span())
        if best:
            name, offset = timezones[best[0]]
            return best[1], name, offset
    return None

A test such as pop_tz_offset_from_string("10 Jan 2020 10:00 CET/CEST", as_offset=False) == ("10 Jan 2020 10:00 CET/", "CEST") would pin it.

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review (claude code-review, at 6daf009). No wrong-output bug found: old and new pop_tz_offset_from_string, word_is_tz and is_timezone_token gave identical results on 60,000 random strings built from table names, case variants, UTC+5:30-style offsets, trailing text and newlines. The full suite passes on this head (24481 passed).

Non-blocking notes:

  • build_tz_offsets has no leading underscore, and _tz_offsets, _search_regex and _search_regex_ignorecase are removed. If external code imports them it will fail with ImportError, so a small lazy shim (module __getattr__) would keep this non-breaking.
  • @cache on _tz_regexes() has no lock, so threads hitting the first call at the same time each compile the whole table (the result is the same, but the first request is slow). The repo treats thread safety as a requirement (#1346), so a lock around the build is worth considering.
  • The old quick gate on the combined regex is gone, so short strings without a timezone are slower (about 39 us against 21 us for a 17-character string; longer strings are faster).
  • Picking the reverse search via template.endswith("$") ties behaviour to the template text; an explicit field in timezone_info_list would be safer. _tz_regexes() returns a positional 3-tuple that is indexed as [1] and [2]; a NamedTuple would be safer.
  • The benchmark calls re.purge() and regex.purge(), which clears the global caches for the other benchmarks in the same run.
  • utils now relies on _find_tz tie-breaking and the leading-space contract with no direct test.

@AdrianAtZyte
AdrianAtZyte merged commit 663348c into scrapinghub:master Oct 21, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import very slow

2 participants