Skip to content

tests: add CodSpeed benchmarks - #1365

Merged
AdrianAtZyte merged 1 commit into
masterfrom
add-cod_speed-1336
Aug 13, 2026
Merged

AdrianAtZyte merged 1 commit into
masterfrom
add-cod_speed-1336

Conversation

@serhii73

@serhii73 serhii73 commented Aug 10, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1336

Why

#1335 fixed a serious performance problem in the relative-date regexes. The
number was spelled as the greedy \d+[.,]?\d*, whose adjacent digit quantifiers
backtrack without ever changing the match, so the cost grew superlinearly with
the length of a digit run β€” PATTERN.findall('9' * 3200) took ~23s. Making the
quantifiers possessive (\d++[.,]?\d*+) brought that back to ~10ms.

Nothing in CI would have caught that regression when it was introduced: the
suite still passed, it just got slow. #1336 asks for CodSpeed so this class of
change is measured, following the precedent of scrapy/w3lib#247.

Benchmarks

tests/benchmarks/, 7 benchmarks. Each one runs a list of inputs rather than a
single string, because CodSpeed's simulation mode measures a single call and one
match on a short string is too small a unit of work to be meaningful.

test_benchmark_freshness_date_parser.py β€” PATTERN, the regex #1335
changed, on both of its call sites:

  • test_freshness_pattern_findall β€” typical relative phrases (get_kwargs())
  • test_freshness_pattern_findall_long_digit_run β€” the make relative-date regex digit quantifiers possessiveΒ #1335 input shape
  • test_freshness_pattern_sub_long_digit_run β€” same, via the sub() in _parse_time()
  • test_freshness_get_kwargs

The typical inputs are the translated forms the freshness parser actually
receives ("2 hour 50 minute ago", not "2 hours, 50 minutes ago"), since
PATTERN only matches singular units β€” the untranslated strings would have
benchmarked a failed scan.

test_benchmark_languages.py β€” the second path #1335 touched: the relative-split
regex Dictionary compiles from the locale's relative-type-regex entries,
applied by re.split() across the whole input.

test_benchmark_parse.py β€” dateparser.parse() end to end for relative and
absolute strings, so a slowdown anywhere in the pipeline is caught too.

Assertions sit outside the measured region and guard against a benchmark
silently degrading into measuring a no-op β€” a regex that stops matching would
otherwise show up as an improvement.

Would a #1335 revert actually be caught?

Measured locally, old greedy pattern vs. current possessive one, same inputs and
identical match results:

digits old (greedy) new (possessive) ratio
100 1.7 ms 24.6 Β΅s 69x
400 55 ms 224 Β΅s 248x
800 381 ms 719 Β΅s 530x
3200 20.5 s 9.7 ms 2109x

LONG_DIGIT_RUN is 800 digits rather than the 3200 of the original repro. 800 is
already a ~500x signal that no threshold can miss, and a regressed run still
finishes in seconds under instrumentation so CodSpeed reports it. At 3200 a
regressed run would need ~20 minutes per benchmark and the job would time out
without reporting anything.

CI

New .github/workflows/codspeed.yml: push to master, pull requests (path
filtered), and workflow_dispatch so CodSpeed can collect the initial baseline.
id-token: write on the job for CodSpeed's OIDC authentication, contents: read
at the top level, and timeout-minutes: 30 so a regressed run is capped rather
than burning hours of runner time.

Benchmarks run through tox, as in w3lib: tox -e benchmark --notest prepares the
environment first, then CodSpeedHQ/action@v5 runs
tox -e benchmark --skip-pkg-install under instrumentation, so the measured
command only runs pytest instead of rebuilding and reinstalling the package.

New [testenv:benchmark] in tox.ini, pinned to Python 3.13 because instruction
counts are not comparable between interpreter versions. It also sets
PYTHONHASHSEED = 0: tox otherwise picks a random hash seed per run, which would
show up as noise in the counts CodSpeed compares. Unlike w3lib this does not
hardcode --codspeed-mode, so the action's mode input stays authoritative and
cannot silently disagree with the tox command.

Impact on the existing test suite

None. No existing test was modified. The benchmarks are guarded by
pytest.importorskip("pytest_codspeed"), so they are collected only in the
benchmark environment and skipped everywhere else. pytest-codspeed is a
dependency of that tox environment only, not of the package.

Status of the first CI run

The Build workflow is green across the whole matrix, including pre-commit,
so the benchmarks skip cleanly on 3.10–3.14 and nothing else moved.

The CodSpeed job measured successfully β€” the log shows valgrind being installed,
skip building and installing the package taking effect,
codspeed: 5.0.3 (enabled, mode: simulation, callgraph: enabled) and
7 benchmarked / 7 passed, with the whole job taking 18s. So the instrumentation
really does engage through tox rather than quietly running plain tests.

It then fails on the last step only:

Uploading results
##[error]Failed to retrieve upload data: 401 Unauthorized
  -> Reason: Repository not found or the user does not have access to it.

That is the expected outcome until scrapinghub/dateparser is connected on
codspeed.io (CodSpeed GitHub App installed on the repository). Once it is
onboarded, OIDC authentication covers a public repo and no upload token is
needed; workflow_dispatch is there so CodSpeed can then backfill a baseline.
Nothing in this branch needs to change for that β€” it is a one-off account-side
step, and the red check will clear on the next run afterwards.

@serhii73
serhii73 force-pushed the add-cod_speed-1336 branch from b002117 to 55a4217 Compare August 10, 2026 12:56
@codecov

codecov Bot commented Aug 10, 2026 •

Copy link
Copy Markdown

Codecov Report

βœ… All modified and coverable lines are covered by tests.
βœ… Project coverage is 97.10%. Comparing base (b1f603e) to head (01cdfc4).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1365   +/-   ##
=======================================
  Coverage   97.10%   97.10%           
=======================================
  Files         236      236           
  Lines        3008     3008           
=======================================
  Hits         2921     2921           
  Misses         87       87           

β˜” View full report in Codecov by Harness.
πŸ“’ Have feedback on the report? Share it here.

πŸš€ New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Add CodSpeed benchmarks and a GitHub Actions workflow that runs them, so a
performance regression like the one fixed in #1335 is reported on the pull
request that introduces it instead of being noticed as a hang.

The benchmarks cover the paths #1335 touched: PATTERN.findall()/sub() in
freshness_date_parser, the relative-split regex Dictionary builds from the
locale's relative-type-regex entries, and dateparser.parse() end to end.
Each long-digit-run benchmark is sized so that a reintroduced regression is
a ~500x jump that still finishes and reports rather than timing out.

The benchmarks live in a dedicated tox environment and are skipped when
pytest-codspeed is not installed, leaving the regular test runs unchanged.

Closes #1336

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@serhii73
serhii73 force-pushed the add-cod_speed-1336 branch from 55a4217 to 01cdfc4 Compare August 10, 2026 13:04
@codspeed

codspeed Bot commented Aug 13, 2026

Copy link
Copy Markdown

Congrats! CodSpeed is installed πŸŽ‰

πŸ†• 7 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


Open in CodSpeed

@serhii73
serhii73 requested a review from AdrianAtZyte August 13, 2026 12:53
@AdrianAtZyte
AdrianAtZyte merged commit de7b986 into master Aug 13, 2026
20 of 21 checks passed
@AdrianAtZyte

Copy link
Copy Markdown
Member

πŸŽ‰

@serhii73
serhii73 deleted the add-cod_speed-1336 branch August 13, 2026 13:06
@serhii73 serhii73 mentioned this pull request Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CodSpeed

2 participants