Repository navigation
tests: add CodSpeed benchmarks - #1365
Merged
Merged
Conversation
serhii73
force-pushed
the
add-cod_speed-1336
branch
from
August 10, 2026 12:56
b002117 to
55a4217
Compare
Codecov Reportβ
All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1365 +/- ##
=======================================
Coverage 97.10% 97.10%
=======================================
Files 236 236
Lines 3008 3008
=======================================
Hits 2921 2921
Misses 87 87 β View full report in Codecov by Harness. π New features to boost your workflow:
|
Add CodSpeed benchmarks and a GitHub Actions workflow that runs them, so a performance regression like the one fixed in #1335 is reported on the pull request that introduces it instead of being noticed as a hang. The benchmarks cover the paths #1335 touched: PATTERN.findall()/sub() in freshness_date_parser, the relative-split regex Dictionary builds from the locale's relative-type-regex entries, and dateparser.parse() end to end. Each long-digit-run benchmark is sized so that a reintroduced regression is a ~500x jump that still finishes and reports rather than timing out. The benchmarks live in a dedicated tox environment and are skipped when pytest-codspeed is not installed, leaving the regular test runs unchanged. Closes #1336 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
serhii73
force-pushed
the
add-cod_speed-1336
branch
from
August 10, 2026 13:04
55a4217 to
01cdfc4
Compare
Congrats! CodSpeed is installed π
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
Member
|
π |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1336
Why
#1335 fixed a serious performance problem in the relative-date regexes. The
number was spelled as the greedy
\d+[.,]?\d*, whose adjacent digit quantifiersbacktrack without ever changing the match, so the cost grew superlinearly with
the length of a digit run β
PATTERN.findall('9' * 3200)took ~23s. Making thequantifiers possessive (
\d++[.,]?\d*+) brought that back to ~10ms.Nothing in CI would have caught that regression when it was introduced: the
suite still passed, it just got slow. #1336 asks for CodSpeed so this class of
change is measured, following the precedent of scrapy/w3lib#247.
Benchmarks
tests/benchmarks/, 7 benchmarks. Each one runs a list of inputs rather than asingle string, because CodSpeed's simulation mode measures a single call and one
match on a short string is too small a unit of work to be meaningful.
test_benchmark_freshness_date_parser.pyβPATTERN, the regex #1335changed, on both of its call sites:
test_freshness_pattern_findallβ typical relative phrases (get_kwargs())test_freshness_pattern_findall_long_digit_runβ the make relative-date regex digit quantifiers possessiveΒ #1335 input shapetest_freshness_pattern_sub_long_digit_runβ same, via thesub()in_parse_time()test_freshness_get_kwargsThe typical inputs are the translated forms the freshness parser actually
receives (
"2 hour 50 minute ago", not"2 hours, 50 minutes ago"), sincePATTERNonly matches singular units β the untranslated strings would havebenchmarked a failed scan.
test_benchmark_languages.pyβ the second path #1335 touched: the relative-splitregex
Dictionarycompiles from the locale'srelative-type-regexentries,applied by
re.split()across the whole input.test_benchmark_parse.pyβdateparser.parse()end to end for relative andabsolute strings, so a slowdown anywhere in the pipeline is caught too.
Assertions sit outside the measured region and guard against a benchmark
silently degrading into measuring a no-op β a regex that stops matching would
otherwise show up as an improvement.
Would a #1335 revert actually be caught?
Measured locally, old greedy pattern vs. current possessive one, same inputs and
identical match results:
LONG_DIGIT_RUNis 800 digits rather than the 3200 of the original repro. 800 isalready a ~500x signal that no threshold can miss, and a regressed run still
finishes in seconds under instrumentation so CodSpeed reports it. At 3200 a
regressed run would need ~20 minutes per benchmark and the job would time out
without reporting anything.
CI
New
.github/workflows/codspeed.yml: push to master, pull requests (pathfiltered), and
workflow_dispatchso CodSpeed can collect the initial baseline.id-token: writeon the job for CodSpeed's OIDC authentication,contents: readat the top level, and
timeout-minutes: 30so a regressed run is capped ratherthan burning hours of runner time.
Benchmarks run through tox, as in w3lib:
tox -e benchmark --notestprepares theenvironment first, then
CodSpeedHQ/action@v5runstox -e benchmark --skip-pkg-installunder instrumentation, so the measuredcommand only runs pytest instead of rebuilding and reinstalling the package.
New
[testenv:benchmark]intox.ini, pinned to Python 3.13 because instructioncounts are not comparable between interpreter versions. It also sets
PYTHONHASHSEED = 0: tox otherwise picks a random hash seed per run, which wouldshow up as noise in the counts CodSpeed compares. Unlike w3lib this does not
hardcode
--codspeed-mode, so the action'smodeinput stays authoritative andcannot silently disagree with the tox command.
Impact on the existing test suite
None. No existing test was modified. The benchmarks are guarded by
pytest.importorskip("pytest_codspeed"), so they are collected only in thebenchmarkenvironment and skipped everywhere else.pytest-codspeedis adependency of that tox environment only, not of the package.
Status of the first CI run
The
Buildworkflow is green across the whole matrix, includingpre-commit,so the benchmarks skip cleanly on 3.10β3.14 and nothing else moved.
The CodSpeed job measured successfully β the log shows valgrind being installed,
skip building and installing the packagetaking effect,codspeed: 5.0.3 (enabled, mode: simulation, callgraph: enabled)and7 benchmarked / 7 passed, with the whole job taking 18s. So the instrumentationreally does engage through tox rather than quietly running plain tests.
It then fails on the last step only:
That is the expected outcome until
scrapinghub/dateparseris connected oncodspeed.io (CodSpeed GitHub App installed on the repository). Once it is
onboarded, OIDC authentication covers a public repo and no upload token is
needed;
workflow_dispatchis there so CodSpeed can then backfill a baseline.Nothing in this branch needs to change for that β it is a one-off account-side
step, and the red check will clear on the next run afterwards.