Skip to content

Fix negative subsecond timestamp parsing - #1387

Merged
serhii73 merged 2 commits into
scrapinghub:masterfrom
jakezwang:fix/negative-subsecond-timestamps
Oct 20, 2026
Merged

serhii73 merged 2 commits into
scrapinghub:masterfrom
jakezwang:fix/negative-subsecond-timestamps

Conversation

@jakezwang

@jakezwang jakezwang commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Negative millisecond and microsecond timestamps apply the fractional digits with a positive sign. For example, -1570308760263111 parses as 1920-03-29 03:07:20.263111 in UTC instead of 1920-03-29 03:07:19.736889.

Normalize negative timestamps into whole seconds and a nonnegative microsecond remainder before converting the timezone. The calculation uses integers to preserve microsecond precision. Add seven parser regressions covering milliseconds, microseconds, zero fractions, and the microsecond boundaries.

Validation after syncing with current master:

  • TZ=UTC tox -e all on Python 3.14.7, with calendar and language detection extras: 24,627 passed, 4 skipped, 1 expected failure.
  • tox -e mypy: no issues in 285 source files (Python 3.10).
  • tox -e pre-commit: Ruff lint/format and zizmor passed.
  • git diff --check passed.

The initial local run in America/New_York hit two unchanged upstream clean-API tests that assume the default timezone differs from EST. The full suite passes with TZ=UTC, matching the CI configuration.

@codspeed

codspeed Bot commented Sep 21, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

βœ… 7 untouched benchmarks


Comparing jakezwang:fix/negative-subsecond-timestamps (861d522) with master (5e77cac)

Open in CodSpeed

@codecov

codecov Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

Codecov Report

βœ… All modified and coverable lines are covered by tests.
βœ… Project coverage is 97.21%. Comparing base (5e77cac) to head (861d522).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1387   +/-   ##
=======================================
  Coverage   97.20%   97.21%           
=======================================
  Files         236      236           
  Lines        3081     3083    +2     
=======================================
+ Hits         2995     2997    +2     
  Misses         86       86           

β˜” View full report in Codecov by Harness.
πŸ“’ Have feedback on the report? Share it here.

πŸš€ New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jakezwang

jakezwang commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Merged current master in 2db57e7 and resolved the test conflict by preserving both the timestamp regressions and upstream type annotations.

Validation: TZ=UTC tox -e all passed 24,627 tests (4 skipped, 1 expected failure) on Python 3.14.7. The full mypy and pre-commit environments also pass. The initial run under America/New_York hit two unrelated upstream timezone assumptions; the successful full run uses the same UTC setting as CI.

The new workflows are waiting for approval; could a maintainer approve the CI runs?

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff. Negative sub-second timestamps are now floored correctly via divmod, so -1015673450000001 gives 12:29:09.999999 instead of 12:29:10.000001. The updated expectation is right, and the new tests cover the boundaries. Full suite passes on this branch merged with current master (24793 passed).

@serhii73
serhii73 merged commit c6cfdff into scrapinghub:master Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants