Skip to content

tests: unblock nightly sync integ tests on pytest 9.1 - #9189

Merged
roger-zhangg merged 2 commits into
developfrom
fix-pytest91-class-scoped-fixture-ignore
Aug 18, 2026
Merged

roger-zhangg merged 2 commits into
developfrom
fix-pytest91-class-scoped-fixture-ignore

Conversation

@roger-zhangg

Copy link
Copy Markdown
Member

Which issue(s) does this change fix?

N/A

Why is this change necessary?

sync-code and sync-watch have failed on every scheduled integration run since pytest was bumped 9.0.3 β†’ 9.1.1 (#9095, 2026-07-31).

Run pytest sync-code
2026-07-30 (last pre-bump) 9.0.3 success β€” 89 passed, 1 rerun
2026-08-14 9.1.1 failure β€” 37 passed, 52 errors, 156 rerun

The logs show only pytest's own internal assertion, assert not self._finalizers (_pytest/fixtures.py:1221), with nothing pointing at the real cause. Reproducible in ten lines, no AWS needed:

class TestFixt:
    @pytest.fixture(scope="class")
    def fixt(self): yield
    def test_1(self, fixt): pass
    def test_2(self, fixt): pass
pytest 9.0.3           -> 2 passed
pytest 9.1.1 -W error  -> 2 errors

Four links:

  1. pytest 9.1 deprecated class-scoped fixtures declared as instance methods (PytestRemovedIn10Warning). TestSyncCodeBase.execute_infra_sync / sync_code_base are exactly this shape.
  2. pytest.ini sets filterwarnings = error, turning that warning into a setup failure on the first test of every affected class.
  3. pytest then raises its own internal AssertionError for the remaining tests in the class β€” upstream pytest-dev/pytest#14775, still open. Root cause is fixtures.py:1147: finish() early-returns when cached_result is None without clearing _finalizers.
  4. --reruns 3 hides the cause. pytest-rerunfailures reports only the final attempt, so the real warning is discarded and only the internal AssertionError survives β€” which is why the CI logs contain zero occurrences of the actual error.

Confirmed link 4 directly, same test with and without --reruns:

without --reruns:  test_1 => PytestRemovedIn10Warning   <- real cause
                   test_2 => AssertionError
with --reruns 3:   test_1 => AssertionError             <- cause gone

The counts line up exactly: 52 errors Γ— 3 reruns = 156 reruns (sync-watch: 12 Γ— 3 = 36).

How does it address the issue?

Marks the warning as ignored on the three classes that declare these fixtures β€” TestSyncCodeBase, LogsIntegTestCases, TestTracesCommand. Six lines total.

Safe here specifically: these fixtures assign to the class (TestSyncCodeBase.stack_name = ...), not to self, so the hazard the deprecation warns about β€” attributes set on self being invisible to tests β€” does not apply to them.

A class mark rather than a pytest.ini entry, so the rest of the suite still fails on this warning. And a class mark rather than a module-level pytestmark, because pytestmark is module-scoped while TestSyncCodeBase is subclassed from other modules (test_sync_build_in_source.py, test_sync_adl.py) β€” which is precisely why one base class took out both sync-code and sync-watch.

What side effects does this change have?

Deferred, not avoided: converting these fixtures is required before pytest 10 removes the behaviour. Left out on purpose β€” it is not mechanical. execute_infra_sync derives its stack name from self._method_to_stack_name(self.id()), and unittest's id() needs an instance, so any conversion changes live CloudFormation stack names. All three autouse fixtures also assert via self.assertEqual, which under @classmethod would bind the first argument as self rather than fail loudly. That belongs in its own PR, verified against real AWS.

Testing

  • Against the repo's real pytest.ini, the same parameterized class-scoped shape goes 6 errors β†’ 6 passed with the mark
  • An unmarked equivalent still errors, so suite-wide protection is intact
  • Confirmed pytest resolves the mark through the MRO for subclasses carrying their own marks (TestLogsCommandWithRegularStack)
  • tests/integration/{sync,logs,traces} collect 266 tests; black and ruff clean

Mandatory Checklist

PRs will only be reviewed after checklist is complete

@roger-zhangg
roger-zhangg requested a review from a team as a code owner August 17, 2026 19:09

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: e1f4bf6..166ee89
Files: 3
Comments: 1

LOG = logging.getLogger(__name__)


# pytest 9.1 deprecation, inapplicable here: these fixtures assign to the class, not self. Remove before pytest 10.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[GENERAL] The justification in the comment is only half true, and that matters for what happens next.

The fixtures do assign results to the class (TestSyncCodeBase.template_path, TestSyncCodeBase.stack_name), but they also read instance state and call unittest methods on self:

@pytest.fixture(scope="class")
def execute_infra_sync(self):
   if self.test_data_path is None: ...
   TestSyncCodeBase.stack_name = self._method_to_stack_name(self.id())   # instance-derived
   ... dependency_layer=self.dependency_layer, image_repository=self.ecr_repo_name, kms_key_id=self.kms_key

@pytest.fixture(autouse=True, scope="class")
def sync_code_base(self, execute_infra_sync):
   self.assertEqual(...)   # unittest assertion on a per-test instance

The same holds in the other two files: deploy_testing_stack reads self.test_template_folder and logs_base / sync_code_base call self.assertEqual. That is exactly the binding the deprecation targets β€” the self a class-scoped fixture receives is an instance created for one particular test in the class (here, the first one), so self.id() and the parameterized_class attributes come from an arbitrary member of the class rather than from the class itself. So the pattern is being used, not "inapplicable"; the suppression defers the migration rather than documenting a false positive.

Two consequences worth addressing before merge:

  • Remove before pytest 10 reads as "delete this line," but deleting it without converting the fixtures will hard-fail the suite. Please state the required follow-up explicitly (e.g., move the shared setup into setUpClass/a request.cls-based fixture, or make the fixture a @classmethod/@staticmethod and replace self.assertEqual with plain assert), and ideally link a tracking issue.
  • Because the mark sits on the class, it silences this warning for every class-scoped fixture added to these classes later, including cases where the instance binding genuinely is a bug. Applying the filter narrowly (only where needed) or keeping it in one place with a pointer to the follow-up work limits that blast radius.

The unblock itself is reasonable; the ask is that the comment describe why the suppression is temporary rather than assert the deprecation does not apply.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the bot. If the extra info we're adding is not correct, let's just change to something simpler so it's not confusing.

# Deprecated in pytest 9.1. Fix before pytest 10.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion, applied the fix

sync-code and sync-watch have failed on every scheduled run since pytest was bumped
9.0.3 -> 9.1.1 (#9095, 2026-07-31). Latest: 37 passed, 52 errors. The last pre-bump run
passed with 89 passed, 1 rerun.

Reproducible in ten lines, no AWS:

    class TestFixt:
        @pytest.fixture(scope="class")
        def fixt(self): yield
        def test_1(self, fixt): pass
        def test_2(self, fixt): pass

  pytest 9.0.3          -> 2 passed
  pytest 9.1.1 -W error -> 2 errors

1. pytest 9.1 deprecated class-scoped fixtures declared as instance methods.
2. `filterwarnings = error` in pytest.ini makes that a setup failure on the first test
   of every affected class.
3. pytest then raises its own internal AssertionError ("assert not self._finalizers",
   fixtures.py:1221) for the remaining tests in the class -- upstream
   pytest-dev/pytest#14775, still open. Root cause is fixtures.py:1147: finish()
   early-returns when cached_result is None without clearing _finalizers.
4. `--reruns 3` reports only the final attempt, so the real warning is discarded and only
   the internal AssertionError survives. That is why the CI logs contain zero occurrences
   of the actual cause. Counts line up: 52 errors x 3 reruns = 156 reruns (sync-watch:
   12 x 3 = 36).

Suppressing is safe here specifically: these fixtures assign to the class
(`TestSyncCodeBase.stack_name = ...`), not to `self`, so the hazard the deprecation warns
about -- attributes set on self being invisible to tests -- does not apply.

Marked on the three classes that declare these fixtures rather than in pytest.ini, so the
rest of the suite still fails on this warning. A class mark rather than a module-level
pytestmark because TestSyncCodeBase is subclassed from other modules
(test_sync_build_in_source.py, test_sync_adl.py) and a module mark would not reach them,
which is why one base class took out both sync-code and sync-watch.

Converting the fixtures is required before pytest 10 and is deliberately not done here:
execute_infra_sync derives its stack name from self._method_to_stack_name(self.id()), and
unittest's id() needs an instance, so it changes live CloudFormation stack names.

Testing: verified against the repo's real pytest.ini that the same parameterized
class-scoped shape goes 6 errors -> 6 passed with the mark, and that an unmarked one still
errors, so suite-wide protection is intact. Confirmed pytest resolves the mark through the
MRO for subclasses that carry their own marks. tests/integration/{sync,logs,traces}
collect 266 tests. black and ruff clean.
@roger-zhangg
roger-zhangg force-pushed the fix-pytest91-class-scoped-fixture-ignore branch from 166ee89 to d373c81 Compare August 17, 2026 21:28

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: e1f4bf6..d373c81
Files: 3
Comments: 1

Review feedback: keying the filter on the message prose means it stops applying if
upstream rewords the warning, and that failure is silent.

Confirmed the failure mode rather than assuming it. `filterwarnings` marks reach
warnings.filterwarnings, where the message becomes re.compile(msg, re.I).match(...), so a
stale phrase simply never matches: with a deliberately reworded pattern the probe goes
back to 4 errors / 8 reruns showing only the bare "assert not self._finalizers", exactly
the opaque failure this change exists to prevent.

Also checked the suggested compromise of combining message and category. It does not
help -- warnings.filterwarnings requires both fields to match, so the combined form is as
brittle as message-only:

  stale message, message only                    -> error
  stale message, message + category              -> error
  stale message, category only                   -> passes

So the filter is now category-only. `pytest.PytestRemovedIn10Warning` is exported by
pytest 9.1.1, and this matches existing convention: test_build_cmd.py:58 uses
"ignore::ResourceWarning" and pytest.ini has six ignore::Category entries.

The tradeoff is that it also suppresses the other eight PytestRemovedIn10Warning messages
in pytest 9.1.1 inside these three classes. That is acceptable: they cover config.inicfg,
pytest.console_main, FixtureDef.has_location, fixture baseid/nodeid, parsefactories
nodeid, monkeypatch.syspath_prepend and --pastebin -- plugin and config level APIs these
integration test classes do not touch.

Testing: unmarked probe still errors (6 errors, 18 rerun) so suite-wide protection is
intact; the category-marked equivalent passes. tests/integration/{sync,logs,traces}
collect 266 tests. black and ruff clean.
@roger-zhangg
roger-zhangg enabled auto-merge August 17, 2026 23:36
@roger-zhangg
roger-zhangg added this pull request to the merge queue Aug 17, 2026
Merged via the queue into develop with commit fc586de Aug 18, 2026
53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants