Repository navigation
[RFC] Explicit mechanism for supporting xfail and skip in [third-party] plugins #7327
Description
While working on pytest-dev/pytest-forked#34 I had to go through source code in the skipping plugin to figure out how to make pytest show xfailed tests for runs made by an external plugin. And here's my observations:
- It's undocumented how to integrate xfail with test reports from plugins.
- I learned that if you set
outcome='xfailed'andwasxfail = 'reason', pytest will show normal xfailed output in the failures and summary sections but the progress section will show the failure (Fletter instead ofX/x). - I discovered that I have to set
outcome='skipped'(along withwasxfail = 'reason'!) to make it show consistent output but I need to set it only if I discover that the test has xfail marker. Wait, what?skipped? Really? But I really ran it... - I need to inspect all the marks and their conditionals. I need to figure out if the conditional expression of any mark is true or it raises the expected exception.
- Internal implementation details leak into the public API... (at least this is how I see the need to set certain magic combination of values and do manual condition matching)
So here's what I think could be improved:
- Docs. It should be explicitly documented how to make your plugin work with xfail.
- There could be some sort of public API to make the test result xfailed. (I don't really like this so see (3))
- Implement a better xfail processing mechanism on the pytest internals side.
- Plugins should cleanly report failures and passes
- pytest should convert the test reports to have proper attributes related to xfail after receiving results from plugins
- There should be a mechanism to convey an exception that caused the failure in the test run from plugin to pytest so that it could also match it against the
raisesarg
Activity
[-][RFC] Explicit mechanism for supporting xfail in [third-party] plugins[/-][+][RFC] Explicit mechanism for supporting xfail and skip in [third-party] plugins[/+]webknjaz commentedon Jul 20, 2020
The issue I faced today (pytest-dev/pytest-forked#44) made me rethink this a bit.
skip, not onlyxfail(and maybe something else too).skipifthat has a truthy condition, it'd be evaluated first and would exclude the test from the execution plan, not giving any chance to the plugins to start running it. This would also help plugins to be independent of any unknown markers, not having to implement support for checking all those skips and xfails...RonnyPfannschmidt commentedon Jul 21, 2020
It is a problem that forked does move the complete evaluation into a subprocess
So the pytest parts that ought to handle marker trigger in the subprocess instead of the parent.
This flaw is part of why I'd like to see a forked alike that's based running a complete pytest session in the subprocess and not just forking inside the runtestprotocol hook
webknjaz commentedon Jul 21, 2020
Interesting... Though, this should be somehow documented too.