Getting pytest to ignore Test* classes in the source folder?

[wpietri]

I'm working on a project where we're building safety tests for LLMs. Inevitably, we have many real domain objects starting with Test. They all live in our src folder, so I would like pytest to never discover anything in src. After much googling, I have tried a number of things in pyproject.toml without success. Here's an example project:

https://github.com/wpietri/pytest-collection-issue

I should note that I could tag each individual object starting with __test__ = False. However, I'd like to solve this problem once and at a high level.

I'm sure this must be doable, and would appreciate any suggestions. Thanks!

[nicoddemus]

Hi @wpietri,

See the python_classes configuration value.

You might also configure testpaths to point to tests only.

[wpietri]

Hey, @nicoddemus! Thanks for the speedy reply.

Regarding python_classes, I'm not quite sure what I'd do with that. I want it to find Test* classes in the tests directory, but not in the src directory. Because my actual project is about a different kind of tests than pytest is thinking of, the names aren't currently distinct.

Regarding testpaths, I tried adding testpaths = ["tests"] to the appropriate pyproject.toml section. I still get the warning. As I do when I run pytest tests. The specific warning message is:

src/pytest_collection_issue/domain.py:6
  src/pytest_collection_issue/domain.py:6: PytestCollectionWarning: cannot collect test class 'Testament' because it has a __init__ constructor (from: tests/test_domain.py)
    class Testament(object):

That makes me think that it is discovering the Testament class via its inclusion in tests/test_domain.py, so it doesn't seem to be a problem with the initial paths.

I figure I must just be getting something basic wrong here, as separating by folder seems like an obvious need, but I'm not sure what I'm missing.

[nicoddemus]

Ahh sorry, I quickly glanced at the question and missed the fine print.

I think the easiest way in your case is to change the test classes to use some other prefix (say Suite*) together with python_classes. pytest collects tests by importing the module and looking at the classes available in that module, and given the model classes are imported in the test module (naturally), they end up being candidates for "test classes".

[wpietri]

Ah, got it. Thanks!

Given that the warning message recognizes that the file with the objected-to class is in the src directory, it seems at least theoretically possible for pytest to skip looking at those. Would you like me to add that as a feature request?

[nicoddemus]

Would you like me to add that as a feature request?

Currently pytest does not consider where a certain class is defined in order to decide if it should be collected or not, it just looks at the symbols in the collected module -- this is not an explicit feature, but I'm certain it is used by test suites.

It would be possible for pytest to only consider classes/functions defined in the collected module, as opposed to looking at all the symbols in the module (which can be imported from somewhere else).

Feel free to open a feature request to foment discussion. 👍

[wpietri]

Thanks!

And in case anybody else needs a temporary workaround, putting this after my src imports but before my test classes seems to fix things for me:

for a_class in [i[1] for i in (globals().items()) if inspect.isclass(i[1])]:
    if a_class.__name__.startswith('Test'):
        a_class.__test__ = False

[wpietri]

In case anybody is interested, here's the feature request: #12749