-
Notifications
You must be signed in to change notification settings - Fork 228
Description
I just realized that omit = tests/* is in the very first example in the docs (https://pytest-cov.readthedocs.io/en/latest/config.html). This probably makes people think this is the way to go. It is, however, under [run] and not [report], which makes coveragepy pretend those files don't exist and it doesn't measure them. This is probably because coveragepy has the same-named setting under two different sections, one is stronger but people would often land on one of them via search, not realizing there's another.
I'm pretty sure this contributes to misconfigurations like #735 (comment) or stuff in https://github.com/pytest-dev/pytest-cov/issues?q=is%3Aissue%20state%3Aopen%20omit.
But my main problem with this is that such suggestions contradict what seems to be the best practice — actually measuring the test modules and enforcing a high coverage level to make sure there's no tests that never run or dead code in there. @nedbat documented this a bunch of times over the years: https://nedbatchelder.com/blog/202008/you_should_include_your_tests_in_coverage.html / https://nedbatchelder.com/blog/201908/dont_omit_tests_from_coverage.html / https://nedbatchelder.com/blog/201106/running_coverage_on_your_tests.html
So my belief is that the docs should replace the example with something else and teach better practices since people often copy things from examples blindly. First because they aren't familiar with the configuration methods, to see something that actually works in front of them, but then I think many would just keep those config bits w/o cleaning them. I think most don't really see any obvious consequences there until they face the confusion. It's in the "unknown unknowns" knowledge space for them.
How about we additionally show some example where enforcing 100% coverage on the test folder is used to find accidentally skipped tests (copied function names with the author forgetting to remove them), and link those blog posts as a time-proven practice?