(optionally) run the full test suite on any "surviving" mutants to confirm

[ejfine]

Sometimes for weird 3rd-party-library doing something funky stuff reasons, the way mutmut uses coverage to determine the subset of unit tests to run on a mutant isn't fully accurate. (I personally ran into a case where FastAPI was generating the OpenAPI schema file, and my tests did some assertions about that, but mutmut didn't autodetect that the mutations it made would impact the OpenAPI schema file)

It seems like it might be helpful to have an option for mutmut to automatically run the full test suite on any mutants that survived the initial screen in order to confirm they are true survivors, and not just that they survive the subset of tests most likely determined to be able to kill them.

[Otto-AA]

Sometimes for weird 3rd-party-library doing something funky stuff reasons, the way mutmut uses coverage to determine the subset of unit tests to run on a mutant isn't fully accurate. (I personally ran into a case where FastAPI was generating the OpenAPI schema file, and my tests did some assertions about that, but mutmut didn't autodetect that the mutations it made would impact the OpenAPI schema file)

I think if your tests are deterministic, mutmut should be able to detect which tests execute which mutants. We simply execute each test and check if the mutated function is called or not. Whether a test calls a function or not should hopefully be deterministic, but for instance could depend on the test order if you have a cache that is not cleared between tests.

It seems like it might be helpful to have an option for mutmut to automatically run the full test suite on any mutants that survived the initial screen in order to confirm they are true survivors, and not just that they survive the subset of tests most likely determined to be able to kill them.

The option should be rather easy to add, I think. tests_by_mangled_function_name is the relevant object, that maps mutants to the tests. We could fill this with all tests for all mutants, if a flag is enabled.

[ejfine]

I think with FastAPI it generates the OpenAPI schema when the module declaring the app is first imported 🤷 . that's the only concrete instance I've run into, but I'm still new-ish to mutmut and in the process of rolling it out to my different projects---so not sure if there are going to be more edge-cases like that

[Otto-AA]

I think for caching globally, we would need to change more than just re-running all tests. Currently, we do a full run of all tests to load the modules (and verify that tests without mutation pass), and then based on this pre-loaded state fork the process to test the mutations. But then the forks would all have the same cached OpenAPI spec.

We'd need to disable module caching additionally to running all tests, which makes the process slower.