How to deal with infinite loops?

[theofidry]

If we take this simple piece of code:

$counter = 0;
while (true) {
    if ($counter > 10) {
        echo "done.";
        return;
    }

    $counter++;
}

one of the mutator will change $counter++ for $counter--. The issue is that this will make the process stop once the maximum nested function level is reached, making the whole analysis fail. Do you have any idea on how to account for it?

[MarkRedeman]

You can either refactor the function so that you remove any mutations that would result in an infinite loop, or you can ignore mutations that create infinite loops.

Humbug has an option that lets you configure a timeout meaning that if a test would take longer than 10 seconds than the mutation is marked as timeout (T).
I consider mutations that have been marked as timeout as a killed mutation.

One way of interpreting mutation testing is that you simulate mistakes (mutations) and you want to determine whether your test suite detects these changes. In this case your test suite would either throw an error since a maximum nested function level is reached, or it would timeout, hence in both cases your test suite failed under this mutation, thus the mutation is killed.

[theofidry]

I agree with the last part, but for the timeout it's a workaround at best. If your loop executes too fast, you may reach the maximum nested level before the timout, which either leaves you making the maximum nested level much higher or the timeout lower. The issue with the former is you may get your process killed because will consume to much memory, and the later may timeout some mutations where you don't want it to :/

[MarkRedeman]

I don't think that scenario could happen in the way Humbug is written. For each mutation Humbug starts a new child process (see here) so that it is completely separated from the other mutations that will be tested afterwards.

[theofidry]

By "that scenario" you mean exhausting the memory? Even though it is a new process it can exhaust all memory depending of the case, recursion can be heavy after all. But yeah it's probably greatly mitigated by that, however I'm still a bit uncomfortable with this idea as it's a balance between maximum nesting level and time, which both depends on your hardware.

[MarkRedeman]

Hmm yeah, there are probably quite a few edge cases that couldn't be solved using the timeout etc.

However whenever you encounter such an edge case it is probably not that difficult to mitigate it. One other thing you could do instead of refactoring the code is simply excluding the part that produces infinite loops from Humbug (using the excludes option), though refactoring the code is probably a better idea in most cases.

[theofidry]

Yeah in the meantime I just excluded it this way. But maybe it would be more interesting to be able to exclude mutants for some piece of codes, for example in this case, exclude the appropriate mutant to mutate $counter++. It could end up being rather brittle though, but this could be done by:

• In the humbug.json, with a check that the mutant can be applied to this line so that you can a warning if you change this piece of code (e.g. I'm adding a blank line, the $counter++ line changes
• Having an annotation of some kind like you can see for some coverage tools to exclude some pieces of code from the code coverage analysis. Less brittle, but less pretty as you have comments/annotations related to your testing tool in your source code.

[juliangut]

+1 for @theofidry on his second options.

We are currently used to annotations everywhere, Doctrine, Symfony, PHPUnit has @codeCoverageIgnore, PHPMD has @SuppressWarnings, ... I don't see that as a real problem

[theofidry]

@MarkRedeman side note: I plan to free some time in a few months to focus on humbug, I'll ping you at that time if you're interested to join