Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support test execution order for JUnit 5 #46

Open
anatoliy-balakirev opened this issue Jan 21, 2020 · 0 comments
Open

Support test execution order for JUnit 5 #46

anatoliy-balakirev opened this issue Jan 21, 2020 · 0 comments

Comments

@anatoliy-balakirev
Copy link

Test execution order is available in Junit 5 since version 5.4 and seems to be stable enough to start supporting it by the PIT (ordered Junit 4 tests are already supported).

Here is a sample repo: https://github.com/anatoliy-balakirev/pitest_ordered_tests
There you can find two tests:

  • pitest.sample.controller.SampleControllerTest - junit 5
  • pitest.sample.controller.SampleControllerJunit4Test - junit 4

If you switch to junit4 branch there (or just remove testPlugin = "junit5" from the build.gradle) and run pitest - tests are executed properly, everything is green. If you try to execute pitest with Junit 5 (default in master) - tests are failed with:

14:10:30 PIT >> INFO : MINION : 14:10:30 PIT >> SEVERE : Description [testClass=pitest.sample.controller.SampleControllerTest, name=validate_2_internalError()]
org.opentest4j.AssertionFailedError: 
Expecting:
 <204>
to be equal to:
 <500>
but was not.
	at java.base/jdk.internal.reflect.N
14:10:30 PIT >> INFO : MINION : ativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(De
14:10:30 PIT >> INFO : MINION : legatingConstructorAccessorImpl.java:45)
	at pitest.sample.controller.SampleControllerTest.validate_2_internalError(SampleControllerTest.java:59)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.
14:10:30 PIT >> INFO : MINION : reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.ju
14:10:30 PIT >> INFO : MINION : nit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:515)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestM
14:10:30 PIT >> INFO : MINION : ethod$6(TestMethodTestDescriptor.java:171)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.
14:10:30 PIT >> INFO : MINION : java:167)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:114)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:59)
	at org.junit.platform.engine.s
14:10:30 PIT >> INFO : MINION : upport.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:105)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.ex
14:10:30 PIT >> INFO : MINION : ecuteRecursively(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:71)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at org.junit.platform.engine.support.hierarchical.SameThrea
14:10:30 PIT >> INFO : MINION : dHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:110)
	at org.junit.platform.engine.support.hierarchi
14:10:30 PIT >> INFO : MINION : cal.ThrowableCollector.execute(ThrowableCollector.java:72)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:71
14:10:30 PIT >> INFO : MINION : )
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support
14:10:30 PIT >> INFO : MINION : .hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:110)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeR
14:10:30 PIT >> INFO : MINION : ecursively(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:71)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExe
14:10:30 PIT >> INFO : MINION : cutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	a
14:10:30 PIT >> INFO : MINION : t org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams
14:10:30 PIT >> INFO : MINION : (DefaultLauncher.java:202)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
	at org.pitest.junit5.JUnit5TestUnit.execute(JUnit5Test
14:10:30 PIT >> INFO : MINION : Unit.java:95)
	at org.pitest.coverage.execute.CoverageDecorator.execute(CoverageDecorator.java:50)
	at org.pitest.testapi.execute.containers.UnContainer.execute(UnContainer.java:31)
	at org.pitest.testapi.execute.Pitest.executeTests(Pitest.java:57)
	at org
14:10:30 PIT >> INFO : MINION : .pitest.testapi.execute.Pitest.run(Pitest.java:48)
	at org.pitest.coverage.execute.CoverageWorker.run(CoverageWorker.java:51)
	at org.pitest.coverage.execute.CoverageMinion.main(CoverageMinion.java:90)

14:10:30 PIT >> SEVERE : Description [testClass=pitest.sample.controller.SampleControllerTest, name=validate_2_internalError()] did not pass without mutation.
14:10:30 PIT >> FINE : Coverage generator Minion exited ok
14:10:30 PIT >> INFO : Calculated coverage in 2 seconds.
14:10:30 PIT >> SEVERE : Tests failing without mutation: 
Description [testClass=pitest.sample.controller.SampleControllerTest, name=validate_2_internalError()]
Exception in thread "main" org.pitest.help.PitHelpError: 1 tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.
See http://pitest.org for more details.
	at org.pitest.coverage.execute.DefaultCoverageGenerator.verifyBuildSuitableForMutationTesting(DefaultCoverageGenerator.java:114)
	at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:96)
	at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:51)
	at org.pitest.mutationtest.tooling.MutationCoverage.runReport(MutationCoverage.java:115)
	at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:120)
	at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:50)
	at org.pitest.mutationtest.commandline.MutationCoverageReport.runReport(MutationCoverageReport.java:87)
	at org.pitest.mutationtest.commandline.MutationCoverageReport.main(MutationCoverageReport.java:45)

FAILURE: Build failed with an exception.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant