[TEST][windows] Seeing new Junit failure from 16th Feb build onwards

[deepika-u]

Describe the bug
test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom test case is failing.

Expected behavior
This junit was passing in the previous builds, so did anything change recently regarding this?

Detailed error

Multiple Failures (2 failures) org.junit.ComparisonFailure: expected:<...ressListener: Found [1] h1 tag(s)> but was:<...ressListener: Found [0] h1 tag(s)> java.lang.RuntimeException: AwtScreenshot VM finished with exit code 1: Exception in thread "main" java.awt.AWTError: no screen devices at java.desktop/sun.awt.Win32GraphicsEnvironment.getDefaultScreenDevice(Win32GraphicsEnvironment.java:101) at java.desktop/java.awt.Robot.<init>(Robot.java:135) at org.eclipse.test.AwtScreenshot.main(AwtScreenshot.java:43)

org.opentest4j.MultipleFailuresError: Multiple Failures (2 failures)
org.junit.ComparisonFailure: expected:<...ressListener: Found [1] h1 tag(s)> but was:<...ressListener: Found [0] h1 tag(s)>
java.lang.RuntimeException: AwtScreenshot VM finished with exit code 1:
Exception in thread "main" java.awt.AWTError: no screen devices
at java.desktop/sun.awt.Win32GraphicsEnvironment.getDefaultScreenDevice(Win32GraphicsEnvironment.java:101)
at java.desktop/java.awt.Robot.<init>(Robot.java:135)
at org.eclipse.test.AwtScreenshot.main(AwtScreenshot.java:43)

at org.eclipse.test.TracingSuite$TracingRunNotifier.fireTestFinished(TracingSuite.java:176)
at org.eclipse.test.TracingSuite.runChild(TracingSuite.java:287)
Suppressed: org.junit.ComparisonFailure: expected:<...ressListener: Found [1] h1 tag(s)> but was:<...ressListener: Found [0] h1 tag(s)>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.eclipse.swt.tests.junit.Test_org_eclipse_swt_browser_Browser.test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom(Test_org_eclipse_swt_browser_Browser.java:1181)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
... 1 more
Suppressed: java.lang.RuntimeException: AwtScreenshot VM finished with exit code 1:
Exception in thread "main" java.awt.AWTError: no screen devices
at java.desktop/sun.awt.Win32GraphicsEnvironment.getDefaultScreenDevice(Win32GraphicsEnvironment.java:101)
at java.desktop/java.awt.Robot.<init>(Robot.java:135)
at org.eclipse.test.AwtScreenshot.main(AwtScreenshot.java:43)

at org.eclipse.test.AwtScreenshot.dumpAwtScreenshot(AwtScreenshot.java:113)
at org.eclipse.test.Screenshots.takeScreenshot(Screenshots.java:102)
at org.eclipse.test.Screenshots$ScreenshotOnFailure.failed(Screenshots.java:40)
... 1 more 

Environment:

1. Select the platform(s) on which the behavior is seen:

• • All OS
• • Windows
• • Linux
• • macOS

2. Additional info
   Link to the doc -> https://download.eclipse.org/eclipse/downloads/drops4/I20250216-1800/testresults/html/org.eclipse.swt.tests_ep435I-unit-win32-x86_64-java21_win32.win32.x86_64_21.html

3. JRE/JDK version

Version since
Seeing this junit failing is failing since "I20250216-1800" and on "I20250217-0600"

Workaround (or) Additional context
Not aware of.

[deepika-u]

This passing on I20250218-0600
https://download.eclipse.org/eclipse/downloads/drops4/I20250218-0600/testResults.php

[deepika-u]

This is passing on I20250223-1800

https://download.eclipse.org/eclipse/downloads/drops4/I20250223-1800/testResults.php

So closing this issue as of now.

[deepika-u]

Again seeing the above failure -> https://download.eclipse.org/eclipse/downloads/drops4/I20250224-1800/testresults/html/org.eclipse.swt.tests_ep435I-unit-win32-x86_64-java21_win32.win32.x86_64_21.html

So opening this issue back.

@here : does anyone has any idea on what can cause this failure?

[HeikoKlare]

The failing test is a browser test: test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0]
The browser flag being 0 means that the test is executed without any flag, i.e., using Edge/WebView2.

The failing line of code is this:

eclipse.platform.swt/tests/org.eclipse.swt.tests/JUnit Tests/org/eclipse/swt/tests/junit/Test_org_eclipse_swt_browser_Browser.java

Line 1181 in 9f60279

assertEquals("ProgressListener: Found 1 h1 tag(s)", title.get());

It indicates that the progress listener is called before the actual browser content has been set.

[deepika-u]

The failing test is a browser test: test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0] The browser flag being 0 means that the test is executed without any flag, i.e., using Edge/WebView2.

Sorry i didnt understand it clearly. So do you mean it is expected failure?
But in between it was passing? so is it a transient failure? or it can be fixed?

[HeikoKlare]

Sorry i didnt understand it clearly. So do you mean it is expected failure?
But in between it was passing? so is it a transient failure? or it can be fixed?

No, sorry for being imprecise here. I don't think this is an expected failure. I just wanted to clarify what happens (i.e., what kind of failure the the failing test expresses), but one would have to analyze whether "only" the test is not properly implemented or whther something is wrong with the progress handling in the Edge implementation.

From the logs, we can at least see that it does not seem to be a timeout issue, as the failure occurs just one second after the test started (and timeout values are fairly higher):

[2025-02-24 23:51:32 +0000] org.eclipse.swt.tests.junit.Test_org_eclipse_swt_browser_Browser.test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0]()
Running Test_org_eclipse_swt_browser_Browser#test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0]
[2025-02-24 23:51:33 +0000] org.eclipse.swt.tests.junit.Test_org_eclipse_swt_browser_Browser.test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0]() failed:
org.junit.ComparisonFailure: expected:<...ressListener: Found [1] h1 tag(s)> but was:<...ressListener: Found [0] h1 tag(s)>
	at org.junit.Assert.assertEquals(Assert.java:117)
	at org.junit.Assert.assertEquals(Assert.java:146)
	at org.eclipse.swt.tests.junit.Test_org_eclipse_swt_browser_Browser.test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom(Test_org_eclipse_swt_browser_Browser.java:1181)