Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GD-220: Add flaky test handling #220

Open
MikeSchulze opened this issue Jun 19, 2023 · 0 comments
Open

GD-220: Add flaky test handling #220

MikeSchulze opened this issue Jun 19, 2023 · 0 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@MikeSchulze
Copy link
Owner

MikeSchulze commented Jun 19, 2023

Is your feature request related to a problem? Please describe.

A flaky test is a test which could fail or pass for the same configuration. Flaky tests could be harmful for developers because their failures do not always indicate bugs in the code.

Describe the solution you'd like
Add support to handle flaky tests by add new options to the GdUnit settings.

  • enable flaky test handling
  • retry count
    Mark such tests as flaky and report the failure and success report.

Describe alternatives you've considered
Without automated retry of the failed test, we need to manually retry the entire test environment.

Additional context

This test are flaky

func test_2s():
	var timer_start := Time.get_ticks_msec()
	await await_millis(2000)
	# subtract an offset of 100ms because the time is not accurate
	assert_int(Time.get_ticks_msec()-timer_start).is_between(2*SECOND-100, 2*SECOND+100)

image
or
image

@MikeSchulze MikeSchulze added the enhancement New feature or request label Jun 19, 2023
@MikeSchulze MikeSchulze self-assigned this Jun 19, 2023
@MikeSchulze MikeSchulze added this to the V4.2.0 milestone Jun 19, 2023
@MikeSchulze MikeSchulze changed the title Add flaky test handling GD-220: Add flaky test handling Jun 19, 2023
@MikeSchulze MikeSchulze modified the milestones: V4.2.0, v4.2.1 Nov 29, 2023
@MikeSchulze MikeSchulze modified the milestones: v4.2.1, v4.3.0 Jan 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: No status
Development

No branches or pull requests

1 participant