Fix flaky e2e tests #14319

wzieba · 2025-07-14T08:50:54Z

Closes: AINFRA-903
Reverts: #14302

Description

This PR is an attempt to address flakiness of e2e tests that the team lately observed. It will

update Firebase Test Lab device to non-deprecated
enable Android orchestrator
bump test dependencies
decrease tests timeout
repeat a test once if it fails

Testing information

You can repeat instrumentation tests job on CI, or run ./gradlew assembleVanillaDebug assembleVanillaDebugAndroidTest && ./gradlew runFlank and see that tests don't fail.

The tests that have been performed

I've executed instrumented tests 5 times (+ more during work on this PR) and they all passed

I have considered if this change warrants release notes and have added them to RELEASE-NOTES.txt if necessary. Use the "[Internal]" label for non-user-facing changes.

To reduce shared state and decrease flakiness

Our tests are not longer than max 2m30s

> The number of times to retry failed tests. Default is 0. Max is 10. Setting the value to 1 will mean that test are retried once. If the test fails then succeeds after the retry the run will be marked as "successful". The matrix with a flaky test will be marked as flaky.

wpmobilebot · 2025-07-14T08:56:06Z

Project dependencies changes

list

! Upgraded Dependencies
com.automattic.tracks:crashlogging:6.0.4, (changed from 6.0.3)
com.automattic.tracks:experimentation:6.0.4, (changed from 6.0.3)
com.automattic:Automattic-Tracks-Android:6.0.4, (changed from 6.0.3)
io.sentry:sentry:8.17.0, (changed from 7.17.0)
io.sentry:sentry-android-core:8.17.0, (changed from 7.17.0)
io.sentry:sentry-android-fragment:8.17.0, (changed from 7.17.0)
io.sentry:sentry-android-navigation:8.17.0, (changed from 7.17.0)
io.sentry:sentry-bom:8.17.0, (changed from 7.17.0)
io.sentry:sentry-compose-android:8.17.0, (changed from 7.17.0)
io.sentry:sentry-okhttp:8.17.0, (changed from 7.17.0)
org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.9.24, (changed from 1.9.10)
org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.24, (changed from 1.9.10)

tree

 +--- androidx.databinding:viewbinding:8.8.0
 |    \--- androidx.annotation:annotation:1.0.0 -> 1.9.1
 |         \--- androidx.annotation:annotation-jvm:1.9.1
 |              \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.24 -> 2.1.10
-|                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.0 -> 1.9.10 (c)
+|                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.0 -> 1.9.24 (c)
-|                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.0 -> 1.9.10 (c)
+|                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.0 -> 1.9.24 (c)
 +--- com.google.firebase:firebase-messaging -> 24.1.0
 |    +--- com.google.firebase:firebase-common-ktx:21.0.0
-|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10
-|    |         +--- org.jetbrains.kotlin:kotlin-stdlib:1.9.10 -> 2.1.10 (*)
-|    |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.9.10
-|    |              \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.10 -> 2.1.10 (*)
+|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24
+|    |         +--- org.jetbrains.kotlin:kotlin-stdlib:1.9.24 -> 2.1.10 (*)
+|    |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.9.24
+|    |              \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.24 -> 2.1.10 (*)
 |    \--- com.google.firebase:firebase-datatransport:18.2.0
 |         \--- com.google.android.datatransport:transport-backend-cct:3.1.9
 |              \--- com.google.firebase:firebase-encoders-json:18.0.0 -> 18.0.1
-|                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.7.10 -> 1.9.10 (*)
+|                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.7.10 -> 1.9.24 (*)
 +--- com.google.android.material:material:1.12.0
 |    \--- org.jetbrains.kotlin:kotlin-bom:1.8.22
-|         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (c)
+|         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (c)
-|         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (c)
+|         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (c)
 +--- androidx.work:work-runtime-ktx:2.10.2
 |    \--- androidx.work:work-runtime:2.10.2
 |         \--- androidx.room:room-ktx:2.6.1
 |              \--- androidx.room:room-common:2.6.1
-|                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
-+--- com.automattic.tracks:experimentation:6.0.3
-|    +--- com.squareup.okhttp3:okhttp:4.12.0
-|    |    +--- com.squareup.okio:okio:3.6.0 -> 3.7.0
-|    |    |    \--- com.squareup.okio:okio-jvm:3.7.0
-|    |    |         \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.21 -> 2.1.10 (*)
-|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.21 -> 1.9.10 (*)
-|    +--- com.squareup.moshi:moshi:1.15.1
-|    |    +--- com.squareup.okio:okio:3.7.0 (*)
-|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.21 -> 1.9.10 (*)
-|    +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4 -> 1.10.2 (*)
-|    \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.24 -> 2.1.10 (*)
++--- com.automattic.tracks:experimentation:6.0.4
+|    +--- com.squareup.okhttp3:okhttp:4.12.0
+|    |    +--- com.squareup.okio:okio:3.6.0 -> 3.7.0
+|    |    |    \--- com.squareup.okio:okio-jvm:3.7.0
+|    |    |         \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.21 -> 2.1.10 (*)
+|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.21 -> 1.9.24 (*)
+|    +--- com.squareup.moshi:moshi:1.15.1
+|    |    +--- com.squareup.okio:okio:3.7.0 (*)
+|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.21 -> 1.9.24 (*)
+|    +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4 -> 1.10.2 (*)
+|    \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.24 -> 2.1.10 (*)
-+--- com.automattic:Automattic-Tracks-Android:6.0.3
++--- com.automattic:Automattic-Tracks-Android:6.0.4
-+--- com.automattic.tracks:crashlogging:6.0.3
-|    +--- io.sentry:sentry-bom:7.17.0
-|    |    +--- io.sentry:sentry-android-core:7.17.0 (c)
-|    |    +--- io.sentry:sentry-android-fragment:7.17.0 (c)
-|    |    +--- io.sentry:sentry-compose-android:7.17.0 (c)
-|    |    +--- io.sentry:sentry-okhttp:7.17.0 (c)
-|    |    +--- io.sentry:sentry:7.17.0 (c)
-|    |    \--- io.sentry:sentry-android-navigation:7.17.0 (c)
-|    +--- io.sentry:sentry-android-core -> 7.17.0
-|    |    +--- androidx.lifecycle:lifecycle-process:2.2.0 -> 2.8.7 (*)
-|    |    +--- androidx.lifecycle:lifecycle-common-java8:2.2.0 -> 2.8.7 (*)
-|    |    +--- androidx.core:core:1.3.2 -> 1.13.1 (*)
-|    |    \--- io.sentry:sentry:7.17.0
-|    +--- io.sentry:sentry-okhttp -> 7.17.0
-|    |    +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.0 -> 1.9.10 (*)
-|    |    \--- io.sentry:sentry:7.17.0
-|    +--- io.sentry:sentry-android-fragment -> 7.17.0
-|    |    \--- io.sentry:sentry:7.17.0
-|    +--- io.sentry:sentry-compose-android -> 7.17.0
-|    |    +--- io.sentry:sentry:7.17.0
-|    |    +--- io.sentry:sentry-android-navigation:7.17.0
-|    |    |    \--- io.sentry:sentry:7.17.0
-|    |    \--- androidx.lifecycle:lifecycle-common-java8:2.2.0 -> 2.8.7 (*)
-|    +--- androidx.annotation:annotation:1.1.0 -> 1.9.1 (*)
-|    +--- com.squareup.okhttp3:okhttp:4.9.0 -> 4.12.0 (*)
-|    +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4 -> 1.10.2 (*)
-|    \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.24 -> 2.1.10 (*)
++--- com.automattic.tracks:crashlogging:6.0.4
+|    +--- io.sentry:sentry-bom:8.17.0
+|    |    +--- io.sentry:sentry-android-core:8.17.0 (c)
+|    |    +--- io.sentry:sentry-android-fragment:8.17.0 (c)
+|    |    +--- io.sentry:sentry-compose-android:8.17.0 (c)
+|    |    +--- io.sentry:sentry-okhttp:8.17.0 (c)
+|    |    +--- io.sentry:sentry:8.17.0 (c)
+|    |    \--- io.sentry:sentry-android-navigation:8.17.0 (c)
+|    +--- io.sentry:sentry-android-core -> 8.17.0
+|    |    +--- androidx.lifecycle:lifecycle-common-java8:2.2.0 -> 2.8.7 (*)
+|    |    +--- androidx.lifecycle:lifecycle-process:2.2.0 -> 2.8.7 (*)
+|    |    +--- androidx.core:core:1.3.2 -> 1.13.1 (*)
+|    |    \--- io.sentry:sentry:8.17.0
+|    +--- io.sentry:sentry-okhttp -> 8.17.0
+|    |    +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.24 (*)
+|    |    \--- io.sentry:sentry:8.17.0
+|    +--- io.sentry:sentry-android-fragment -> 8.17.0
+|    |    \--- io.sentry:sentry:8.17.0
+|    +--- io.sentry:sentry-compose-android -> 8.17.0
+|    |    +--- io.sentry:sentry:8.17.0
+|    |    +--- io.sentry:sentry-android-navigation:8.17.0
+|    |    |    \--- io.sentry:sentry:8.17.0
+|    |    \--- androidx.lifecycle:lifecycle-common-java8:2.2.0 -> 2.8.7 (*)
+|    +--- androidx.annotation:annotation:1.1.0 -> 1.9.1 (*)
+|    +--- com.squareup.okhttp3:okhttp:4.9.0 -> 4.12.0 (*)
+|    +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4 -> 1.10.2 (*)
+|    \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.24 -> 2.1.10 (*)
 +--- project :libs:fluxc
 |    \--- com.squareup.okhttp3:okhttp-urlconnection:4.9.0
-|         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.4.10 -> 1.9.10 (*)
+|         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.4.10 -> 1.9.24 (*)
 +--- project :libs:login
 |    \--- androidx.credentials:credentials-play-services-auth:1.2.0
 |         \--- com.google.android.libraries.identity.googleid:googleid:1.1.0
-|              \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.0 -> 1.9.10 (*)
+|              \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.0 -> 1.9.24 (*)
 +--- project :libs:cardreader
 |    \--- com.stripe:stripeterminal-taptopay:4.3.1
 |         +--- com.squareup.okhttp3:okhttp-tls:4.12.0
-|         |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.21 -> 1.9.10 (*)
+|         |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.21 -> 1.9.24 (*)
 |         +--- com.squareup.wire:wire-moshi-adapter:4.9.11
 |         |    +--- com.squareup.wire:wire-runtime:4.9.11
 |         |    |    \--- com.squareup.wire:wire-runtime-jvm:4.9.11
-|         |    |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (*)
+|         |    |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 -> 1.9.24 (*)
-|         |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 (*)
+|         |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.9.10 -> 1.9.24 (*)
 |         \--- com.stripe:stripeterminal-internal-common:4.3.1
 |              +--- com.squareup.moshi:moshi-adapters:1.15.1
-|              |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.21 -> 1.9.10 (*)
+|              |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.21 -> 1.9.24 (*)
 |              +--- io.ktor:ktor-client-core:2.3.11
 |              |    \--- io.ktor:ktor-client-core-jvm:2.3.11
-|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              |         +--- io.ktor:ktor-http:2.3.11
 |              |         |    \--- io.ktor:ktor-http-jvm:2.3.11
-|              |         |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |         |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |         |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |         |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              |         |         \--- io.ktor:ktor-utils:2.3.11
 |              |         |              \--- io.ktor:ktor-utils-jvm:2.3.11
-|              |         |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |         |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |         |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |         |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              |         |                   \--- io.ktor:ktor-io:2.3.11
 |              |         |                        \--- io.ktor:ktor-io-jvm:2.3.11
-|              |         |                             +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |         |                             +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |         |                             \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |         |                             \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              |         +--- io.ktor:ktor-events:2.3.11
 |              |         |    \--- io.ktor:ktor-events-jvm:2.3.11
-|              |         |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |         |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |         |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |         |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              |         \--- io.ktor:ktor-websocket-serialization:2.3.11
 |              |              \--- io.ktor:ktor-websocket-serialization-jvm:2.3.11
-|              |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              |                   \--- io.ktor:ktor-serialization:2.3.11
 |              |                        \--- io.ktor:ktor-serialization-jvm:2.3.11
-|              |                             +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |                             +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |                             +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |                             +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              |                             \--- io.ktor:ktor-websockets:2.3.11
 |              |                                  \--- io.ktor:ktor-websockets-jvm:2.3.11
-|              |                                       +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |                                       +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |                                       \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |                                       \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              +--- io.ktor:ktor-client-content-negotiation:2.3.11
 |              |    \--- io.ktor:ktor-client-content-negotiation-jvm:2.3.11
-|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              +--- io.ktor:ktor-client-serialization:2.3.11
 |              |    \--- io.ktor:ktor-client-serialization-jvm:2.3.11
-|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              |         \--- io.ktor:ktor-client-json:2.3.11
 |              |              \--- io.ktor:ktor-client-json-jvm:2.3.11
-|              |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              +--- io.ktor:ktor-serialization-kotlinx-json:2.3.11
 |              |    \--- io.ktor:ktor-serialization-kotlinx-json-jvm:2.3.11
-|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |         +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              |         \--- io.ktor:ktor-serialization-kotlinx:2.3.11
 |              |              \--- io.ktor:ktor-serialization-kotlinx-jvm:2.3.11
-|              |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|              |                   +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|              |                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|              |                   \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 |              \--- io.ktor:ktor-client-okhttp:2.3.11
 |                   \--- io.ktor:ktor-client-okhttp-jvm:2.3.11
-|                        +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.10 (*)
+|                        +--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.22 -> 1.9.24 (*)
-|                        \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.10 (*)
+|                        \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.22 -> 1.9.24 (*)
 +--- com.automattic:about:1.7.0
 |    \--- com.google.accompanist:accompanist-drawablepainter:0.20.2 -> 0.23.1
-|         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.10 -> 1.9.10 (*)
+|         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.10 -> 1.9.24 (*)
 +--- com.zendesk:support:5.0.8
 |    +--- com.zendesk:support-providers:5.0.8
 |    |    +--- com.zendesk:core:4.0.6
-|    |    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.10 (*)
+|    |    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.24 (*)
 |    |    +--- com.zendesk:guide-providers:1.0.7
-|    |    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.10 (*)
+|    |    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.24 (*)
-|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.10 (*)
+|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.24 (*)
 |    +--- com.zendesk:guide:1.0.7
 |    |    +--- com.zendesk:sdk-configurations:2.0.1
-|    |    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.10 (*)
+|    |    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.24 (*)
 |    |    +--- com.zendesk:messaging-api:5.2.3
-|    |    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.10 (*)
+|    |    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.24 (*)
 |    |    +--- com.zendesk:messaging:5.2.3
-|    |    |    +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.10 (*)
+|    |    |    +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.24 (*)
 |    |    |    \--- com.zendesk:common-ui:4.0.3
-|    |    |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.10 (*)
+|    |    |         \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.24 (*)
-|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.10 (*)
+|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.24 (*)
-|    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.10 (*)
+|    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.21 -> 1.9.24 (*)
 +--- com.tinder.statemachine:statemachine:0.2.0
-|    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.3.21 -> 1.9.10 (*)
+|    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.3.21 -> 1.9.24 (*)
 +--- androidx.constraintlayout:constraintlayout-compose:1.0.1
-|    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.10 -> 1.9.10 (*)
+|    \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.10 -> 1.9.24 (*)
 \--- com.airbnb.android:lottie-compose:5.2.0
-     \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.10 -> 1.9.10 (*)
+     \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.10 -> 1.9.24 (*)

wpmobilebot · 2025-07-14T09:01:04Z

📲 You can test the changes from this Pull Request in WooCommerce-Wear Android by scanning the QR code below to install the corresponding build.

	App Name	WooCommerce-Wear Android
	Platform	⌚️ Wear OS
	Flavor	Jalapeno
	Build Type	Debug
	Commit	`a39668e`
	Direct Download	`woocommerce-wear-prototype-build-pr14319-a39668e.apk`

wpmobilebot · 2025-07-14T09:12:34Z

📲 You can test the changes from this Pull Request in WooCommerce Android by scanning the QR code below to install the corresponding build.

	App Name	WooCommerce Android
	Platform	📱 Mobile
	Flavor	Jalapeno
	Build Type	Debug
	Commit	`a39668e`
	Direct Download	`woocommerce-prototype-build-pr14319-a39668e.apk`

codecov-commenter · 2025-07-14T09:16:50Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 37.64%. Comparing base (dd4b000) to head (a39668e).
Report is 17 commits behind head on trunk.

Additional details and impacted files

@@             Coverage Diff              @@
##              trunk   #14319      +/-   ##
============================================
- Coverage     37.64%   37.64%   -0.01%     
+ Complexity     9082     9081       -1     
============================================
  Files          1980     1980              
  Lines        111396   111396              
  Branches      14680    14680              
============================================
- Hits          41940    41939       -1     
  Misses        65618    65618              
- Partials       3838     3839       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wzieba · 2025-07-14T11:20:14Z

WooCommerce/build.gradle

    ]
    localResultsDir = "$rootDir/build/instrumented-tests"

+    useOrchestrator = true


Orchestrator will help with test runs isolation. Any performance drawbacks should be mitigated by tests sharding.

wzieba · 2025-07-14T11:21:42Z

WooCommerce/build.gradle

    ]
    localResultsDir = "$rootDir/build/instrumented-tests"

+    useOrchestrator = true
+    testTimeout = "5m"


Our longest e2e tests are no longer than ~2m30s so I think 5m timeout is good enough (instead of default 15m).

wzieba · 2025-07-14T11:24:08Z

WooCommerce/build.gradle

    ]
    localResultsDir = "$rootDir/build/instrumented-tests"

+    useOrchestrator = true
+    testTimeout = "5m"
+    flakyTestAttempts = 1


docs

I think it's acceptable to repeat an e2e test once in our case: they rely (at least partially) on backend sites and this might be one of the sources of flakiness.

wzieba · 2025-07-14T11:24:48Z

gradle/libs.versions.toml

@@ -39,7 +39,7 @@ apache-commons-text = '1.10.0'
 apache-http-client-android = '4.3.5.1'
 automattic-about = '1.7.0'
 automattic-measure-builds = '3.2.1'
-automattic-tracks = '6.0.3'
+automattic-tracks = '6.0.4'


This bumps the underlying Sentry SDK in hope to address a native crash during tests:

kidinov · 2025-07-14T12:54:23Z

@wzieba Thanks for taking care of this!

I reviewed the changes, and they make sense to me. However, I’m puzzled as to why they were failing - could it be an issue with the outdated emulator and its network settings?

wzieba · 2025-07-15T14:55:22Z

No problem @kidinov !

I’m puzzled as to why they were failing - could it be an issue with the outdated emulator and its network settings?

Yes, I think you're right. As @ParaskP7 mentioned in a Slack thread, this flakiness started to appear around the time of some other changes in FTL - e.g. Nexus5 device was completely decommissioned (we had to fix this in Aztec). It seems there might've been some changes to Pixel2.arm, but I can't find the source of this.

I hope that changing device to default + other changes will help eliminate (or reduce to reasonable levels) flakiness of these e2e tests. If no, we'll dig more 👍 .

Do you mind approving the PR?

kidinov

LGTM!

dangermattic · 2025-07-15T19:26:26Z

	1 Error
🚫	This PR is tagged with `status: do not merge` label(s).

Generated by 🚫 Danger

wzieba · 2025-07-15T19:26:37Z

Thanks! I'll wait a bit with merging this, as we discussed with Petros on DM.

wzieba added 7 commits July 11, 2025 14:46

Update Fladle

214972a

Switch device to a default "medium phone" and sdk 33

e5b7c1c

Update Tracks to update Sentry to 8.17.0

b79df61

Use Android test orchestrator

aab4a2f

To reduce shared state and decrease flakiness

Timeout tests after 5m instead of default 15m

3b8489f

Our tests are not longer than max 2m30s

Bring back e2e tests

2b9b5ad

wzieba commented Jul 14, 2025

View reviewed changes

wzieba added the category: ui tests Related to UI testing. label Jul 14, 2025

wzieba added this to the 22.9 milestone Jul 14, 2025

wzieba requested a review from ParaskP7 July 14, 2025 11:30

wzieba marked this pull request as ready for review July 14, 2025 11:30

wzieba requested a review from a team as a code owner July 14, 2025 11:30

wzieba requested review from kidinov and removed request for a team July 14, 2025 11:30

kidinov approved these changes Jul 15, 2025

View reviewed changes

wzieba added the status: do not merge Dependent on another PR, ready for review but not ready for merge. label Jul 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix flaky e2e tests #14319

Fix flaky e2e tests #14319

wzieba commented Jul 14, 2025 •

edited

Loading

Uh oh!

wpmobilebot commented Jul 14, 2025

Uh oh!

wpmobilebot commented Jul 14, 2025

Uh oh!

wpmobilebot commented Jul 14, 2025

Uh oh!

codecov-commenter commented Jul 14, 2025

Uh oh!

wzieba Jul 14, 2025

Uh oh!

wzieba Jul 14, 2025

Uh oh!

wzieba Jul 14, 2025

Uh oh!

wzieba Jul 14, 2025

Uh oh!

kidinov commented Jul 14, 2025

Uh oh!

wzieba commented Jul 15, 2025

Uh oh!

kidinov left a comment

Uh oh!

dangermattic commented Jul 15, 2025

Uh oh!

wzieba commented Jul 15, 2025

Uh oh!

Uh oh!

Fix flaky e2e tests #14319

Are you sure you want to change the base?

Fix flaky e2e tests #14319

Conversation

wzieba commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing information

The tests that have been performed

Uh oh!

wpmobilebot commented Jul 14, 2025

Project dependencies changes

Uh oh!

wpmobilebot commented Jul 14, 2025

Uh oh!

wpmobilebot commented Jul 14, 2025

Uh oh!

codecov-commenter commented Jul 14, 2025

Codecov Report

Uh oh!

wzieba Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

wzieba Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

wzieba Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

wzieba Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

kidinov commented Jul 14, 2025

Uh oh!

wzieba commented Jul 15, 2025

Uh oh!

kidinov left a comment

Choose a reason for hiding this comment

Uh oh!

dangermattic commented Jul 15, 2025

Uh oh!

wzieba commented Jul 15, 2025

Uh oh!

Uh oh!

wzieba commented Jul 14, 2025 •

edited

Loading