Running Tests

Unit Tests

yarn test:unit

E2E Tests Overview

Our end-to-end (E2E) testing strategy leverages a combination of technologies to ensure robust test coverage for our mobile applications. We use Wix/Detox for the majority of our automation tests, Appium for specific non-functional testing like app upgrades and launch times, and Bitrise as our CI platform. All tests are written in JavaScript using Jest and Cucumber frameworks.

Wix/Detox Tests

Detox serves as our primary mobile automation framework, with most of our tests written using it. Learn more about Wix/Detox here.

Supported Platforms: iOS and Android
Test Location: e2e/specs

Setup and Execution

Test Wallet: Requires a wallet with access to testnet and mainnet. On Bitrise CI, this wallet is created using a secret recovery phrase from environment variables. For local testing, retrieve the phrase from the .e2e.env file.
Environment Variable: Set IS_TEST='true' to enable the test environment. Refer to the .e2e.env file in the mobile 1Password vault for the complete list of environment variables.
Warning Logs: Warning logs may sometimes cause test failures by interfering with automation interactions. To prevent this, disable warning logs during test execution.

Default Devices

iOS: iPhone 15 Pro
Android: Pixel 5 API 34

Ensure that these devices are set up. You can change the default devices at any time by updating the device.type in the Detox config located at e2e/.detoxrc.js.

Commands

Start Metro Server: Ensure the Metro server is running before executing tests:
```
yarn watch:clean
```

Build the Apps for Testing:

iOS Debug:
```
yarn test:e2e:ios:debug:build
```
Android Debug:
```
yarn test:e2e:android:debug:build
```

Run All Tests Locally:

iOS Debug:
```
yarn test:e2e:ios:debug:run
```
Android Debug:
```
yarn test:e2e:android:debug:run
```

NOTE: EXPO DOESN'T SUPPORT DETOX OUT OF THE BOX SO IT IS POSSIBLE THAT, IN SLOWER COMPUTERS, LOADING FROM THE BUNDLER TAKES TOO LONG WHICH MAKES THE VERY FIRST TEST FAIL. THE FAILED TEST WILL THEN AUTOMATICALLY RESTART AND IT SHOULD WORK FROM THEN ON.

Run Specific Tests:
- iOS:
```
yarn test:e2e:ios:debug:run e2e/specs/TEST_NAME.spec.js
```
- Android:
  
  UPDATE: with the implementation of Expo, mobile app will need to be manually loaded on emulator before running automated E2E tests.
  - install a build on the emulator
    - either install the apk or keep an existing install on the emulator
  - on the metro server hit 'a' on the keyboard as indicated by metro for launching emulator
  - if emulator fails to launch you can launch emulator in another terminal
```
emulator -avd <emulator-name>
```
    - on the metro server hit 'a' on the keyboard as indicated by metro for launching emulator
  - you don't need to repeat these steps unless emulator or metro server is restarted
```
yarn test:e2e:android:debug:run e2e/specs/TEST_NAME.spec.js
```

Run Tests by Tag (e.g., Smoke):

iOS:

yarn test:e2e:ios:debug:run --testNamePattern="Smoke"

Android:

yarn test:e2e:android:debug:run --testNamePattern="Smoke"

Appium

Platform: Android
Test Location: wdio

We currently utilize Appium, Webdriver.io, and Cucumber to test the application launch times and the upgrade between different versions. As a brief explanation, webdriver.io is the test framework that uses Appium Server as a service. This is responsible for communicating between our tests and devices, and cucumber as the test framework.

Configuration for Testing

We have two separate configurations for testing the different variants of our applications:

QA Variant (local): Runs in debug mode on your local machine.
QA Variant (production): Runs in production mode on BrowserStack.

We use the QA variant for Appium tests because of our screen-blocking mechanism, which would otherwise prevent tests from getting past the wallet setup screen.

Capabilities Setup

We require two sets of capabilities to handle app upgrade tests, leading to the creation of two configurations: defaultCapabilities and upgradeCapabilities.

Default Capabilities

const defaultCapabilities = [
  {
    platformName: 'Android',
    noReset: false,
    fullReset: false,
    maxInstances: 1,
    build: 'Android App Launch Times Tests',
    device: process.env.BROWSERSTACK_DEVICE || 'Google Pixel 6',
    os_version: process.env.BROWSERSTACK_OS_VERSION || '12.0',
    app: process.env.BROWSERSTACK_APP_URL,
    'browserstack.debug': true,
    'browserstack.local': true,
  }
];

This configuration is our standard, as it only requires one app per install.

Upgrade Capabilities

const upgradeCapabilities = [
  {
    platformName: 'Android',
    noReset: false,
    fullReset: false,
    maxInstances: 1,
    build: 'Android App Upgrade Tests',
    device: process.env.BROWSERSTACK_DEVICE || 'Google Pixel 6',
    os_version: process.env.BROWSERSTACK_OS_VERSION || '12.0',
    app: process.env.PRODUCTION_APP_URL || process.env.BROWSERSTACK_APP_URL,
    'browserstack.debug': true,
    'browserstack.local': true,
    'browserstack.midSessionInstallApps': [process.env.BROWSERSTACK_APP_URL],
  }
];

This configuration requires two applications: the current production app and the app built from the branch.

Note: You can, if you choose to, run the tests against any one of the devices and operating systems mentioned in the browserstack device list.

Flag-Based Capability Selection

We use flags like --performance and --upgrade to determine which capabilities to use for specific tests.

const { selectedCapabilities, defaultTagExpression } = (() => {
  if (isAppUpgrade) {
    return {
      selectedCapabilities: upgradeCapabilities,
      defaultTagExpression: '@upgrade and @androidApp',
    };
  } else if (isPerformance) {
    return {
      selectedCapabilities: defaultCapabilities,
      defaultTagExpression: '@performance and @androidApp',
    };
  } else {
    return {
      selectedCapabilities: defaultCapabilities,
      defaultTagExpression: '@smoke and @androidApp',
    };
  }
})();

Initial Setup

Before running tests, you need to perform an initial setup:

yarn setup

Running Tests Locally Against QA Build

You can run your E2E tests on local simulators either in development mode (with automatic code refresh) or without it.

iOS

To start an iOS QA build:

yarn start:ios:qa

Android

To start an Android QA build:

yarn start:android:qa

Ensure that the bundler compiles all files before running the tests to avoid build breaks. Use:

yarn watch:clean

Then, run the tests on the simulator:

iOS

yarn test:wdio:ios

Android

yarn test:wdio:android

To run specific tests, use the --spec option:

yarn test:wdio:android --spec ./wdio/features/performance/ColdStartLaunchTimes.feature

Note: Ensure that your installed simulator names match the configurations in wdio/config/android.config.debug.js and wdio/config/ios.config.debug.js.

Running Tests on BrowserStack

To trigger tests locally on BrowserStack:

Retrieve your BrowserStack username and access key from the App Automate section.
Update config.user and config.key in android.config.browserstack with your BrowserStack credentials.
Upload your app to BrowserStack via the create_qa_builds_pipeline. Grab the app_url from browserstack_uploaded_apps.json.
Update process.env.BROWSERSTACK_APP_URL with the correct app_url.
Run your tests using the appropriate flag (e.g., for performance tests):

yarn test:wdio:android:browserstack --performance

Running Appium Tests on CI (Bitrise)

You can also run Appium tests on CI using Bitrise pipelines:

app_launch_times_pipeline
app_upgrade_pipeline

For more details on our CI pipelines, see the Bitrise Pipelines Overview.

API Spec Tests

Platform: iOS
Test Location: e2e/api-specs/json-rpc-coverage.js

The API Spec tests use the @open-rpc/test-coverage tool to generate tests from our api-specs OpenRPC Document. These tests are currently executed only on iOS and use the same build as the Detox tests for iOS.

Test Coverage Tool: The test-coverage tool uses Rules and Reporters to generate and report test results. These are passed as parameters in the test coverage tool call located in e2e/api-specs/json-rpc-coverage.js. For more details on Rules and Reporters, refer to the OpenRPC test coverage documentation.

Commands

Build the App:
```
yarn test:e2e:ios:debug:build
```
Run API Spec Tests:
```
yarn test:api-specs
```

Bitrise Pipelines Overview

Our CI/CD process is automated through various Bitrise pipelines, each designed to streamline and optimize different aspects of our E2E testing.

1. PR_Smoke_e2e_Pipeline

Triggers:
- When "Run Smoke E2E" label is applied to a Pull request: Automatically runs smoke tests.
Manual Trigger: Select the desired branch in the Bitrise dashboard and choose pr_smoke_e2e_pipeline from the pipeline dropdown menu.

2. PR_Regression_e2e_Pipeline

Triggers:
- Nightly: Automatically runs all regression tests against main branch.
Manual Trigger: Select the main branch (or another branch of choice) in the Bitrise dashboard and choose pr_regression_e2e_pipeline from the pipeline dropdown menu.

3. Release_e2e_Pipeline

Workflows:
- Build: Creates iOS and Android artifacts.
- Test: Executes regression tests across both platforms.
Manual Trigger: Typically run on release branches but can be manually triggered in the Bitrise dashboard.

4. App Launch Times Pipeline

Function: Measures and monitors app launch times on real devices using BrowserStack to ensure consistent performance over time.
Nightly: Automatically runs on the main branch.
Manual Trigger: Select the desired branch in the Bitrise dashboard and choose app_upgrade_pipeline from the pipeline dropdown menu.

5. App Upgrade Pipeline

Function: Automates testing of app upgrades to verify smooth transitions between versions.
Configuration: Requires the PRODUCTION_APP_URL environment variable to be set with the current production build's BrowserStack URL.You would need to search and update PRODUCTION_APP_URL in the bitrise.yml with the production browserstack build URL.
Manual Trigger: Select the desired branch in the Bitrise dashboard and choose app_upgrade_pipeline from the pipeline dropdown menu.

Test Reports in Bitrise

Detox Tests: Test reports are displayed directly in the Bitrise UI, offering a visual representation of test results and execution details. Screenshots on test failures are also captured and stored in a zip file. You can download these screenshots from the Artifacts tab in Bitrise.
API Spec and Appium Tests: HTML reporters generate and display test results. Access these HTML reports through the Bitrise build artifacts section for detailed analysis.

Debugging Failed Tests

Example:

FAIL e2e/specs/swaps/swap-action-smoke.spec.js (232.814 s)
  SmokeSwaps Swap from Actions
    ✓ should Swap .05 'ETH' to 'USDT' (90488 ms)
    ✕ should Swap 100 'USDT' to 'ETH' (50549 ms)
  ● SmokeSwaps Swap from Actions › should Swap 100 'USDT' to 'ETH'
    Test Failed: Timed out while waiting for expectation: TOBEVISIBLE WITH MATCHER(id == “swap-quote-summary”) TIMEOUT(15s)
    HINT: To print view hierarchy on failed actions/matches, use log-level verbose or higher.
      163 |     return await waitFor(element(by.id(elementId)))
      164 |       .toBeVisible()
    > 165 |       .withTimeout(15000);
        |        ^
      166 |   }
      167 |
      168 |   static async checkIfNotVisible(elementId) {
    at Function.withTimeout (e2e/helpers.js:165:8)
    ...

In this example, the test failed because the swap-quote-summary ID was not found. This issue could be due to a changed testID or the swap quotes not being visible. To confirm whether either case is true, we then look at the screenshots on failure.

Here we can see that the swaps quotes in fact did not load hence why the tests failed.

Smoke Tests Breakdown

Per Team: Smoke tests are divided by team, allowing targeted verification of core functionalities pertinent to each team's responsibilities.
Benefits:
- Faster Feedback: Running a subset of tests on PRs provides quicker feedback, ensuring critical functionalities are validated without the overhead of executing all tests.
- Efficient Resource Use: Limits resource consumption and test execution time, optimizing CI/CD pipeline performance.

Best Practices

For more guidelines and best practices, refer to our Best Practices Document.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

testing.md

testing.md

Running Tests

Unit Tests

E2E Tests Overview

Wix/Detox Tests

Setup and Execution

Default Devices

Commands

Appium

Configuration for Testing

Capabilities Setup

Default Capabilities

Upgrade Capabilities

Flag-Based Capability Selection

Initial Setup

Running Tests Locally Against QA Build

iOS

Android

iOS

Android

Running Tests on BrowserStack

Running Appium Tests on CI (Bitrise)

API Spec Tests

Commands

Bitrise Pipelines Overview

1. PR_Smoke_e2e_Pipeline

2. PR_Regression_e2e_Pipeline

3. Release_e2e_Pipeline

4. App Launch Times Pipeline

5. App Upgrade Pipeline

Test Reports in Bitrise

Debugging Failed Tests

Smoke Tests Breakdown

Best Practices

Files

testing.md

Latest commit

History

testing.md

File metadata and controls

Running Tests

Unit Tests

E2E Tests Overview

Wix/Detox Tests

Setup and Execution

Default Devices

Commands

Appium

Configuration for Testing

Capabilities Setup

Default Capabilities

Upgrade Capabilities

Flag-Based Capability Selection

Initial Setup

Running Tests Locally Against QA Build

iOS

Android

iOS

Android

Running Tests on BrowserStack

Running Appium Tests on CI (Bitrise)

API Spec Tests

Commands

Bitrise Pipelines Overview

1. PR_Smoke_e2e_Pipeline

2. PR_Regression_e2e_Pipeline

3. Release_e2e_Pipeline

4. App Launch Times Pipeline

5. App Upgrade Pipeline

Test Reports in Bitrise

Debugging Failed Tests

Smoke Tests Breakdown

Best Practices