Skip to content

Arm backend: Add OOTB tests #12392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Arm backend: Add OOTB tests #12392

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d6e2269
Arm backend: Add OOTB tests
tom-arm Jul 2, 2025
d6df766
Merge branch 'main' into add_ootb_testing
tom-arm Jul 11, 2025
05ada50
Correct formatting in trunk.yml
tom-arm Jul 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .github/workflows/trunk.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,60 @@ jobs:
exit 1
fi

test-arm-ootb-linux:
name: test-arm-ootb-linux
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
permissions:
id-token: write
contents: read
with:
runner: linux.2xlarge
docker-image: executorch-ubuntu-22.04-arm-sdk
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
# The generic Linux job chooses to use base env, not the one setup by the image
CONDA_ENV=$(conda env list --json | jq -r ".envs | .[-1]")
conda activate "${CONDA_ENV}"

# Follow the steps required before running the notebooks
# Try to mirror these as closely as possible
install_executorch "--use-pt-pinned-commit"
bash examples/arm/setup.sh --i-agree-to-the-contained-eula
source examples/arm/ethos-u-scratch/setup_path.sh

# Install requirements for converting notebooks
pip install notebook

# Run OOTB tests
backends/arm/test/test_arm_ootb.sh

test-arm-ootb-macos:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the value add for macos tests?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The value add here is this should ensure that the OOTB experience for the Arm backend does not regress/break on Mac OS. At the moment, there is no testing for this, so this should cover that gap

name: test-arm-ootb-macos
uses: pytorch/test-infra/.github/workflows/macos_job.yml@main
with:
runner: macos-m1-stable
python-version: '3.11'
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
timeout: 90
script: |
bash .ci/scripts/setup-conda.sh
eval "$(conda shell.bash hook)"

# Follow the steps required before running the notebooks
# Try to mirror these as closely as possible
${CONDA_RUN} sh install_executorch.sh "--use-pt-pinned-commit"
${CONDA_RUN} sh examples/arm/setup.sh --i-agree-to-the-contained-eula
${CONDA_RUN} source examples/arm/ethos-u-scratch/setup_path.sh

# Install requirements for converting notebooks
${CONDA_RUN} pip install notebook

# Run OOTB tests
${CONDA_RUN} sh backends/arm/test/test_arm_ootb.sh

nxp-build-test:
name: nxp-build-test
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
Expand Down
18 changes: 18 additions & 0 deletions backends/arm/test/test_arm_ootb.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/usr/bin/env bash

# Copyright 2025 Arm Limited and/or its affiliates.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.

set -e

run_ootb_tests_ethos_u() {
echo "$FUNCNAME: Running out-of-the-box tests for Arm Ethos-U"
jupyter nbconvert \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

--to notebook \
--execute examples/arm/ethos_u_minimal_example.ipynb
echo "${FUNCNAME}: PASS"
}

run_ootb_tests_ethos_u
Loading