Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compare screenshots with main on PRs #13248

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,17 @@ jobs:
name: example-run-macos
path: example-run/

compare-macos-screenshots:
name: Compare macOS screenshots
needs: [run-examples-macos-metal]
uses: ./.github/workflows/send-screenshots-to-pixeleagle.yml
with:
commit: ${{ github.sha }}
branch: ${{ github.ref_name }}
artifact: screenshots-macos
os: macos
secrets: inherit

check-doc:
runs-on: ubuntu-latest
timeout-minutes: 30
Expand Down
95 changes: 95 additions & 0 deletions .github/workflows/send-screenshots-to-pixeleagle.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
name: Send Screenshots to Pixel Eagle

on:
workflow_call:
inputs:
artifact:
required: true
type: string
commit:
required: true
type: string
branch:
required: true
type: string
os:
required: true
type: string

jobs:
send-to-pixel-eagle:
name: Send screenshots to Pixel Eagle
# runs on macos because it has the updated curl that will accept the --json flag, and the script is probably not portable to windows
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be changed back to ubuntu-latest once Ubuntu 24 becomes the new default. (Installed software)

I won't block on it, though, because who knows when that will be ready. :)

runs-on: macos-14
steps:

- name: Download artifact
uses: actions/download-artifact@v4
with:
pattern: ${{ inputs.artifact }}

- name: Send to Pixel Eagle
run: |
project="B04F67C0-C054-4A6F-92EC-F599FEC2FD1D"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be set using the env property? I feel like it signals better that this is a hardcoded constant, and may be changed in the future.

- name: Send to Pixel Eagle
  env:
    project: B04F67C0-C054-4A6F-92EC-F599FEC2FD1D
  run: ...


# Create a new run with its associated metadata
metadata='{"os":"${{ inputs.os }}", "commit": "${{ inputs.commit }}", "branch": "${{ inputs.branch }}"}'
run=`curl https://pixel-eagle.vleue.com/$project/runs --json "$metadata" --oauth2-bearer ${{ secrets.PIXELEAGLE_TOKEN }} | jq '.id'`

SAVEIFS=$IFS

cd ${{ inputs.artifact }}

# Read the hashes of the screenshot for fast comparison when they are equal
IFS=$'\n'
# Build a json array of screenshots and their hashes
hashes='[';
for screenshot in $(find . -type f -name "*.png");
do
name=${screenshot:14}
echo $name
hash=`shasum -a 256 $screenshot | awk '{print $1}'`
hashes="$hashes [\"$name\",\"$hash\"],"
done
hashes=`echo $hashes | rev | cut -c 2- | rev`
hashes="$hashes]"

IFS=$SAVEIFS

# Upload screenshots with unknown hashes
curl https://pixel-eagle.vleue.com/$project/runs/$run/hashes --json "$hashes" --oauth2-bearer ${{ secrets.PIXELEAGLE_TOKEN }} | jq '.[]|[.name] | @tsv' |
while IFS=$'\t' read -r name; do
name=`echo $name | tr -d '"'`
echo "Uploading $name"
curl https://pixel-eagle.vleue.com/$project/runs/$run/screenshots -F "data=@./$name" -F "screenshot=$name" --oauth2-bearer ${{ secrets.PIXELEAGLE_TOKEN }}
echo
done

IFS=$SAVEIFS

cd ..

# Trigger comparison with the main branch on the same os
curl https://pixel-eagle.vleue.com/$project/runs/$run/compare/auto --json '{"os":"<equal>", "branch": "main"}' --oauth2-bearer ${{ secrets.PIXELEAGLE_TOKEN }} > pixeleagle.json

# Log results
compared_with=`cat pixeleagle.json | jq '.to'`

status=0
missing=`cat pixeleagle.json | jq '.missing | length'`
if [ ! $missing -eq 0 ]; then
echo "There are $missing missing screenshots"
echo "::warning title=$missing missing screenshots on ${{ inputs.os }}::https://pixel-eagle.vleue.com/$project/runs/$run/compare/$to"
status=1
fi

diff=`cat pixeleagle.json | jq '.diff | length'`
if [ ! $diff -eq 0 ]; then
echo "There are $diff screenshots with a difference"
echo "::warning title=$diff different screenshots on ${{ inputs.os }}::https://pixel-eagle.vleue.com/$project/runs/$run/compare/$to"
status=1
fi

echo "created run $run: https://pixel-eagle.vleue.com/$project/runs/$run/compare/$to"

exit $status
26 changes: 24 additions & 2 deletions .github/workflows/validation-jobs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ jobs:
run: ANDROID_NDK_ROOT=$ANDROID_NDK_LATEST_HOME cargo apk build --package bevy_mobile_example

run-examples-linux-vulkan:
if: ${{ github.event_name == 'merge_group' }}
if: ${{ github.event_name != 'pull_request' }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment on why this is different, since all of the other jobs in this file uses == 'merge_group'?

runs-on: ubuntu-latest
timeout-minutes: 30
steps:
Expand Down Expand Up @@ -125,8 +125,19 @@ jobs:
name: example-run-linux
path: example-run/

compare-linux-screenshots:
name: Compare Linux screenshots
needs: [run-examples-linux-vulkan]
uses: ./.github/workflows/send-screenshots-to-pixeleagle.yml
with:
commit: ${{ github.sha }}
branch: ${{ github.ref_name }}
artifact: screenshots-linux
os: linux
secrets: inherit

run-examples-on-windows-dx12:
if: ${{ github.event_name == 'merge_group' }}
if: ${{ github.event_name != 'pull_request' }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing as above, please!

runs-on: windows-latest
timeout-minutes: 30
steps:
Expand Down Expand Up @@ -169,6 +180,17 @@ jobs:
name: example-run-windows
path: example-run/

compare-windows-screenshots:
name: Compare Windows screenshots
needs: [run-examples-on-windows-dx12]
uses: ./.github/workflows/send-screenshots-to-pixeleagle.yml
with:
commit: ${{ github.sha }}
branch: ${{ github.ref_name }}
artifact: screenshots-windows
os: windows
secrets: inherit

run-examples-on-wasm:
if: ${{ github.event_name == 'merge_group' }}
runs-on: ubuntu-latest
Expand Down