GitHub Actions Passing on Test Failures

[Justintime50]

GitHub Actions is passing builds when there are errors in the test suite. This was reported on another issue:

I'm also having issues with xcpretty and github actions, I'm getting test failures but the pipeline still passes. is this going to be fixed any time soon?

Originally posted by @samshiffman321 in #368 (comment)

I would expect the workflow on GitHub Actions to fail when the test suite fails, instead it passes with flying colors. I'm using the following command per the documentation on the README to no avail:

xcodebuild -project color-craze.xcodeproj -scheme 'color-craze (iOS)' -sdk iphonesimulator -destination 'platform=iOS Simulator,name=$(IPHONE_TYPE),OS=$(IOS_VERSION)' test | xcpretty && exit ${PIPESTATUS[0]}

It appears that ${PIPESTATUS[0]} isn't accessible on GitHub Actions or isn't being set correctly.

[rajivshah3]

Does this work? I think $PIPESTATUS is just on bash, not sh (the default)

- name: Test
  run: |
    set -o pipefail
    xcodebuild ... | xcpretty

[Justintime50]

Sadly no, I have my xcodebuild command in a Makefile if that makes a difference...

I doubt it works since apparently the build couldn't even find the correct SDK to run (it's been a bit since I've updated):

xcodebuild -project crazy-color-clash.xcodeproj -scheme 'crazy-color-clash (iOS)' -sdk iphonesimulator -destination 'platform=iOS Simulator,name=iPhone 13 Pro,OS=15.5' test | xcpretty && exit 
2022-12-12 17:31:08.349 xcodebuild Writing error result bundle to /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/ResultBundle_2022-12-12_1[7]-31-0008.xcresult
xcodebuild: error: Unable to find a destination matching the provided destination specifier:
		{ platform:iOS Simulator, OS:15.5, name:iPhone 13 Pro }

	The requested device could not be found because no available devices matched the request.

	Available destinations for the "crazy-color-clash (iOS)" scheme:
		{ platform:iOS, id:dvtdevice-DVTiPhonePlaceholder-iphoneos:placeholder, name:Any iOS Device }
		{ platform:iOS Simulator, id:dvtdevice-DVTiOSDeviceSimulatorPlaceholder-iphonesimulator:placeholder, name:Any iOS Simulator Device }
		{ platform:iOS Simulator, id:4D46E7BA-4485-4DF0-9E13-E20B5FA882E7, OS:16.0, name:iPad Air (4th generation) }
		{ platform:iOS Simulator, id:C40E6AB6-B91D-4EBB-8533-01B823FC84D6, OS:16.0, name:iPad Air (5th generation) }
		{ platform:iOS Simulator, id:EF33BB73-F3C6-4EFB-8156-38C2AB57D26B, OS:16.0, name:iPad Pro (9.7-inch) }
		{ platform:iOS Simulator, id:A06DFE39-F237-41C1-A2B3-387CCF9FBEBA, OS:16.0, name:iPad mini (6th generation) }
		{ platform:iOS Simulator, id:4B5C8C20-A8BC-4B6E-ABB7-D271D258834C, OS:16.0, name:iPhone 8 Plus }
		{ platform:iOS Simulator, id:C83D554F-F03B-4527-BD66-A2E2E3E7AB4D, OS:16.0, name:iPhone 11 }
		{ platform:iOS Simulator, id:5683BAE8-ED8A-45E6-8EB0-5CD7086F5F08, OS:16.0, name:iPhone 11 Pro }
		{ platform:iOS Simulator, id:62D2E07D-5BF0-4055-BBDA-B7615E885926, OS:16.0, name:iPhone 11 Pro Max }
		{ platform:iOS Simulator, id:637A8EB8-DFE7-40F7-8071-890F65202F4F, OS:16.0, name:iPhone 12 }
		{ platform:iOS Simulator, id:7D5769D5-BA50-4747-9F1B-768F5C718C08, OS:16.0, name:iPhone 12 Pro }
		{ platform:iOS Simulator, id:95D3F9F8-35B8-486B-A702-39DF48780208, OS:16.0, name:iPhone 12 Pro Max }
		{ platform:iOS Simulator, id:EBC1E97A-8BA5-4D16-8F99-650925FB74A5, OS:16.0, name:iPhone 12 mini }
		{ platform:iOS Simulator, id:F296500D-7ED9-4402-9AF7-6E024BD93D96, OS:16.0, name:iPhone 13 Pro }
		{ platform:iOS Simulator, id:205A5E8D-9D64-4896-BCEB-AC553E20E0A0, OS:16.0, name:iPhone 13 Pro Max }
		{ platform:iOS Simulator, id:D36D6F80-AC7C-4674-B980-F6ABB8E896CC, OS:16.0, name:iPhone 13 mini }
		{ platform:iOS Simulator, id:BD4F3B05-B4C4-479E-81B9-47144F7F81C2, OS:16.0, name:iPhone 14 }
		{ platform:iOS Simulator, id:978B9C8B-9E09-47CC-9C1F-92BF3D100F3E, OS:16.0, name:iPhone 14 Plus }
		{ platform:iOS Simulator, id:DF4792B4-1D7D-432A-BD4E-B8C44A9F12DC, OS:16.0, name:iPhone 14 Pro }
 }

This failure to even run the test suite when xcpretty was used and set pipefail still resulted in a passing build without errors.