Feature Request: stop using xcpretty junit for scan retry

[yakimant]

Please switch to other junit generation tool

Motivation Behind Feature

xcprerry is not updated for several years, looks like it's loosing support:
xcpretty/xcpretty#360

It generates junit from xcodebuild log, which can cause issues.
Sometimes it misses some failed tests, which is dangerous for CI.

Where are other tools, which rely on xcresults for junit generation and more reliable (fastlane-community/trainer, TitouanVanBelle/XCTestHTMLReport).

Feature Description

I see 2 options:

1. Rely on xcresults for tests rerun logic
2. Run trainer or XCTestHTMLReport to generate junit from xcresults instead of

Alternatives or Workarounds

We are going to switch off xcpretty junit generation and setup XCTestHTMLReport.
When we are going to use suppress_tests_from_junit and trey scan.

[lyndsey-ferguson]

Thank you for the request.

I do not have a lot of time to work on this plugin, so I would have to prioritize it lower than the more critical issues. If you'd like to try your hand at making the changes, I am open to PRs and I can provide guidance (also checkout the CONTRIBUTING guidelines).

I already have the action tests_from_xcresult in this plugin, so you could use that to get the test results rather than the TestsFromJunitAction class.

You would change RetryingScanHelper's method update_only_testing to use the other TestsFromXcresultAction as well as RetryingScanHelper's method tests_from_xcresult

Or, maybe you're lucky and some other generous soul will pick this up.

[yakimant]

Thank you for instructions! I will think about it.

[yakimant]

@lyndsey-ferguson, I tried to use TestsFromXcresultAction instead of TestsFromJunitAction and got the following format difference:

TestsFromXcresultAction
TargetName/TestCase/Test

TestsFromJunitAction
TestCase/TestCase/Test()

What's the best to do? Allign the format or reformat TestsFromXcresultAction output in update_only_testing?

[lyndsey-ferguson]

only_testing is supposed to receive an array of strings of the format TargetName/TestCase/Test, and that is what TestsFromJunitAction should be providing...Did you perhaps mean to write that TestsFromXcresultAction results in TestCase/TestCase/Test()?

Otherwise, TestsFromXcresultAction should not put the () as that will break xcodebuild.

[yakimant]

@lyndsey-ferguson, thanks for quick reply!

Sorry, my bad, It was vice versa:
TestsFromXcresultAction
TestCase/TestCase/Test()

TestsFromJunitAction
TargetName/TestCase/Test

[lyndsey-ferguson]

So, you probably want to put a

t.delete('()')

here

[yakimant]

And wind a way to replace TestCase to TargetName in the same class?

[lyndsey-ferguson]

And wind a way to replace TestCase to TargetName in the same class?

Can you elaborate? It isn't clear to me what the question is.

[yakimant]

@lyndsey-ferguson, sorry for that. I have a question about same name difference:

TestsFromXcresultAction
TestCase/TestCase/Test()

TestsFromJunitAction
TargetName/TestCase/Test

Shall I make TestsFromXcresultAction to have the same format as TestsFromJunitAction?
Meaning not only removeing (), but also changing TestCase to TargetName as a first part of the name.

[lyndsey-ferguson]

Ouch, what!? My action returns testcases for the first segment! How embarrassing!!!

Yes, it should return TargetNames as the first segment.

[yakimant]

We decided to use the following actions instead:

1. tests_from_xcresult
2. collate_junit_reports
3. collate_xcresults

Reason - we want to switch from scan to xcodebuild action to run the tests because of this issue: fastlane/fastlane#17563

[lyndsey-ferguson]

Interesting. I could probably monkeypatch scan so it doesn't do that.

[Cyberbeni]

Sometimes our macOS tests hang when using Xcode12+. When this happens then ruby is at 100% CPU usage and either the app or the test runner or both are stuck in some kind of logging (-[XCTestLog testLogWithFormat:] in the test, _NSLogv in the regular app from the spindumps I have saved)

The feedbackassistant people suggested trying to remove xcpretty, tee, set -o pipefail, and NSUnbufferedIO=YES

[Cyberbeni]

As far as I know NSUnbufferedIO=YES is only needed to make the log appear faster, so I'll try passing xcodebuild_command: "xcodebuild" to multi_scan for a while.

[lyndsey-ferguson]

@Cyberbeni can you elaborate on what you're going to try doing?

[Cyberbeni]

I'm trying to get to a point where tests never hang (which is time consuming because they don't always hang, so I have to run them a lot) just taking out NSUnbufferedIO=YES didn't seem to help, the new number_of_retries parameter in scan seems to break a lot, so I'm going to try running scan without retries and without xcpretty to see if that changes anything.

[Cyberbeni]

The test runs with disabled xcpretty (thus with regular scan, without retries) didn't hang so far (could be a coincident but based on the history, it's 90+% that it's not) so supporting disable_xcpretty would help us too. (These hangs don't affect iOS tests, only macOS ones, and only with Xcode 12+)

[lyndsey-ferguson]

You tested with iOS tests? Similar number of tests?

[Cyberbeni]

Both iOS and macOS run at least once every day on all supported platforms. iOS has around 180 tests and runs 6 simulators parallelly with 18 batches. macOS has around 200 tests, which is randomly split into 2 that run on separate machines.

[Cyberbeni]

I don't know if a ruby spindump is any useful to you, apart from seeing that it spends all of its time in rb_reg_... functions but here it is:
rubyspindump.txt

[Cyberbeni]

One of the tests completed in 5 hours and 45 minutes (compared to the normal 90 minutes), so I'm trying to pipe the log file through xcpretty and it is taking a long time on my local machine too.

We print out the view hierarchy in the failure description and replace newlines with Đ, so it can be easily copied from jenkins which resulted in this case in ~37k character long line. Replacing these lines with shorter ones make xcpretty finish faster. (iOS does the same, earlier macOS versions also)

edit: I took out this description formatting to a single line as the whole description can be accessed when running the tests manually through screen share, so xcpretty won't be that big of a hit for us anymore.

[lyndsey-ferguson]

The Ruby Spindump isn't helpful. I have used rbspy to help me when I was parallelizing the iOS builds; I don't know if that will help you. If xcpretty is the cause of this slowdown, the result is to use xcresult files instead to find out which tests failed. Unfortunately, I won't be free to spend any significant time on this until at least December.

[Cyberbeni]

We removed the code that produced the output that made xcpretty take long in some cases, it took 32 minutes to format a log with 5 failures and half second when replacing the long lines . (The raw log of the last retry is saved, so if we really need to know what was the state at the moment of failure, we can use that instead)

[lyndsey-ferguson]

Note for anyone watching: I'm prioritizing this feature now and am working on the plan. As far as the -showBuildSettings issue slowing the start-up, I will treat that as a separate issue. Actually, it looks like it has already been fixed.

[lyndsey-ferguson]

@Cyberbeni @yakimant I have started working on this feature to use xcresult files to find the failing tests instead of the junit files. If you're interested in alpha testing, can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-292-use-xcresult-vs-xcpretty"

If there are problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

[Cyberbeni]

I'm currently on vacation, will try it out next week.

[Cyberbeni]

12:57:42 ERROR [2021-09-21 12:57:42.20]: Error loading plugin 'fastlane-plugin-test_center': cannot load such file -- pry-byebug
12:57:42 ERROR [2021-09-21 12:57:42.21]: It seems like you wanted to load some plugins, however they couldn't be loaded

[lyndsey-ferguson]

@Cyberbeni sorry about that, I've fixed that issue.

[Cyberbeni]

There is a problem with older Xcodes (10.1 and older). After running the first batch, there is a crash because there is no xcresult file at TestTarget-batch-1/report.xcresult
We use these configurations:

# macOS 11, Xcode 12.5
# macOS 10.15, Xcode 11.7
# macOS 10.14, Xcode 11.3
# macOS 10.13, Xcode 10.1
# macOS 10.12, Xcode 9.2
# macOS 10.11, Xcode 8.2.1

Attached stack trace:
test_center_crash.txt

For iOS we also use Xcode 12.5, that seems to work.

As far as I can tell, the TestSummaries.plist file can be used inside the *.test_result directory for older Xcodes.

[lyndsey-ferguson]

@Cyberbeni yes, the action that is supposed to get tests from an xcresult only works for Xcode 11+. I'll have to put some more work into tests_from_xcresult to support older versions.

[lyndsey-ferguson]

@Cyberbeni so, you're saying that there is no *.xcresult bundle at that path, but there is a *.test_result bundle at that path?

[Cyberbeni]

Yes

[lyndsey-ferguson]

I want to apologize that I have not responded to this issue. A family health issue is requiring me to narrow my focus on the essentials and I don't have the time I need to focus on the plugin more than a minute. If you can get some other contributors to this project to help with this, I would consider merging a resulting PR.