Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Spun off from #4573 (comment) and below
This PR contains two improvements to the progress logging:
Don't show the percentage until the process is "closed", i.e. all tasks have been received. This prevents Nextflow from overestimating the progress before it knows the total task count of a process. Instead a question mark is shown.
This requires a new lifecycle event
onProcessClosed()
, which is also needed to implement array jobs, task grouping, and automatic cleanup, all for the same reason -- to know how many tasks a process will executeDon't include the failed task count in the percentage or the complete/total counts. This prevents the percentage from increasing when a task fails even though no progress was made, and it allows the process to reach 100% if the tasks are successfully retried/ignored.
As an example, consider a process that executes 4 tasks with 1 task that fails on the first attempt.
Before all tasks are received (i.e. process is still "open"):
A task fails and is retried, and succeeds on the second attempt:
If instead the failed task is ignored, it simply reduces the number of total tasks, allowing the process to reach 100% while still noting the number of ignored tasks: