task.exitStatus is unset on timeout with local executor, breaking dynamic errorStrategy #6549
Description
Bug report
When a process exceeds its time limit under the local executor, task.exitStatus remains unset. As a consequence, dynamic errorStrategy logic depending on the exit status cannot detect timeout conditions, preventing controlled retries.
Expected behavior and actual behavior
Dynamic directives can adjust task resources or retry conditions based on task.exitStatus.
A timeout normally results in exit codes in the 137-145 range, which can be tested in errorStrategy to decide on resubmission.
Under the local executor, if the task is killed due to a time directive, task.exitStatus is absent. An expression such as:
errorStrategy { task.exitStatus in 137..145 ? 'retry' : 'terminate' }
evaluates to terminate, causing premature workflow termination instead of retry.
Steps to reproduce the problem
workflow {
hello()
}
process hello {
time { 5.s * task.attempt }
errorStrategy { task.exitStatus in 137..145 ? 'retry' : 'terminate' }
maxRetries 3
script:
"""
sleep 7
"""
}
time enforces a 5-second limit on the first attempt; sleep 7 exceeds it.
- Expected outcome: first attempt times out, second attempt runs with longer time limit.
- Observed outcome: workflow stops after the first timeout;
task.exitStatusis unset.
Program output
N E X T F L O W ~ version 25.10.0
Launching `local-timeout.nf` [sharp_shannon] DSL2 - revision: fa28a6ceb6
executor > local (1)
[46/e1572a] process > hello [ 0%] 0 of 1 ✘
ERROR ~ Error executing process > 'hello'
Caused by:
Process exceeded running time limit (5s)
Command executed:
sleep 7
Command exit status:
-
Command output:
(empty)
Work dir:
/home/mblum/work/46/e1572ad724e5046bf57b29f2d6cf24
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
-- Check '.nextflow.log' file for details(- indicates exit status not recorded)
nextflow.log
Nov-07 17:35:17.536 [main] DEBUG nextflow.cli.Launcher - $> nextflow run local-timeout.nf
Nov-07 17:35:17.631 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 25.10.0
Nov-07 17:35:17.734 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/mblum/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Nov-07 17:35:17.757 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Nov-07 17:35:17.758 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Nov-07 17:35:17.761 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Nov-07 17:35:17.777 [main] DEBUG nextflow.util.RetryConfig - Missing nextflow session - using default retry config
Nov-07 17:35:17.897 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Nov-07 17:35:17.898 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Nov-07 17:35:17.898 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Nov-07 17:35:17.908 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/mblum/.nextflow/secrets/store.json
Nov-07 17:35:17.915 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@4ed38226] - activable => nextflow.secret.LocalSecretsProvider@4ed38226
Nov-07 17:35:17.922 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Nov-07 17:35:17.930 [main] DEBUG nextflow.cli.CmdRun - Launching `local-timeout.nf` [sharp_shannon] DSL2 - revision: fa28a6ceb6
Nov-07 17:35:17.990 [main] DEBUG nextflow.Session - Session UUID: 29844b98-63ce-4397-a40c-b5ef10b275c1
Nov-07 17:35:17.990 [main] DEBUG nextflow.Session - Run name: sharp_shannon
Nov-07 17:35:17.990 [main] DEBUG nextflow.Session - Executor pool size: 12
Nov-07 17:35:17.998 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Nov-07 17:35:18.007 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=36; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Nov-07 17:35:18.034 [main] DEBUG nextflow.cli.CmdRun -
Version: 25.10.0 build 10289
Created: 22-10-2025 16:26 UTC (17:26 BST)
System: Linux 6.8.0-87-generic
Runtime: Groovy 4.0.28 on OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu122.04.1
Encoding: UTF-8 (UTF-8)
Process: 1729539@ubu-n-c6c04f343434 [127.0.1.1]
CPUs: 12 - Mem: 15.3 GB (1.6 GB) - Swap: 4 GB (3.5 GB)
Nov-07 17:35:18.059 [main] DEBUG nextflow.Session - Work-dir: /home/mblum/work [ext2/ext3]
Nov-07 17:35:18.059 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /home/mblum/bin
Nov-07 17:35:18.069 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Nov-07 17:35:18.081 [main] DEBUG nextflow.Session - Observer factory (v2): LinObserverFactory
Nov-07 17:35:18.086 [main] DEBUG nextflow.Session - Observer factory (v2): DefaultObserverFactory
Nov-07 17:35:18.119 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Nov-07 17:35:18.126 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 13; maxThreads: 1000
Nov-07 17:35:18.168 [main] DEBUG nextflow.Session - Session start
Nov-07 17:35:18.613 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Nov-07 17:35:18.668 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Nov-07 17:35:18.668 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Nov-07 17:35:18.674 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Nov-07 17:35:18.678 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=12; memory=15.3 GB; capacity=12; pollInterval=100ms; dumpInterval=5m
Nov-07 17:35:18.679 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Nov-07 17:35:18.701 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'hello': maxForks=0; fair=false; array=0
Nov-07 17:35:18.755 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: hello
Nov-07 17:35:18.756 [main] DEBUG nextflow.Session - Igniting dataflow network (1)
Nov-07 17:35:18.757 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > hello
Nov-07 17:35:18.758 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_dae212b67a2c27a0: /home/mblum/local-timeout.nf
Nov-07 17:35:18.758 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Nov-07 17:35:18.758 [main] DEBUG nextflow.Session - Session await
Nov-07 17:35:18.927 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Nov-07 17:35:18.930 [Task submitter] INFO nextflow.Session - [46/e1572a] Submitted process > hello
Nov-07 17:35:24.016 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: hello; status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (5s); workDir: /home/mblum/work/46/e1572ad724e5046bf57b29f2d6cf24]
Nov-07 17:35:24.017 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=36; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Nov-07 17:35:24.031 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=hello; work-dir=/home/mblum/work/46/e1572ad724e5046bf57b29f2d6cf24
error [nextflow.exception.ProcessFailedException]: Process `hello` failed
Nov-07 17:35:24.066 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'hello'
Caused by:
Process exceeded running time limit (5s)
Command executed:
sleep 7
Command exit status:
-
Command output:
(empty)
Work dir:
/home/mblum/work/46/e1572ad724e5046bf57b29f2d6cf24
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
Nov-07 17:35:24.079 [main] DEBUG nextflow.Session - Session await > all processes finished
Nov-07 17:35:24.080 [TaskFinalizer-1] DEBUG nextflow.Session - Session aborted -- Cause: Process `hello` failed
Nov-07 17:35:24.114 [TaskFinalizer-1] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=5s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=0; ]
Nov-07 17:35:24.132 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Nov-07 17:35:24.133 [main] DEBUG nextflow.Session - Session await > all barriers passed
Nov-07 17:35:24.135 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=5s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=0; ]
Nov-07 17:35:24.135 [TaskFinalizer-1] DEBUG nextflow.Session - null
java.lang.InterruptedException: null
at java.base/java.lang.Object.wait(Native Method)
at java.base/java.lang.Thread.join(Thread.java:1313)
at java.base/java.lang.Thread.join(Thread.java:1381)
at nextflow.trace.AnsiLogObserver.onFlowComplete(AnsiLogObserver.groovy:520)
at nextflow.Session$_notifyFlowComplete_lambda39.doCall(Session.groovy:1121)
at nextflow.Session.notifyEvent(Session.groovy:1151)
at nextflow.Session.notifyFlowComplete(Session.groovy:1121)
at nextflow.Session.shutdown0(Session.groovy:779)
at nextflow.Session.abort(Session.groovy:834)
at nextflow.Session.fault(Session.groovy:796)
at nextflow.processor.TaskPollingMonitor.finalizeTask(TaskPollingMonitor.groovy:753)
at nextflow.processor.TaskPollingMonitor.safeFinalizeTask(TaskPollingMonitor.groovy:736)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1339)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1094)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
at nextflow.processor.TaskPollingMonitor$_checkTaskStatus_lambda8.doCall(TaskPollingMonitor.groovy:726)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Nov-07 17:35:24.284 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Nov-07 17:35:24.292 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
Environment
- Nextflow versions:
25.04.7 build 5955and25.10.0 build 10289 - Java version:
openjdk version "17.0.16" 2025-07-15 - Operating system:
Ubuntu 22.04.5 LTS - Bash version:
GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)