'The termination flag is set, exiting' with docker version 'rnaseq-cgl-pipeline:3.2.1-1'

[zhenzhenyang-psu]

Hello Jvivian,
I tried to run the rnaseq-cgl-pipeline:3.2.1-1 version to process RNA-Seq data. My command is as follows:

docker run -v $(pwd):$(pwd) -v $(pwd)/sample_fq_SLE:/samples -v /home/yangzhzh/database/cgl_RNA-seq_pipeline_inputs:/inputs -v /var/run/docker.sock:/var/run/docker.sock quay.io/ucsc_cgl/rnaseq-cgl-pipeline:3.2.1-1 --star /inputs/starIndex_hg38_no_alt.tar.gz --rsem /inputs/rsem_ref_hg38_no_alt.tar.gz --kallisto /inputs/kallisto_hg38.idx --sample-paired '/samples/SRR11539349_R1.fastq.gz,/samples/SRR11539349_R2.fastq.gz' --work_mount $(pwd)/SRR11539349_out --logDebug --save-bam --no-clean --cores 32 > nohup49 &

Yet it failed and only produced 'SRR11539349_R1merged.sorted.bam'.
Could you help me investigate how I might fix this error?
Thanks very much, Best. - Zhenzhen

The error message is as follows:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.del of <toil.fileStore.FileStore object at 0x7fa6380fef10>> ignored
Exception in thread Thread-7:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-8:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.del of <toil.fileStore.FileStore object at 0x7ff1694e2110>> ignored
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.del of <toil.fileStore.FileStore object at 0x7f81ea55af10>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.del of <toil.fileStore.FileStore object at 0x7f4591e81f10>> ignored

[adamnovak]

The termination flag is set is not the underlying error; it's just other threads responding to one thread having told the workflow to shut down. There should be another error message somewhere, probably earlier in the logs.

One way to try and work around this option would be to rerun the workflow again with the same job store and the --restart option. It should try and pick up where it left off.