MultiOmeGEX on GCP=> no space left on device #456
Comments
|
AFAIK, the disk directive would just increase the size of the cloud job's working directory, and not for the directories within the container image (e.g., |
|
Huh, but the container isn't writable anyway, is it? |
|
Docker overlays a writable layer on top of the base file system. So, software can write to the install directory e.g., Bind-mounting a larger volume to |
|
I'm surprised that no one has run into this issue as of yet. It is reproducible on my end. Maybe it only occurs when using Maybe updating to CellRanger v9 would help? v8 is nearly a year old now. I've contacted 10X about the issue and am waiting to hear back. |
I don't think it writes to the install directory. It certainly shouldn't. Maybe docker by default creates a writable overlay file system, singularity certainly does not unless you specify The error message says specifically
So it's the So I think changing the disk directive should fix that and I believe the reason nobody ran into this so far could be that they either provision their cloud instances with a lot of disk space by default (e.g. ours have 3TB) or run on a HPC with a shared file system. |
|
Yeah, you're right. So, this error shows that the cellranger_count module needs to be updated. I'm guessing that the Maybe a good time to update the entire set of Cell Ranger modules to v9? I could try creating a PR, but getting a sign-off on an update to all Cell Ranger modules will likely be non-trivial for a new contributor. |
|
It appears that adding I first tried 100 Gb, but that was not enough, at least for a sample with ~765 mil paired-end reads. I have another sample that I have yet to process with ~3.5 bil paired-end reads, so I'll let you know how much disk space is needed in that case. |
|
In regards to scaling: For larger datasets, |
|
Thanks for testing this! I'll double check if there are any guidelines within nf-core about the Regarding memory consumption, we could also look into the on-disk concatenation feature of AnnData. It is still experimental, though. For the time being, increasing memory sounds reasonable. |
|
Ok, so apparently, there's nothing against using The point was made that ideally the disk size could be set dynamically based on the input size. |
|
Yeah, |
|
|
|
FYI: it appears that >800 Gb of disk space is needed when the sample includes ~3.5 billion reads. I'm still working on getting this job to actually finish successfully. |
|
The Cell Ranger job took ~21 hours and required somewhere between 0.8 and 1 Tb of disk space for the ~3.5 billion paired-end read input. |
Description of the bug
At least for
--aligner cellranger --protocol ARC-v1, the CellRanger job runs out of temp file space:The job resources need to be modified to provide more disk space, at least for retried of the job.
Command used and terminal output
Relevant files
The job log:
System information
The text was updated successfully, but these errors were encountered: