-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request: multi-threaded DE analysis #35
Comments
@aksarkar I believe this is already implemented with the |
I see. I missed this in the documentation since I was expecting it to be in the function arguments, not in However, when I tried it on my data set (~180K x 100K scATAC-seq count matrix), the process appears to only use one core and hang. |
From what I recall, one issue is that the parallel implementation is pretty memory-hungry (I tried to fix this, but did not have any luck). These were my settings for a scRNA-seq data set with ~90,000 cells. You might try first with a small number of samples, say |
I think the issue is that https://github.com/stephenslab/fastTopics/blob/master/R/lfc.R#L107-L121 |
Memory usage is an issue with |
See this gist. |
@pcarbo After digging into #37 and the details of There is still room to improve the total memory usage, although it's a bit difficult to profile (probably would require something like |
I agree 100%. |
Thank you for the great package! I am also failing to run DE on a larger dataset (~31000 x 35000) matrix. I am using the 'better-multithread' brench. Even on HPC cluster its still not passing 0%. Is there any workaround to run it? |
@hlszlaszlo Is your matrix a sparse matrix ("dgCMatrix")? Could you please share your exact |
Was running as a regular marix class. Now running with counts as "dgCMarix" class and seems much faster, thank you 🙂 Command: |
I'm glad to hear that helped. When you are using |
Yes, and I see using 25 cores. The sparse matrix helped. I can even run with less core on personal computer. |
Bumping up the priority of this issue based on some recent conversations with researchers trying to run |
Testing for differential expression/accessibility is embarrassingly parallel, and could use all available cores by default.
The text was updated successfully, but these errors were encountered: