Can future output file inside loop?

[yuw444]

f.toy <- function(x){
load("path_to_object")
y <- rnorm(100,mean=x)
save(y,file=paste0(x,".Rdata"))
}

foreach(i = 1:3) %dopar%{
f.toy(i)
}

[HenrikBengtsson]

Yes, you can save to files inside a future expression. But, it'll save to the file system where the worker is running, which is fine if you use sequential, multisession, multicore, or other localhost backends.

[yuw444]

Would it work on multiple nodes of HPC?

[yuw444]

If possible, would you mind writing an example by using future for the above code? Sorry, I am new to Future, just realize it might be a solution for my need. Thanks so much!

[HenrikBengtsson]

Would it work on multiple nodes of HPC?

If those HPC compute nodes share the same file system, which is the most common way how HPC environments work, then it'll work. In this sense, R is no different from any other software, or a simple shell script.

[yuw444]

Thanks so much for your clarification, that helps me a lot.

I definitely need more extensive reading about parallel computing in R.

My real-world question is FindAllMarkers( ) in Seurat package, this function support future as you may already know, I like to use FindAllMarkers( ) for different Seurat objects, like 1000 of them. So, I was thinking use foreach to loop through different Seurat objects, then inside each loop, the ideal situation would be FindAllMarkers() can leverage the power of future as well. In this case, it would be nested parallel computing. How can I manage(balance/assign) the workers of foreach and the workers of FindAllMarkers( )?

Sorry, there may not be an easier answer for it. Any recommended reading would be appreciated too.

[yuw444]

I read through some of your vignettes. I am clearer a lot now.

[HenrikBengtsson]

FWIW, independently of futures, I highly recommend using saveRDS() and readRDS() instead of "old school" save()/load() to save and load objects in R.

[yuw444]

Great tips!