futuremice on computing cluster

[jeandigitale]

I'm about to implement a function in R called [futuremice] (https://www.gerkovink.com/miceVignettes/futuremice/Vignette_futuremice.html) (part of the mice package for multiple imputation), which is based on your package future. Before I was running a non-parallelized version of mice, but now I need to run more imputations so I need to speed up the processing. Before I was specifying 12 threads and 10 imputations were taking perhaps 48 hours to run, now I need to trial at least 30 imputations (more if possible). I'm a parallel computing novice and have a few questions so I can write my code correctly:

• How do I choose the plan strategy for future given I will run my code on wynton (computing cluster) - should it always be cluster?
• Should I specify the number of cores as Sys.getenv('nslots') in the futuremice call?
• Should use.logical (a logical indicating whether logical (TRUE) or physical (FALSE) CPU’s on machine should be used) be set to TRUE or FALSE?
• In the SGE call for the computing cluster, how do I know whether to call smp, mpi, mpi_onehost, or mpi-8? (Or is that not necessary?)

[HenrikBengtsson]

The UCSF Wynton HPC environment uses Son of Grid Engine (SGE) as the scheduler. SGE job allocations are automatically recognized by Futureverse (via the parallelly package). For example, if you request a multi-core, single-host job using something like:

$ qsub -pe smp 42 myscript.sh

where myscript.sh then runs R, then:

parallelly::availableCores()

will detect the number of CPU slots that SGE has allocated to your job and return 42. Therefore, if you use:

library(future)
plan(multisession, workers = availableCores())

that will spin up 42 parallel R workers in the background, which then are ready to be used by future_lapply() ,future_map(), ...

Since workers = availableCores() is the default, it's sufficient, and cleaner, to just do:

library(future)
plan(multisession)

• How do I choose the plan strategy for future given I will run my code on wynton (computing cluster) - should it always be cluster?

As long as you're sticking to parallelizing on a single machine, use the above plan(multisession).

• Should I specify the number of cores as Sys.getenv('nslots') in the futuremice call?

Nope, let parallelly::availableCores() worry about such details. The goal of the Futureverse is that you should be able to use the same code everywhere, without having to know the ins and outs of different systems and schedulers.

• Should use.logical (a logical indicating whether logical (TRUE) or physical (FALSE) CPU’s on machine should be used) be set to TRUE or FALSE?

Nope, you can scratch that thought.

• In the SGE call for the computing cluster, how do I know whether to call smp, mpi, mpi_onehost, or mpi-8? (Or is that not necessary?)

I recommend using -l smp <ncores> for requesting a specific number of cores on a single compute node. Personally, I'd like to keep it simple, so I avoid any MPI clusters, especially since it can be a bit tricky to install some of the R package dependencies for MPI. If you don't request multiple cores from SGE, it will give you a single-core job, i.e. availableCores() == 1. So, you need to ask SGE for what you need.

[jeandigitale]

Thank you so much! So helpful.