Future for R packages developers

[orozcoae89]

Dear Henrik,

I am developing an R package and I want to parallelize and/or distribute some processes within the main function using your "future" package.

As you recommend in a vignette (https://cran.r-project.org/web/packages/future/vignettes/future-7-for-package-developers.html) I have to avoid adding the plan() within the main function as follows:

my_fcn <- function(x, parallel = FALSE) {
if (parallel) {
oplan <- plan(multisession)
on.exit(plan(oplan))
y <- future_lapply(x, FUN = analyze) ## from future.apply package
} else {
y <- lapply(x, FUN = analyze)
}
summarize(y)
}

I have a similar function in my package, which is internally called by the main function. When I run this function outside the main function it performs efficiently. However, if I include this function within the main function it takes too long (it seems that R is not using all the threads).

I would be very grateful if you could provide me some solutions.

Erick.

[HenrikBengtsson]

First, note that the call:

oplan <- plan(multisession)

will add overhead, because it launches the background R sessions. The more workers it launches, where nworkers = availableCores(), the longer it will take. This is because the workers are launched one by one(*). Roughly, it'll take twice the time to set up 8 workers on an 8-core machine, that it takes to set up 4 workers on a 4-core machine. (*) Hopefully, soon we'll be able to set up the workers concurrently so 8 workers will take the same time as 4, or 32 workers, cf. futureverse/parallelly#5

Other than that, "it should work". You can convince yourself that things are indeed running in parallel with the number of workers that you'd expect by adding a bit of debug/verbose output, e.g.

my_fcn <- function(x, parallel = FALSE) {
  if (parallel) {
    oplan <- plan(multisession)
    on.exit(plan(oplan))
    message("Number of workers: ", nbrOfWorkers())
    y <- future_lapply(x, FUN = function(z) {
      message("Worker PID: ", Sys.getpid())
      analyze(z)
    }) ## from future.apply package
  } else {
    y <- lapply(x, FUN = analyze)
  }
  summarize(y)
}

Call

y <- my_fcn(x, parallel = TRUE)

and you should see the debug output. You should expect one unique 'Worker PID' per worker, i.e. per core.

[aritz-adin]

Dear Henrik,

I am currently working with @orozcoae89 developing an R package that includes some functions of your future library.

We are quite surprised because the computational time of our functions increases when calling them after building the package. On the contrary, if we locally define the functions in our R session it seems to work "quite" faster.

What do you think could be the problem? Could it be somehow related to global/local R environments?

Thank you very much for your time,
Aritz

[HenrikBengtsson]

Just a guess: It might be that the slowdown is because your packaged code somehow picks up more and larger globals, which adds overhead because they need to be streamed to the parallel workers.

You can turn on options(future.debug = TRUE) to see their sizes. You can also play with limiting options(future.globals.maxSize = 20*1024^2) up and down, to see if you're exporting more globals than you anticipate.

Also, like any other R programming, if you work with large objects, make sure to wipe them ASAP, e.g.

x <- matrix(rnorm(1e8), ncol=8)
y <- sum(x)

# Don't need 'x' anymore
rm(list = "x")

That'll help lower the memory needs, but also, if you happen to define a local function (in the same function as x) that you then you call in parallel, that function will carry x along.

[aritz-adin]

Thank you very much for your reply Henrik,

I have to make additional checks, but from what I've seen the only difference between running the functions locally or after building a package is the current R environment (global versus package environment). I have to admit that working with environments in R is a bit "confusing" for me...

Best regards,
Aritz