retrieve future expression on local machine

[marcosci]

Hi Henrik,

maybe that's utterly basic and the wrong place to ask, but I thought it might be of interest for more people:

Is there a way you can retrieve a future expressions in the following setup:

login <- tweak(remote, workers = "remote@remote")
bsub <- tweak(batchtools_lsf, ...)
plan(list(
    login,
    bsub,
    multiprocess
))

and I submit the following job (just an example):

y %<-% furrr::future_map_dfr(1:5, function(x){
    cores <- availableCores()
    furrr::future_map_dfr(1:5, function(x){
        tibble::tibble(x = x, 
                       nodename = Sys.info()['nodename'],
                       Sys.getpid = Sys.getpid(), 
                       availableCores = cores) 
    })
}) 

... let's imagine this takes an considerable amount of time and I would close my local r session in between submitting the job and its being finished.

After a login then (and rerunning the same plan again) I can't access y, neither can I access y on the remote machine in an R session. Is that possible in general?

If not, we wondered if you can save R objects somehow in an future_apply, submitted from a local machine and evaluated an a remote one?

Sorry for the very basic questions, just started to fiddle around with future.batchtools.

Cheers
Marco

[HenrikBengtsson]

No worries, this is certainly a valid question / feature request.

Unfortunately, saving future objects to file and reviving them later is currently not supported - neither at the core level (the future package) nor at the future.batchtools backend. This is something that will only be possible for some future backends, and therefore will only be available as an optional feature to the Future API.

I believe this has been asked before regarding the batchtools package, but I forgot what the status is. If anyone remembers/got points, please share here. As a starter, in order to add support for this in future.batchtools, it needs to be supported by batchtools itself.