How to use future_lapply's future.lazy?

[xiaodaigh]

I am trying to use future_lapply's future.lazy option. As can be seen below. It doesn't seem to be lazily evaluating it and there is no example in the package on how to do this correctly?

library(future)
plan(multiprocess, workers = 2)

pt <- proc.time()
tmp <- future_lapply(1:4, function(i) {
  Sys.sleep(5)
  i
}, future.lazy = T)
data.table::timetaken(pt) #takes 10 seconds

pt <- proc.time()
tmp # takes no time
data.table::timetaken(pt)

[HenrikBengtsson]

By design, future_lapply() is always going to return resolved futures, i.e. from the outside it will appear to work the same as lapply(). If you want to actually set up a set of lazy futures you need to do something like:

library("future")
plan(sequential) # easier to show it works
fs <- lapply(1:4, function(i) future({
  Sys.sleep(5)
  i
}, lazy = TRUE))

Then you can see that these are yet to be resolved:

> fs[[3]]
MulticoreFuture:
Label: '<none>'
Expression:
{
    Sys.sleep(5)
    i
}
Lazy evaluation: TRUE
Asynchronous evaluation: TRUE
Local evaluation: TRUE
Environment: <environment: 0x55c6030>
Globals: <none>
Packages: <none>
L'Ecuyer-CMRG RNG seed: <none>
Resolved: FALSE
Value: <not collected>
Early signalling: FALSE
Owner process: 0cc1763c-9ac5-09c2-f2ea-fce7a62e1948
Class: 'MulticoreFuture', 'MultiprocessFuture', 'Future', 'environment'

To resolve one of them, do:

> v <- value(fs[[3]])  ## will take 5 sec
> v
[1] 3

or all:

> vs <- values(fs)  # will take 15 sec (3rd is already done)
> unlist(vs)
[1] 1 2 3 4

Details and future plans (sic!)

Now, with the above, is there any use case where one would want to use future_lapply(..., future.lazy = FALSE) over future_lapply(..., future.lazy = TRUE)? I don't know - I have to think about it, but regardless, it might be that there is someone out there who has/will have a need for it and there is no extra cost of supporting the future.lazy argument.

Having said that, there will probably be a corresponding version of future_lapply() that returns a list of futures (rather than resolved values) as in my example above but will at smarter about load balancing that that example code snippet. Adding support for that goes into the plan of creating a separate future.apply package with a much more powerful set of functions. These plans are outlined in Issue #159.

Hope this helps

[xiaodaigh]

So the future.lazy argument currently doesn't work.

[HenrikBengtsson]

No, it works but it doesn't do what you think it does. This is what I'm trying to stress in my first sentence.

[xiaodaigh]

I must be missing something. So whats the difference if I set future.lazy to TRUE or FALSE? It always return resolved futures anyway.

…

On 30 Nov. 2017 4:26 pm, "Henrik Bengtsson" ***@***.***> wrote: No, it works but it doesn't do what you think it does. This is what I'm trying to stress in my first sentence. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#179 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AESfJXX9qlEaVu22oxIgdPq4OyboUd0Zks5s7jyegaJpZM4QvwKf> .

[HenrikBengtsson]

See what I wrote under 'Details and future plans (sic!)'.

Maybe it helps me to clarify if you can explain how you'd expect it work.

[xiaodaigh]

I was thinking future_lapply(..., future.lazy) would return a future that will resolve to the list of results of lapply(....).

[HenrikBengtsson]

Ok, so "Adding support for that goes into the plan of creating a separate future.apply package with a much more powerful set of functions. These plans are outlined in Issue #159 ."

[HenrikBengtsson]

I've moved this issue/discussion over to futureverse/future.apply#1.