multisession "future_lapply" error when R quits: Error in unserialize(node$con) : error reading from connection

[wlandau]

The errors are the same as futureverse/future#143. They only appear when I quit the R session, so I cannot use traceback() to dig further. They affects both "future_lapply" parallelism on the master branch and "future" parallelism on the i227-attempt2 branch (which is almost ready to alpha test).
Targets appear to build normally anyway, and the errors do not appear with drake 5.0.0 (current CRAN version).

library(drake)
library(future)
load_basic_example()
future::plan(multisession)
make(my_plan, parallelism = "future_lapply")
q()

##  Error in unserialize(node$con) : error reading from connection
## Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
## Execution halted
## Error in unserialize(node$con) : error reading from connection
## Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
## Execution halted
## Error in unserialize(node$con) : error reading from connection
## Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
## Execution halted
## ...

[wlandau]

Also, for the non-HPC backends, this one seems specific to future::multisession. The sequential and multicore backends appear unaffected.

[wlandau]

MWE:

library(future)
future::plan(multisession)
f <- future::future(capture.output(1+1, type = "message"))
q()

## Error in unserialize(node$con) : error reading from connection
## Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
## Execution halted

So I think it's because of this capture.output() call from when I fixed #212. Not yet sure how/whether to fix it. Drake should continue to capture and store messages somehow so users can retrieve them with diagnose(whatever_target).

[wlandau]

As @HenrikBengtsson said, users can get around this one either by using future.callr::callr or by creating and cleaning up an explicit PSOCK cluster. See also: HenrikBengtsson/Wishlist-for-R#55. I think archiving this explanation is enough. I believe drake should still continue capturing messages for diagnose().

library(future)
cl <- parallel::makeCluster(2L)
plan("cluster", workers = cl)  ## same as plan(multisession, workers = 2L)
> f <- future(capture.output(message("hello"), type = "message"))
> value(f)
[1] "hello"
> f <- future(capture.output(message("world"), type = "message"))
> value(f)
[1] "world"
> parallel::stopCluster(cl)
> q("no")