future does not invoke winProgressBar

[xiaodaigh]

I have a piece that just shows a progress bar in Windows. However when I try to use future to run it in the background it doesn't show. Perhaps it's not a bug but would be worthwhile to document.

doprog <- function() {
  tkpb = winProgressBar(min = 0, max = 20, initial = 0, width = 500)
  on.exit(close(tkpb))
  for(i in 1:10) {
    setWinProgressBar(tkpb, i)
    Sys.sleep(1)
  }
}
doprog() #shwos progress bar on Windows

# doesn't show progressbar if run in a future background session.
library(future)
plan(multiprocess)
a %<-% doprog()

BTW thank you again for such an awesome package I will try to help with documentation and finding bugs whenever I can.

[HenrikBengtsson]

This is expected and not really related to futures per se but rather to limitations of the different parallel backends we're using.

When on Windows, plan(multiprocess) is effectively the same as plan(multisession) whereas on Unix/macOS it's plan(multicore). The multisession backend encapsulates/builds up the parallel package's PSOCK "cluster" framework (think parLapply() and friend). When using such PSOCK cluster workers, it is not possible to transfer output (standard output and standard error) in a "live" fashion. You're not the first nor the last to try this and discover that it does not work; if you search for "parLapply output not showing" I'm pretty sure you get lots of hits.

So why does it work on Unix/macOS? Because there the multicore (not multisession) backend is used which encapsulates/builds up the parallel package's "mcparallel" framework (think mclapply()). That in turn uses so called forked processes (not possible on Windows), which causes the background workers to share the same output channels as your main R processes. This is why output from those workers appear.

What can be done? This is tricky and with the risk of overwhelming you with details, this is related to:

• Issue CONSISTENCY: Should all futures capture stdout & stderr as well? #67 (capturing workers' output such that you can retrieve it when they're done - so not live - not good for progress bars)

• The future.callr package will potentially/eventually be able to relay output from workers while they're still running although in a polling fashion. That is, the main R process has to ask to the available output and re-output it, which should be good enough for progress bars. The reason why this package will be able to do this is because the callr package was designed to support it.

• From the above it should be clear that progress bars/live output will never be supported by all future backends. For some it may work and for others it won't. I anticipate that there will be an Extended Future API where there will be functions/mechanisms to work with these type of optional features. I've outlined these thoughs in Issue DESIGN: Future API - Minimal/Core/Essential API and Extended/Optional API #172.

Hope this clarifies.

[Jean-Romain]

It is difficult to find an anwser to this question and the doc as well as your answer are pretty clear this is not possible. However I did it relatively easily. Did I miss something ?

plan(multiprocess)

n = 10
f <- list()
for (i in 1:n) 
{
  f[[i]] <- future({ Sys.sleep(runif(1, 0.5, 1)) })
  cat(sprintf("\rProgress: %g%%", i/n*100), file = stderr())
}

[HenrikBengtsson]

Note that you're running the progress bar in the main R session, which works just fine. What OP (and many others) is after is to update the progress bar from within a future. Since such futures can run anywhere, not just (in forked process) on your local machine, it is a complex problem to figure out how to communicate such information from future workers back to the main R session.

[Jean-Romain]

Ok I got it. Indeed this is much complex question. But for my task this solution works fine :-)
Thank you.