DESIGN: Future API - Minimal/Core/Essential API and Extended/Optional API

[HenrikBengtsson]

The future package receives many interesting and handy feature requests. Some of them are straighforward whereas others does not necessarily fit straight in. I'm creating this issue to clarify why it's not straightforward to implement these and what the alternatives going forward are, and to encourage further discussion and ideas.

Minimal Future API (aka Future API)

In it's most minimal and essential form, the Future API provides:

• future() - creates a future (on any future backend)
• value() - collects the value of the future (waits for it to resolve if not already done)
• resolved() - checks whether a future is resolved or not.
• A future is stateless, i.e. just as plain R functions, evaluation of a future expression is purely functional without side effects and the outcome is the value (or a condition) of the evaluated expression.
• The values of futures should not depend in what order they are resolved.

On top of this, we have arguments controlling whether the future should be resolved lazily or eagerly, what or how globals are exported, polling and timeout strategies, etc.

I probably forgot something above, so please feel free to comment.

It is critical that this Minimal Future API can be supported by all future backends (including those not yet implemented by that may show up in the future). Because of this, the Minimal Future API is limited in what it can provide.

Examples of features that probably would fits in the Minimal Future API, but has not yet been added:

• (since future 1.9.0) Capture and relay standard output (in addition to the value) (Issue CONSISTENCY: Should all futures capture stdout & stderr as well? #67)
• (since future 1.11.0) Capture and relay conditions (e.g. messages and warning)
• Capturing of standard error (in addition to the value) (Issue CONSISTENCY: Should all futures capture stdout & stderr as well? #67) unlikely, at least not until R itself supports it fully
• Benchmarking, e.g. collecting total processing time and memory usage for resolving a future (Issue BENCHMARKING: Record timing and memory stats for the various steps in futures #59)
  • timestamps (easy)
  • memory (requires non-base R solutions)
• Mirroring of options (Issue R options: options() are not passed down to futures - should they? (document either way) #134)
  • manually
  • automatically
• Mirroring of environment variables (e.g. Issue Changes to .libPaths() not inherited by plan(multisession) #219)
  • manually
  • automatically
• Just-in-time compilation (Issue JIT: How much of a performance gain, automatic(?), and is it safe everywhere? #133)
• Memoization (Issue Should memoization be added? #127)
• Syntax for specifying resources and declaring them optional or mandatory (see below)
• Hook functions called when a future is, say, created, launched, resolved, finalized
• Merging two or more non-launched futures into a single future

Optional Future API

Any features related to futures that can not be supported by all backends belongs to what I consider an extended / optional API - let's call it the Optional Future API. Some features may be specific to a single backend while others to a majority of backends but not all.

Below is a set of features that fit into this category:

• "Passing" existing futures to an new one, e.g. a <- future(1); b <- future(value(a)) - requires b to be able to "communicate" with a (e.g. different machines)
• Suspending/terminating a future currently being evaluated, e.g. suspend(f) (Issue WISH: Add method for interrupting / terminating a future #93)
• Instant forwarding of the future's standard output and standard error streams to the owner process (Issues Obtain function console output before it is resolved? #141, future_lapply() is silent for the multisession plan #171)
• "Monitoring" of a future, e.g. progress updates / progress bars (Issue Progressbar HenrikBengtsson/doFuture#8)
• Persistent workers, i.e. a future can change the state of an underlying worker that a following future can utilize.
  • efficiency: don't export globals that already exist on the worker (requires a method for asserting identical(local, remote).
  • this can be for efficiency, e.g. futures that share the same global variables may resolve faster if they are resolved by the same worker (this can be optional, i.e. export global if not already available; think memoization)
  • a future preserves a value for a downstream future (not sure if this fits into the concept of futures, but I'll add it here in case someone has thoughts about this)
• Resources specifications typically seen in HPC environments, e.g. how much available memory and wall-time need to be available in order to start resolving a specific future. Other examples are access to a GPU. The future.batchtools package actually provides a little bit of these features under the hood, but such features are currently experimental and exploratory.
• Other resource specifications, such as only running on the local machine, on the local file system, on a given version of R, access to a certain set of files, and so on.
• ...

Some of the referenced issues discuss why it's hard to implement the features in a generic fashion such that they would work with all future backends (i.e. why the cannot be added to the Minimal Future API but belongs to a set of optional features).

[wlandau]

For ropensci/drake#227, I am interested in detecting failed futures without having to look at the value. I think there is a need to learn when a future has crashed, for whatever reason.

library(future)
plan(multicore)
f <- future({
  stop("some kind of error")
})

resolved(f)

# [1] TRUE

f$state

# [1] "running"

As I understand it, f$state is not part of the minimal API.