setting the finalize option in tweak [options future.delete = FALSE]

[yonicd]

looking at the documentation of batchtools_template there is the dots argument which passes additional arguments passed to BatchtoolsFuture(). which in there there is a finalize argument which If TRUE, any underlying registries are deleted when this object is garbage collected, otherwise not. I want to use this argument to keep future from removing the files after a call to a sungrid is complete.

I have tried to use the options below, but can not toggle the garbage collection off.

library(future)
library(batchtools)
library(future.batchtools)
library(future.apply)

options(future.finalize = FALSE)  
options(future.delete = FALSE)

my_sge <- future::tweak(
  future.batchtools::batchtools_sge,
  label = 'test2',
  template = 'batchtools.sge-new.tmpl',
  workers = 3,
  resources = list(slots = 4))

future::plan(list(multiprocess, my_sge))

Y1 %<-% future.apply::future_lapply(rep(800, 20),
                      FUN = function(nr){
                        solve( matrix(rnorm(nr^2), nrow=nr, ncol=nr))
                      })

if i try to set it in tweak i get the following warning, which makes sense too

> future::tweak(
+   future.batchtools::batchtools_sge,
+   label = 'test2',
+   template = 'batchtools.sge-new.tmpl',
+   workers = 3,
+   resources = list(slots = 4),
+   finalize = FALSE)
batchtools_sge:
- args: function (expr, envir = parent.frame(), substitute = TRUE, globals = TRUE, label = "test2", template = "batchtools.sge-new.tmpl", resources = list(slots = 4), workers = 3, ...)
- tweaked: TRUE
- call: NULL
Warning message:
In tweak.future(future.batchtools::batchtools_sge, label = "test2",  :
  Ignored 1 unknown arguments: ‘finalize’

where is the correct place to set this argument in the code?

[HenrikBengtsson]

I just had a look, and yes, using option future.delete = FALSE will only skip the deletion of the batchtools registry if there's a failure. If there's no failure, then the registry (folder) is always removed:

future.batchtools/R/BatchtoolsFuture-class.R

Lines 632 to 660 in ede01c2

	if (!getOption("future.delete", interactive())) {
	status <- status(future)
	res <- future$result
	if (inherits(res, "FutureResult")) {
	if (result_has_errors(res)) status <- unique(c("error", status))
	}
	mdebug("delete(): status(<future>) = %s",
	paste(sQuote(status), collapse = ", "))
	if (any(c("error", "expired") %in% status)) {
	msg <- sprintf("Will not remove batchtools registry, because the status of the batchtools was %s and option 'future.delete' is FALSE or running in an interactive session: %s", paste(sQuote(status), collapse = ", "), sQuote(path)) #nolint
	mdebug("delete(): %s", msg)
	warning(msg)
	return(invisible(FALSE))
	}
	}
	## Control batchtools info output
	oopts <- options(batchtools.verbose = debug)
	on.exit(options(oopts))
	## Try to delete registry
	interval <- delta
	for (kk in seq_len(times)) {
	try(clearRegistry(reg = reg), silent = TRUE)
	try(removeRegistry(wait = 0.0, reg = reg), silent = FALSE)
	if (!file_test("-d", path)) break
	Sys.sleep(interval)
	interval <- alpha * interval
	}

The background for the future.delete option here was for troubleshooting purposes of failed futures/jobs; I never considered the need to keep it otherwise. Shouldn't be hard to add support for this though, so it's now on the todo list.

About your attempt to configure the backend via the plan() function: This is a direction the Future API is moving toward. This far the focus has been on the "frontend" (what future(), value(), ... should support), but more and more use cases show that there is need to better control backend-specific properties. This is in line with where the future package is heading/aiming, cf. futureverse/future#172

[yonicd]

Thank you for the explanation! glad to hear the enhancements are in the works :)

[yonicd]

so the action to remove the contents is here in practice?

https://github.com/HenrikBengtsson/future.batchtools/blob/ede01c270487ee1371ed2b65968bd238bdf8547c/R/BatchtoolsFuture-class.R#L653_L660

which propagates to this https://github.com/mllg/batchtools/blob/3b0b1a9a59e377bb4d827e355d6955d66849c9e6/R/clearRegistry.R

[HenrikBengtsson]

Correct, plus some tweaks of assertions following that.

[HenrikBengtsson]

I've implemented this in the develop branch. After installing:

remotes::install_github("HenrikBengtsson/future.batchtools@develop")

The following now works:

library(future.batchtools)
plan(batchtools_local)

options(future.delete = FALSE)  ## <== DISABLE CLEANUP

f <- future(42)
v <- value(f)

print(f)
# BatchtoolsFuture:
# Label: '<none>'
# Expression:
# [1] 42
# Lazy evaluation: FALSE
# Asynchronous evaluation: TRUE
# Local evaluation: TRUE
# Environment: <environment: R_GlobalEnv>
# Capture standard output: TRUE
# Capture condition classes: 'message', 'warning'
# Globals: <none>
# Packages: <none>
# L'Ecuyer-CMRG RNG seed: <none>
# Resolved: TRUE
# Value: 56 bytes of class 'numeric'
# Early signalling: FALSE
# Owner process: a7e7b546-3d74-3d40-2c40-8de865c540e6
# Class: 'BatchtoolsFuture', 'Future', 'environment'
# batchtools cluster functions: 'Interactive'
# batchtools status: 'defined', 'finished', 'started', 'submitted'
# batchtools Registry:
#   Job Registry
#   Backend  : Interactive
#   File dir : /home/alice/.future/20190502_141813-obkGJW/batchtools_549386655
#   Work dir : /home/alice
#   Jobs     : 1
#   Seed     : 9954
#   Writeable: TRUE
#   File dir exists: TRUE
#   Work dir exists: TRUE

dir("/home/alice/.future/20190502_141813-obkGJW/batchtools_549386655")
# [1] "exports"           "external"          "jobs"             
# [4] "logs"              "more.args.rds"     "registry.rds"     
# [7] "results"           "updates"           "user.function.rds"

[yonicd]

Thank you!