Add support for controlling the submission rate of jobs

[HenrikBengtsson]

Add support for controlling the submission rate of jobs, by exposing the sleep argument of batchtools::submitJobs() in one way or the other, e.g. plan(batchtools_slurm, sleep = 5) for 5 seconds between each job submission.

It's possible that this should/could be implemented in the Future API (futureverse/future#172) and if so, then maybe the default should be controllable via ?future.options.

Related to Issue #13.

[HenrikBengtsson]

For the record: The future.batchtools framework creates one batchtools registry per future. Because of this, there will only be a single job submitted via batchtools::submitJobs() for each future. The batchtools sleep argument does therefore not come in play when using batchtools futures; the sleep only happens when multiple jobs are submitted to batchtools::submitJobs().

So, by the current future.batchtools design, it needs to roll it's own "sleep-on-submission" mechanism.

[HenrikBengtsson]

The scheduler.latency argument for batchtools::makeClusterFunctions...() can be used to control the submission rate. For example,

plan(batchtools_sge, scheduler.latency = 10.0)

will sleep 10.0 seconds after submitting a job to the scheduler.

The scheduler.latency argument is documented in batchtools as: "Time to sleep after important interactions with the scheduler to ensure a sane state. Currently only triggered after calling submitJobs". Code inspection of batchtools::submitJobs(), confirms this behavior:

> batchtools::submitJobs
function (ids = NULL, resources = list(), sleep = NULL, reg = getDefaultRegistry()) 
{
    assertRegistry(reg, writeable = TRUE, sync = TRUE)
    ...
    Sys.sleep(reg$cluster.functions$scheduler.latency)
    runHook(reg, "post.submit")
    return(invisible(ids))
}