Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problem with running example parallel futures using batchtools_lsf #76

Open
haibol2016 opened this issue Dec 20, 2021 · 0 comments
Open

Comments

@haibol2016
Copy link

haibol2016 commented Dec 20, 2021

I am trying to run the example code in the future.batchtools Git Hub README, as follows.
But I failed due to error message shown below.

Could you help me out?
Thank you.

Haibo

library(future.batchtools)

plan(batchtools_lsf(template = "lsf.tmpl", workers = 2,
                    resources = list(walltime = 3600*8, num.cpu = 4,
                                        mpp = 1024*4, queue = "long")))

x %<-% { Sys.sleep(5); 3.14 }
y %<-% { Sys.sleep(5); 2.71 }
x + y

I got the following error message inconsistently

  1. err message 1:

Error: Log file '/project/umw_arthur_mercurio/Haibo/InPAS/.future/20211220_104348-sqIN0l/batchtools_2107070426/logs/job74e5f3ce70dd44fe91a775b79b221d03.log'
for job with id 1 not available

  1. err message 2:
    Error: BatchtoolsExpiration: Future ('') expired (registry path /project/umw_arthur_mercurio/Haibo/InPAS/.future/20211220_104348-sqIN0l/batchtools_290284087)
    .. No logged output exist.

The contents of my lsf.tmpl file is as follow:

## Default resources can be set in your .batchtools.conf.R by defining the variable
## 'default.resources' as a named list.

#BSUB -n <%= resources$num.cpu %> # minmal numbers of processors required for a parallel job
#BSUB -R rusage[mem=<%= resources$mpp %>] # ask for memory per cpu
#BSUB -J <%= job.name %> # Name of the job
#BSUB -o <%= log.file %> # Output is sent to logfile, stdout + stderr by default
#BSUB -q <%= resources$queue %> # Job queue
#BSUB -W <%= round(resources$walltime / 60, 1) %> # Walltime (LSF requires minutes, batchtools uses seconds)
##BSUB -M <%= resources$memory %> # Memory requirements, e.g. "5000KB", "500MB", "5GB" etc.
#BSUB -R "span[hosts=1]" # All hosts on the same chassis"

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

>  sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux 8.4 (Ootpa)

Matrix products: default
BLAS/LAPACK: /project/umw_mccb/bin/miniconda2/envs/seurat4/lib/libopenblasp-r0.3.15.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] future.batchtools_0.10.0 future_1.23.0           

loaded via a namespace (and not attached):
 [1] parallelly_1.29.0 magrittr_2.0.1    hms_1.1.1         progress_1.2.2   
 [5] rappdirs_0.3.3    R6_2.5.1          brew_1.0-6        rlang_0.4.12     
 [9] fansi_0.5.0       globals_0.14.0    tools_4.1.0       parallel_4.1.0   
[13] checkmate_2.0.0   data.table_1.14.2 utf8_1.2.2        withr_2.4.3      
[17] ellipsis_0.3.2    base64url_1.4     digest_0.6.29     tibble_3.1.6     
[21] lifecycle_1.0.1   crayon_1.4.2      fs_1.5.2          vctrs_0.3.8      
[25] batchtools_0.9.15 codetools_0.2-18  stringi_1.7.6     pillar_1.6.4     
[29] compiler_4.1.0    backports_1.4.1   prettyunits_1.1.1 listenv_0.8.0    
[33] pkgconfig_2.0.3  
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant