Fail to receive from connection Slurm RStudio Server

[matthewgson]

Hi,

I've been struggling to find relevant answers that might help solve the problem, but after hundreds of googling I felt I was out of luck. I'm wondering if someone could point out where might the problem is.

I'm running parallel with future + doparallel (doFuture) with simple data.table code.

I'm using one note with 122 cores on the slurm server, using

#SBATCH --ntasks=1
#SBATCH --output=rserver.log
#SBATCH --nodes=1
#SBATCH --cpus-per-task=122
#SBATCH --mem=1000gb

...

rserver

This launches R Server (opensource version), and I connect to it with ssh, with connection info generated on rserver.log file:

Create an SSH tunnel with:
ssh -N -L 8080:c0706a-s27.dsfcf:33209 [email protected]
Then, open in the local browser:
http://localhost:8080

Below is the setting in my R:

library(tidyverse)
library(data.table)
library(doFuture)
library(progressr)
library(fst)
library(fasttime)

handlers(global = TRUE)
handlers("progress")
options(future.globals.maxSize= 1e20)
options(future.gc=TRUE) 
availableCores()
availableWorkers()
plan(cluster, workers = 120)
# plan(multisession, workers = 120) # Also tried with multisession as well
registerDoFuture()

And the process is basically reading lots of csv files and filtering it in parallel. Here's my code:

csv_parser = function(folder_address, root_symbol = NULL, out_path = NULL,  test = FALSE, type = 1){
  
  # unzip command for each file
  filepath_list = str_c('unzip -p ', list.files(folder_address, full.names = T))
  
  if (test==TRUE) {filepath_list = filepath_list[1:5]}
  
  # read file as data.table and append into list
  p <- progressor(along = filepath_list)

  list_df <- foreach(x = seq_along(filepath_list)) %dopar% {
    
    p(sprintf("x=%g", x))
    
    DT = fread(cmd = filepath_list[[x]], fill=TRUE)

    if (!is.null(root_symbol)) {
      DT = DT[root %chin% root_symbol]
    }
    
    gc()
    return(DT)
  }

  if (is.null(out_path)){
    result = rbindlist(list_df, fill=TRUE)
    # setnames(result, clean_names)
    return(result)
  } else {
    full_DT = rbindlist(list_df, fill=TRUE)
    # setnames(full_DT, clean_names)
    write.fst(full_DT, out_path, compress=100)
  }
}

cboe_parser(folder_address, root_symbol = snp500_tickers, out_path = 'some/path')

It seems it starts up multiple subprocesses but crashes in couple minutes.
The error says:

Error in unserialize(node$con) :                                                                                                                                                                                                                                         
  ClusterFuture (doFuture-2) failed to receive results from cluster RichSOCKnode HenrikBengtsson/doFuture#2 (PID 56721 on ‘localhost’). 
The reason reported was ‘error reading from connection’. Post-mortem diagnostic: 
The total size of the 9 globals exported is 145.86 KiB. 
The three largest globals are ‘filepath_list’ (101.57 KiB of class ‘character’), ‘root_symbol’ (35.55 KiB of class ‘character’) and ‘p’ (5.38 KiB of class ‘function’)

It doesn't seem related to the size but I made sure with options(future.globals.maxSize= 1e20)
I tried with plan(multisession, workers=120) and plan(cluster, workers=120) but it yielded the same errors.

Here's sessionInfo:

> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] RPostgres_1.4.4   dbplyr_2.1.1      fasttime_1.1-0    fst_0.9.8         progressr_0.10.0  doFuture_0.12.2   future_1.26.1     foreach_1.5.2     data.table_1.14.3 forcats_0.5.1     stringr_1.4.0     dplyr_1.0.9       purrr_0.3.4       readr_2.1.2      
[15] tidyr_1.2.0       tibble_3.1.7      ggplot2_3.3.6     tidyverse_1.3.1  

loaded via a namespace (and not attached):
 [1] nlme_3.1-157        matrixStats_0.62.0  fs_1.5.2            lubridate_1.8.0     bit64_4.0.5         progress_1.2.2      webshot_0.5.3       httr_1.4.3          dreamerr_1.2.3      numDeriv_2016.8-1.1 tools_4.1.2         backports_1.4.1     utf8_1.2.2         
[14] R6_2.5.1            DBI_1.1.2           colorspace_2.0-3    withr_2.5.0         prettyunits_1.1.1   tidyselect_1.1.2    bit_4.0.4           compiler_4.1.2      cli_3.2.0           rvest_1.0.2         xml2_1.3.3          ggthemr_1.1.0       sandwich_3.0-1     
[27] fstcore_0.9.12      scales_1.2.0        fixest_0.10.4       systemfonts_1.0.4   digest_0.6.29       rmarkdown_2.14      svglite_2.1.0       pkgconfig_2.0.3     htmltools_0.5.2     parallelly_1.31.1   fastmap_1.1.0       collapse_1.7.6      rlang_1.0.2        
[40] readxl_1.4.0        rstudioapi_0.13     generics_0.1.2      zoo_1.8-10          jsonlite_1.8.0      magrittr_2.0.3      kableExtra_1.3.4    Formula_1.2-4       Rcpp_1.0.8.3        munsell_0.5.0       fansi_1.0.3         lifecycle_1.0.1     stringi_1.7.6      
[53] yaml_2.3.5          blob_1.2.3          grid_4.1.2          parallel_4.1.2      listenv_0.8.0       crayon_1.5.1        lattice_0.20-45     haven_2.5.0         hms_1.1.1           knitr_1.39          pillar_1.7.0        codetools_0.2-18    reprex_2.0.1       
[66] glue_1.6.2          evaluate_0.15       modelr_0.1.8        vctrs_0.4.1         tzdb_0.3.0          cellranger_1.1.0    gtable_0.3.0        assertthat_0.2.1    xfun_0.31           broom_0.8.0         viridisLite_0.4.0   iterators_1.0.14    globals_0.15.0     
[79] ellipsis_0.3.2     

and here's number of cores

> availableCores()
nproc 
  122 
> availableWorkers()
  [1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
 [22] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
 [43] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
 [64] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
 [85] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[106] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[127] "localhost" "localhost"

I'd greatly appreciate if you could point where the problem might be. I've been using future (doParallel) for a while in the same cluster setting, and it's been working great, but somehow it started to give this error message recently.

[HenrikBengtsson]

Hi. Can you please install the develop version of parallelly;

remotes::install_github("HenrikBengtsson/parallelly", ref="develop")

Then rerun, and share the error message you get. This updated parallelly version will make the error message include a tad more info on what the problem is.

[matthewgson]

@HenrikBengtsson Here's code and error messages I received after installing dev version parallelly.

#remotes::install_github("HenrikBengtsson/parallelly", ref="develop")
library(parallelly)
library(tidyverse)
library(data.table)
library(doFuture)
library(progressr)
library(fst)
library(fasttime)

handlers(global = TRUE)
handlers("progress")
options(future.globals.maxSize= 1e20)
options(future.gc=TRUE) 
availableCores()
availableWorkers()

plan(cluster, workers = 120)
registerDoFuture()

csv_parser(folder_address, root_symbol = snp500_tickers, out_path = 'data/Originals/SNP500_transactions.fst') #SNP500
- [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------]   0% 
*** longjmp causes uninitialized stack frame ***: terminated
*** longjmp causes uninitialized stack frame ***: terminated
Error in unserialize(node$con) :                                                                                                                                                                                                            
  ClusterFuture (doFuture-4) failed to receive results from cluster RichSOCKnode #4 (PID 51821 on localhost ‘localhost’). 
The reason reported was ‘error reading from connection’. 
Post-mortem diagnostic: No process exists with this PID, i.e. the localhost worker is no longer alive. 
The total size of the 9 globals exported is 145.91 KiB. 
The three largest globals are ‘filepath_list’ (101.57 KiB of class ‘character’), 
‘root_symbol’ (35.55 KiB of class ‘character’) and ‘p’ (5.38 KiB of class ‘function’)
*** stack smashing detected ***: terminated

[HenrikBengtsson]

From the output:

csv_parser(folder_address, root_symbol = snp500_tickers, out_path = 'data/Originals/SNP500_transactions.fst') #SNP500
- [----------...------------------------------------------------------]   0% 
*** longjmp causes uninitialized stack frame ***: terminated
*** longjmp causes uninitialized stack frame ***: terminated
Error in unserialize(node$con) :
  ClusterFuture (doFuture-4) failed to receive results from cluster RichSOCKnode #4 (PID 51821 on localhost ‘localhost’). 
The reason reported was ‘error reading from connection’. 
Post-mortem diagnostic: No process exists with this PID, i.e. the localhost worker is no longer alive. 
The total size of the 9 globals exported is 145.91 KiB. 
The three largest globals are ‘filepath_list’ (101.57 KiB of class ‘character’), 
‘root_symbol’ (35.55 KiB of class ‘character’) and ‘p’ (5.38 KiB of class ‘function’)
*** stack smashing detected ***: terminated

there are a few things I can tell:

1. There is a progress bar, meaning it reached p(sprintf("x=%g", x)) (at least some of the parallel workers)

2. There's a very suspicous *** longjmp causes uninitialized stack frame ***: terminated output

3. At least parallel worker 4 has completely crashed, i.e. its R process has terminated. This is based on No process exists with this PID, i.e. the localhost worker is no longer alive., which is triggered when the main R process tries to query the parallel worker for its status or its results

So, there's something causing your parallel worker(s) to crash. My first guess would be that it's related to the *** longjmp causes uninitialized stack frame ***: terminated output. It looks like a message related to compiled code (e.g. C, C++), so it could be from some R package involved.

To track down the problematic code/function/package, you could try to use law of elemination, e.g. drop write.fst() and see if it makes a difference, which I guess would rule out the fst package.

Another, wild guess, is that you could be running out of memory on the machine, because you're running 120 parallel workers and each of them needs consumes RAM. OTH, I see you request 1,000 GiB, so you should be good. But still, I would try to decrease the number of parallel workers to see if that makes a difference. Try to see if you can reproduce it with, say, workers = 24.

Use the de-facto standard multisession. No need to use cluster (although it's almost the same).

FYI 1: The extra information in the error message on The total size of the 9 globals exported is 145.91 KiB. ... can sometimes give hints at non-exportable objects (https://future.futureverse.org/articles/future-4-non-exportable-objects.html), but in this case it doesn't look to be the case (at least not among the three largest ones). So, if that part made you think there was an issue with too large globals, and you tried with:

options(future.globals.maxSize= 1e20)

That is not necessary. BTW, if you want to disable the size check, set the limit to +Inf (https://future.futureverse.org/reference/future.options.html).

FYI 2: When using parallelization on the current machine (e.g. multisession), availableCores() gives you the number of cores that you've got available. And, availableWorkers() is identical to rep("localhost", times = availableCores()), so no need to look at that.

FYI 3: You can use plan(multisession, workers = availableCores(omit = 2)) to get 122-2=120 workers. If you resubmit the RStudio Server job with, say, --cpus-per-task=24, then it'll end up using 22 workers, and so on. Cleaner and one less thing to have to think about.

[matthewgson]

When R was shut down and terminated, the server status was OUT OF MEMORY, so it seems to be related to your guess. I put gc() so that it may clean up memory after reading on csv file though. Is it not enough for this?

[matthewgson]

@HenrikBengtsson I appreciate your comments. I'll examine it step by step and share the result here soon.

There's one thing I'd like to share. I was guessing that the above crash could be coming from using a burst request on the server (which allows me to use more resources allocated to my subscription, by using idle cores with low-priority), so I used with small cluster setting with priority (that I'm allocated, which is 16 cores 120GB ram).

Well, surprisingly without changing much of code, it ran great until at 73% of progress, then R (Rstudio) was crashed without previous error messages. I used plan(multisession, workers = 16) and library(parallelly) It seems the message is coming from data.table::fread. I'll update here.

> csv_parser(folder_address, root_symbol = snp500_tickers, out_path = '../data/Originals/SNP500_transactions.fst') #SNP500
Avoidable 0.872 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.578 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.549 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.574 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.676 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.514 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.963 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.578 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.502 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 1.105 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.633 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
Avoidable 0.607 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
| [===============================================================================================================================================================>-------------------------------------------------------------]  73% x=177> 

[matthewgson]

I'm not entirely sure if this is related, but something strange I found during debugging:

when data.table::fread was called in R subprocess, it behaves differently.

library(parallelly)
library(tidyverse)
library(data.table)
library(doFuture)
library(progressr)
library(fst)
library(fasttime)

handlers(global = TRUE)
handlers("progress")
options(future.globals.maxSize= +Inf)


filepath_vector = str_c('unzip -p ', list.files(folder_address, full.names = T))
set.seed(1)
filepath_vector = filepath_vector[sample(1:length(filepath_vector), 3)]

print(filepath_vector) # random sample of 3 csv files
[1] "unzip -p ../data/Originals/Price_data/UnderlyingOptionsTrades_2019-07-08.zip" "unzip -p ../data/Originals/Price_data/UnderlyingOptionsTrades_2021-01-08.zip" "unzip -p ../data/Originals/Price_data/UnderlyingOptionsTrades_2020-11-12.zip"

Strangely, the argument fread(..., fill=TRUE) does not work in the subprocess. Below is the code.

1. Sequential mode

plan(sequential)
list_df1 = foreach(x = seq_along(filepath_vector)) %dopar% {
  
  DT = fread(cmd = filepath_vector[[x]], fill=TRUE)
  gc()
  DT
}
Avoidable 1.237 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.

2. Multisession mode

plan(multisession, workers = availableCores(omit=2))
list_df2 = foreach(x = seq_along(filepath_vector)) %dopar% {
  
  DT = fread(cmd = filepath_vector[[x]], fill=TRUE)
  gc()
  DT
}

Warning message:
In fread(cmd = filepath_vector[[x]], fill = TRUE) :
  Stopped early on line 2116081. Expected 97 fields but found 144. Consider fill=TRUE and comment.char=. First discarded non-empty line: <<GLD,2021-01-08 13:27:05.576,644536368,GLD,2021-02-19,180.000,C,7,1,1.9600,18,0,1.9400,1.9600,171.7700,171.7800,16,1,36,1.9400,1,1.9900,65,26,1.8800,6,2.1600,4,18,1.9300,18,2.0090,5,33,1.9300,34,2.0090,69,33,1.9300,33,2.0000,6,1,1.9400,20,2.0090,7,41,1.9300,1,1.9600,71,4,1.9400,11,2.0800,119,0,0.0800,0.0900,23.0600,23.0700,16,1,276,0.0800,174,0.0900,65,1,0.0400,87,0.1100,4,90,0.0700,130,0.1000,5,361,0.0700,1003,0.1000,69,154,0.0800,87,0.0900,6,87,0.0800,198,0.1000,7,93,0.0800,87,0.0900,71,43,0.08>>

Compare results: subprocess' fread truncated the data.

> identical(list_df1, list_df2)
[1] FALSE

> lapply(list_df1, nrow)
[[1]]
[1] 1379076

[[2]]
[1] 5073362

[[3]]
[1] 4761699

> lapply(list_df2, nrow)
[[1]]
[1] 1379076

[[2]]
[1] 2116079

[[3]]
[1] 2987855

> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] RPostgres_1.4.4        dbplyr_2.1.1           fasttime_1.1-0         fst_0.9.8              progressr_0.10.0       doFuture_0.12.2        future_1.26.1          foreach_1.5.2          data.table_1.14.3      forcats_0.5.1         
[11] stringr_1.4.0          dplyr_1.0.9            purrr_0.3.4            readr_2.1.2            tidyr_1.2.0            tibble_3.1.7           ggplot2_3.3.6          tidyverse_1.3.1        parallelly_1.32.0-9002

loaded via a namespace (and not attached):
 [1] nlme_3.1-157        matrixStats_0.62.0  fs_1.5.2            lubridate_1.8.0     bit64_4.0.5         progress_1.2.2      webshot_0.5.3       httr_1.4.3          dreamerr_1.2.3      numDeriv_2016.8-1.1 tools_4.1.2         backports_1.4.1    
[13] utf8_1.2.2          R6_2.5.1            DBI_1.1.2           colorspace_2.0-3    withr_2.5.0         prettyunits_1.1.1   tidyselect_1.1.2    bit_4.0.4           compiler_4.1.2      cli_3.2.0           rvest_1.0.2         xml2_1.3.3         
[25] ggthemr_1.1.0       sandwich_3.0-1      fstcore_0.9.12      scales_1.2.0        fixest_0.10.4       systemfonts_1.0.4   digest_0.6.29       rmarkdown_2.14      svglite_2.1.0       pkgconfig_2.0.3     htmltools_0.5.2     fastmap_1.1.0      
[37] collapse_1.7.6      rlang_1.0.2         readxl_1.4.0        rstudioapi_0.13     generics_0.1.2      zoo_1.8-10          jsonlite_1.8.0      magrittr_2.0.3      kableExtra_1.3.4    Formula_1.2-4       Rcpp_1.0.8.3        munsell_0.5.0      
[49] fansi_1.0.3         lifecycle_1.0.1     stringi_1.7.6       yaml_2.3.5          blob_1.2.3          grid_4.1.2          parallel_4.1.2      listenv_0.8.0       crayon_1.5.1        lattice_0.20-45     haven_2.5.0         hms_1.1.1          
[61] knitr_1.39          pillar_1.7.0        codetools_0.2-18    reprex_2.0.1        glue_1.6.2          evaluate_0.15       modelr_0.1.8        vctrs_0.4.1         tzdb_0.3.0          cellranger_1.1.0    gtable_0.3.0        assertthat_0.2.1   
[73] xfun_0.31           broom_0.8.0         viridisLite_0.4.0   iterators_1.0.14    globals_0.15.0      ellipsis_0.3.2     

[HenrikBengtsson]

Those warnings on "incomplete" files doesn't sound like a problem with future, but rather some kind of race conditions in data.table::fread() with unzip -p ... paths. Maybe worth reaching out to the data.table and ask about running it in parallel like that (FYI, future multisession is basically just a parallel PSOCK cluster)

[matthewgson]

Good news - it seems to be related to the specific RStudio Server version I was running.
This morning I used an updated version of RStudio and the above code started to work seamlessly.
The RStudio Server version I used previously was build 352 (2019 September), and updated is 548.

[matthewgson]

@HenrikBengtsson Also I found the reason for messages from data.table::fread(), which was due to unzipped file size being too big, which took too much of tmp folder - unzipped .csv file size was 1.5GB on average, and 100 parallel workers quickly generates 150GB of data on /tmp folder on the server. When the limit was reached, the server truncated the file and hence generated error.

[matthewgson]

Well, it turns out my issue was not much related to future, except for conflict with some RStudio Server version... Hope this might help someone.