Errors: vector memory exhausted (limit reached?)

[robohall]

Brief problem description

My dataset appears to be too large for streamMetabolizer to run on my computer. I am estimating metabolism from the Columbia River. Data set size is ~ 66,000 entries at 15 min intervals so about 690 days of metabolism, not too long. It runs roughly overnight (24,000 seconds) and returns the following error (error 1-3 are about tidyverse deprecations). It saves a small output file, but there are no mcmc samples in the file.

What you saw on your computer

4: In metab_fun(specs = specs, data = data, data_daily = data_daily, :
Modeling failed
Warnings:
error in running command
There were 4 chains where the estimated Bayesian Fraction of Missing Information was low. See
http://mc-stan.org/misc/warnings.html#bfmi-low
Examine the pairs() plot to diagnose sampling problems Errors:
vector memory exhausted (limit reached?)

This last error suggests a memory problem. I am running a 2015 Macbook Pro with 16 GB memory. Run is on 4 cores. I am sampling 1000 warm up and 1000 saved steps. When I run a file with about 1/10 of the days and 500 warm up and saved steps, it works fine, and I get the expected results. That the small file works suggest that the data or model output is too big for my computer. Thus there is no problem per se, rather this is a PSA that 700 days of metabolism and 100 steps might be too much for my laptop. Time to run on a cluster.

Include all code you ran (a minimal example) and all console output, errors, and warnings. Include a data file if needed.

load("CRall_sm.RData")  #load data fram ready to go for sM, Bob can supply if needed

cr_name <- mm_name(type='bayes', pool_K600='normal', err_obs_iid = T, err_proc_iid =T)
cr_specs <- specs(cr_name, K600_daily_meanlog_meanlog=1, K600_daily_meanlog_sdlog=0.75, K600_daily_sdlog_sigma=0.1, burnin_steps=1000, 
                  saved_steps=1000)
CRall_fit <- metab(cr_specs, data=CRall_sm, info=c(site='Columbia, date/time correction', source='Bob Hall'))

Session information

Run the following code line (or sessionInfo() if that doesn't work) and paste in your output.

> devtools::session_info()

─ Session info ────────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 4.0.3 (2020-10-10)
 os       macOS Mojave 10.14.6        
 system   x86_64, darwin17.0          
 ui       RStudio                     
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       America/Denver              
 date     2021-01-21                  

─ Packages ────────────────────────────────────────────────────────────────────────────────
 package           * version  date       lib source                                   
 assertthat          0.2.1    2019-03-21 [1] CRAN (R 4.0.2)                           
 backports           1.1.10   2020-09-15 [1] CRAN (R 4.0.2)                           
 base64enc           0.1-3    2015-07-28 [1] CRAN (R 4.0.2)                           
 bayesplot           1.7.2    2020-05-28 [1] CRAN (R 4.0.2)                           
 callr               3.5.1    2020-10-13 [1] CRAN (R 4.0.2)                           
 cli                 2.1.0    2020-10-12 [1] CRAN (R 4.0.2)                           
 codetools           0.2-16   2018-12-24 [1] CRAN (R 4.0.3)                           
 colorspace          1.4-1    2019-03-18 [1] CRAN (R 4.0.2)                           
 colourpicker        1.1.0    2020-09-14 [1] CRAN (R 4.0.2)                           
 crayon              1.3.4    2017-09-16 [1] CRAN (R 4.0.2)                           
 crosstalk           1.1.0.1  2020-03-13 [1] CRAN (R 4.0.2)                           
 curl                4.3      2019-12-02 [1] CRAN (R 4.0.1)                           
 desc                1.2.0    2018-05-01 [1] CRAN (R 4.0.2)                           
 deSolve             1.28     2020-03-08 [1] CRAN (R 4.0.2)                           
 devtools            2.3.2    2020-09-18 [1] CRAN (R 4.0.2)                           
 digest              0.6.26   2020-10-17 [1] CRAN (R 4.0.2)                           
 dplyr             * 1.0.2    2020-08-18 [1] CRAN (R 4.0.2)                           
 DT                  0.16     2020-10-13 [1] CRAN (R 4.0.2)                           
 dygraphs            1.1.1.6  2018-07-11 [1] CRAN (R 4.0.2)                           
 ellipsis            0.3.1    2020-05-15 [1] CRAN (R 4.0.2)                           
 fansi               0.4.1    2020-01-08 [1] CRAN (R 4.0.2)                           
 farver              2.0.3    2020-01-16 [1] CRAN (R 4.0.2)                           
 fastmap             1.0.1    2019-10-08 [1] CRAN (R 4.0.2)                           
 fs                  1.5.0    2020-07-31 [1] CRAN (R 4.0.2)                           
 generics            0.1.0    2020-10-31 [1] CRAN (R 4.0.2)                           
 ggplot2           * 3.3.2    2020-06-19 [1] CRAN (R 4.0.2)                           
 ggridges            0.5.2    2020-01-12 [1] CRAN (R 4.0.2)                           
 glue                1.4.2    2020-08-27 [1] CRAN (R 4.0.2)                           
 gridExtra           2.3      2017-09-09 [1] CRAN (R 4.0.2)                           
 gtable              0.3.0    2019-03-25 [1] CRAN (R 4.0.2)                           
 gtools              3.8.2    2020-03-31 [1] CRAN (R 4.0.2)                           
 hms                 0.5.3    2020-01-08 [1] CRAN (R 4.0.2)                           
 htmltools           0.5.0    2020-06-16 [1] CRAN (R 4.0.2)                           
 htmlwidgets         1.5.2    2020-10-03 [1] CRAN (R 4.0.2)                           
 httpuv              1.5.4    2020-06-06 [1] CRAN (R 4.0.2)                           
 igraph              1.2.6    2020-10-06 [1] CRAN (R 4.0.2)                           
 inline              0.3.16   2020-09-06 [1] CRAN (R 4.0.2)                           
 jsonlite            1.7.1    2020-09-07 [1] CRAN (R 4.0.2)                           
 labeling            0.4.2    2020-10-20 [1] CRAN (R 4.0.3)                           
 LakeMetabolizer     1.5.0    2016-06-23 [1] CRAN (R 4.0.2)                           
 later               1.1.0.1  2020-06-05 [1] CRAN (R 4.0.2)                           
 lattice             0.20-41  2020-04-02 [1] CRAN (R 4.0.3)                           
 lazyeval            0.2.2    2019-03-15 [1] CRAN (R 4.0.2)                           
 lifecycle           0.2.0    2020-03-06 [1] CRAN (R 4.0.2)                           
 loo                 2.3.1    2020-07-14 [1] CRAN (R 4.0.2)                           
 lubridate         * 1.7.9.2  2020-11-13 [1] CRAN (R 4.0.2)                           
 magrittr            1.5      2014-11-22 [1] CRAN (R 4.0.2)                           
 markdown            1.1      2019-08-07 [1] CRAN (R 4.0.2)                           
 matrixStats         0.57.0   2020-09-25 [1] CRAN (R 4.0.2)                           
 memoise             1.1.0    2017-04-21 [1] CRAN (R 4.0.2)                           
 mime                0.9      2020-02-04 [1] CRAN (R 4.0.2)                           
 miniUI              0.1.1.1  2018-05-18 [1] CRAN (R 4.0.2)                           
 munsell             0.5.0    2018-06-12 [1] CRAN (R 4.0.2)                           
 pillar              1.4.6    2020-07-10 [1] CRAN (R 4.0.2)                           
 pkgbuild            1.1.0    2020-07-13 [1] CRAN (R 4.0.2)                           
 pkgconfig           2.0.3    2019-09-22 [1] CRAN (R 4.0.2)                           
 pkgload             1.1.0    2020-05-29 [1] CRAN (R 4.0.2)                           
 plyr              * 1.8.6    2020-03-03 [1] CRAN (R 4.0.2)                           
 prettyunits         1.1.1    2020-01-24 [1] CRAN (R 4.0.2)                           
 processx            3.4.4    2020-09-03 [1] CRAN (R 4.0.2)                           
 promises            1.1.1    2020-06-09 [1] CRAN (R 4.0.2)                           
 ps                  1.4.0    2020-10-07 [1] CRAN (R 4.0.2)                           
 purrr               0.3.4    2020-04-17 [1] CRAN (R 4.0.2)                           
 R6                  2.4.1    2019-11-12 [1] CRAN (R 4.0.2)                           
 Rcpp                1.0.5    2020-07-06 [1] CRAN (R 4.0.2)                           
 RcppParallel        5.0.2    2020-06-24 [1] CRAN (R 4.0.2)                           
 readr             * 1.4.0    2020-10-05 [1] CRAN (R 4.0.2)                           
 remotes             2.2.0    2020-07-21 [1] CRAN (R 4.0.2)                           
 reshape2            1.4.4    2020-04-09 [1] CRAN (R 4.0.2)                           
 rLakeAnalyzer       1.11.4.1 2019-06-09 [1] CRAN (R 4.0.2)                           
 rlang               0.4.8    2020-10-08 [1] CRAN (R 4.0.2)                           
 rprojroot           1.3-2    2018-01-03 [1] CRAN (R 4.0.2)                           
 rsconnect           0.8.16   2019-12-13 [1] CRAN (R 4.0.2)                           
 rstan             * 2.21.3   2020-10-20 [1] Github (stan-dev/rstan@ae36e6f)          
 rstudioapi          0.11     2020-02-07 [1] CRAN (R 4.0.2)                           
 scales              1.1.1    2020-05-11 [1] CRAN (R 4.0.2)                           
 sessioninfo         1.1.1    2018-11-05 [1] CRAN (R 4.0.2)                           
 shiny             * 1.5.0    2020-06-23 [1] CRAN (R 4.0.2)                           
 shinyjs             2.0.0    2020-09-09 [1] CRAN (R 4.0.2)                           
 shinystan         * 2.5.0    2018-05-01 [1] CRAN (R 4.0.2)                           
 shinythemes         1.1.2    2018-11-06 [1] CRAN (R 4.0.2)                           
 StanHeaders       * 2.21.0-6 2020-08-16 [1] CRAN (R 4.0.2)                           
 streamMetabolizer * 0.11.4   2020-10-22 [1] Github (USGS-R/streamMetabolizer@4a7ba9c)
 stringi             1.5.3    2020-09-09 [1] CRAN (R 4.0.2)                           
 stringr             1.4.0    2019-02-10 [1] CRAN (R 4.0.2)                           
 testthat            2.3.2    2020-03-02 [1] CRAN (R 4.0.2)                           
 threejs             0.3.3    2020-01-21 [1] CRAN (R 4.0.2)                           
 tibble              3.0.4    2020-10-12 [1] CRAN (R 4.0.2)                           
 tidyr             * 1.1.2    2020-08-27 [1] CRAN (R 4.0.2)                           
 tidyselect          1.1.0    2020-05-11 [1] CRAN (R 4.0.2)                           
 unitted             0.2.9    2020-10-22 [1] Github (appling/unitted@d1f1172)         
 usethis             2.0.0    2020-12-10 [1] CRAN (R 4.0.2)                           
 V8                  3.2.0    2020-06-19 [1] CRAN (R 4.0.2)                           
 vctrs               0.3.4    2020-08-29 [1] CRAN (R 4.0.2)                           
 withr               2.3.0    2020-09-22 [1] CRAN (R 4.0.2)                           
 xfun                0.18     2020-09-29 [1] CRAN (R 4.0.2)                           
 xtable              1.8-4    2019-04-21 [1] CRAN (R 4.0.2)                           
 xts                 0.12.1   2020-09-09 [1] CRAN (R 4.0.2)                           
 yaml                2.2.1    2020-02-01 [1] CRAN (R 4.0.2)                           
 zoo                 1.8-8    2020-05-02 [1] CRAN (R 4.0.2)    

[ahaha123589413]

I just ran into the same problem, I found a reference for the description and solution of this problem, but I haven't found the answer yet, I hope it will help you, if you find a solution, you can also share it, the original text is as follows:
Bayesian models require the rstan
interface to Stan. Sometimes this is as simple as
installing Rtools and calling the above install.packages or install_github command, but other
times everything seems fine until you try to run a Bayesian model in
streamMetabolizer. Symptoms of an imperfect rstan installation are probably
diverse. Here's one we've seen:

> bayes_fit <- metab(specs('bayes'), data=mydat)
Warning message:
In metab_fun(specs = specs, data = data, data_daily = data_daily,  :
  Modeling failed: argument is of length zero

> get_fit(bayes_fit)
...
$warnings
[1] "running command ''/Library/Frameworks/R.framework/Resources/bin/R' CMD config CXX 2>/dev/null' had status 1"

$errors
[1] "argument is of length zero"

In such cases you should refer to the detailed instructions on the rstan website for
Mac and Linux
or Windows.