reduce memory footprint

[jairideout]

fresco eats up quite a bit of memory. With the moving pictures dataset (currently the largest one we have), fresco requires more than 256GB of memory for a single process (--n_procs 1). I'm not sure exactly how much memory it requires, as this is the maximum amount of memory that Minerva can allocate for a single node. fresco's memory consumption will need to be reduced.

My guess (without delving into this too far) is that each OTU map is being parsed and stored in memory (dictionaries mapping OTU IDs to sequence IDs, and sequence IDs to OTU IDs). This is likely the largest factor driving memory consumption.

[rybern]

Okay. How did you measure the memory consumption? I've had problems before where Minerva would list an astronomical memory footprint for a program that runs fine on my laptop.

[jairideout]

The job gets killed by the job manager with an error message indicating that more memory was requested than was available.

I've also seen the strange memory footprint issue when using time on Minerva. If a job successfully finishes, Minerva seems to record a reasonable number in the stderr log file, so I've been using that.

[cleme]

I wonder if the scheduler in Minerva is just killing the job without it actually using that much memory. I've asked Ryan to work on profiling memory with moving pictures (study_550)