Memory issue with test profile with --ancestry

[AWS-crafter]

With 64 GB of RAM on Ubuntu, this occurs: "63288 MiB RAM detected; reserving 6144 MiB for main workspace," then it crashes due to not enough RAM. I think the expected behavior would be to allocate more than ~10% for main workspace.

The test profile without the --ancestry flag works fine.

Command ran: nextflow run pgsc_calc/main.nf -profile test,docker --run_ancestry pgsc_1000G_v1.tar.zst

Running with a custom.config with more RAM did not change the behavior.

Oct-04 01:21:30.930 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 9; name: PGSCATALOG_PGSCALC:PGSCALC:ANCESTRY_PROJECT:FILTER_VARIANTS (cineca GRCh37); status: COMPLETED; exit: 2; error: -; workDir: /home/ubuntu/work/68/a270c5e9f8df847078180043444783]
Oct-04 01:21:30.934 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=PGSCATALOG_PGSCALC:PGSCALC:ANCESTRY_PROJECT:FILTER_VARIANTS (cineca GRCh37); work-dir=/home/ubuntu/work/68/a270c5e9f8df847078180043444783
  error [nextflow.exception.ProcessFailedException]: Process `PGSCATALOG_PGSCALC:PGSCALC:ANCESTRY_PROJECT:FILTER_VARIANTS (cineca GRCh37)` terminated with an error exit status (2)
Oct-04 01:21:30.943 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'PGSCATALOG_PGSCALC:PGSCALC:ANCESTRY_PROJECT:FILTER_VARIANTS (cineca GRCh37)'

Caused by:
  Process `PGSCATALOG_PGSCALC:PGSCALC:ANCESTRY_PROJECT:FILTER_VARIANTS (cineca GRCh37)` terminated with an error exit status (2)

Command executed:

  # 1. Get QC'd variant set & unrelated samples from REFERENCE data for PCA --
  
  # ((IS_MA_REF == FALSE) && (IS_MA_TARGET == FALSE)) && (((IS_INDEL == FALSE) && (STRANDAMB == FALSE)) || ((IS_INDEL == TRUE) && (SAME_REF == TRUE)))
  awk '((($6 == 0) && ($9 == 0)) && ((($4 == 0) && ($5 == 0)) || (($4 == 1) && ($10 == 1)))) {print $2}' <(zcat cineca_22_matched.txt.gz) | gzip -c > shared.txt.gz
  
  plink2             --threads 2             --memory 6144             --pfile GRCh37_1000G_ALL vzs             --remove GRCh37_1000G.king.cutoff.out.id             --extract shared.txt.gz             --max-alleles 2             --snps-only just-acgt             --rm-dup exclude-all             --geno 0.1             --mind 0.1             --maf 0.05             --hwe 0.0001             --make-pgen vzs             --allow-extra-chr --autosome             --freq zs             --out GRCh37_reference
  
  # 2. LD-thin variants in REFERENCE (filtered variants & samples) for input
  # into PCA -----------------------------------------------------------------
  plink2             --threads 2             --memory 6144             --pfile vzs GRCh37_reference             --indep-pairwise 1000 50 0.05             --exclude range high-LD-regions-hg19-GRCh37.txt             --out GRCh37_1000G_ALL_thinned
  
  gzip -f *.prune.in *.prune.out
  
  cat <<-END_VERSIONS > versions.yml
  FILTER_VARIANTS:
      plink2: $(plink2 --version 2>&1 | sed 's/^PLINK v//; s/ 64.*$//' )
  END_VERSIONS

Command exit status:
  2

Command output:
  PLINK v2.00a3.3 SSE4.2 (3 Jun 2022)            www.cog-genomics.org/plink/2.0/
  (C) 2005-2022 Shaun Purcell, Christopher Chang   GNU General Public License v3
  Logging to GRCh37_reference.log.
  Options in effect:
    --allow-extra-chr
    --autosome
    --extract shared.txt.gz
    --freq zs
    --geno 0.1
    --hwe 0.0001
    --maf 0.05
    --make-pgen vzs
    --max-alleles 2
    --memory 6144
    --mind 0.1
    --out GRCh37_reference
    --pfile GRCh37_1000G_ALL vzs
    --remove GRCh37_1000G.king.cutoff.out.id
    --rm-dup exclude-all
    --snps-only just-acgt
    --threads 2
  
  Start time: Wed Oct  4 01:21:23 2023
  63288 MiB RAM detected; reserving 6144 MiB for main workspace.
  Using up to 2 compute threads.
  2504 samples (1270 females, 1234 males; 2497 founders) loaded from
  GRCh37_1000G_ALL.psam.
  77818345 out of 84805772 variants loaded from GRCh37_1000G_ALL.pvar.zst.
  2 categorical phenotypes loaded.
  
  End time: Wed Oct  4 01:21:30 2023

Command error:
  Error: Out of memory.  The --memory flag may be helpful.
  PLINK v2.00a3.3 SSE4.2 (3 Jun 2022)            www.cog-genomics.org/plink/2.0/
  (C) 2005-2022 Shaun Purcell, Christopher Chang   GNU General Public License v3
  Logging to GRCh37_reference.log.
  Options in effect:
    --allow-extra-chr
    --autosome
    --extract shared.txt.gz
    --freq zs
    --geno 0.1
    --hwe 0.0001
    --maf 0.05
    --make-pgen vzs
    --max-alleles 2
    --memory 6144
    --mind 0.1
    --out GRCh37_reference
    --pfile GRCh37_1000G_ALL vzs
    --remove GRCh37_1000G.king.cutoff.out.id
    --rm-dup exclude-all
    --snps-only just-acgt
    --threads 2
  
  Start time: Wed Oct  4 01:21:23 2023
  63288 MiB RAM detected; reserving 6144 MiB for main workspace.
  Using up to 2 compute threads.
  2504 samples (1270 females, 1234 males; 2497 founders) loaded from
  GRCh37_1000G_ALL.psam.
  77818345 out of 84805772 variants loaded from GRCh37_1000G_ALL.pvar.zst.
  2 categorical phenotypes loaded.
  
  End time: Wed Oct  4 01:21:30 2023

Work dir:
  /home/ubuntu/work/68/a270c5e9f8df847078180043444783

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
Oct-04 01:21:30.945 [Task monitor] INFO  nextflow.Session - Execution cancelled -- Finishing pending tasks before exit
Oct-04 01:21:30.947 [Actor Thread 24] ERROR nextflow.Nextflow - ERROR: No scores calculated!
Oct-04 01:21:30.949 [Actor Thread 25] ERROR nextflow.Nextflow - ERROR: Projection subworkflow failed
Oct-04 01:21:30.949 [Actor Thread 4] ERROR nextflow.Nextflow - ERROR: No results report written!
Oct-04 01:21:30.953 [main] DEBUG nextflow.Session - Session await > all processes finished
Oct-04 01:21:30.954 [main] DEBUG nextflow.Session - Session await > all barriers passed

[AWS-crafter]

The solution here: #63 does not change the behavior. However, it works great with real genomes (not the test profile) so I think it is some quirk of the test profile.

[nebfield]

Yes, the test profile won't work well with the ancestry workflow. The ancestry workflow needs a lot more resources.

pgsc_calc/conf/test.config

Lines 17 to 20 in c4117db

	// Limit resources so that this can run on GitHub Actions
	max_cpus = 2
	max_memory = '6.GB'
	max_time = '6.h'

Setting --max_memory and --max_cpus manually might help, but the test data isn't biologically meaningful and probably won't interact well with the reference panel. I'll add a note to the docs 🤠