401 Unauthorized error from Singularity gatk

[cakirb]

Description of the bug

I’m running the Sarek pipeline and can run most of the steps well, but I’m getting an error from Singularity, as shown below.

I’m confused by this error because Singularity and the images for the previous steps worked perfectly. I’m wondering if I'm missing something or if the image itself has some access issues.

Command used and terminal output

#!/bin/bash

set -euo pipefail

sample_table=actions/samplesheet_fq.csv

[[ -e "$sample_table" ]] || (echo "File $sample_table not found" && false)

nextflow run nf-core/sarek -r 3.6.0 \
  --input $sample_table \
  --outdir ./sarek-results \
  --tools deepvariant,haplotypecaller,strelka \
  -profile singularity,sanger \
  -c actions/custom.singularity.config \
  -resume

---

ERROR ~ Error executing process > 'NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:VCF_VARIANT_FILTERING_GATK:CNNSCOREVARIANTS (1)'

Caused by:
  Failed to pull singularity image
    command: singularity pull  --name nf-core-gatk-4.5.0.0.img.pulling.1761213387673 docker://nf-core/gatk:4.5.0.0 > /dev/null
    status : 255
    hint   : Try and increase singularity.pullTimeout in the config (current is "1h")
    message:
      WARNING: Couldn't use cached digest for registry: HEAD https://index.docker.io/v2/nf-core/gatk/manifests/4.5.0.0: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
      WARNING: Falling back to direct digest.
      FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://nf-core/gatk:4.5.0.0: GET https://index.docker.io/v2/nf-core/gatk/manifests/4.5.0.0: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:nf-core/gatk Type:repository]]



 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details
-[nf-core/sarek] Pipeline completed with errors-

Relevant files

# custom.singularity.config
singularity.pullTimeout = '1h'

nextflow.log

System information

• Nextflow version: 25.04.4
• Hardware: Sanger's HPC
• Executor: lsf
• Container engine: Singularity
• OS: Ubuntu
• nf-core/sarek version: 3.6.0

[famosab]

We run into these timeouts in the CI sometimes as well. Did you try re-running the pipeline? And if yes, did the error persist?

[cakirb]

Hi, thanks for getting back to me. Yes, I’ve re-run the pipeline multiple times, but I keep encountering the same error. I suspect it’s happening because I can’t successfully pull the Singularity image due to an authentication issue.

I don’t think the problem is with Singularity itself, none of the steps would run if it were. Since the previous steps completed successfully, it makes me wonder whether this particular image might have an issue.

Thanks!

[famosab]

Have you tried just running

singularity pull  --name nf-core-gatk-4.5.0.0.img.pulling.1761213387673 docker://nf-core/gatk:4.5.0.0

and is this also failing then?

Edit:

$ singularity pull  --name nf-core-gatk-4.5.0.0.img.pulling.1761213387673 docker://nf-core/gatk:4.5.0.0
FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://nf-core/gatk:4.5.0.0: reading manifest 4.5.0.0 in docker.io/nf-core/gatk: requested access to the resource is denied

Seems like there is something wrong with the image itself as you said. We will investigate!