v1.3.1 Docker credentials not read from ~/.docker/config.json

[kshakir]

Version of Apptainer

$ apptainer --version
apptainer version 1.3.1-1.el7
$ 

Expected behavior

If one only has a ~/.docker/config.json then the credentials should be read from this file path, as stated in the apptainer documentation.

Actual behavior

Only the file ~/.apptainer/docker-config.json is searched for credentials.

Steps to reproduce this behavior

Remove your apptainer docker-config.json

# for example: rm ~/.apptainer/docker-config.json

Remove your docker configuration directory

# for example: rm ~/.docker/config.json

Login to DockerHub

$ apptainer registry login -u [elided] docker://docker.io
Password / Token: 
INFO:    Token stored in /[elided]/.apptainer/remote.yaml
$ 

NOTE: As we'll see, the token is NOT stored in that file… but that message isn't really the issue.

Try to pull a private docker image (this works)

# Don't need to pull the whole image, so using `timeout` which logs in, but then produces an error when terminated
$ timeout 3s apptainer -s build temp.sif docker://somerepo/someimg:sometag
FATAL:   While performing build: while creating squashfs: create command failed: exit status 1: 
$ 

Move the file containing the token to ~/.docker/config.json

$ mkdir ~/.docker
$ mv ~/.apptainer/docker-config.json ~/.docker/config.json
$ 

(Optional) Verify your username and password are stored within the ~/.docker/config.json

$ jq -r '.auths["docker.io"].auth' ~/.docker/config.json | base64 -d && echo
[elided]
$ 

Try to pull the same private image, but using ~/.docker/config.json (should work but doesn't)

$ timeout 3s apptainer -s build temp.sif docker://somerepo/someimg:sometag
FATAL:   While performing build: conveyor failed to get: while converting reference: reading manifest sometag in docker.io/someimg/sometag: requested access to the resource is denied
$ 

What OS/distro are you running

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"
$ 

How did you install Apptainer

Was probably installed via RPM by our I/T department.

Other notes:

If this feels like a repeat, it is. This was filed as #1616 and then fixed in v1.2.3. I can only guess that PR #1866 recreated the issue.

For anyone else looking for a workaround, I recommend users on our cluster link the path apptainer is looking for credential info back to the place docker and others are looking for credential info:

$ ln -sf ~/.docker/config.json ~/.apptainer/docker-config.json 

I'm not sure of the full extent of the side effects of this symlink on apptainer registry login. But apptainer seems happy with the linking on our system, along with third-party tools like gcloud auth configure-docker that only write to the file ~/.docker/config.json.

Alternatively, it's possible that apptainer only wants to support the ~/.apptainer/docker-config.json, in which case this is a documentation bug, as referred to in the "expected behavior" above.

[DrDaveD]

@JasonYangShadow please take a look at this

[JasonYangShadow]

@kshakir thanks so much for the detailed investigation and report. Yeah, this issue is recreated by upgrading the oras-go library in 1.3.x. I will take a look at this issue and will update here.

[JasonYangShadow]

@DrDaveD cc @ikaneshiro @cclerget
I checked the code, apptainer uses oci pull func to pull images from docker registry
https://github.com/apptainer/apptainer/blob/main/internal/pkg/client/oci/pull.go#L41
the code is pretty much not changed in the last year.
PR mentioned above #1669 are changes for oras protocol.
Every time when I try pulling images from docker hub, I always get the following error

FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://jasonyangshadow/alpine:latest: reading manifest latest in docker.io/jasonyangshadow/alpine: requested access to the resource is denied

it looks like the ocitypes.SystemContext has the correct value, but the pulling always fails.
Any ideas?

[DrDaveD]

The only idea I have is to try it with older versions to see if it used to work, and if so figure out when it changed.