Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

COPY fails on some base-images (cern/cc7-base) #19

Open
lukasheinrich opened this issue Jan 18, 2018 · 17 comments
Open

COPY fails on some base-images (cern/cc7-base) #19

lukasheinrich opened this issue Jan 18, 2018 · 17 comments

Comments

@lukasheinrich
Copy link

Hi,

i'm continuing to workon integrating this with some of the CERN infrastructure. skopeo / orca-build / umoci (with https://github.com/openSUSE/umoci/issues/223) works nicely and I can build a new image from this Dockerfile

lheinric@lxplus071:/tmp/lheinric/test% cat Dockerfile
FROM busybox
RUN echo hello > /mydata
COPY datatocopy /datatocopy
lheinric@lxplus071:/tmp/lheinric/test% ls
datatocopy  Dockerfile

changing the FROM line to cern/cc7-base fails with a python error on shutil.copy when it tries to copy the datatocopy file

this is the log. Could this be related to some permissions set in the base image?

orca-build[INFO] Using existing image for build: ../outputimage
orca-build[INFO] BUILD[1 of 3]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:ad4325b2dd179e2103046df76bfe6efa7ac85e9ed6ab52edb33b8db399a9bd02
 81.41 MB / 81.41 MB [======================================================] 8s
Copying config sha256:a6644c6917335ef1b1285b7e6eb0e08445a798e4c230f3fd3601aa67f9c1ddd6
 280 B / 280 B [============================================================] 3s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 3]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
  ---> [runc]
  <--- [runc]
  ---> [umoci]
  <--- [umoci]
orca-build[INFO] BUILD[3 of 3]: copy ['datatocopy', '/datatocopy'] [json=False]
  ---> [umoci]
  <--- [umoci]
  ---> [umoci]
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
Traceback (most recent call last):
  File "/tmp/lheinric/bin/orca-build", line 741, in <module>
    __wrapped_main__()
  File "/tmp/lheinric/bin/orca-build", line 739, in __wrapped_main__
    main(ctx, config)
  File "/tmp/lheinric/bin/orca-build", line 705, in main
    builder.build(output=config.output, tags=config.tags, clean=config.clean, gc=config.gc)
  File "/tmp/lheinric/bin/orca-build", line 666, in build
    getattr(self, fn)(*args, isjson=step.isjson)
  File "/tmp/lheinric/bin/orca-build", line 563, in _dispatch_copy
    shutil.copy2(src, dst)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/shutil.py", line 257, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/tmp/lheinric/orca-bundle.xkqko1km/rootfs/datatocopy'
@lukasheinrich
Copy link
Author

lukasheinrich commented Jan 18, 2018 •
edited

I notice that the rootfs directory is created without any write permissions (dr-xr-xr-x), which I think are set by umoci, so maybe this issue should be moved there, tough this line seems to suggest the rootfs path is created with a 755 mask

https://github.com/openSUSE/umoci/blob/52afb0fb11000b6229f9bc297ad85fb210f0572f/oci/layer/unpack.go#L129

lheinric@lxplus071:/tmp/lheinric/orca-bundle.uy10pjk4% ls -lr
total 2388
-rw-r--r--.  1 lheinric zp     542 Jan 18 14:29 umoci.json
-rw-r--r--.  1 lheinric zp 2429008 Jan 18 14:29 sha256_e063fcf061ac201da9bf323aef6b70a10b51721e9b005e881eef9063d4128d12.mtree
dr-xr-xr-x. 17 lheinric zp    4096 Jan 18 14:29 rootfs
-rw-r--r--.  1 lheinric zp    3588 Jan 18 14:29 config.json

@cyphar
Copy link
Owner

cyphar commented Mar 1, 2018

I will take a look at this bug today/tomorrow -- sorry for not looking at this in a while (for some reason I got un-subscribed from this repository...). Just to verify -- is cern/cc7-base publicly available?

I notice that the rootfs directory is created without any write permissions (dr-xr-xr-x), which I think are set by umoci, so maybe this issue should be moved there, tough this line seems to suggest the rootfs path is created with a 755 mask

That's very strange -- umoci should be making it with a 755 mask... I haven't seen that before.

@lukasheinrich
Copy link
Author

lukasheinrich commented Mar 1, 2018 •
edited

yes it's available. Thanks for looking into it!

@cyphar
Copy link
Owner

cyphar commented Mar 22, 2018 •
edited

I haven't been able to reproduce this problem for some reason:

orca-build[INFO] Created new image for build: /tmp/orca-build.v6_8qzzo
orca-build[INFO] BUILD[1 of 3]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:297ef71a1de0345dfd52a966e8072ea4f1c38194c86df1ec46605ad56d958a69
 81.45 MB / 81.45 MB [=====================================================] 39s
Copying config sha256:8a8dfdb81bfd41a0346c19564f11444ac1a3e8905206b92d79e5f9a38f131c92
 280 B / 280 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 3]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
  <--- [umoci]
  ---> [runc]
  <--- [runc]
  ---> [umoci]
  <--- [umoci]
orca-build[INFO] BUILD[3 of 3]: copy ['datatocopy', '/datatocopy'] [json=False]
  ---> [umoci]
  <--- [umoci]
  ---> [umoci]
  <--- [umoci]
  ---> [umoci]
  <--- [umoci]
orca-build[INFO] BUILD: finished
orca-build[INFO] BUILD: created tags ['e60688b797f95c7bab02fad19bd7910021f963975cdf2dffc28b290d006163ff-dest']

Do you have a strange umask set in your shell that could be causing this -- or something like that? What is the output of umask? I tested this with:

% runc version
runc version 1.0.0-rc5
spec: 1.0.0
% umoci --version
umoci version 0.4.0
% skopeo version
skopeo version 0.1.26

@lukasheinrich
Copy link
Author

hm this is my setup

umask gives

% umask
022

the tool versions are

% umoci --version
umoci version 0.4.0+dev~git8b89fabfac6c176b95078cb106c2704303564e6d
% runc --version
runc version spec: 1.0.0
% skopeo --version
skopeo version 0.1.26

% orca-build -t what .
orca-build[INFO] Created new image for build: /tmp/lheinric/orca-build.aqeez1ak
orca-build[INFO] BUILD[1 of 5]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:297ef71a1de0345dfd52a966e8072ea4f1c38194c86df1ec46605ad56d958a69
 81.45 MB / 81.45 MB [======================================================] 3s
Copying config sha256:8a8dfdb81bfd41a0346c19564f11444ac1a3e8905206b92d79e5f9a38f131c92
 280 B / 280 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 5]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
   ⨯ create runtime bundle: chown rootfs: lchown /tmp/lheinric/orca-bundle.hyumu_7p/rootfs: operation not permitted
  <--- [umoci]
orca-build[CRITICAL] Error executing subprocess: umoci unpack --image=/tmp/lheinric/orca-build.aqeez1ak:2c1a8d7ff8c4c1c7b2b4211768e79103fa4b0d6bbe5a4433df03fe8a3446861c-src /tmp/lheinric/orca-bundle.hyumu_7p failed with error code 1

@cyphar
Copy link
Owner

cyphar commented Mar 23, 2018

Ah, I misunderstood. I was running orca-build as root. You need to use --rootless, but I will take another look...

@lukasheinrich
Copy link
Author

lukasheinrich commented Mar 23, 2018 •
edited

Ah right this was just a mistake in the post above in the OP I used rootless.

this is what ai get

lheinric@lxplus021:/tmp/lheinric% orca-build --rootless -t what .
orca-build[INFO] Created new image for build: /tmp/lheinric/orca-build.q75819d9
orca-build[INFO] BUILD[1 of 3]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:297ef71a1de0345dfd52a966e8072ea4f1c38194c86df1ec46605ad56d958a69
 81.45 MB / 81.45 MB [======================================================] 2s
Copying config sha256:8a8dfdb81bfd41a0346c19564f11444ac1a3e8905206b92d79e5f9a38f131c92
 280 B / 280 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 3]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
  ---> [runc]
  <--- [runc]
  ---> [umoci]
  <--- [umoci]
orca-build[INFO] BUILD[3 of 3]: copy ['datatocopy', '/datatocopy'] [json=False]
  ---> [umoci]
  <--- [umoci]
  ---> [umoci]
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
Traceback (most recent call last):
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 741, in <module>
    __wrapped_main__()
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 739, in __wrapped_main__
    main(ctx, config)
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 705, in main
    builder.build(output=config.output, tags=config.tags, clean=config.clean, gc=config.gc)
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 666, in build
    getattr(self, fn)(*args, isjson=step.isjson)
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 563, in _dispatch_copy
    shutil.copy2(src, dst)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/shutil.py", line 257, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/tmp/lheinric/orca-bundle.zjpxpr91/rootfs/datatocopy'

@cyphar
Copy link
Owner

cyphar commented Mar 23, 2018

It's almost certainly caused by orca-build not being as clever as umoci when it comes to unprivileged copying (it doesn't know how to emulate CAP_DAC_OVERRIDE like umoci does). But I will investigate this one further...

@lukasheinrich
Copy link
Author

thanks for looking into it!

@cyphar
Copy link
Owner

cyphar commented Apr 6, 2018 •
edited

Sorry for not responding earlier --

The underlying issue is that orca-build isn't as clever as umoci when it comes to dealing with "bad" directories. Unfortunately the only real fix for this is to rewrite in Go so we can directly use umoci's packages.

The core problem is that the cern/cc7-base images have / as chmod 0555 -- which is not writable. You can fix this by copying to a sub-directory which is writable (so something like COPY datacopy /var/datacopy). Another option is to make your images not have this issue (you might need to add RUN chmod o+w / inside the image build you have set up).

The plan going forward for orca-build is that it will be part of the umoci project (but still a separate tool) and be written in Go. It will take a while unfortunately. But the above workarounds should help in the meantime.

@lukasheinrich
Copy link
Author

lukasheinrich commented Apr 6, 2018 •
edited

Thanks @cyphar

I just chose / as it seemed the most minimal of examples. It's very reasonable I think to request people to copy stuff elsewhere. Usually we use a pattern like

FROM ...
WORKDIR /code
ADD . /code

so from your description it seems this should work. Unfortunately it seems CERNs interactive node regressed into a state that makes nsenter fail

I wonder if it's related to this opencontainers/runc#1513

The node I'm working on should have unprivileged namespaces enabled though, maybe some update rolled that back. @davidlt did you try rootless runc recently on lxplus? I do see

cat /sys/module/namespace/parameters/unpriv_enable
Y

lheinric@lxplus021:/tmp/lheinric/wha% orca-build --rootless -t what .
orca-build[INFO] Created new image for build: /tmp/lheinric/orca-build.rym_2ou0
orca-build[INFO] BUILD[1 of 3]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:297ef71a1de0345dfd52a966e8072ea4f1c38194c86df1ec46605ad56d958a69
 81.45 MB / 81.45 MB [======================================================] 3s
Copying config sha256:8a8dfdb81bfd41a0346c19564f11444ac1a3e8905206b92d79e5f9a38f131c92
 280 B / 280 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 3]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
  ---> [runc]
nsenter: failed to unshare namespaces: Invalid argument
container_linux.go:348: starting container process caused "process_linux.go:279: running exec setns process for init caused \"exit status 40\""
  <--- [runc]
orca-build[CRITICAL] Error executing subprocess: runc --root=/tmp/lheinric/orca-runcroot.49u4p_ir run --bundle=/tmp/lheinric/orca-bundle.q7tuoacp orca-build-CcsvqYx301em4non1wp1f5FkCev4krQ0 failed with error code 1

@lukasheinrich
Copy link
Author

lukasheinrich commented Apr 6, 2018 •
edited

it seems due to this CVE https://access.redhat.com/security/cve/cve-2018-1000001 we now have have max_user_namespaces = 0 (temprarily at least) :( I'll try to resume testing once they are enabled again.

we're following the discussions here:

https://bugzilla.redhat.com/show_bug.cgi?id=1533836

@cyphar
Copy link
Owner

cyphar commented Apr 7, 2018

I just chose / as it seemed the most minimal of examples. It's very reasonable I think to request people to copy stuff elsewhere. Usually we use a pattern like so from your description it seems this should work.

In principle it might work -- though the writable issue still might occur. Really I would recommend not making your images have / un-writeable by the owner. But when I finish the Go port of this project and merge it into umoci, then this should no longer be an issue. In the meantime, you can try doing COPY to a directory which is writeable in the rootfs.

@lukasheinrich
Copy link
Author

thanks. I'll raise it with the people responsible for the image.

@lukasheinrich
Copy link
Author

turns out one can just add a chmod +w to it and that works

FROM cern/cc7-base
RUN mkdir /code && chmod +w /code
COPY . /code

@cyphar
Copy link
Owner

cyphar commented Jul 17, 2018

I guess that implies that the default uname is wrong somehow and so the directory is not created with 755 permissions? It's nice that this is a work around for now -- but as I said we really need to have orca-build be using pkg/unpriv from umoci.

@lukasheinrich
Copy link
Author

yeah the workaround also only works sometimes. If i switch the image

from cern/slc6-base to one build with this Dockerfile

FROM cern/slc6-base:latest
RUN adduser atlas && \
    usermod -aG wheel atlas

building a new image based on it fails (while it worked with cern/slc6-base. The /etc/gshadow file created during user creation seems to break it

FROM lukasheinrich/cern-slc6-breaking:latest
RUN  mkdir /code && chmod +w /code
RUN  echo hello > /code/mydata
COPY datatocopy /code/datatocopy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lukasheinrich @cyphar