Speeding up Image build on kubernetes / openshift

[o4rz3l]

Hi,
i'm using this buildah-Image (https://catalog.redhat.com/software/containers/ubi9/buildah/61959488b0df17a5d66395f6?architecture=amd64&image=6571633ec743cccfe7615c69) to build images inside an Openshift Cluster.
This works quite well, but it is a little bit slow.
I tried using caching from/to an external registry, but that did not speed up things as expected and introduced other problems, like API-Limit hiting...
Then i tried to use internal openshift registry but got a lot of HTTP 500 errors, so i decided not to dig deeper in this direction.
Now, i want to add a PVC Storage for layer caching, but can't figure out which directories i need.
I tried /var/run/containers/storage, /var/lib/containers/storage and /var/lib/shared in different combinations.
When buildah starts using the cache, there where several strang errors like Permission Denied inside the build, directories not found etc.
So my questions would be, what do i need the cache via PVC in order to get successful builds while using the cache.

Thanks in advance!
Oli

[flouthoc]

@o4rz3l Could you paste output of buildah info it shows the path of graphroot. You can use path of PVC mount path as graphroot and all your images/layers can be stored there and buildah also has cli flag --root which lets you do this for every run :)

Could you please try this and update back here.

[o4rz3l]

Hi @flouthoc,
thanks for your reply. Here is the buildah info output:

{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "\"rhel\"",
            "version": "9.3"
        },
        "MemFree": 23102844928,
        "MemTotal": 126717054976,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 24,
        "hostname": "runner-9cimsvzbz-project-43-concurrent-0-lgj5bqv4",
        "kernel": "4.18.0-372.59.1.el8_6.x86_64",
        "os": "linux",
        "rootless": false,
        "uptime": "7h 7m 44.55s (Approximately 14.04 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.imagestore=/var/lib/shared",
            "overlay.mount_program=/usr/bin/fuse-overlayfs",
            "overlay.mountopt=nodev,metacopy=on"
        ],
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "overlayfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}

I tried using the --root option and pointed it, to a pvc mounted directory (/var/lib/image-cache). After this, my build start to fail with this error

Failed to store expired repos cache: [Errno 13] Permission denied: '/var/cache/dnf'

I have no idea, where the connction is between caching and this error. disbaling the cache fixes the image build.
Any ideas?

Thanks in advance!

[flouthoc]

Can you also share buildah --root <with-your-root> info , also could you please share if you have selinux enabled or something like apparmor ?

[flouthoc]

If selinux is enabled could you check if something is showing in audit logs ?

[o4rz3l]

Can you also share buildah --root <with-your-root> info , also could you please share if you have selinux enabled or something like apparmor ?

{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "\"rhel\"",
            "version": "9.3"
        },
        "MemFree": 34156642304,
        "MemTotal": 12671705[49](https://gitlab-inte.rz.bankenit.de/atruvia-early-adopters/DOM00020/DOM00097/DOM00109/2a69db8e-1a19-4bcc-bf18-26f2de2a20b3/gl-image-build/-/jobs/86347#L49)76,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 24,
        "hostname": "runner-9cimsvzbz-project-43-concurrent-0-ymgq0zq8",
        "kernel": "4.18.0-372.59.1.el8_6.x86_64",
        "os": "linux",
        "rootless": false,
        "uptime": "428h 34m 28.9s (Approximately 17.83 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 1
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.imagestore=/var/lib/shared",
            "overlay.mount_program=/usr/bin/fuse-overlayfs",
            "overlay.mountopt=nodev,metacopy=on"
        ],
        "GraphRoot": "/var/lib/image-cache",
        "GraphStatus": {
            "Backing Filesystem": "nfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 5
        },
        "RunRoot": "/run/containers/storage"
    }
}

I'm not aware of selinux or appamor, so i can't provide any logs of that :/

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[gewei2000]

+1

[github-actions]

A friendly reminder that this issue had no activity for 30 days.