Caching in CI/CD

[ryancausey]

I'm seeing some unexpected behavior when using the --layers option in GitLab CI/CD. When the build gets to the ARG command in the Dockerfile, the cache is never used, and every step from then on in the build does not use the already existing cached layers.

I'm running buildah by using this container image: quay.io/buildah/stable

$ buildah --version
buildah version 1.28.0 (image-spec 1.0.2-dev, runtime-spec 1.0.2-dev)

The command I am running in CI/CD is roughly the following:

buildah build \
  --layers \
  --cache-from $CI_REGISTRY_IMAGE/cache \
  --cache-to $CI_REGISTRY_IMAGE/cache \
  --tag $REPOSITORY_URL:$IMAGE_TAG \
  --secret id=aws-creds,src=/root/.aws/credentials \
  --secret id=aws-config,src=/root/.aws/config \
  --build-arg AWS_PROFILE=default \
  .

$CI_REGISTRY_IMAGE is a predefined variable that doesn't change, whose description can be found here: https://docs.gitlab.com/ee/ci/variables/predefined_variables.html

The relevant snippet of the dockerfile is as follows:

FROM public.ecr.aws/lambda/python:3.9 AS requirements_stage

# Install the AWS OTEL lambda layer and unpack it into /opt.
# Roughly following this guide:
# https://aws.amazon.com/blogs/compute/working-with-lambda-layers-and-extensions-in-container-images/
#
# Lambda layer ARN pulled from here:
# https://aws-otel.github.io/docs/getting-started/lambda/lambda-python#add-the-arn-of-the-lambda-layer
RUN yum install --assumeyes aws-cli curl unzip
RUN mkdir -p /opt
# The AWS_PROFILE variable needs to be set in order for the config profile to be picked
# up.
ARG AWS_PROFILE
# This mounts in the AWS credentials and AWS config file as secrets so they do not get
# stored in the final image.
# https://docs.docker.com/engine/reference/builder/#run---mounttypesecret
RUN --mount=type=secret,id=aws-creds,target=/root/.aws/credentials,required=true \
    --mount=type=secret,id=aws-config,target=/root/.aws/config,required=true \
    curl $( \
        aws lambda get-layer-version-by-arn \
        --arn arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-14-0:1 \
        --query 'Content.Location' \
        --output text \
    ) --output layer.zip
RUN unzip layer.zip -d /opt
RUN rm layer.zip

I tried running this same build locally using the quay.io/buildah/stable image and podman. I was able to reproduce the issue if I did not use a bind mount to preserve the information in /var/lib/containers. If I use a bind mount, something like --mount type=bind,source=$HOME/.local/share/containers,destination=/var/lib/containers, the ARG step will properly be recognized as being able to use the cache, and all subsequent steps can use the cache as well.

The issue is that I cannot cache /var/lib/containers across run in GitLab CI/CD as the cache path must reside within the project's root directory. Most other tools I use allow changing the cache directory via environment variables or otherwise, but I cannot find a similar docs for how to change the location of /var/lib/containers for buildah.

Is there not a way to change where buildah stores its intermediate containers/layers/cache from /var/lib/containers to somewhere else?

[ryancausey]

The solution I found was to modify /etc/containers/storage.conf inside the quay.io/buildah/stable container. I found the information on this file at the following links:

https://github.com/containers/buildah/blob/main/vendor/github.com/containers/storage/storage.conf

https://github.com/containers/storage/blob/main/docs/containers-storage.conf.5.md

Overriding the graphroot to point to a in project cache directory allowed the caching to start working properly.

I also discovered that all the intermediate layers seem to be cached here. This means that using --cache-from and --cache-to was no longer necessary and removing them resulted in a much quicker build. However, some folks might still want to leverage them to have a centralized cache repository shared among multiple projects, but this is not my personal use case.

[ryancausey]

@rhatdan I was thinking about this more today and was wondering if the behavior stated above is something worthy of an issue, or is this expected behavior? Summarized issue: Caching with --cache-from and --cache-to doesn't appear to work as expected with build args when the container storage location is not persistent across builds.

[rhatdan]

I am not an expert on this. WDYT @flouthoc @vrothberg

[ryancausey]

If I don't hear back within the next week I'm going to open an issue for this. Caching the graphroot works, but is quickly becoming unsustainable as it grows quite large quite quickly, and there's no good way to automatically clean it up that I can find. I don't have a way to mount a volume in the runner being used for CI/CD.

This makes me think that the --cache-from and the --cache-to should be preferred, but they don't appear to work when using an ARG AND starting with an empty graphroot.

[flouthoc]

Hi @ryancausey , sure could you please open an issue for this with small reproducer. I'll take this.

[ryancausey]

@flouthoc created #4621

[ryancausey]

This issue appears resolved by buildah version 1.29.1, though it may have been resolved by an earlier version. See: #4621