Can I replace Dockerfile with pypline ?

[blaisep]

(RTFM welcomed)

So, I have to impose consistency and discipline at work (for my sins). It seems like every Dockerfile is written a bit differently, and it is easy to write bad Dockerfiles.
Moreover, I showed my devs how to use https://www.testcontainers.org and they really like them.
Does anyone have experience using pypyr to replace dockerfiles (perhaps using buildah, testcontainers-python or the Docker SDK ?

[yaythomas]

There are a few ways of going about this (sorry I took a minute to respond, the week got away from me!).

I take the heart of your question to be standardizing or normalising Dockerfiles. This is. . . tricky.

But depending on what exactly you're doing you can find a solution that Works Well(Tm) for your purposes :-D

1. Use Dockerfiles as templates, with {substitutions} to swap in build-time values via filereplace or fileformat

So, let's say you have ops/docker/templates/my-app-dockerfile.template:

FROM {base_image}

RUN {image_cmds}
WORKDIR /usr/src/app

COPY arbdir/blah ./

RUN pip install --no-cache-dir {pip_deps}

CMD [ "python", "{default_pyscript}" ]

So the idea is that you can now hot-swap anything in {curlies} via pypyr substitutions. Let's use fileformat so pypyr will match {keyname} in the dockerfile template to the same variable names in the pypyr context:

Let's put the dynamic values in pyproject.toml.

# ./pyproject.toml
[tool.pypyr.vars]
base_image = "python:3.10.5"
image_cmds = """set -ex; \
  apt-get update; \
  apt-get install -y --no-install-recommends \
  autoconf \
  automake"""
pip_deps = "requests pylint autopep8==1.5.7"
default_pyscript = "./your-daemon-or-script.py"
image_repo = "myrepo"
image_name = "myimagename"

No real reason to use pyproject.toml other than for the sake of example, you could use a json or yaml or custom toml file instead depending on which you prefer. In that case you'd use fetchjson or fetchyaml or fetchtoml instead of configvars to initialise the pypyr context. Or you could of course just hard-code the values into the pipeline, but I'm trying to show that in principle you can pull those values from somewhere else.

steps:
# this loads the values from the table tool.pypyr.vars in pyproject.toml into context
- pypyr.steps.configvars

- name: pypyr.steps.fileformat
   comment: swap out all substitution expressions in the Dockerfile template and write to output
   in:
     fileFormat:
       in: ops/docker/templates/my-app-dockerfile.template
       out: ops/out/Dockerfile

- name: pypyr.steps.set
   comment: arbitrarily set branch + build no. you prob want to get these values from the env rather than just setting here as example.
   in:
     set:
       branch: mybranch
       build_number: 1234
              
- name: pypyr.steps.cmd
   comment: build the image specified by the newly created Dockerfile at ops/out/Dockerfile
   in:
     cmd: docker build -t {image_repo}/{image_name}:{branch}.{build_number} -f ./ops/out/Dockerfile .

[blaisep]

(So hard to pick one of these as "the answer")

[yaythomas]

on a more, ah, governance related note. . . it's very probably not a bad idea that an image build is both deterministic and repeatable - so although in theory "npm install latest" or "pip install blah" means you can just re-run your docker build when you want to create a new image with all the latest packages on it without having to edit the Dockerfile. . . BUT if you run into some compatibility issues down the line (let's say latest version of package x doesn't work with latest version of package y) and you want/need to re-build your image from scratch (let's say you accidentally overwrote the previous version's tag in the docker repository, or maybe you need to hunt down a bug on an older version of the product) then you don't have an easy way to roll-back to the previous known image.

So pinning version numbers might be a good thing, depending on your ops practices/testing regime/release management etc.

Of course, if you encode your image creation process in a sensibly tested repeatable pypyr pipeline and your container naming/tagging conventions are good you hopefully won't end up with accidents like these 😁 But even so, personally I like the warm safe feeling I get when I know I can re-build my container images from scratch for any specific release version and it'll work. . .

So while "dynamic" Dockerfiles or docker image building can be convenient, it might be a good idea to keep a record of the actual generated output Dockerfile in source control somewhere (i.e maybe commit the output Dockerfile you generate into source-control with some sort of build number/version id in the filename; or tag it alongside its various source files in git; or park it in a branch; or however you're doing versioning/history), or if you're building it via buildah that the generating script/pipeline is in source-control too.

[yaythomas]

2. If you're using buildah to build your image, you can use cmd to pass whatever cmds you like to buildah:

steps:
# some sort of step to fetch or set the necessary variables in context here

- name: pypyr.steps.set
   in:
     set:
       container_id: mycontainer from fedora:latest
          
- name: pypyr.steps.cmd
   in:
     cmd:
       - buildah config --env GOPATH=/root/mydir {container_id}
       - buildah run {container_id} /bin/sh -c 'dnf -y install --enablerepo=updates-testing make golang; mkdir -p /root/blah; make install;'
       - buildah commit {container_id} buildahupstream

[blaisep]

Oh, yes cmd is so handy, Thank you !

[yaythomas]

3. For testcontainers, I'm guessing you're more likely to want to inject your test connection string or connection object into your pytest/unittest runner - so in your integration test module(s) setup/teardown you'd instantiate/clean-up the testcontainer. Given that testcontainer so nicely exposes the container setup/teardown as python, it's probably more productive just to take advantage of that directly in python.

Alternatively, you could of course use pypyr to manage an integration test scenario something like this:

steps:
  - name: pypyr.steps.cmd
     in:
       cmd: docker run -d -p 8000:8000 --name test-rest-server node-docker
       
  - name: pypyr.steps.cmd
     in:
       cmd: pytest tests/integration

  - name: pypyr.steps.cmd
     in:
       cmd: docker stop test-rest-server

Note: you prob want to be a bit more careful about error handling to make sure the stop command always runs at the end. . . As given is just a quick-and-dirty example to demonstrate the point.

[blaisep]

Very nice, thank you.
BTW,

Given that testcontainer so nicely exposes the container setup/teardown as python, it's probably more productive just to take advantage of that directly in python.

IMHO, the brilliance of pypyr is that it exposes the structure of the pipeline to a Javascript dev, and allows the devops crew to do their testcontainer magic in python.

[yaythomas]

4. For the docker python api, the build method requires a Dockerfile, so this does not quite solve the problem you speak about re the Dockerfiles themselves. Given how easily pypyr can just invoke the docker cli cmd itself, I'm not sure that docker python api is giving you all that much extra.