Add support for building with buildkit

[felipecrs]

Requires passing version='2' like:

stream = self.client.build(
    fileobj=dockerfile, tag='myimage', version='2'
)

Refs https://github.com/moby/moby/blob/3ff85c73436f1c4f8d9764a0d72e41a03b4116f5/api/swagger.yaml#L9343-L9352.

Closes #2230

[felipecrs]

@thaJeztah do you think you can take a look at this one?

I know the PR backlog is full, but in my opinion this PR is a no-brainer and is of huge help (#2230 has 200+ upvotes).

[AkihiroSuda]

Bumping up the version number doesn't magically implement the BuildKit API.
You have to dial /grpc to call the BuildKit gRPC API.
https://github.com/docker/buildx/blob/v0.25.0/driver/docker/driver.go
https://github.com/moby/buildkit/blob/v0.23.1/api/services/control/control.proto

In addition to implementing the gRPC client, you also have to implement several "attachable" servers (auth, secret, ssh) via the reverse-gRPC connection.
https://github.com/moby/buildkit/blob/v0.23.1/cmd/buildctl/build.go#L191-L214

This is quite complicated than you might imagine; I suggest just shelling out docker buildx and call it a day.

[felipecrs]

Thanks a lot for reviewing it.

Bumping up the version number doesn't magically implement the BuildKit API.

Tests prove you are wrong. No?

In addition to implementing the gRPC client, you also have to implement several "attachable" servers (auth, secret, ssh) via the reverse-gRPC connection.

Exposing additional options can be done in follow-up PRs. This PR is sufficient for my use case which is building Dockerfiles that depend on BuilKit-features that are not auth, secret, or ssh.

I suggest just shelling out docker buildx and call it a day.

That means installing docker buildx, which means installing docker cli. It's a big downside comparing to just calling the Rest API.

[AkihiroSuda]

Only basic BuildKit features can be enabled by just bumping up the version:
https://github.com/moby/moby/blob/v28.3.0/api/server/backend/build/backend.go#L55-L73

The scope of the available features should be documented

[felipecrs]

Do you mean I should clarify that squash and forcerm will error when version=2?

Otherwise, I expect all options currently supported by client.build() to be supported.

[felipecrs]

I will test it more, and let you know.

[felipecrs]

Ok, this is worse than I expected.

I got lucky with the tests because I only tried with FROM scratch.

Even a basic Dockerfile like below does not work:

    @requires_api_version('1.38')
    def test_build_buildkit_alpine(self):
        script = io.BytesIO('\n'.join([
            'FROM alpine',
        ]).encode('ascii'))

        self.tmp_imgs.append('buildkitalpine')

        stream = self.client.build(
            fileobj=script, tag='buildkitalpine',
            version='2'
        )

        for _chunk in stream:
            pass

        assert self.client.inspect_image('buildkitalpine')

But it's because of a bug:

• API /build doesn't pass AuthConfig to BuildKit moby/moby#48112

Still, it means not even basic use cases like pulling an image that doesn't require auth will not be able to use this feature.

@AkihiroSuda do you have any idea? Otherwise, I think it may be the end of this PR (and a dream). :(

This comment says creating a basic session is enough for pull to work. Do you think it would be acceptable to build a solution around it?

[AkihiroSuda]

This moby/moby#48112 (comment) says creating a basic session is enough for pull to work. Do you think it would be acceptable to build a solution around it?

SGTM

[felipecrs]

I haven't been able to build anything so far, but I will update when I have any news.

[AkihiroSuda]

This PR doesn't work

https://github.com/docker/docker-py/pull/3344/files#r2174151883