Invoking a frontend using the gRPC API

[cowwoc]

Description

I am trying to build a Java client for BuildKit using the gRPC API.

The "dockerfile.v0" frontend seems to expect us to pass inputs in LLB format, but (as far as I can tell) there is no way to generate LLB outside of the Go code.

Is this functionality that needs to be added or am I missing something?

To be clear, I am looking for an API to invoke as opposed to running buildkitd over command-line. Thank you.

[jsternberg]

Hi. If I'm understanding correctly, you're attempting to invoke the gRPC API for Buildkit as a client. There are two APIs that use gRPC and I suspect you might be looking at the wrong one.

There's the frontend API which is very heavily based on the current Go implementation. It's possible to use that API from outside of Go, but it's primarily meant as a way to create custom frontends. A custom frontend is invoked by buildkit and buildkit acts as the client to your frontend which is the server. The most typical usage of this is with the # syntax line at the top of a dockerfile.

I suspect you may be looking for the control API which is the outwards facing API. That's the gRPC API where buildkit is the server and your client is making an outside request for a build. This API is located in api/services/control/control.proto and you can use a standard gRPC client to connect with it over a Unix socket. This API also has a Solve method and, while you can send LLB directly, you can also specify a frontend in the solve request.

Rather than looking at anything in the frontend package, you might want to look at the implementation of buildctl. buildctl is a generic client for buildkit that uses the gRPC API directly while docker buildx is a very opinionated usage of buildkit to match with how people used the docker build command.

I hope that helps. If I've misunderstood your intention, please correct me. Thanks.

[cowwoc]

Hi @jsternberg

Thank you for the detailed response.

I am actually already using api/services/control/control.proto. The problem was that I couldn't figure out how to access the buildkit instance that is bundled with Docker for Desktop. Fortunately, I was able to reach Tonis on Slack and he told me about a undocumented (!!) /grpc endpoint in the Docker REST API. That was the missing piece.

I still have some open questions about the BuildKit gRPC API. If you take a look at https://github.com/cowwoc/docker/blob/3c7aa4849e63d188c45c5e94d9819cf0238ba56a/pom.xml#L403 I am totally guessing at what proto files I need to download, especially when it comes to dependencies of BuildKit's proto files.

Where can I find a list of all the proto files used to build BuildKit (and any dependent proto files) to make sure that I'm using the same versions as you guys?

For example, I'm not sure if this is the right version of ext.proto: https://github.com/cowwoc/docker/blob/3c7aa4849e63d188c45c5e94d9819cf0238ba56a/pom.xml#L692

And as you can see at https://github.com/cowwoc/docker/blob/3c7aa4849e63d188c45c5e94d9819cf0238ba56a/pom.xml#L723 I am having to modify the downloaded proto file to avoid a compile-time error... so something doesn't smell right.

[jsternberg]

I think the easiest way to get the correct protobuf files may be to use our builder at hack/dockerfiles/generated-files.Dockerfile and then copy/paste them into your own repository after retrieving them and then manually updating them when you need to. Updating them should be very infrequent as you would also only need to do that when you add a new feature since, even if we add a new feature, the old protobuf is compatible with any new protobuf.

I modified the file a little bit to hopefully make things easier for you.

#5911

The file previously copied the protobuf files for generation into a golang based image so it could then run the generator. It aggregated multiple different dependencies. I've modified it so it now aggregates into a separate stage so you can do something like:

docker buildx build -f hack/dockerfiles/generated-files.Dockerfile --target protobuf -o ../protobuf-out

This will also include the bin/protoc binary but you can just ignore that.