ci: workflow to build and publish the stacks-node image used internally for integration tests

[zone117x]

PR against master branch so that the github "run workflow" UI is available.

The API pulls a pre-built image for the stacks-node, used for both local development and in CI tests. Currently, it's pulling from another repo:

stacks-blockchain-api/stacks-blockchain/docker/Dockerfile

Line 2 in 5ee9d34

FROM --platform=linux/amd64 zone117x/stacks-api-e2e:stacks2.1-21295d3 as build

This PR creates a workflow dispatch to build and publish the testing stacks-node docker image. Once merged, a follow-up PR will change the above code to use the image produced by this repo.

Cross-compilation optimizations are implemented to build both x86 and arm64 images (without taking several hours).

Image tags use the short git commit hash -- this can be improved later once the stacks-blockchain next branch stabilizes (e.g. with an RC).

See demo run here: https://github.com/hirosystems/stacks-blockchain-api/actions/runs/3759601436/jobs/6389358450

[CharlieC3]

Can you clarify what the difference is between the docker image being built in this workflow compared to the image built in the stacks-blockchain repo for the desired branch? I may be missing some context, but they appear to be getting built the same way, with the only difference possibly being the supported platforms.

If the workflow in the stacks-blockchain repo does not currently fit your need, it may make more sense to modify it in the origin repo rather than maintain an orphaned build process in a separate repository. Additionally, if this is for integration testing we should try to use the same images built and published from the stacks-blockchain repo if possible for them to be considered valid.

What are your thoughts?

[CharlieC3]

Seems like this step can be used to checkout the stacks-blockchain repo instead of the api repo

[zone117x]

It checks out the API repo to grab the dockerfile (in this repo) used in a job later down. That could be changed to pull the dockerfile via a git uri but the syntax isn't very nice.

[zone117x]

the only difference possibly being the supported platforms.

Yep that's the primary difference (arm64 in particular).

If the workflow in the stacks-blockchain repo does not currently fit your need, it may make more sense to modify it in the origin repo rather than maintain an orphaned build process in a separate repository. Additionally, if this is for integration testing we should try to use the same images built and published from the stacks-blockchain repo if possible for them to be considered valid.

Yeah, I think that makes sense. One barrier here would be that even with the cross-arch compilation technique used in this PR, it still adds around 15 minutes to the build time for the image (and we'd still want it building imagines on each branch, at least for now). Perhaps the blockchain repo maintainers would be alright with that though. I'll try to get a PR out on that repo, then we can re-consider this PR if that doesn't pan out.

[CharlieC3]

Perhaps the blockchain repo maintainers would be alright with that though. I'll try to get a PR out on that repo, then we can re-consider this PR if that doesn't pan out.

If I'm not mistaken, I think jwiley has already completed this work here: stacks-network/stacks-core#3199
I believe he's planning on merging it after 2.1 considering its size. If you're able to hold off until then, I believe it should work for you. If not, then I think this PR would be a good workaround in the short-term with the intention to switch over once jwiley's PR is merged.

[zone117x]

We're still blocked on upstream adding ARM64 support. And we might also need this workflow for the upcoming Stacks v2.2 release. Leaving this PR as-is for now.

[zone117x]

This should be re-addressed closer to Stacks 3.0 release to see if cross-platform stacks-node images are available from core repo

[smcclellan]

@zone117x Are we still waiting for this one?