Rebasing a container image from GCP Buildpacks

[jama22]

As some of you know, rebase is one of the more exciting features behing using CNBs. It allows you to swap out base images super quickly without having to rebuild the entire container image. This can be beneficial when you need to apply a security patch to a fleet of images.

source: Cloud Native buildpacks documentation on rebase

Some folks have asked (e.g. #169) how you would set up rebasing with GCP, so I decided to give it a shot as a Friday project

The Plan

The Runtimes team publishes new builders under gcr.io/buildpacks/builder. There are three tags worth looking at:

• :latest points to v1, but will eventually move to google22 (see Ubuntu 22 builders now available for GCP Buildpacks #271)
• v1 an Ubuntu 18 based builder
• google-22 an Ubuntu 22 based builder

Limitations

• gcr.io/buildpacks/builder doesn't have a webhook or global push queue that we can tap into. Ideally I'd want to set up a trigger based off a webhook, but that might not be possible
• For simplicity, I'm going to rebase within a major base-image version (no jumping between Ubuntu 18 -> 22 unfortunately)
• To minimize the complexity, I'll do this by working directly with Cloud Build. If there's enough interest I can try to hook this up to a CD system like Cloud Deploy or something.

Approach

1. Set up a test project on GitHub
2. Create a manual Cloud Build Trigger
3. Create a cloudbuild.yaml to configure our rebase
4. Test it out
5. Set our manual trigger (4) to run on a schedule

Steps

Set up Test Project on GitHub

For the sake of this demonstration, the complexity of this project doesn't actually matter. Just make sure you're using one of the supported languages. I grabbed a copy of the Python sample app from https://github.com/GoogleCloudPlatform/buildpack-samples/tree/master/sample-functions-framework-python

• Commit your code to a GitHub repo
• Build your app using Cloud Build
• (Optional) Deploy your image to a target like Cloud Run
• Make a note of where the image is stored in Artifact Registry. You'll need this information when creating your cloudbuild.yaml

Create a Cloud Build Trigger

Fire up the GCP Cloud Console web interface and load up the Cloud Build interface. From the sidebar, we're looking to bring up the Triggers menu

From there we're going to Create a Trigger in your preferred region

Fill in the important details:

• Name - name of the trigger
• Event - pick Manual invocation
• Source - Add your code Repo from the dropdown. You may need to use the `Connect New Repository" flow to authenticate and add your Github repository
• Branch - I chose main, but you can use whatever branch you'd like
• Configuration
  • Type - Cloud Build configuration file (yaml or jason). We're going to be writing out the steps in a later step
  • Location - Repository, under /cloudbuild.yaml

I left the rest of the options to their defaults. Go ahead and CREATE the trigger

Create a cloudbuild.yaml

Next we'll need to create a cloudbuild.yaml to live alongside your source code.

options:
  logging: LEGACY
  pool: {}
projectId: <YOUR PROJECT ID>
steps:  
- name: gcr.io/k8s-skaffold/pack
  entrypoint: pack
  args:
  - rebase
  - <URI TO YOUR IMAGE IN ARTIFACT REGISTRY>
  - --run-image
  - gcr.io/buildpacks/google-22/run:latest
  - --publish
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk' 
  entrypoint: gcloud
  args:
   - 'run'
   - 'deploy'
   - 'rebase-app'
   - '--image'
   - <URI TO YOUR IMAGE IN ARTIFACT REGISTRY>
   - '--region'
   - us-east1

In the first step, we're using gcr.io/k8s-skaffold/pack to run our rebase command against gcr.io/buildpacks/google-22/run:latest. With the --publish flag we'll be pushing those changes back to our Artifact Registry bucket

(Optional) In the second step, we're using 'gcr.io/google.com/cloudsdktool/cloud-sdk' to redeploy our app to Cloud Run.

• make your cloudbuild.yaml
• commit cloudbuild.yaml to your repo

Test it out

Going back to your Cloud Build Triggers page, you should now be able to manually invoke a rebase by click Run

Inspecting the build logs, you should see it rebase the image and redeploy to Cloud Run!

Setting up a timer

Following the instructions on Cloud Build's documentation, we can turn the manual trigger into a scheduled build that runs on your preferred cadence

[iamacarpet]

Thanks @jama22 , this look like a great way to apply security updates!

Interestingly, I can see that the rebase is done against the run image, which makes sense.

In the main builder, the PHP runtime appears to be applying a compiled version of PHP to a generic base image from a tar.gz file, which would mean rebasing against the generic run image would update the underlying operating system, but not the PHP language runtime.

It is not done this way for App Engine Standard - there is a run image specifically for PHP that already contains the PHP language runtime, presumably as some additional Docker layers on top of the generic run image and I'm guessing it's for this very reason - so you can update the language runtime with a rebase.

Is there a reason the main builder is designed in a way that makes this impossible?

[jama22]

By main builder i'm assuming you're referencing to gcr.io/buildpacks/builder? I think there's some room for improvement on that specific builder. It was originally shipped to support basic use cases for Cloud Run source deployments. Instead of shipping it with GAE / GCF system package, its much more stripped down and "basic".

We're getting more requests from folks to use buildpacks in Cloud Run while being able to take advantage of rebasing. I personally would love it if we had more options on the builders, something with the runtime-specialization of the GAE images, but without all of the additional system packages that folks might not need to use

[iamacarpet]

Thanks James, yes, gcr.io/buildpacks/builder is what I mean and you've got it exactly.

I personally would love it if we had more options on the builders, something with the runtime-specialization of the GAE images, but without all of the additional system packages that folks might not need to use

This is really what we want to see - I mean, not even stripped out system packages necessarily, but just buildpack runtimes that are as supported, tested & feature packed as the App Engine Standard runtimes.

Don't get me wrong, we love App Engine and would prefer to stick with it, but I've been told multiple times by the App Engine PMs that they're wanting everyone to move to Cloud Run instead, however, we are yet to see it as a true viable option as a like-for-like replacement, with source deploys as a first class citizen with build packs.

If the revenue stream is a problem, could you even make "Build Packs Enterprise", where we pay for supported runtimes with build packs on Cloud Run, potentially where you handle the rebase / security updates for us, and make it equal or slightly less than App Engine currently costs, so we're keeping the level of support & features, but moving for all the new features on Cloud Run that aren't being implemented on App Engine... ?