Skip to content

Commit 73356a6

Browse files
authored
Add regional location support to GCP Image Builder (#4268)
* Add regional location support to GCP Image Builder Users can now specify a `location` parameter to run Cloud Build in a specific GCP region instead of the global endpoint. This enables: - Data residency compliance (GDPR, etc.) - Lower latency when building near GCS buckets/Artifact Registry - Required for Cloud Build private pools The Service Connector region does not control Cloud Build region, so this must be set explicitly on the Image Builder component. * Add validator to normalize empty location strings Adds a Pydantic field_validator that strips whitespace and converts empty strings to None at parse time. This prevents edge cases where location="" would produce an invalid endpoint like "-cloudbuild.googleapis.com".
1 parent 3a3324d commit 73356a6

File tree

3 files changed

+56
-4
lines changed

3 files changed

+56
-4
lines changed

docs/book/component-guide/image-builders/gcp.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,11 @@ To use the Google Cloud image builder, we need:
3838
* the Docker image used by Google Cloud Build to execute the steps to build and push the Docker image. By default, the builder image will be `'gcr.io/cloud-builders/docker'`.
3939
* The network to which the container used to build the ZenML pipeline Docker image will be attached. More information: [Cloud build network](https://cloud.google.com/build/docs/build-config-file-schema#network).
4040
* The build timeout for the build, and for the blocking operation waiting for the build to finish. More information: [Build Timeout](https://cloud.google.com/build/docs/build-config-file-schema#timeout_2).
41+
* The location to run Cloud Build (e.g., `us-central1`, `europe-west1`) when you need regional data residency, lower latency to nearby GCS buckets or Artifact Registry, or to use Cloud Build private pools.
42+
43+
{% hint style="info" %}
44+
Even if your GCP Service Connector is scoped to a specific region, the GCP Image Builder uses the **global** Cloud Build endpoint by default. To run builds in a specific region, set the `location` parameter on the Image Builder. The Service Connector only supplies authentication and does not influence which Cloud Build region is used.
45+
{% endhint %}
4146

4247
We can register the image builder and use it in our active stack:
4348

@@ -46,7 +51,8 @@ zenml image-builder register <IMAGE_BUILDER_NAME> \
4651
--flavor=gcp \
4752
--cloud_builder_image=<BUILDER_IMAGE_NAME> \
4853
--network=<DOCKER_NETWORK> \
49-
--build_timeout=<BUILD_TIMEOUT_IN_SECONDS>
54+
--build_timeout=<BUILD_TIMEOUT_IN_SECONDS> \
55+
--location=<GCP_REGION>
5056
5157
# Register and activate a stack with the new image builder
5258
zenml stack register <STACK_NAME> -i <IMAGE_BUILDER_NAME> ... --set
@@ -127,7 +133,8 @@ zenml image-builder register <IMAGE_BUILDER_NAME> \
127133
--flavor=gcp \
128134
--cloud_builder_image=<BUILDER_IMAGE_NAME> \
129135
--network=<DOCKER_NETWORK> \
130-
--build_timeout=<BUILD_TIMEOUT_IN_SECONDS>
136+
--build_timeout=<BUILD_TIMEOUT_IN_SECONDS> \
137+
--location=<GCP_REGION>
131138
132139
# Connect the GCP Image Builder to GCP via a GCP Service Connector
133140
zenml image-builder connect <IMAGE_BUILDER_NAME> -i
@@ -175,6 +182,7 @@ zenml image-builder register <IMAGE_BUILDER_NAME> \
175182
--service_account_path=<PATH_TO_SERVICE_ACCOUNT_KEY> \
176183
--cloud_builder_image=<BUILDER_IMAGE_NAME> \
177184
--network=<DOCKER_NETWORK> \
185+
--location=<GCP_REGION> \
178186
--build_timeout=<BUILD_TIMEOUT_IN_SECONDS>
179187
180188
# Register and set a stack with the new image builder

src/zenml/integrations/gcp/flavors/gcp_image_builder_flavor.py

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515

1616
from typing import TYPE_CHECKING, Optional, Type
1717

18-
from pydantic import PositiveInt
18+
from pydantic import Field, PositiveInt, field_validator
1919

2020
from zenml.image_builders import BaseImageBuilderConfig, BaseImageBuilderFlavor
2121
from zenml.integrations.gcp import (
@@ -53,11 +53,41 @@ class GCPImageBuilderConfig(
5353
about this parameter:
5454
https://cloud.google.com/build/docs/build-config-file-schema#timeout_2
5555
Defaults to `3600`.
56+
location: Optional GCP region for running Cloud Build (e.g.,
57+
'us-central1', 'europe-west1'). Controls data residency and latency
58+
and is required when using Cloud Build private pools. If not set,
59+
the global endpoint is used.
5660
"""
5761

5862
cloud_builder_image: str = DEFAULT_CLOUD_BUILDER_IMAGE
5963
network: str = DEFAULT_CLOUD_BUILDER_NETWORK
6064
build_timeout: PositiveInt = DEFAULT_CLOUD_BUILD_TIMEOUT
65+
location: Optional[str] = Field(
66+
default=None,
67+
description=(
68+
"GCP region for Cloud Build execution to control data residency and "
69+
"latency. Examples: 'us-central1', 'europe-west1'. Required when "
70+
"using Cloud Build private pools. If omitted, the global Cloud Build "
71+
"endpoint is used."
72+
),
73+
)
74+
75+
@field_validator("location", mode="before")
76+
@classmethod
77+
def validate_location(cls, v: Optional[str]) -> Optional[str]:
78+
"""Normalize location field, treating empty strings as unset.
79+
80+
Args:
81+
v: The location to validate.
82+
83+
Returns:
84+
The validated location.
85+
"""
86+
if v is not None:
87+
v = v.strip()
88+
if not v:
89+
return None
90+
return v
6191

6292

6393
class GCPImageBuilderFlavor(BaseImageBuilderFlavor):

src/zenml/integrations/gcp/image_builders/gcp_image_builder.py

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
from typing import TYPE_CHECKING, Optional, Tuple, cast
1717
from urllib.parse import urlparse
1818

19+
from google.api_core.client_options import ClientOptions
1920
from google.cloud.devtools import cloudbuild_v1
2021

2122
from zenml.enums import StackComponentType
@@ -216,7 +217,20 @@ def _run_cloud_build(self, build: cloudbuild_v1.Build) -> str:
216217
RuntimeError: If the Cloud Build run has failed.
217218
"""
218219
credentials, project_id = self._get_authentication()
219-
client = cloudbuild_v1.CloudBuildClient(credentials=credentials)
220+
client_options = None
221+
if self.config.location:
222+
endpoint = f"{self.config.location}-cloudbuild.googleapis.com"
223+
client_options = ClientOptions(api_endpoint=endpoint)
224+
logger.info(
225+
"Using regional Cloud Build endpoint `%s`.",
226+
endpoint,
227+
)
228+
else:
229+
logger.info("Using global Cloud Build endpoint.")
230+
231+
client = cloudbuild_v1.CloudBuildClient(
232+
credentials=credentials, client_options=client_options
233+
)
220234

221235
operation = client.create_build(project_id=project_id, build=build)
222236
log_url = operation.metadata.build.log_url

0 commit comments

Comments
 (0)