[dagster-gcp] live tests #26761
Open
[dagster-gcp] live tests #26761
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
dpeng817
force-pushed
the
dpeng817/gcs-live-tests
branch
from
54adede to
d619329
Compare
dpeng817
commented
Dec 30, 2024
There was a problem hiding this comment.
this is a direct copy paste from azure tests
dpeng817
commented
Dec 30, 2024
| module: dagster_gcp.gcs.compute_log_manager | ||
| class: GCSComputeLogManager | ||
| config: | ||
| json_credentials_envvar: GCP_LIVE_TEST_CREDENTIALS |
There was a problem hiding this comment.
these are new creds I created for this test suite
dpeng817
commented
Dec 30, 2024
Comment on lines
+29
to
+48
| subprocess.run( | ||
| ["dagster", "asset", "materialize", "--select", "my_asset", "-m", "gcp_test_proj.defs"], | ||
| check=True, | ||
| ) | ||
| logs_captured_data = check.not_none( | ||
| DagsterInstance.get() | ||
| .get_event_records( | ||
| EventRecordsFilter( | ||
| event_type=DagsterEventType.LOGS_CAPTURED, | ||
| ) | ||
| )[0] | ||
| .event_log_entry.dagster_event | ||
| ).logs_captured_data | ||
|
|
||
| assert logs_captured_data.external_stderr_url | ||
| assert logs_captured_data.external_stdout_url | ||
| stderr = get_content_from_url(logs_captured_data.external_stderr_url) | ||
| stdout = get_content_from_url(logs_captured_data.external_stdout_url) | ||
| assert stdout.count("Printing without context") == 10 | ||
| assert stderr.count("Logging using context") == 10 |
There was a problem hiding this comment.
this is all mostly the same as azure
dpeng817
commented
Dec 30, 2024
|
|
||
| def _parse_gcs_url_into_uri(url: str) -> str: | ||
| return url.removeprefix( | ||
| "https://console.cloud.google.com/storage/browser/_details/computelogmanager-tests/" |
There was a problem hiding this comment.
the url format can't be directly downloaded; so we need to do some gross munging to pull out the downloadable log key.
We actually hard code a similar string in the compute log manager itself, we should turn that into a constant and use it in both places ideally.
dpeng817
commented
Dec 30, 2024
|
|
||
|
|
||
| def get_bucket_client(credentials: Mapping) -> gcs.Bucket: | ||
| return gcs.Client.from_service_account_info(credentials).get_bucket("computelogmanager-tests") |
There was a problem hiding this comment.
this is pulled straight from the compute log manager implementation
dpeng817
force-pushed
the
dpeng817/complogmngr-delete-blobs
branch
from
a4f8c2e to
02f06c7
Compare
dpeng817
force-pushed
the
dpeng817/gcs-live-tests
branch
from
d619329 to
8021849
Compare
dpeng817
force-pushed
the
dpeng817/gcs-live-tests
branch
from
8021849 to
9449650
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Similar to what we introduced for azure, introduce live testing for gcp. Starting with the compute log manager.
Just like the azure tests, this runs on branches which touch dagster-gcp, as well as on the oss nightly pipeline.
Most of the files here are a relative copy-paste job from the azure tests. There is some new functionality, and I've added comments pointing to where that lives.
How I Tested These Changes
New tests pass.