-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize worker behavior for object store writes #650
Merged
robertbartel
merged 11 commits into
NOAA-OWP:master
from
robertbartel:f/optimize_job_io_output/main
Jul 2, 2024
Merged
Optimize worker behavior for object store writes #650
robertbartel
merged 11 commits into
NOAA-OWP:master
from
robertbartel:f/optimize_job_io_output/main
Jul 2, 2024
Commits on Jun 6, 2024
-
Alter worker entrypoints to optimize output write.
Adjusting to account for slowness of writes for output when output dataset directory is actually a mounted object store bucket (the default at the moment), such that we work in container-specific dirs and archive things before writing to such a place; also adding TODO comments to come back and add more general behavior later, when output datasets may be somewhere else.
Configuration menu - View commit details
-
Copy full SHA for 7565ee7 - Browse repository at this point
Copy the full SHA 7565ee7View commit details -
Adjusting data formats for CSV/archive output.
Renaming NGEN_OUTPUT to NGEN_CSV_OUTPUT, introducing ARCHIVED_NGEN_CSV_OUTPUT format for when CSVs are bundled into tar file before being placed in dataset, and proactively add NGEN_NETCDF_OUTPUT format for eventual use; also, switch to ARCHIVED_NGEN_CSV_OUTPUT as current job default.
Configuration menu - View commit details
-
Copy full SHA for 9a05f15 - Browse repository at this point
Copy the full SHA 9a05f15View commit details -
Configuration menu - View commit details
-
Copy full SHA for e38df7a - Browse repository at this point
Copy the full SHA e38df7aView commit details
Commits on Jun 12, 2024
-
Configuration menu - View commit details
-
Copy full SHA for b667dd6 - Browse repository at this point
Copy the full SHA b667dd6View commit details
Commits on Jun 13, 2024
-
Configuration menu - View commit details
-
Copy full SHA for f93fef9 - Browse repository at this point
Copy the full SHA f93fef9View commit details -
Bump package versions and internal dependencies.
Bumping versions of dmod-core and dmod-communication for direct changes, and then bumping other library and service packages to use the updated versions for their dependencies (along with bumping their own versions).
Configuration menu - View commit details
-
Copy full SHA for 874589f - Browse repository at this point
Copy the full SHA 874589fView commit details -
Configuration menu - View commit details
-
Copy full SHA for af243c4 - Browse repository at this point
Copy the full SHA af243c4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6b37f28 - Browse repository at this point
Copy the full SHA 6b37f28View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6d3c685 - Browse repository at this point
Copy the full SHA 6d3c685View commit details
Commits on Jun 14, 2024
-
Use RuntimeError for bad command in py_funcs.py.
Switching to use this instead of ValueError if command arg doesn't match something supported by main(), and making message reflect the situation a little better.
Configuration menu - View commit details
-
Copy full SHA for e643a96 - Browse repository at this point
Copy the full SHA e643a96View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4d30a05 - Browse repository at this point
Copy the full SHA 4d30a05View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.