Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize worker behavior for object store writes #650

Merged

Commits on Jun 6, 2024

  1. Alter worker entrypoints to optimize output write.

    Adjusting to account for slowness of writes for output when output
    dataset directory is actually a mounted object store bucket (the default
    at the moment), such that we work in container-specific dirs and archive
    things before writing to such a place; also adding TODO comments to come
    back and add more general behavior later, when output datasets may be
    somewhere else.
    robertbartel committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    7565ee7 View commit details
    Browse the repository at this point in the history
  2. Adjusting data formats for CSV/archive output.

    Renaming NGEN_OUTPUT to NGEN_CSV_OUTPUT, introducing
    ARCHIVED_NGEN_CSV_OUTPUT format for when CSVs are bundled into tar file
    before being placed in dataset, and proactively add NGEN_NETCDF_OUTPUT
    format for eventual use; also, switch to ARCHIVED_NGEN_CSV_OUTPUT as
    current job default.
    robertbartel committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    9a05f15 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e38df7a View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Configuration menu
    Copy the full SHA
    b667dd6 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Configuration menu
    Copy the full SHA
    f93fef9 View commit details
    Browse the repository at this point in the history
  2. Bump package versions and internal dependencies.

    Bumping versions of dmod-core and dmod-communication for direct changes,
    and then bumping other library and service packages to use the updated
    versions for their dependencies (along with bumping their own versions).
    robertbartel committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    874589f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    af243c4 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6b37f28 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    6d3c685 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Use RuntimeError for bad command in py_funcs.py.

    Switching to use this instead of ValueError if command arg doesn't match
    something supported by main(), and making message reflect the situation
    a little better.
    robertbartel committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    e643a96 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4d30a05 View commit details
    Browse the repository at this point in the history