Description
Ray applications often use different working_dir packages while sharing the same large model or data files.
Today, this may require bundling the same model into multiple packages:
job_a_with_model.zip = application A + model files
job_b_with_model.zip = application B + model files
Because the package URIs are different, working_dir caching cannot reuse the model contents across them. This duplicates storage and transfer, and changing application code requires rebuilding the package containing the model.
Applications can download the model themselves, but then each application must implement local path management, synchronization, retries, and cleanup.
Ray should provide an independent archives RuntimeEnv field for shared, non-Python resources. Different RuntimeEnvs should be able to reference the same archive URI and reuse its node-local cache independently of working_dir.
The field should:
- Accept one remote archive URI or a mapping of names to URIs.
- Download and unpack archives on nodes where the RuntimeEnv is used.
- Expose local paths through ray.runtime_env.get_archive_paths().
- Avoid changing the worker's current directory or PYTHONPATH.
- Treat extracted contents as read-only and manage them through the RuntimeEnv lifecycle.
- Initially support .zip, .tar.gz, and .tgz archives.
Concurrent download deduplication, safe retries, validation, and cleanup are secondary benefits of integrating this with the existing RuntimeEnv plugin and cache infrastructure.
Use case
Two applications can use different working directories while sharing the same large model archive:
runtime_env_a = {
"working_dir": "s3://bucket/jobs/job_a.zip",
"archives": {
"model": "s3://bucket/models/model.tar.gz",
},
}
runtime_env_b = {
"working_dir": "s3://bucket/jobs/job_b.zip",
"archives": {
"model": "s3://bucket/models/model.tar.gz",
},
}
A worker can access the shared model without knowing the cache location:
from pathlib import Path
from ray.runtime_env import get_archive_paths
@ray.remote
def load_model():
model_dir = Path(get_archive_paths()["model"])
return model_dir / "weights.bin"
Description
Ray applications often use different
working_dirpackages while sharing the same large model or data files.Today, this may require bundling the same model into multiple packages:
Because the package URIs are different, working_dir caching cannot reuse the model contents across them. This duplicates storage and transfer, and changing application code requires rebuilding the package containing the model.
Applications can download the model themselves, but then each application must implement local path management, synchronization, retries, and cleanup.
Ray should provide an independent archives RuntimeEnv field for shared, non-Python resources. Different RuntimeEnvs should be able to reference the same archive URI and reuse its node-local cache independently of working_dir.
The field should:
Concurrent download deduplication, safe retries, validation, and cleanup are secondary benefits of integrating this with the existing RuntimeEnv plugin and cache infrastructure.
Use case
Two applications can use different working directories while sharing the same large model archive:
A worker can access the shared model without knowing the cache location: