[RFC] Feedback on Ray Train and Ray Tune API revamp proposals

[justinvyu]

Hi Ray community!

We have some plans for some revamps to the Ray Train and Ray Tune libraries that we want feedback on from the community.

At a high level, the changes include:

• Ray Train is decoupled from Ray Tune, with a few removals of low-usage APIs and inherited Tune APIs that are not compatible with Ray Train usage.
• Ray Tune will stay the same, but some API imports will be changed in order to separate it from Ray Train (ex: changing ray.train.RunConfig -> ray.tune.RunConfig).
• We’re planning a 6 month long deprecation period to allow users to migrate gradually.

Many common Ray Train user pain points that we’ve seen in issues and forum posts are either solved or can be more easily addressed in the new version. For example:

• Confusing “slow experiment checkpoint syncing” logs indicating that some file uploading is taking a long time.
• Users want to set a custom, deterministic checkpoint directory name rather than the current behavior, which puts checkpoints in a randomized nested “Tune trial directory.”
• User experience papercuts including confusing configurations such as resume_from_checkpoint and trainer_resources [1, 2].
• Take a look at this section in the REP for more motivation for the changes.

We are hoping the usability improvements here outweigh a minimal amount of migration. This upgrade is designed to improve API clarity and stability, which will enhance the usability and experience for developers.

REP to review:

• Feel free to leave comments directly on the REP PR or in this issue.
• Key sections to review: API changes and the feature parity summary.