Skip to content

Add optional s3fs / gcfs as optional dependencies #1256

@praateekmahajan

Description

@praateekmahajan

Is your feature request related to a problem? Please describe.
If customers want to use NeMo Curator on Cloud we should add optional groups as s3 or gc that installs the correct version inline with our existing fsspec installation. If users install themselves using pip, it might be possible that Curator needs fsspec=2024.12 while pip as an additional install of s3fs installs 2025.12

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions