Skip to content

Feature request: Allow use of arbitrary network filesystems with the AWS batch executor #6554

@dvg-p4

Description

@dvg-p4

New feature

Currently, executor 'awsbatch' requires the use of a bucketDir / -bucket-dir. It looks like there's built-in support for Fusion, and a licenseable extenstion for FSx-for-Lustre and AWS EFS. It would be nice to be able to use any network filesystem with AWS batch (perhaps with less-rich support in the open-source version of nextflow).

Use case

We have a network filesystem that's none of the above (FlexFS) that's mounted in our AWS batch templates, and would like to use it with the nextflow awsbatch executor.

Suggested implementation

Add a setting for the awsbatch executor to skip the check that a -bucket-dir / bucketDir is defined, and in that case simply copy/link/etc. files into the standard workDir to stage them (assuming the workDir is on some kind of network filesystem).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions