Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create AWS Batch job queue for running nextflow #150

Open
kurtwheeler opened this issue Nov 22, 2021 · 0 comments
Open

Create AWS Batch job queue for running nextflow #150

kurtwheeler opened this issue Nov 22, 2021 · 0 comments

Comments

@kurtwheeler
Copy link
Contributor

Right now @jashapiro runs nextflow locally as his user and it submits jobs which run within Batch. However, if Josh's temporary AWS credentials expire, nextflow is unable to continue submitting jobs.

To prevent this from happening, we should create a separate job queue and compute environment which runs on On Demand instances to run nextflow. This can then use a specific bot user which won't be affected by MFA.

The new flow would then be submit a job to the nextflow-runner queue, which then sends all the nextflow sub-jobs to the currently existing job queue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant