feat: job arrays #174
feat: job arrays #174
Conversation
|
Warning Rate limit exceeded@johanneskoester has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 10 minutes and 28 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (1)
WalkthroughThe changes introduce a new method, Changes
Sequence Diagram(s)sequenceDiagram
participant Executor
participant JobExecutorInterface
Executor->>JobExecutorInterface: run_jobs(jobs)
alt Single Job
Executor->>JobExecutorInterface: run_job(job)
else Multiple Jobs
Executor->>Executor: // TODO: Implement job array submission
end
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (1)
snakemake_executor_plugin_slurm/__init__.py (1)
111-124: Enhance error handling and loggingThe new method should follow the same error handling and logging patterns as the rest of the codebase. Consider adding:
- Error handling for empty job lists
- Logging for job grouping operations
- Consistent error propagation
Here's a suggested implementation:
def run_jobs(self, jobs: List[JobExecutorInterface]): + if not jobs: + return + + self.logger.debug(f"Grouping {len(jobs)} jobs by rule name") try: for rule_name, group in groupby(jobs, key=lambda job: job.rule.name): same_rule_jobs = list(group) + self.logger.debug( + f"Processing {len(same_rule_jobs)} jobs for rule '{rule_name}'" + ) if len(same_rule_jobs) == 1: self.run_job(same_rule_jobs[0]) else: # Temporary implementation: submit jobs individually for job in same_rule_jobs: self.run_job(job) + except Exception as e: + raise WorkflowError(f"Failed to process job group: {str(e)}")🧰 Tools
🪛 Ruff (0.8.0)
114-114: Using the generator returned from
itertools.groupby()more than once will do nothing on the second usage(B031)
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
📒 Files selected for processing (1)
snakemake_executor_plugin_slurm/__init__.py(2 hunks)
🧰 Additional context used
🪛 Ruff (0.8.0)
snakemake_executor_plugin_slurm/__init__.py
114-114: Using the generator returned from itertools.groupby() more than once will do nothing on the second usage
(B031)
🔇 Additional comments (1)
snakemake_executor_plugin_slurm/__init__.py (1)
8-8: LGTM!
The groupby import is correctly placed with other standard library imports and is required for the new job array functionality.
| else: | ||
| # TODO submit as array | ||
| # share code with run_job | ||
|
|
||
| # TODO in the future: give a hint to the scheduler to select preferably | ||
| # many jobs from the same rule if possible, in order to have | ||
| # more efficient array jobs. This should be somehow tunable, because | ||
| # it might contradict other efficiency goals. | ||
| ... | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Implement placeholder for array jobs
The current implementation uses an ellipsis (...) as a placeholder, which will raise a NotImplementedError. Until the array job submission is implemented, we should handle these jobs individually.
Apply this improvement to handle multi-job groups temporarily:
else:
# TODO submit as array
# share code with run_job
- ...
+ # Temporary implementation: submit jobs individually until array support is added
+ for job in same_rule_jobs:
+ self.run_job(job)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| else: | |
| # TODO submit as array | |
| # share code with run_job | |
| # TODO in the future: give a hint to the scheduler to select preferably | |
| # many jobs from the same rule if possible, in order to have | |
| # more efficient array jobs. This should be somehow tunable, because | |
| # it might contradict other efficiency goals. | |
| ... | |
| else: | |
| # TODO submit as array | |
| # share code with run_job | |
| # TODO in the future: give a hint to the scheduler to select preferably | |
| # many jobs from the same rule if possible, in order to have | |
| # more efficient array jobs. This should be somehow tunable, because | |
| # it might contradict other efficiency goals. | |
| # Temporary implementation: submit jobs individually until array support is added | |
| for job in same_rule_jobs: | |
| self.run_job(job) |
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
|
Plan: support aggregation of jobs in the Snakemake scheduler before they are send to the executor. This can be done by supporting a new standard resource This way we avoid the case that jobs drop into the executor in a too fine-grained way. |
Summary by CodeRabbit