Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update looper and pipestat changelogs #40

Merged
merged 4 commits into from
Jan 16, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/eido/writing-a-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ This document defines the required an optional sample attributes for this pipeli

In the above example, we listed `read1` and `read2` attributes as *required*. This will enforce that these attributes must be defined on the samples, but for this example, this is not enough -- these also must *point to files that exist*. Checking for files is outside the scope of JSON Schema, which only validates JSON documents, so eido extends JSON Schema with the ability to specify which attributes should point to files.

Eido provides two ways to do it: `files` and `required_files`. The basic `files` is simply used to specify which attributes point to files, which are not required to exist. This is useful for tools that want to calculate the total size of any provided inputs, for example. The `required_files` list specifies that the attributes point to files that *must exist*, otherwise the PEP doesn't validate. Here's an example of specifying an optional and required input attribute:
Eido provides two ways to do it: `sizing` and `tangible`. The basic `sizing` is simply used to specify which attributes point to files, which are not required to exist. This is useful for tools that want to calculate the total size of any provided inputs, for example. The `tangible` list specifies that the attributes point to files that *must exist*, otherwise the PEP doesn't validate. Here's an example of specifying an optional and required input attribute:

```yaml
description: A PEP for ATAC-seq samples for the PEPATAC pipeline.
Expand Down Expand Up @@ -109,7 +109,7 @@ properties:
read2:
type: string
description: "Fastq file for read 2 (for paired-end experiments)"
required_files:
tangible:
- read1
files:
- read1
Expand Down
7 changes: 5 additions & 2 deletions docs/looper/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.

## [2.0.0a2] -- 2024-09-30
## [2.0.0] -- 2025-01-16

This release breaks backwards compatibility for Looper versions < 2.0.0

Expand All @@ -12,14 +12,17 @@ This release breaks backwards compatibility for Looper versions < 2.0.0
- looper cli parameters now take priority as originally intended [#518](https://github.com/pepkit/looper/issues/518)
- fix divvy inspect
- remove printed dictionary at looper finish [#511](https://github.com/pepkit/looper/issues/511)
- fix [#536](https://github.com/pepkit/looper/issues/536)
- fix [#522](https://github.com/pepkit/looper/issues/522)
- fix [#537](https://github.com/pepkit/looper/issues/537)
- fix [#534](https://github.com/pepkit/looper/issues/534)

### Changed
- `--looper-config` is now `--config`, `-c`. [#455](https://github.com/pepkit/looper/issues/455)
- A pipeline interface now consolidates a `sample_interface` and a `project_interface` [#493](https://github.com/pepkit/looper/issues/493)
- Updated documentation for Looper 2.0.0, removing previous versions [pepspec PR #34](https://github.com/pepkit/pepspec/pull/34)
- remove position based argument for divvy config, must use --config or run as default config


### Added
- `looper init` tutorial [#466](https://github.com/pepkit/looper/issues/466)
- looper config allows for `pephub_path` in pipestat config section of `.looper.yaml` [#519](https://github.com/pepkit/looper/issues/519)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -207,12 +207,15 @@ properties:
sample_name:
type: string
description: "Name of the sample"
minLength: 1 # set a minimum length required for this required attribute
file_path:
type: string
description: "Path to the input file to count"
minLength: 1 # set a minimum length required for this required attribute
area_type:
type: string
description: "Name of the components of the country"
minLength: 1 # set a minimum length required for this required attribute
required:
- sample_name
- file_path
Expand All @@ -223,7 +226,8 @@ required:

This file specifies what inputs the pipeline uses, and what type they are.
It is a [JSON Schema](https://json-schema.org/), which allows us to use this file to validate the inputs, which we'll cover later.
For now, it defines that our input samples have 3 properties: `sample_name`, `file_path`, and `area_type`.
For now, it defines that our input samples have 3 properties: `sample_name`, `file_path`, and `area_type`.
We also specify a minimum length required for these inputs with `minLength: 1`.

### Adapt the pipeline interface to use the input schema

Expand Down
47 changes: 47 additions & 0 deletions docs/looper/how-to/rerun.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# How to rerun samples

## Introduction

It is recommended that you have read and completed the basic [pipestat tutorial](../user-tutorial/user-pipestat.md) as well as [configuring pipestat backends](configure-pipestat-backends.md).

This document will briefly cover details about how to configure pipestat in looper:

!!! success "Learning objectives"
- How do I rerun samples?

There may be a time when you need to re-submit samples. Looper's `rerun` command can be used to achieve this. By default, the `rerun` command will gather any samples with a `failed` status and submit again.

However, Looper does not create flags for samples. It only reads them. Creating flags is the job of the pipeline via either the pipeline manager (e.g. [pypiper](../../pypiper/README.md)) or pipestat.

Therefore, to create submission flags, you must:

1. Configure looper to use pipestat
2. Add the `flag_file_dir` key and a flag directory path under the pipestat key within the looper config:
```yaml title=".looper.yaml" hl_lines="6"
pep_config: project_config_pipestat.yaml
output_dir: results
pipeline_interfaces:
- ./pipeline_interface1_sample_pipestat.yaml
pipestat:
flag_file_dir: output/results_pipeline
results_file_path: tmp_pipestat_results.yaml
```
3. Ensure your pipeline is using pipestat to set the [status](../../pipestat/code/python-tutorial.md#pipeline-status-management) upon sample processing.

```python
currentPipestatManager.set_status(record_identifier="sample1", status_identifier="running")
currentPipestatManager.set_status(record_identifier="sample2", status_identifier="failed")
```
More info on setting status can be found [here](../developer-tutorial/developer-pipestat.md#setting-and-checking-status).


Now that your pipeline is setting statuses, you can resubmit failed jobs with:
```shell
looper rerun
```

By default, this command will _only_ submit failed jobs. However, you can use the `--ignore-flags` flag to resubmit all jobs.
```shell
looper rerun --ignore-flags
```

7 changes: 7 additions & 0 deletions docs/pipestat/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.

## [0.12.0] - 2025-01-16
### Fixed
- portable report now has proper file extension in messaging.
- add exception to pipestat summarize if there are no results to report [#210](https://github.com/pepkit/pipestat/issues/210)
- fix spaces in html files [#211](https://github.com/pepkit/pipestat/issues/211)
- add output_dir parameter to psm.table

## [0.11.0] - 2024-10-02
### Fixed
- for output schema, make samples an array type and nest under items [#204](https://github.com/pepkit/pipestat/issues/204)
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ nav:
- Running project-level pipelines: looper/how-to/project-level-pipelines.md
- Link similar results: looper/how-to/link-objects.md
- Report objects: looper/how-to/report-objects.md
- Rerun samples: looper/how-to/rerun.md
- Pipeline Developer Docs:
- Developer tutorial:
- Writing a pipeline interface: looper/developer-tutorial/writing-a-pipeline-interface.md
Expand Down
Loading