diff --git a/docs/eido/writing-a-schema.md b/docs/eido/writing-a-schema.md index 1f713629..628f3caf 100644 --- a/docs/eido/writing-a-schema.md +++ b/docs/eido/writing-a-schema.md @@ -75,7 +75,7 @@ This document defines the required an optional sample attributes for this pipeli In the above example, we listed `read1` and `read2` attributes as *required*. This will enforce that these attributes must be defined on the samples, but for this example, this is not enough -- these also must *point to files that exist*. Checking for files is outside the scope of JSON Schema, which only validates JSON documents, so eido extends JSON Schema with the ability to specify which attributes should point to files. -Eido provides two ways to do it: `files` and `required_files`. The basic `files` is simply used to specify which attributes point to files, which are not required to exist. This is useful for tools that want to calculate the total size of any provided inputs, for example. The `required_files` list specifies that the attributes point to files that *must exist*, otherwise the PEP doesn't validate. Here's an example of specifying an optional and required input attribute: +Eido provides two ways to do it: `sizing` and `tangible`. The basic `sizing` is simply used to specify which attributes point to files, which are not required to exist. This is useful for tools that want to calculate the total size of any provided inputs, for example. The `tangible` list specifies that the attributes point to files that *must exist*, otherwise the PEP doesn't validate. Here's an example of specifying an optional and required input attribute: ```yaml description: A PEP for ATAC-seq samples for the PEPATAC pipeline. @@ -109,7 +109,7 @@ properties: read2: type: string description: "Fastq file for read 2 (for paired-end experiments)" - required_files: + tangible: - read1 files: - read1 diff --git a/docs/looper/changelog.md b/docs/looper/changelog.md index 5668812a..254f8386 100644 --- a/docs/looper/changelog.md +++ b/docs/looper/changelog.md @@ -2,7 +2,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format. -## [2.0.0a2] -- 2024-09-30 +## [2.0.0] -- 2025-01-16 This release breaks backwards compatibility for Looper versions < 2.0.0 @@ -12,6 +12,10 @@ This release breaks backwards compatibility for Looper versions < 2.0.0 - looper cli parameters now take priority as originally intended [#518](https://github.com/pepkit/looper/issues/518) - fix divvy inspect - remove printed dictionary at looper finish [#511](https://github.com/pepkit/looper/issues/511) +- fix [#536](https://github.com/pepkit/looper/issues/536) +- fix [#522](https://github.com/pepkit/looper/issues/522) +- fix [#537](https://github.com/pepkit/looper/issues/537) +- fix [#534](https://github.com/pepkit/looper/issues/534) ### Changed - `--looper-config` is now `--config`, `-c`. [#455](https://github.com/pepkit/looper/issues/455) @@ -19,7 +23,6 @@ This release breaks backwards compatibility for Looper versions < 2.0.0 - Updated documentation for Looper 2.0.0, removing previous versions [pepspec PR #34](https://github.com/pepkit/pepspec/pull/34) - remove position based argument for divvy config, must use --config or run as default config - ### Added - `looper init` tutorial [#466](https://github.com/pepkit/looper/issues/466) - looper config allows for `pephub_path` in pipestat config section of `.looper.yaml` [#519](https://github.com/pepkit/looper/issues/519) diff --git a/docs/looper/developer-tutorial/writing-a-pipeline-interface.md b/docs/looper/developer-tutorial/writing-a-pipeline-interface.md index a8281257..84b26800 100644 --- a/docs/looper/developer-tutorial/writing-a-pipeline-interface.md +++ b/docs/looper/developer-tutorial/writing-a-pipeline-interface.md @@ -207,12 +207,15 @@ properties: sample_name: type: string description: "Name of the sample" + minLength: 1 # set a minimum length required for this required attribute file_path: type: string description: "Path to the input file to count" + minLength: 1 # set a minimum length required for this required attribute area_type: type: string description: "Name of the components of the country" + minLength: 1 # set a minimum length required for this required attribute required: - sample_name - file_path @@ -223,7 +226,8 @@ required: This file specifies what inputs the pipeline uses, and what type they are. It is a [JSON Schema](https://json-schema.org/), which allows us to use this file to validate the inputs, which we'll cover later. -For now, it defines that our input samples have 3 properties: `sample_name`, `file_path`, and `area_type`. +For now, it defines that our input samples have 3 properties: `sample_name`, `file_path`, and `area_type`. +We also specify a minimum length required for these inputs with `minLength: 1`. ### Adapt the pipeline interface to use the input schema diff --git a/docs/looper/how-to/rerun.md b/docs/looper/how-to/rerun.md new file mode 100644 index 00000000..c35cf974 --- /dev/null +++ b/docs/looper/how-to/rerun.md @@ -0,0 +1,47 @@ +# How to rerun samples + +## Introduction + +It is recommended that you have read and completed the basic [pipestat tutorial](../user-tutorial/user-pipestat.md) as well as [configuring pipestat backends](configure-pipestat-backends.md). + +This document will briefly cover details about how to configure pipestat in looper: + +!!! success "Learning objectives" + - How do I rerun samples? + +There may be a time when you need to re-submit samples. Looper's `rerun` command can be used to achieve this. By default, the `rerun` command will gather any samples with a `failed` status and submit again. + +However, Looper does not create flags for samples. It only reads them. Creating flags is the job of the pipeline via either the pipeline manager (e.g. [pypiper](../../pypiper/README.md)) or pipestat. + +Therefore, to create submission flags, you must: + +1. Configure looper to use pipestat +2. Add the `flag_file_dir` key and a flag directory path under the pipestat key within the looper config: +```yaml title=".looper.yaml" hl_lines="6" +pep_config: project_config_pipestat.yaml +output_dir: results +pipeline_interfaces: + - ./pipeline_interface1_sample_pipestat.yaml +pipestat: + flag_file_dir: output/results_pipeline + results_file_path: tmp_pipestat_results.yaml +``` +3. Ensure your pipeline is using pipestat to set the [status](../../pipestat/code/python-tutorial.md#pipeline-status-management) upon sample processing. + +```python +currentPipestatManager.set_status(record_identifier="sample1", status_identifier="running") +currentPipestatManager.set_status(record_identifier="sample2", status_identifier="failed") +``` +More info on setting status can be found [here](../developer-tutorial/developer-pipestat.md#setting-and-checking-status). + + +Now that your pipeline is setting statuses, you can resubmit failed jobs with: +```shell +looper rerun +``` + +By default, this command will _only_ submit failed jobs. However, you can use the `--ignore-flags` flag to resubmit all jobs. +```shell +looper rerun --ignore-flags +``` + \ No newline at end of file diff --git a/docs/pipestat/changelog.md b/docs/pipestat/changelog.md index bc9e7d83..a4de036c 100644 --- a/docs/pipestat/changelog.md +++ b/docs/pipestat/changelog.md @@ -2,6 +2,13 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format. +## [0.12.0] - 2025-01-16 +### Fixed +- portable report now has proper file extension in messaging. +- add exception to pipestat summarize if there are no results to report [#210](https://github.com/pepkit/pipestat/issues/210) +- fix spaces in html files [#211](https://github.com/pepkit/pipestat/issues/211) +- add output_dir parameter to psm.table + ## [0.11.0] - 2024-10-02 ### Fixed - for output schema, make samples an array type and nest under items [#204](https://github.com/pepkit/pipestat/issues/204) diff --git a/mkdocs.yml b/mkdocs.yml index 76a8f967..122e42d4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -153,6 +153,7 @@ nav: - Running project-level pipelines: looper/how-to/project-level-pipelines.md - Link similar results: looper/how-to/link-objects.md - Report objects: looper/how-to/report-objects.md + - Rerun samples: looper/how-to/rerun.md - Pipeline Developer Docs: - Developer tutorial: - Writing a pipeline interface: looper/developer-tutorial/writing-a-pipeline-interface.md