Skip to content

Add Assay Development to workflow#30

Closed
ErinWeisbart wants to merge 9 commits intodevfrom
assaydevelopment
Closed

Add Assay Development to workflow#30
ErinWeisbart wants to merge 9 commits intodevfrom
assaydevelopment

Conversation

@ErinWeisbart
Copy link
Copy Markdown
Collaborator

Very much in draft form at this point.

Outstanding questions:
• Do we want to include logic to support auto-pipeline creation/manipulation based on input channels? (I have some json and python code that I can pull from elsewhere as a starting point if so).
• In assaydevelopment/main.nf do the outputs of the illum step need to be moved to the input folder so a single input folder flag can be passed to CellProfiler for file location?
• In the workflows/cellpainting.nf we were not sure how to properly create the load_data.csv

ErinWeisbart and others added 2 commits July 25, 2025 13:39
Co-authored-by: kate-bowers-broad <kbowers@broadinstitute.org>
Co-authored-by: kate-bowers-broad <kbowers@broadinstitute.org>
@nf-core-bot
Copy link
Copy Markdown
Member

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.2.1.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jul 25, 2025

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 11a2442

+| ✅ 204 tests passed       |+
#| ❔   9 tests were ignored |#
!| ❗  52 tests had warnings |!
Details

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in README.md: TODO nf-core:
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline 2. Present QC for raw reads (MultiQC)
  • pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
  • pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
  • pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in nextflow.config: Update the field with the details of the contributors to your pipeline. New with Nextflow version 24.10.0
  • pipeline_todos - TODO string in analysis.nf: If in doubt look at other nf-core/modules to see how we are doing things! :)
  • pipeline_todos - TODO string in analysis.nf: A module file SHOULD only define input and output files as command-line parameters.
  • pipeline_todos - TODO string in analysis.nf: Software that can be piped together SHOULD be added to separate module files
  • pipeline_todos - TODO string in analysis.nf: Optional inputs are not currently supported by Nextflow. However, using an empty
  • pipeline_todos - TODO string in analysis.nf: List required Conda package(s).
  • pipeline_todos - TODO string in analysis.nf: See section in main README for further information regarding finding and adding container addresses to the section below.
  • pipeline_todos - TODO string in analysis.nf: Where applicable all sample-specific information e.g. "id", "single_end", "read_group"
  • pipeline_todos - TODO string in analysis.nf: Where applicable please provide/convert compressed files as input/output
  • pipeline_todos - TODO string in analysis.nf: Named file extensions MUST be emitted for ALL output channels
  • pipeline_todos - TODO string in analysis.nf: List additional required output channels/values here
  • pipeline_todos - TODO string in analysis.nf: Where possible, a command MUST be provided to obtain the version number of the software e.g. 1.10
  • pipeline_todos - TODO string in analysis.nf: It MUST be possible to pass additional parameters to the tool as a command-line string via the "task.ext.args" directive
  • pipeline_todos - TODO string in analysis.nf: If the tool supports multi-threading then you MUST provide the appropriate parameter
  • pipeline_todos - TODO string in analysis.nf: Please replace the example samtools command below with your module's command
  • pipeline_todos - TODO string in analysis.nf: Please indent the command appropriately (4 spaces!!) to help with readability ;)
  • pipeline_todos - TODO string in analysis.nf: A stub section should mimic the execution of the original module as best as possible
  • pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
  • pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in test.config: Specify the paths to your test data on nf-core/test-datasets
  • pipeline_todos - TODO string in test.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in meta.yml: #Add a list of the modules and/or subworkflows used in the subworkflow
  • pipeline_todos - TODO string in meta.yml: #List all of the channels used as input with a description and their structure
  • pipeline_todos - TODO string in meta.yml: #List all of the channels used as output with a descriptions and their structure
  • pipeline_todos - TODO string in main.nf.test: Once you have added the required tests, please run the following command to build this file:
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in main.nf.test: Once you have added the required tests, please run the following command to build this file:
  • schema_description - No description provided in schema for parameter: cellprofiler_mode
  • schema_description - No description provided in schema for parameter: cellprofiler_illumination_cppipe
  • schema_description - No description provided in schema for parameter: cellprofiler_analysis_cppipe
  • schema_description - No description provided in schema for parameter: cellprofiler_assaydevelopment_cppipe
  • schema_description - No description provided in schema for parameter: stop_after

❔ Tests ignored:

  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: conf/igenomes_ignored.config
  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: conf/igenomes_ignored.config
  • nextflow_config - Config default ignored: params.cellprofiler_illumination_cppipe
  • nextflow_config - Config default ignored: params.cellprofiler_analysis_cppipe
  • nextflow_config - Config default ignored: params.cellprofiler_assaydevelopment_cppipe
  • template_strings - Ignoring Jinja template strings in file /home/runner/work/cellpainting/cellpainting/assets/cellprofiler/illumination.cppipe.jinja
  • template_strings - Ignoring Jinja template strings in file /home/runner/work/cellpainting/cellpainting/modules/local/cellprofiler/illuminationcorrection/main.nf

✅ Tests passed:

Run details

  • nf-core/tools version 3.3.2
  • Run at 2025-09-04 18:53:11

Co-authored-by: kate-bowers-broad <kbowers@broadinstitute.org>
@kenibrewer
Copy link
Copy Markdown
Member

@kate-bowers-broad @ErinWeisbart
I've polished off my PR in #27 . It now includes the subworkflows for creating load_data_csv with illum files as well. I also wired up everything in the main workflow to a dummy version of the assay_development step.

If you merge in the changes from that branch then you should be able get your step working pretty quickly.

Copy link
Copy Markdown
Member

@kenibrewer kenibrewer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I merged the changes in from the dev branch and hooked up the process you two wrote. You can see the non-merge fixes I made in commit 9ce279f .

I've got some additional comments with next steps to get this PR ready for review. Key points are:

  • Fix cppipe file input staging
  • Fix nf-test file
  • Run nf-test and create snapshots
  • (if-minor) consider preserving metadata in output file

Comment thread assets/cellprofiler/assaydevelopment.cppipe Outdated
Comment thread modules/local/cellprofiler/assaydevelopment/main.nf Outdated
Comment thread assets/cellprofiler/assaydevelopment.cppipe Outdated
output:
output:
tuple val(meta), path("assaydevelopment/*.png"), emit: png
tuple val(meta), path("assaydevelopment/Image.csv"), emit: csv, optional: true
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion
As a continuation of the above comment, can we get this to pick up the file at: ${meta.id}.csv ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the .cppipe to output the .csv file as "_ImageQCMeasurements.csv".
I'm not sure how to make that match ${meta.id}

touch ${prefix}.bam
mkdir -p assaydevelopment
echo 'this is not assay development' > assaydevelopment/mock_segmentedimage.png
echo 'this is not assay development' > assaydevelopment/mock_Image.csv
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion
If we change the output file pattern/name we should mirror it here.

Suggested change
echo 'this is not assay development' > assaydevelopment/mock_Image.csv
echo 'this is not assay development' > assaydevelopment/${meta.id}.csv

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue
This file needs to be updated with the new process format

@ErinWeisbart
Copy link
Copy Markdown
Collaborator Author

In

['batch', 'plate','well'],
we make the load_data.csv, grouping by Batch, Plate, and Well. We need to add in some additional logic to only include a single site (we usually use site 2).

kenibrewer added a commit that referenced this pull request Apr 1, 2026
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
kenibrewer added a commit that referenced this pull request Apr 1, 2026
- assaydevelopment.cppipe from #30
- analysis.cppipe from cpg0000-jump-pilot (CPJUMP1_analysis_without_batchfile)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@FloWuenne FloWuenne closed this Apr 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants