Package directive #6342
Draft
Package directive #6342
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for nextflow-docs-staging canceled.
|
pditommaso
reviewed
Aug 14, 2025
| The conda provider supports: | ||
| - Package specifications with version constraints | ||
| - Custom channels | ||
| - Environment files (`.yml`, `.yaml`) |
There was a problem hiding this comment.
I'm tempted to get rid of env files with the new directive. Tool requirements should be strictly the ones required by the task, not bloated env files
There was a problem hiding this comment.
Which part of env files do you not like?
- The extra file
- The separation of the packages from the process
People having bloated Nextflow processes is just a part of migrating old scripts to Nextflow. Because their scripts, prior the "business" logic, was mixed with the actual implementation logic, it can be a pain to separate out.
Task batching or grouping would help people organize the logic; sometimes, it's a performance concern, like piping BWA into Samtools.
I personally like the environment.yamls, and I find once you get over three packages(which happens in an R script) and add in the channels, you can't read the conda directive. 😅
So, compromise, what if we made the environment.yml magically picked up like Dockerfiles are with wave? I just want to get rid of the conda "${moduleDir/environment.yml}" line.
Closed
edmundmiller
force-pushed
the
package-directive
branch
3 times, most recently
from
31ca478 to
89f69bc
Compare
- Introduced PixiConfig and PixiCache classes for managing Pixi environments. - Updated Session class to include PixiConfig retrieval. - Enhanced CmdRun with options to enable/disable Pixi environments. - Modified ConfigBuilder to apply Pixi environment settings based on command-line options. - Updated BashWrapperBuilder to include Pixi activation in the execution script. - Added pixiEnv property to TaskBean and methods in TaskRun for Pixi environment handling. - Updated ProcessConfig to include 'pixi' in process configuration options. This commit integrates Pixi environment management into the Nextflow framework, allowing users to specify and control Pixi environments during execution. Signed-off-by: Edmund Miller <[email protected]>
- Introduced PixiCacheTest and PixiConfigTest classes to validate Pixi environment functionality. - Added tests for TOML and lock file detection, environment prefix path creation, and command execution handling. - Implemented checks for configuration options and cache directory retrieval in PixiConfig. This commit enhances the test coverage for Pixi environment management, ensuring robust functionality within the Nextflow framework. Signed-off-by: Edmund Miller <[email protected]>
- Introduced PixiCacheIntegrationTest class to validate PixiCache behavior with various environment specifications. - Implemented tests for creating environments from package specifications, TOML files, and lock files. - Added checks for handling custom cache directories and validating TOML and lock file detection. - Enhanced coverage for existing prefix directory handling and environment variable cache directory usage. This commit strengthens the testing framework for PixiCache, ensuring reliable environment management within the Nextflow ecosystem. Signed-off-by: Edmund Miller <[email protected]>
- Changed the environment specification from 'python=3.8' to 'cowpy'. - Updated the script to utilize cowpy for output instead of a Python command. This modification enhances the testing of Pixi environments by leveraging cowpy for greeting functionality. Signed-off-by: Edmund Miller <[email protected]>
Introduces the `nextflow.preview.package` feature flag to enable the unified package management system. This allows users to opt-in to the new `package` directive while maintaining backward compatibility with existing `conda` and `pixi` directives. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Edmund Miller <[email protected]>
Creates the foundation for unified package management with: - PackageSpec: Unified specification for packages across providers - PackageProvider: Interface for package manager implementations - PackageManager: Central coordinator for package providers - PackageProviderExtension: Plugin extension point for providers This abstraction layer enables consistent package management across different tools (conda, pixi, mamba, etc.) while supporting both simple and advanced configuration patterns. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Edmund Miller <[email protected]>
Extends process configuration to support the unified package directive: - ProcessConfig: Adds 'package' to the list of valid directives - TaskRun: Implements getPackageSpec() method with caching - TaskBean: Adds packageSpec field for task execution Also adds deprecation warnings when conda/pixi directives are used with the preview.package feature enabled, encouraging migration to the new unified syntax. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Edmund Miller <[email protected]>
Enhances the bash wrapper system to support the new package directive:
- BashWrapperBuilder: Adds getPackageActivateSnippet() method that
creates package environments using the PackageManager
- command-run.txt: Adds {{package_activate}} template variable
for package environment activation
The new system works alongside existing conda/pixi activation,
enabling gradual migration while maintaining backward compatibility.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
Signed-off-by: Edmund Miller <[email protected]>
Creates the nf-conda plugin that implements conda support for the unified package management system: - CondaPlugin: Main plugin class - CondaPackageProvider: Implements PackageProvider interface - CondaProviderExtension: Registers the provider with Nextflow - Includes existing CondaCache and CondaConfig implementations This plugin enables conda package management through the new unified 'package' directive while maintaining all existing conda functionality and configuration options. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Edmund Miller <[email protected]>
Creates the nf-pixi plugin that implements pixi support for the unified package management system: - PixiPlugin: Main plugin class - PixiPackageProvider: Implements PackageProvider interface - PixiProviderExtension: Registers the provider with Nextflow - Includes existing PixiCache and PixiConfig implementations This plugin enables pixi package management through the new unified 'package' directive, providing fast conda-compatible package resolution with lockfile support. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Edmund Miller <[email protected]>
edmundmiller
force-pushed
the
package-directive
branch
from
89f69bc to
ee856fc
Compare
Integrates the unified package management system with Wave for container building: - Add PackageSpec to Wave PackagesSpec conversion in WaveClient - Support provider mapping (conda/mamba/pixi -> CONDA type in Wave) - Enable package directive processing through Wave container builds - Use HxClient API for HTTP requests (aligned with lib-httpx refactor) - Add comprehensive package management documentation - Remove deprecated Pixi implementation in favor of unified system - Add integration tests for package management workflows This enables the `package` directive to work seamlessly with Wave for building containers with the specified package dependencies. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
edmundmiller
force-pushed
the
package-directive
branch
from
ee856fc to
ed16697
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PoC of a package directive.
Moved conda into a plugin, and the Pixi code from #6157. Hoping that by making them plugins, we can support things like #1526 in the community, etc.
I also wanted to keep
condafor now(it might throw a deprecation, but it shouldn't yet) and tried to gate it behind a feature flag.