Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explain Nextflow config order of priority #279

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions platform_versioned_docs/version-24.1/compute-envs/aws-batch.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,10 @@ Batch Forge automatically creates resources that you may be charged for in your
1. Apply [**Resource labels**](../resource-labels/overview.mdx) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.
1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Configuration settings in this field override the same values in the pipeline Nextflow config file.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::
1. Specify custom **Environment variables** for the **Head job** and/or **Compute jobs**.
1. Configure any advanced options described in the next section, as needed.
1. Select **Create** to finalize the compute environment setup. It will take a few seconds for all the AWS resources to be created before you are ready to launch pipelines.
Expand Down Expand Up @@ -324,7 +327,10 @@ Your Seqera compute environment uses resources that you may be charged for in yo
1. Apply [**Resource labels**](../resource-labels/overview.mdx) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.
1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Configuration settings in this field override the same values in the pipeline Nextflow config file.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::
1. Specify custom **Environment variables** for the **Head job** and/or **Compute jobs**.
1. Configure any advanced options described in the next section, as needed.
1. Select **Create** to finalize the compute environment setup.
Expand Down
12 changes: 10 additions & 2 deletions platform_versioned_docs/version-24.1/compute-envs/azure-batch.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,12 @@ Create a Batch Forge Azure Batch compute environment:
1. Enable **Dispose resources** for Seqera to automatically delete the Batch pool if the compute environment is deleted on the platform.
1. Select or create [**Container registry credentials**](../credentials/azure_registry_credentials.mdx) to authenticate a registry (used by the [Wave containers](https://www.nextflow.io/docs/latest/wave.html) service). It is recommended to use an [Azure Container registry](https://azure.microsoft.com/en-gb/products/container-registry) within the same region for maximum performance.
1. Apply [**Resource labels**](../resource-labels/overview.mdx). This will populate the **Metadata** fields of the Azure Batch pool.
1. Expand **Staging options** to include optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre-and-post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::
1. Specify custom **Environment variables** for the **Head job** and/or **Compute jobs**.
1. Configure any advanced options you need:
- Use **Jobs cleanup policy** to control how Nextflow process jobs are deleted on completion. Active jobs consume the quota of the Azure Batch account. By default, jobs are terminated by Nextflow and removed from the quota when all tasks succesfully complete. If set to _Always_, all jobs are deleted by Nextflow after pipeline completion. If set to _Never_, jobs are never deleted. If set to _On success_, successful tasks are removed but failed tasks will be left for debugging purposes.
Expand Down Expand Up @@ -301,7 +306,10 @@ Create a manual Seqera Azure Batch compute environment:
1. Apply [**Resource labels**](../resource-labels/overview.mdx). This will populate the **Metadata** fields of the Azure Batch pool.
1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Configuration settings in this field override the same values in the pipeline Nextflow config file.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::
:::info
To use managed identities, Platform requires Nextflow version 24.06.0-edge or later. Add `export NXF_VER=24.06.0-edge` to the **Global Nextflow config** field for your compute environment to use this Nextflow version by default.
:::
Expand Down
5 changes: 4 additions & 1 deletion platform_versioned_docs/version-24.1/compute-envs/eks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,10 @@ After you have prepared your Kubernetes cluster and assigned a service account r
1. Apply [**Resource labels**](../resource-labels/overview.mdx) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.
1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Configuration settings in this field override the same values in the pipeline Nextflow config file.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::
1. Specify custom **Environment variables** for the **Head job** and/or **Compute jobs**.
1. Configure any advanced options described in the next section, as needed.
1. Select **Create** to finalize the compute environment setup.
Expand Down
5 changes: 4 additions & 1 deletion platform_versioned_docs/version-24.1/compute-envs/gke.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,10 @@ After you've prepared your Kubernetes cluster and granted cluster access to your
1. Apply [**Resource labels**](../resource-labels/overview.mdx) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.
1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Configuration settings in this field override the same values in the pipeline Nextflow config file.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::
1. Specify custom **Environment variables** for the **Head job** and/or **Compute jobs**.
1. Configure any advanced options described in the next section, as needed.
1. Select **Create** to finalize the compute environment setup.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -196,12 +196,15 @@ Apply [**Resource labels**][resource-labels] to the cloud resources consumed by

#### Scripting and environment variables

Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Configuration settings in this field override the same values in the pipeline Nextflow config file.

1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.

:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::

Specify custom **Environment variables** for the head and compute jobs.
1. Specify custom **Environment variables** for the head and compute jobs.

#### Advanced options

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,10 @@ You can create multiple credentials in your Seqera workspace. See [Credentials](
11. Apply [**Resource labels**](../resource-labels/overview.mdx) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.
12. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Configuration settings in this field override the same values in the pipeline Nextflow config file.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::
13. Use the **Environment variables** option to specify custom environment variables for the Head job and/or Compute jobs.
14. Configure any advanced options you need:
- Enable **Use Private Address** to ensure that your Google Cloud VMs aren't accessible to the public internet.
Expand Down
5 changes: 4 additions & 1 deletion platform_versioned_docs/version-24.1/compute-envs/hpc.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,10 @@ To create a new **HPC** compute environment:
1. Enter the **Compute queue name**. This is the [default](https://www.nextflow.io/docs/latest/process.html#queue) cluster queue to which the Nextflow job will submit tasks.
1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Configuration settings in this field override the same values in the pipeline Nextflow config file.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::
1. Specify custom **Environment variables** for the head job and/or compute jobs.
1. Configure any advanced options needed:
- Use the **Nextflow queue size** to limit the number of jobs that Nextflow can submit to the scheduler at the same time.
Expand Down
5 changes: 4 additions & 1 deletion platform_versioned_docs/version-24.1/compute-envs/k8s.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,10 @@ After you've prepared your Kubernetes cluster for Seqera integration, create a c
1. Apply [**Resource labels**](../resource-labels/overview.mdx) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.
1. Expand **Staging options** to include:
- Optional [pre- or post-run Bash scripts](../launch/advanced.mdx#pre--post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Configuration settings in this field override the same values in the pipeline Nextflow config file.
- Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch.
:::info
Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced.mdx#nextflow-config-file) for more information on configuration priority.
:::
1. You can use the **Environment variables** option to specify custom environment variables for the Head job and/or Compute jobs.
1. Configure any advanced options described below, as needed.
1. Select **Create** to finalize the compute environment setup.
Expand Down
73 changes: 67 additions & 6 deletions platform_versioned_docs/version-24.1/launch/advanced.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,6 @@ You can modify the configuration and execution of a pipeline with advanced launc

Add settings to the Nextflow configuration file. This must follow the same syntax as the [Nextflow configuration file](https://www.nextflow.io/docs/latest/config.html#config-syntax).


:::note
Nextflow parameters defined in this field override the same parameters in your `nextflow.config` file.
:::

For example, modify the **manifest** section to give the pipeline a name and description that will show up in the Seqera monitoring section:

```ini
Expand All @@ -27,6 +22,64 @@ For example, modify the **manifest** section to give the pipeline a name and des
}
```

#### Nextflow configuration order of priority

When launcing pipelines in Platform, Nextflow configuration values can be supplied from the `nextflow.config` file in the pipeline repository and the **Nextflow config file** field in the pipeline launch form. If different values of the same configuration parameter are defined, Nextflow parameters defined in the launch form **Nextflow config file** field override the same parameters in your `nextflow.config` file.
llewellyn-sl marked this conversation as resolved.
Show resolved Hide resolved

Configuration values set in the **Global Nextflow config** field during compute environment creation are pre-filled in the **Nextflow config file** field during pipeline launch. These pre-filled values from the compute environment can be overridden manually during launch.

| Nextflow configuration order of priority (highest to lowest) |
| ------------------------------------------------------------ |
| Pipeline launch form **Nextflow config file** field |
| Compute environment **Global Nextflow config** field |
| Pipeline repository `nextflow.config` file |

For example, if:

1. The `nextflow.config` file in your pipeline repository contains this manifest:

```ini title="Pipeline repository config file"
manifest {
llewellyn-sl marked this conversation as resolved.
Show resolved Hide resolved
name = 'A'
description = 'Pipeline description A'
mainScript = 'main.nf'
defaultBranch = 'master'
}
```

2. Your compute environment **Global Nextflow config** field contains this manifest:

```ini title="Compute environment Global Nextflow config field"
manifest {
name = 'B'
description = 'Pipeline description B'
mainScript = 'main.nf'
defaultBranch = 'master'
}
```

3. You specify this manifest in the **Nextflow config file** field on the pipeline launch form:

```ini title="Pipeline launch form Nextflow config file field"
manifest {
name = 'C'
description = 'Pipeline description C'
mainScript = 'main.nf'
defaultBranch = 'master'
}
```

The resolved configuration will contain the **Nextflow config file** field's manifest:

```ini title="Pipeline launch form Nextflow config file field"
manifest {
name = 'C'
description = 'Pipeline description C'
mainScript = 'main.nf'
defaultBranch = 'master'
}
```

### Seqera Cloud config file

Configure per-pipeline Seqera reporting behavior. Settings specified here override the same settings in the `tower.yml` [configuration file](../enterprise/configuration/overview.mdx) for this execution. Use the `reports` key to specify report paths, titles, and MIME types:
Expand All @@ -43,7 +96,15 @@ reports:
Run custom code either before or after the execution of the Nextflow script. These fields allow you to enter shell commands.

Pre-run scripts are executed in the nf-launch script prior to invoking Nextflow processes. Pre-run scripts are useful for:
- Specifying an alternate Nextflow version to use for the run. For example, use `export NXF_VER=24.04.2` to take advantage of [new Nextflow features](https://nextflow.io/blog/2024/nextflow-2404-highlights.html) before a Platform version is released with 24.04.2 as its baseline.
- Specifying an alternate Nextflow version to use for the run:

```bash
nextflow self-update
export NXF_VER=24.10.0
```
:::info
`nextflow self-update` is required before updating a pre-24.10.0 version of Nextflow to version 24.10.0 or later.
:::
- Executor setup, such as loading a private CA certificate.
- Troubleshooting. For example, add `sleep 3600` to your pre-run script to instruct Nextflow to wait 3600 seconds (60 minutes) before process execution after the nf-launcher container is started, to create a window in which to test connectivity and other issues before your Nextflow processes execute.

Expand Down
Loading