From f97e3d752ab6ad59a794b141ef677d22e531bc5f Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 9 Jun 2026 19:15:53 -0500 Subject: [PATCH 01/13] ADR: Meta-pipelines Signed-off-by: Ben Sherman --- adr/20251114-module-system.md | 5 +- adr/20260608-remote-pipeline-inclusion.md | 283 ++++++++++++++++++++++ 2 files changed, 285 insertions(+), 3 deletions(-) create mode 100644 adr/20260608-remote-pipeline-inclusion.md diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index de764142b2..a41bccc5f3 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -1,9 +1,9 @@ -# Module System for Nextflow +# Module system - Authors: Paolo Di Tommaso - Status: approved - Date: 2025-01-06 -- Tags: modules, dsl, registry, versioning, architecture +- Tags: modules, dsl, registry, versioning - Version: 2.7 ## Updates @@ -927,4 +927,3 @@ output: description: Software versions pattern: "versions.yml" ``` - diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md new file mode 100644 index 0000000000..46ca179307 --- /dev/null +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -0,0 +1,283 @@ +# Remote pipeline inclusion + +- Authors: Ben Sherman +- Status: draft +- Date: 2026-06-08 +- Tags: pipelines, modules, dsl, registry + +## Summary + +Add the ability to include a remote pipeline into a *meta-pipeline*. + +## Problem Statement + +Nextflow supports reusing process definitions via remote *module* inclusion (e.g. `include { BWA_MEM } from 'nf-core/bwa/mem'`), but there is no standard mechanism to reuse an entire *pipeline* as a building block. Users must either fork and copy code, or compose/chain multiple `nextflow run` sessions which forfeits dataflow composition. + +The natural unit of reuse for a pipeline is its *core workflow* -- the named workflow that takes and emits channels (e.g. `NFCORE_RNASEQ` in `nf-core/rnaseq`) -- as distinct from the deployment shell around it (the `params`, entry workflow, `output` block, and config). The nf-core community has already structured their pipelines around this split in anticipation of meta-pipelines. + +The decision to make: how should a remote pipeline be distributed, resolved, stored, and included so that it can be composed into a meta-pipeline while preserving dataflow composition. + +## Goals + +- **Preserve dataflow composition**: the included pipeline participates in the meta-pipeline's dataflow graph (same session, same DAG, same work dir), enabling incremental reaction to emitted outputs. + +- **Preserve reproducibility**: an included pipeline should produce the exact same results as it would when executed directly. Transitive dependencies should not be silently altered to reduce duplication. + +- **Reuse existing conventions**: follow the conventions established by the module system (e.g. include syntax) as much as possible rather than introducing parallel conventions. + +## Non-goals + +- **Nested pipeline execution**: avoid Nextflow-in-Nextflow execution, which forfeits dataflow composition. + +- **Pipeline execution via registry**: out of scope for first iteration. Registry-based execution (e.g. `nextflow pipeline run nf-core/rnaseq@3.0.0`) may be investigated in the future. + +## Decision + +Allow remote pipelines to be included into meta-pipelines, using the same namespacing conventions and include syntax as modules. Store the included pipeline in the meta-pipeline repository under `workflows///` with its own subdirectories for modules and subworkflows. The meta-pipeline owns all top-level concerns (entry workflow, params, outputs, config). Pipelines should be written with a self-contained core workflow to make importing as easy as possible. + +## Core Capabilities + +### Composition over orchestration + +The included pipeline must be incorporated into the meta-pipeline's dataflow graph in order to maximize dataflow concurrency. Existing approaches like pipeline chaining and Nextflow-in-Nextflow impose a synchronization barrier between each pipeline run. + +To this end, the included pipeline should be written in a way that separates the *core workflow* from the rest of the pipeline (entry workflow, params, publishing, config). Only the core workflow is included into the meta-pipeline; the rest is discarded. + +### Meta-pipeline owns all top-level concerns + +The meta-pipeline owns the entry workflow, `params` block, `output` block, and config. An included pipeline contributes none of these. This approach aligns with existing include semantics, which only supports composition of processes and named workflows. + +As a result, if a user wants to preserve any top-level concerns from the included pipeline, they must be explicitly replicated in the meta-pipeline. For example, params exposed by the included pipeline must be replicated in the meta-pipeline params and passed to the included pipeline's core workflow. + +### Best practices for included pipeline + +To be importable in practice, a pipeline's core workflow (and its dependent modules/workflows) should be free of external context: + +1. No `params` usage outside the entry workflow -- pass values as explicit process/workflow inputs. +2. No `publishDir` -- use the `output` block. +3. No use of project-level assets (`projectDir`, `bin`, `lib`) within the core workflow. Module-level assets can be used through the module `resources/` bundle and `moduleDir`. +4. Declare software dependencies (`container`, `conda`) in the process definition, not in config. +5. No default `ext` settings in config -- specify these defaults in the process definition or use explicit process inputs. Otherwise, any default `ext` settings must be replicated manually in the meta-pipeline. +6. No plugin functions within the core workflow. + +For process directives, it is helpful to distinguish *what* is executed vs *how* it is executed. Directives that affect the *what* (`container`, `ext` settings) should be owned by the process definition. Directives that affect the *how* (`cpus`, `memory`, `executor`, `queue`, `errorStrategy`) should be owned by the meta-pipeline. + +These constraints are not absolute -- it is possible to import a pipeline that does not adhere to any of these rules. Following these constraints simply makes it easier to import a pipeline with minimal extra work (manual replication, cross-cutting concerns). + +### Pipeline inclusion and storage + +Modules and pipelines share the same include syntax and naming conventions: + +```groovy +// module +include { BWA_MEM } from 'nf-core/bwa/mem' + +// pipeline +include { NFCORE_RNASEQ } from 'nf-core/rnaseq' +``` + +Including a remote pipeline is equivalent to including the top-level `main.nf` of that pipeline; any named workflow defined there can be included by name. By convention, the main script defines only the entry workflow and the core workflow (e.g. `NFCORE_RNASEQ` in nf-core/rnaseq). From this point, the included workflow can be called like any other workflow. + +When a pipeline is included, it is vendored into the meta-pipeline project under `workflows///`. Included pipelines are isolated -- each included pipeline has its own `modules/` and `workflows/` directories. This way, two pipelines can use different versions of the same module without compromising reproducibility. + +Included pipelines should be committed to the meta-pipeline repository. The pipeline should have a *pipeline spec* (`nextflow_spec.json`) which specifies the pipeline version, so that Nextflow can track local changes. + +## Open Questions + +### Sourcing from Nextflow registry vs Git repositories + +Pipelines could be stored in the Nextflow registry (as a new artifact type) or fetched directly from Git repositories. The pipeline registry is a potential long-term goal with other use cases, but it likely introduces additional scope that is not strictly related to meta-pipelines. Using existing Git repositories would be an expedient solution for the first iteration. + +### Pipeline CLI + +Sourcing remote pipelines from the Nextflow registry implies a `nextflow pipeline` command group for publishing and installing pipelines, similar to `nextflow module`. Even with a Git-based approach, a CLI is likely still needed to install and update remote pipelines. + +### Using plugin functions in included pipeline + +If an included pipeline uses plugin functions in the core workflow, these plugins must be explicitly declared in the meta-pipeline config, since the included pipeline config is not inherited. + +Alternatively, these core plugin dependencies could be specified in the pipeline spec under `requires.plugins`. When installing a pipeline, Nextflow could copy these plugin declarations into the meta-pipeline config and/or spec. + +Since this use case is rare -- plugin functions are typically used in the entry workflow outside the core workflow -- it can be deferred in the first iteration. + +## Alternatives + +### Pipeline chaining + +An alternative to a meta-pipeline is a *pipeline chain*, such as a shell script that simply calls multiple Nextflow pipelines in sequence. + +For example, a fetchngs -> rnaseq pipeline chain could be implemented as follows: + +```bash +# fetch FASTQ samples from NCBI SRA +nextflow -q run nf-core/fetchngs \ + --input samplesheet.csv \ + -output-format json \ + > results/output-fetchngs.json + +# adapt fetchngs output to rnaseq input (add strandedness column) +nextflow -q run ./fetchngs-rnaseq.nf \ + -params-file results/output-fetchngs.json \ + --strandedness auto \ + -output-format json \ + > results/output-fetchngs-rnaseq.json + +# perform RNAseq analysis +nextflow -q run nf-core/rnaseq \ + -params-file results/output-fetchngs-rnaseq.json \ + -output-format json \ + > results/output-rnaseq.json +``` + +While pipeline chaining has always been possible in theory, new language features such as [workflow outputs](20251020-workflow-outputs.md) and [record types](20260306-record-types.md) make it much more practical. Each pipeline can define a structured output which can be passed to the next pipeline via JSON. Mismatches between an upstream output and downstream input (e.g. missing columns, different column names) can be resolved by a small adapter pipeline. + +Pipeline chaining remains a valid solution for certain use cases, such as simple chains (A -> B -> C) of off-the-shelf pipelines with little to no additional coding. For compositions that are more complex and/or require maximum dataflow concurrency, meta-pipelines are the general solution. + +### Nextflow-in-Nextflow + +Another solution that sits between pipeline chaining and meta-pipelines is a Nextflow pipeline that runs other Nextflow pipelines as tasks. + +The [nf-cascade](https://github.com/mahesh-panchal/nf-cascade) project explores this approach to its limits. It defines a native process called `NEXTFLOW_RUN`, which takes a pipeline with inputs and runs it via `nextflow run`. The pipeline developer then writes a kind of meta-pipeline with aliases of `NEXTFLOW_RUN`. + +For example, rewriting the fetchngs -> rnaseq chain described above as a Nextflow pipeline: + +```groovy +include { NEXTFLOW_RUN as NFCORE_FETCHNGS } from "./modules/local/nextflow/run/main" +include { NEXTFLOW_RUN as NFCORE_RNASEQ } from "./modules/local/nextflow/run/main" + +workflow { + NFCORE_FETCHNGS ( + 'nf-core/fetchngs', + // nextflow opts, pipeline inputs, etc ... + ) + NFCORE_RNASEQ ( + 'nf-core/rnaseq', + // nextflow opts, pipeline inputs, etc ... + ) +} +``` + +This approach works with any Nextflow pipeline out of the box, because it simply executes the pipeline directly. However, this approach sacrifices dataflow composition, i.e. each pipeline must complete before the next pipeline can start. + +### Best of both: runtime inheritance + +The meta-pipeline approach treats the included pipeline as a *white box* -- it composes the pipeline like any included workflow, producing a single dataflow graph. It also imposes several constraints on how the included pipeline is written, and it imposes development overhead. For example, any params / outputs that need to be exposed from the included pipeline must be replicated in the meta-pipeline. + +The Nextflow-in-Nextflow approach treats the included pipeline as a *black box* -- it preserves the exact pipeline behavior (core workflow + entry workflow + config) while forfeiting dataflow composition (separate dataflow graphs). + +An ideal solution might combine the best of both: compose pipelines into a single dataflow graph (white box) while inheriting each pipeline's params, outputs, and config so they need not be replicated (black box). We considered such a model, where an included pipeline contributes its shell as namespaced, overridable defaults, but rejected it. Dataflow composition fundamentally requires exposing the core workflow as a set of channel ports, so the white-box mechanism is unavoidable; inheritance would only layer implicit behavior on top of it. That behavior comes at a steep cost: it relocates a one-time *write* cost (boilerplate) into a recurring *read* cost (hidden defaults, auto-bound arguments, auto-published outputs), burdens every tool that must now understand it (linter, type checker, config resolution, resume), and conflicts with the frozen-island philosophy that otherwise governs vendored code. + +Instead, we keep the meta-pipeline fully explicit and address the boilerplate at write time. The replicated params, outputs, and workflow call are mechanical transcriptions of the included pipeline's shell -- precisely the kind of task a coding agent can generate from the pipeline definition and a description of which params and outputs to expose, leaving the developer to write only the composition logic that carries novel intent. + +## Links + +- Related: [Module system](20251114-module-system.md) +- Related: [Workflow params](20250825-workflow-params.md) +- Related: [Workflow outputs](20251020-workflow-outputs.md) + +## Appendix + +### Example: fetchngs -> rnaseq + +This section walks through the aforementioned `fetchngs -> rnaseq` example as a meta-pipeline. + +> NOTE: This example uses simplified and idealized versions of `nf-core/fetchngs` and `nf-core/rnaseq` and may not match the actual implementations. + +**Project layout** + +The meta-pipeline is an ordinary Nextflow project with `nf-core/fetchngs` and `nf-core/rnaseq` vendored under `workflows/`: + +``` +fetchngs-rnaseq/ +├── main.nf +├── nextflow.config +└── workflows/ + └── nf-core/ + ├── fetchngs/ + │ ├── main.nf + │ ├── nextflow_spec.json + │ ├── modules/ + │ └── workflows/ + └── rnaseq/ + ├── main.nf + ├── nextflow_spec.json + ├── modules/ + └── workflows/ +``` + +Each pipeline has its own `modules/` and `workflows/`, so the two pipelines can depend on different versions of the same module without conflict. Both pipelines are committed to the meta-pipeline repository. + +**Pipeline code** + +The meta-pipeline includes the core workflow from each pipeline and composes them into an entry workflow with params and outputs: + +```groovy +include { NFCORE_FETCHNGS } from 'nf-core/fetchngs' +include { NFCORE_RNASEQ } from 'nf-core/rnaseq' + +params { + input: Path + strandedness: String = 'auto' +} + +workflow { + main: + // fetch FASTQ samples from NCBI SRA + ids = channel.fromPath(params.input).splitCsv() + ch_samples = NFCORE_FETCHNGS( ids ) + + // adapt fetchngs output to rnaseq input (add strandedness) + ch_samples = ch_samples.map { r -> + r + record(strandedness: params.strandedness) + } + + // perform RNAseq analysis + multiqc_report = NFCORE_RNASEQ( ch_samples ) + + publish: + multiqc_report = multiqc_report +} + +output { + multiqc_report { + path '.' + } +} +``` + +Notes about the white-box approach: + +- **The handoff is a channel, not a file.** The black-box approaches block until fetchngs finishes before rnaseq starts. Here, `ch_samples` is a live channel: rnaseq begins aligning each sample the moment fetchngs emits it. This is the dataflow composition that motivates the meta-pipeline. + +- **The adapter is an operator, not a pipeline.** The strandedness gap that required a separate `fetchngs-rnaseq.nf` adapter in the chaining example collapses to a single `map` operator in the meta-pipeline. + +- **Params and outputs are replicated, not inherited.** `--input` and `--strandedness` are declared in the meta-pipeline's own `params` block and passed explicitly into the core workflows. Similarly, any outputs must be declared as such in the meta-pipeline's `output` block. The included pipelines do not contribute any of their own params, entry workflows, or output blocks. + +**Configuration** + +Resource directives -- the *how* -- live in the meta-pipeline config. Because each included pipeline is part of the dataflow graph, its processes can be targeted like normal: + +```groovy +process { + withName: 'NFCORE_FETCHNGS:.*:SRATOOLS_FASTERQDUMP' { + cpus = 6 + memory = 24.GB + } + withName: 'NFCORE_RNASEQ:.*:STAR_ALIGN' { + cpus = 12 + memory = 72.GB + } +} +``` + +The *what* directives (`container`, `conda`) are defined in each pipeline's process definitions and do not need to be replicated here. Directives like `ext` can still be overridden by the meta-pipeline as needed. + +**Trade-offs** + +| Concern | Chaining (Black-box) | Meta-pipeline (white-box) | +| --- | --- | --- | +| Concurrency | Synchronous (each `run` completes first) | Asynchronous (rnaseq reacts to each fetchngs sample) | +| Params and outputs | Owned by each pipeline | Replicated in the meta-pipeline | +| Resource config | Per-pipeline config files | Unified meta-pipeline config | + +The most notable trade-off is the replication of params and outputs: anything the included pipelines exposed at the top level (params, published outputs) must be re-declared in the meta-pipeline. From 98411519a1ea7a2ee720566ee2a5e75f5baf52be Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 10 Jun 2026 08:10:32 -0500 Subject: [PATCH 02/13] Define fetchngs/rnaseq pipelines in appendix example [ci skip] Signed-off-by: Ben Sherman --- adr/20260608-remote-pipeline-inclusion.md | 86 +++++++++++++++++++++-- 1 file changed, 79 insertions(+), 7 deletions(-) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index 46ca179307..466378cd12 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -209,6 +209,74 @@ Each pipeline has its own `modules/` and `workflows/`, so the two pipelines can **Pipeline code** +The included pipelines are defined as follows, with a clear separation of *core workflow* from *entry workflow*: + +```groovy +// nf-core/fetchngs — main.nf +params { + input: Path // file of SRA/ENA accessions +} +workflow { + main: + ch_ids = channel.fromPath(params.input).splitCsv() + ch_samples = NFCORE_FETCHNGS( ch_ids ) + publish: + samples = ch_samples +} +output { + samples: Channel { path 'fastq' } +} + +workflow NFCORE_FETCHNGS { + take: + ids: Channel + + main: + // ... + + emit: + samples: Channel +} +``` + +```groovy +// nf-core/rnaseq — main.nf +params { + input: Path // samplesheet + aligner: String = 'star_salmon' + fasta: Path +} +workflow { + main: + ch_samples = channel.fromPath(params.input).splitCsv() + rnaseq = NFCORE_RNASEQ( ch_samples, params.aligner, params.fasta ) + publish: + multiqc = rnaseq.multiqc + bams = rnaseq.bams + counts = rnaseq.counts +} +output { + multiqc: Path { path 'multiqc' } + bams: Channel { path 'bams' } + counts: Channel { path 'counts' } +} + +workflow NFCORE_RNASEQ { + take: + samples: Channel + aligner: String + fasta: Path + + main: + // ... + + emit: + multiqc: Value + bams: Channel + counts: Channel +} +``` + The meta-pipeline includes the core workflow from each pipeline and composes them into an entry workflow with params and outputs: ```groovy @@ -218,13 +286,15 @@ include { NFCORE_RNASEQ } from 'nf-core/rnaseq' params { input: Path strandedness: String = 'auto' + aligner: String = 'star_salmon' + fasta: Path } workflow { main: // fetch FASTQ samples from NCBI SRA - ids = channel.fromPath(params.input).splitCsv() - ch_samples = NFCORE_FETCHNGS( ids ) + ch_ids = channel.fromPath(params.input).splitCsv() + ch_samples = NFCORE_FETCHNGS( ch_ids ) // adapt fetchngs output to rnaseq input (add strandedness) ch_samples = ch_samples.map { r -> @@ -232,16 +302,18 @@ workflow { } // perform RNAseq analysis - multiqc_report = NFCORE_RNASEQ( ch_samples ) + rnaseq = NFCORE_RNASEQ( ch_samples, params.aligner, params.fasta ) publish: - multiqc_report = multiqc_report + multiqc = rnaseq.multiqc + bams = rnaseq.bams + counts = rnaseq.counts } output { - multiqc_report { - path '.' - } + multiqc: Path { path 'multiqc' } + bams: Channel { path 'bams' } + counts: Channel { path 'counts' } } ``` From c0911de8740e4908bd9b712919a2225b0ff40dc2 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 10 Jun 2026 08:17:51 -0500 Subject: [PATCH 03/13] Improve description of configuration in appendix example Signed-off-by: Ben Sherman --- adr/20260608-remote-pipeline-inclusion.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index 466378cd12..1f09aae73c 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -327,7 +327,7 @@ Notes about the white-box approach: **Configuration** -Resource directives -- the *how* -- live in the meta-pipeline config. Because each included pipeline is part of the dataflow graph, its processes can be targeted like normal: +Since each included pipeline is just part of the dataflow graph, configuration works like normal. Processes in an included pipeline can be targeted via config selector: ```groovy process { @@ -342,7 +342,7 @@ process { } ``` -The *what* directives (`container`, `conda`) are defined in each pipeline's process definitions and do not need to be replicated here. Directives like `ext` can still be overridden by the meta-pipeline as needed. +Both the meta-pipeline developer and users can override whatever they want from config. In practice, the process definitions should own the *what* (`container`, `conda`) while the meta-pipeline config should own the *how* (`cpus`, `memory`). **Trade-offs** From 286ac8db2e6e9287d76662fb12ceb8c7988395f6 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 10 Jun 2026 09:54:00 -0500 Subject: [PATCH 04/13] Relax language in "Best practices for included pipelines" [ci skip] Signed-off-by: Ben Sherman --- adr/20260608-remote-pipeline-inclusion.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index 1f09aae73c..87e8835737 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -53,14 +53,14 @@ As a result, if a user wants to preserve any top-level concerns from the include To be importable in practice, a pipeline's core workflow (and its dependent modules/workflows) should be free of external context: -1. No `params` usage outside the entry workflow -- pass values as explicit process/workflow inputs. -2. No `publishDir` -- use the `output` block. -3. No use of project-level assets (`projectDir`, `bin`, `lib`) within the core workflow. Module-level assets can be used through the module `resources/` bundle and `moduleDir`. +1. Avoid `params` usage outside the entry workflow -- pass values as explicit process/workflow inputs. +2. Avoid `publishDir` -- use the `output` block. +3. Avoid use of project-level assets (`projectDir`, `bin`, `lib`) within the core workflow. Module-level assets can be safely used through the module `resources/` bundle and `moduleDir`. 4. Declare software dependencies (`container`, `conda`) in the process definition, not in config. -5. No default `ext` settings in config -- specify these defaults in the process definition or use explicit process inputs. Otherwise, any default `ext` settings must be replicated manually in the meta-pipeline. -6. No plugin functions within the core workflow. +5. Avoid default `ext` settings in config -- specify these defaults in the process definition or use explicit process inputs. Otherwise, any default `ext` settings must be replicated manually in the meta-pipeline. +6. Avoid plugin functions within the core workflow. -For process directives, it is helpful to distinguish *what* is executed vs *how* it is executed. Directives that affect the *what* (`container`, `ext` settings) should be owned by the process definition. Directives that affect the *how* (`cpus`, `memory`, `executor`, `queue`, `errorStrategy`) should be owned by the meta-pipeline. +For process directives, it is helpful to distinguish *what* is computed vs *how* it is computed. Directives that affect the *what* (`container`, `ext` settings) should be owned by the process definition. Directives that affect the *how* (`cpus`, `memory`, `executor`, `queue`, `errorStrategy`) should be owned by the meta-pipeline. These constraints are not absolute -- it is possible to import a pipeline that does not adhere to any of these rules. Following these constraints simply makes it easier to import a pipeline with minimal extra work (manual replication, cross-cutting concerns). From 09c9e9680b5deacf3d9c429e4bebf911d52a5698 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Thu, 11 Jun 2026 11:57:20 -0500 Subject: [PATCH 05/13] Consolidate pipeline chaining, nf-cascade into one section Signed-off-by: Ben Sherman --- adr/20260608-remote-pipeline-inclusion.md | 34 ++++++++++------------- 1 file changed, 15 insertions(+), 19 deletions(-) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index 87e8835737..c974e3e5aa 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -104,9 +104,9 @@ Since this use case is rare -- plugin functions are typically used in the entry ### Pipeline chaining -An alternative to a meta-pipeline is a *pipeline chain*, such as a shell script that simply calls multiple Nextflow pipelines in sequence. +An alternative to a meta-pipeline is a *pipeline chain*, in which multiple Nextflow pipelines are called in sequence via `nextflow run`. -For example, a fetchngs -> rnaseq pipeline chain could be implemented as follows: +For example, a fetchngs -> rnaseq pipeline chain can be implemented in a shell script: ```bash # fetch FASTQ samples from NCBI SRA @@ -129,21 +129,11 @@ nextflow -q run nf-core/rnaseq \ > results/output-rnaseq.json ``` -While pipeline chaining has always been possible in theory, new language features such as [workflow outputs](20251020-workflow-outputs.md) and [record types](20260306-record-types.md) make it much more practical. Each pipeline can define a structured output which can be passed to the next pipeline via JSON. Mismatches between an upstream output and downstream input (e.g. missing columns, different column names) can be resolved by a small adapter pipeline. - -Pipeline chaining remains a valid solution for certain use cases, such as simple chains (A -> B -> C) of off-the-shelf pipelines with little to no additional coding. For compositions that are more complex and/or require maximum dataflow concurrency, meta-pipelines are the general solution. - -### Nextflow-in-Nextflow - -Another solution that sits between pipeline chaining and meta-pipelines is a Nextflow pipeline that runs other Nextflow pipelines as tasks. - -The [nf-cascade](https://github.com/mahesh-panchal/nf-cascade) project explores this approach to its limits. It defines a native process called `NEXTFLOW_RUN`, which takes a pipeline with inputs and runs it via `nextflow run`. The pipeline developer then writes a kind of meta-pipeline with aliases of `NEXTFLOW_RUN`. - -For example, rewriting the fetchngs -> rnaseq chain described above as a Nextflow pipeline: +Or a Nextflow pipeline: ```groovy -include { NEXTFLOW_RUN as NFCORE_FETCHNGS } from "./modules/local/nextflow/run/main" -include { NEXTFLOW_RUN as NFCORE_RNASEQ } from "./modules/local/nextflow/run/main" +include { NEXTFLOW_RUN as NFCORE_FETCHNGS } from "./modules/local/nextflow/run" +include { NEXTFLOW_RUN as NFCORE_RNASEQ } from "./modules/local/nextflow/run" workflow { NFCORE_FETCHNGS ( @@ -157,13 +147,19 @@ workflow { } ``` -This approach works with any Nextflow pipeline out of the box, because it simply executes the pipeline directly. However, this approach sacrifices dataflow composition, i.e. each pipeline must complete before the next pipeline can start. +The `NEXTFLOW_RUN` process simply calls `nextflow run` in a native process. See [nf-cascade](https://github.com/mahesh-panchal/nf-cascade) for more information about this approach. + +Pipeline chains can also be implemented in Seqera Platform using actions (e.g. when a fetchngs run completes -> launch rnaseq on the fetchngs output). + +Pipeline chaining works with any Nextflow pipeline out of the box, because it simply executes each pipeline directly. Chaining often requires glue logic to adapt upstream outputs to downstream outputs -- missing columns, different column names, etc -- but language features such as [workflow outputs](20251020-workflow-outputs.md) and [record types](20260306-record-types.md) make it easier by allowing pipelines to defined structured inputs and outputs. + +The downside of pipeline chaining is that it sacrifices dataflow concurrency -- each pipeline must complete before the next pipeline can start. As a result, pipeline chaining is a categorically different solution from meta-pipelines. It remains a valid option for certain use cases, such as simple chains (A -> B -> C) of off-the-shelf pipelines. For compositions that are more complex and/or require maximum dataflow concurrency, meta-pipelines are the general solution. ### Best of both: runtime inheritance The meta-pipeline approach treats the included pipeline as a *white box* -- it composes the pipeline like any included workflow, producing a single dataflow graph. It also imposes several constraints on how the included pipeline is written, and it imposes development overhead. For example, any params / outputs that need to be exposed from the included pipeline must be replicated in the meta-pipeline. -The Nextflow-in-Nextflow approach treats the included pipeline as a *black box* -- it preserves the exact pipeline behavior (core workflow + entry workflow + config) while forfeiting dataflow composition (separate dataflow graphs). +Pipeline chaining treats the included pipeline as a *black box* -- it preserves the exact pipeline behavior (core workflow + entry workflow + config) while forfeiting dataflow composition (separate dataflow graphs). An ideal solution might combine the best of both: compose pipelines into a single dataflow graph (white box) while inheriting each pipeline's params, outputs, and config so they need not be replicated (black box). We considered such a model, where an included pipeline contributes its shell as namespaced, overridable defaults, but rejected it. Dataflow composition fundamentally requires exposing the core workflow as a set of channel ports, so the white-box mechanism is unavoidable; inheritance would only layer implicit behavior on top of it. That behavior comes at a steep cost: it relocates a one-time *write* cost (boilerplate) into a recurring *read* cost (hidden defaults, auto-bound arguments, auto-published outputs), burdens every tool that must now understand it (linter, type checker, config resolution, resume), and conflicts with the frozen-island philosophy that otherwise governs vendored code. @@ -319,7 +315,7 @@ output { Notes about the white-box approach: -- **The handoff is a channel, not a file.** The black-box approaches block until fetchngs finishes before rnaseq starts. Here, `ch_samples` is a live channel: rnaseq begins aligning each sample the moment fetchngs emits it. This is the dataflow composition that motivates the meta-pipeline. +- **The handoff is a channel, not a file.** A pipeline chain blocks until fetchngs finishes before rnaseq starts. Here, `ch_samples` is a live channel: rnaseq begins aligning each sample the moment fetchngs emits it. This is the dataflow composition that motivates the meta-pipeline. - **The adapter is an operator, not a pipeline.** The strandedness gap that required a separate `fetchngs-rnaseq.nf` adapter in the chaining example collapses to a single `map` operator in the meta-pipeline. @@ -346,7 +342,7 @@ Both the meta-pipeline developer and users can override whatever they want from **Trade-offs** -| Concern | Chaining (Black-box) | Meta-pipeline (white-box) | +| Concern | Chaining (black-box) | Meta-pipeline (white-box) | | --- | --- | --- | | Concurrency | Synchronous (each `run` completes first) | Asynchronous (rnaseq reacts to each fetchngs sample) | | Params and outputs | Owned by each pipeline | Replicated in the meta-pipeline | From 86ed3b6cce31a11df980784d2673cdfbce525d3e Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 23 Jun 2026 13:56:20 -0500 Subject: [PATCH 06/13] Separate ADR for workflow modules, replace "core workflow" distinction with implicit pipeline inclusion Signed-off-by: Ben Sherman --- adr/20260608-remote-pipeline-inclusion.md | 201 +++++++++++----------- adr/20260608-workflow-modules.md | 199 +++++++++++++++++++++ 2 files changed, 296 insertions(+), 104 deletions(-) create mode 100644 adr/20260608-workflow-modules.md diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index c974e3e5aa..53642dc25f 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -13,9 +13,7 @@ Add the ability to include a remote pipeline into a *meta-pipeline*. Nextflow supports reusing process definitions via remote *module* inclusion (e.g. `include { BWA_MEM } from 'nf-core/bwa/mem'`), but there is no standard mechanism to reuse an entire *pipeline* as a building block. Users must either fork and copy code, or compose/chain multiple `nextflow run` sessions which forfeits dataflow composition. -The natural unit of reuse for a pipeline is its *core workflow* -- the named workflow that takes and emits channels (e.g. `NFCORE_RNASEQ` in `nf-core/rnaseq`) -- as distinct from the deployment shell around it (the `params`, entry workflow, `output` block, and config). The nf-core community has already structured their pipelines around this split in anticipation of meta-pipelines. - -The decision to make: how should a remote pipeline be distributed, resolved, stored, and included so that it can be composed into a meta-pipeline while preserving dataflow composition. +The [module system](20251114-module-system.md) and [workflow module](20260608-workflow-modules.md) ADRs define how standalone *processes* and *workflows* should be distributed as modules through the Nextflow registry. This ADR defines how *pipelines* -- workflows with a deployment shell -- should be composed into larger *meta-pipelines*. ## Goals @@ -33,73 +31,101 @@ The decision to make: how should a remote pipeline be distributed, resolved, sto ## Decision -Allow remote pipelines to be included into meta-pipelines, using the same namespacing conventions and include syntax as modules. Store the included pipeline in the meta-pipeline repository under `workflows///` with its own subdirectories for modules and subworkflows. The meta-pipeline owns all top-level concerns (entry workflow, params, outputs, config). Pipelines should be written with a self-contained core workflow to make importing as easy as possible. +Allow pipelines to be published and installed through the Nextflow registry, using the same namespacing conventions as modules. Store the included pipeline in the meta-pipeline repository under `pipelines///` with its own subdirectories for modules and workflows. Provide a way to include an entire pipeline (`params` block, entry workflow, `output` block) as a named workflow to facilitate workflow composition. ## Core Capabilities -### Composition over orchestration - -The included pipeline must be incorporated into the meta-pipeline's dataflow graph in order to maximize dataflow concurrency. Existing approaches like pipeline chaining and Nextflow-in-Nextflow impose a synchronization barrier between each pipeline run. +### Pipeline composition -To this end, the included pipeline should be written in a way that separates the *core workflow* from the rest of the pipeline (entry workflow, params, publishing, config). Only the core workflow is included into the meta-pipeline; the rest is discarded. +A pipeline -- that is, a `params` / `workflow` / `output` trio -- can be included and called like a named workflow. This way, pipelines can be composed using regular dataflow logic. -### Meta-pipeline owns all top-level concerns +For example, given the following pipeline: -The meta-pipeline owns the entry workflow, `params` block, `output` block, and config. An included pipeline contributes none of these. This approach aligns with existing include semantics, which only supports composition of processes and named workflows. - -As a result, if a user wants to preserve any top-level concerns from the included pipeline, they must be explicitly replicated in the meta-pipeline. For example, params exposed by the included pipeline must be replicated in the meta-pipeline params and passed to the included pipeline's core workflow. +```groovy +// pipelines/rnaseq.nf +params { + input: Path + aligner: String = 'star_salmon' + fasta: Path +} +workflow { + // ... +} +output { + bams: Channel { path 'bams' } + multiqc: Path { path 'multiqc' } +} +``` -### Best practices for included pipeline +It can be included and called as follows: -To be importable in practice, a pipeline's core workflow (and its dependent modules/workflows) should be free of external context: +```groovy +// main.nf +include { workflow as RNASEQ } from './pipelines/rnaseq.nf' -1. Avoid `params` usage outside the entry workflow -- pass values as explicit process/workflow inputs. -2. Avoid `publishDir` -- use the `output` block. -3. Avoid use of project-level assets (`projectDir`, `bin`, `lib`) within the core workflow. Module-level assets can be safely used through the module `resources/` bundle and `moduleDir`. -4. Declare software dependencies (`container`, `conda`) in the process definition, not in config. -5. Avoid default `ext` settings in config -- specify these defaults in the process definition or use explicit process inputs. Otherwise, any default `ext` settings must be replicated manually in the meta-pipeline. -6. Avoid plugin functions within the core workflow. +workflow { + rnaseq = RNASEQ( + input: file('input.csv'), + fasta: file('index.fasta') + ) + rnaseq.bams.view() // Channel + rnaseq.multiqc.view() // Value +} +``` -For process directives, it is helpful to distinguish *what* is computed vs *how* it is computed. Directives that affect the *what* (`container`, `ext` settings) should be owned by the process definition. Directives that affect the *how* (`cpus`, `memory`, `executor`, `queue`, `errorStrategy`) should be owned by the meta-pipeline. +Notes: -These constraints are not absolute -- it is possible to import a pipeline that does not adhere to any of these rules. Following these constraints simply makes it easier to import a pipeline with minimal extra work (manual replication, cross-cutting concerns). +- The pipeline must included using the `workflow` keyword and aliased to a specific name (`RNASEQ`). +- The `params` block becomes the `take:` section and the `output` block becomes the `emit:` section. +- The workflow is called using named arguments so that defaults can be omitted. +- All outputs are either a `Channel` or wrapped as `Value`, allowing them to be used in regular dataflow logic. -### Pipeline inclusion and storage +### Remote pipeline inclusion and storage -Modules and pipelines share the same include syntax and naming conventions: +Pipelines can be published, installed, and included through the Nextflow registry: ```groovy // module include { BWA_MEM } from 'nf-core/bwa/mem' // pipeline -include { NFCORE_RNASEQ } from 'nf-core/rnaseq' +include { workflow as NFCORE_RNASEQ } from 'nf-core/rnaseq' ``` -Including a remote pipeline is equivalent to including the top-level `main.nf` of that pipeline; any named workflow defined there can be included by name. By convention, the main script defines only the entry workflow and the core workflow (e.g. `NFCORE_RNASEQ` in nf-core/rnaseq). From this point, the included workflow can be called like any other workflow. - -When a pipeline is included, it is vendored into the meta-pipeline project under `workflows///`. Included pipelines are isolated -- each included pipeline has its own `modules/` and `workflows/` directories. This way, two pipelines can use different versions of the same module without compromising reproducibility. +When a pipeline is included from the registry, it is vendored into the including project under `pipelines///`. Included pipelines are isolated -- each included pipeline has its own `modules/` directory. This way, two pipelines can use different versions of the same module without compromising reproducibility. Included pipelines should be committed to the meta-pipeline repository. The pipeline should have a *pipeline spec* (`nextflow_spec.json`) which specifies the pipeline version, so that Nextflow can track local changes. -## Open Questions +### Best practices for including pipelines -### Sourcing from Nextflow registry vs Git repositories +Pipeline inclusion only captures the pipeline's main script and included modules -- it does not capture external context such as config or the `lib` directory. As a result, the pipeline should be written in a way that works when included by a meta-pipeline: + +1. Pipeline parameters should be defined in the script `params` block. The config should only declare *config params* (params that only affect config settings). + +2. Project-level assets (`projectDir`, `bin`, `lib`) should not be used since the meta-pipeline will have a different project root. Module-level assets can be safely used through the module `resources/` bundle and `moduleDir`. + +3. Default `ext` settings should be specified in the process definition or avoided in favor of process inputs. + +4. Software dependencies (`container`, `conda`) should be declared in the process definition, not in config. + +5. Workflow outputs should be published using the `output` block, not `publishDir`. + +None of these constraints are absolute. All of them can be circumvented by manually replicating the external context in the meta-pipeline. Following these constraints simply makes it easier to import a pipeline with minimal extra work. + +## Open Questions -Pipelines could be stored in the Nextflow registry (as a new artifact type) or fetched directly from Git repositories. The pipeline registry is a potential long-term goal with other use cases, but it likely introduces additional scope that is not strictly related to meta-pipelines. Using existing Git repositories would be an expedient solution for the first iteration. +### Pipeline registry and CLI -### Pipeline CLI +Sourcing remote pipelines from the Nextflow registry implies a pipeline registry API and a `nextflow pipeline` command group for publishing and installing pipelines. This infrastructure can be largely inferred from existing patterns established for modules. -Sourcing remote pipelines from the Nextflow registry implies a `nextflow pipeline` command group for publishing and installing pipelines, similar to `nextflow module`. Even with a Git-based approach, a CLI is likely still needed to install and update remote pipelines. +One aspect that remains open is the pipeline spec (`nextflow_spec.json` or `nextflow_schema.json`) which may have a different shape from the module spec (`meta.yml`). A minimal pipeline spec could be introduced to enable remote pipelines without bloating scope. Alternatively, the pipeline version could be managed by a helper file (e.g. `.pipeline-info`) until the pipeline spec is finalized. ### Using plugin functions in included pipeline -If an included pipeline uses plugin functions in the core workflow, these plugins must be explicitly declared in the meta-pipeline config, since the included pipeline config is not inherited. +If an included pipeline uses plugins, these plugins must be explicitly declared in the meta-pipeline config since they cannot be inferred from the pipeline inclusion. Alternatively, these core plugin dependencies could be specified in the pipeline spec under `requires.plugins`. When installing a pipeline, Nextflow could copy these plugin declarations into the meta-pipeline config and/or spec. -Since this use case is rare -- plugin functions are typically used in the entry workflow outside the core workflow -- it can be deferred in the first iteration. - ## Alternatives ### Pipeline chaining @@ -135,35 +161,43 @@ Or a Nextflow pipeline: include { NEXTFLOW_RUN as NFCORE_FETCHNGS } from "./modules/local/nextflow/run" include { NEXTFLOW_RUN as NFCORE_RNASEQ } from "./modules/local/nextflow/run" +params { + // ... +} + workflow { - NFCORE_FETCHNGS ( + // fetch FASTQ samples from NCBI SRA + fetchngs = NFCORE_FETCHNGS ( 'nf-core/fetchngs', // nextflow opts, pipeline inputs, etc ... ) - NFCORE_RNASEQ ( + // adapt fetchngs output to rnaseq input (add strandedness column) + ch_samples = fetchngs2rnaseq(fetchngs) + // perform RNAseq analysis + rnaseq = NFCORE_RNASEQ ( 'nf-core/rnaseq', // nextflow opts, pipeline inputs, etc ... ) } + +output { + // ... +} ``` The `NEXTFLOW_RUN` process simply calls `nextflow run` in a native process. See [nf-cascade](https://github.com/mahesh-panchal/nf-cascade) for more information about this approach. Pipeline chains can also be implemented in Seqera Platform using actions (e.g. when a fetchngs run completes -> launch rnaseq on the fetchngs output). -Pipeline chaining works with any Nextflow pipeline out of the box, because it simply executes each pipeline directly. Chaining often requires glue logic to adapt upstream outputs to downstream outputs -- missing columns, different column names, etc -- but language features such as [workflow outputs](20251020-workflow-outputs.md) and [record types](20260306-record-types.md) make it easier by allowing pipelines to defined structured inputs and outputs. - -The downside of pipeline chaining is that it sacrifices dataflow concurrency -- each pipeline must complete before the next pipeline can start. As a result, pipeline chaining is a categorically different solution from meta-pipelines. It remains a valid option for certain use cases, such as simple chains (A -> B -> C) of off-the-shelf pipelines. For compositions that are more complex and/or require maximum dataflow concurrency, meta-pipelines are the general solution. +Pipeline chaining works with any Nextflow pipeline out of the box, because it simply executes each pipeline directly. Language features such as [workflow outputs](20251020-workflow-outputs.md) and [record types](20260306-record-types.md) make pipeline chaining easier by allowing each pipeline to define structured inputs and outputs. -### Best of both: runtime inheritance +However, there are a number of downsides: -The meta-pipeline approach treats the included pipeline as a *white box* -- it composes the pipeline like any included workflow, producing a single dataflow graph. It also imposes several constraints on how the included pipeline is written, and it imposes development overhead. For example, any params / outputs that need to be exposed from the included pipeline must be replicated in the meta-pipeline. +- It forfeits native dataflow composition. The developer must serialize/deserialize samplesheet files instead of passing channels directly between pipelines. Each pipeline must complete before the next pipeline can start. -Pipeline chaining treats the included pipeline as a *black box* -- it preserves the exact pipeline behavior (core workflow + entry workflow + config) while forfeiting dataflow composition (separate dataflow graphs). +- It requires an external workflow system instead of reusing the language that pipeline developers already know. Even the Nextflow-in-Nextflow approach shown above requires many tricks to orchestrate nested pipeline runs via the `NEXTFLOW_RUN` process. -An ideal solution might combine the best of both: compose pipelines into a single dataflow graph (white box) while inheriting each pipeline's params, outputs, and config so they need not be replicated (black box). We considered such a model, where an included pipeline contributes its shell as namespaced, overridable defaults, but rejected it. Dataflow composition fundamentally requires exposing the core workflow as a set of channel ports, so the white-box mechanism is unavoidable; inheritance would only layer implicit behavior on top of it. That behavior comes at a steep cost: it relocates a one-time *write* cost (boilerplate) into a recurring *read* cost (hidden defaults, auto-bound arguments, auto-published outputs), burdens every tool that must now understand it (linter, type checker, config resolution, resume), and conflicts with the frozen-island philosophy that otherwise governs vendored code. - -Instead, we keep the meta-pipeline fully explicit and address the boilerplate at write time. The replicated params, outputs, and workflow call are mechanical transcriptions of the included pipeline's shell -- precisely the kind of task a coding agent can generate from the pipeline definition and a description of which params and outputs to expose, leaving the developer to write only the composition logic that carries novel intent. +Pipeline chaining can be practical for certain use cases, such as simple chains (A -> B -> C) of off-the-shelf pipelines. But the general solution is to compose pipelines using dataflow logic, just like any other Nextflow pipeline. ## Links @@ -181,31 +215,29 @@ This section walks through the aforementioned `fetchngs -> rnaseq` example as a **Project layout** -The meta-pipeline is an ordinary Nextflow project with `nf-core/fetchngs` and `nf-core/rnaseq` vendored under `workflows/`: +The meta-pipeline is an ordinary Nextflow project with `nf-core/fetchngs` and `nf-core/rnaseq` vendored under `pipelines/`: ``` fetchngs-rnaseq/ ├── main.nf ├── nextflow.config -└── workflows/ +└── pipelines/ └── nf-core/ ├── fetchngs/ │ ├── main.nf │ ├── nextflow_spec.json - │ ├── modules/ - │ └── workflows/ + │ └── modules/ └── rnaseq/ ├── main.nf ├── nextflow_spec.json - ├── modules/ - └── workflows/ + └── modules/ ``` -Each pipeline has its own `modules/` and `workflows/`, so the two pipelines can depend on different versions of the same module without conflict. Both pipelines are committed to the meta-pipeline repository. +Each pipeline has its own `modules/` directory, so the two pipelines can depend on different versions of the same module without conflict. Both pipelines are committed to the meta-pipeline repository. **Pipeline code** -The included pipelines are defined as follows, with a clear separation of *core workflow* from *entry workflow*: +The included pipelines are defined as follows: ```groovy // nf-core/fetchngs — main.nf @@ -215,24 +247,13 @@ params { workflow { main: ch_ids = channel.fromPath(params.input).splitCsv() - ch_samples = NFCORE_FETCHNGS( ch_ids ) + ch_samples = // ... publish: samples = ch_samples } output { samples: Channel { path 'fastq' } } - -workflow NFCORE_FETCHNGS { - take: - ids: Channel - - main: - // ... - - emit: - samples: Channel -} ``` ```groovy @@ -245,7 +266,7 @@ params { workflow { main: ch_samples = channel.fromPath(params.input).splitCsv() - rnaseq = NFCORE_RNASEQ( ch_samples, params.aligner, params.fasta ) + rnaseq = // ... publish: multiqc = rnaseq.multiqc bams = rnaseq.bams @@ -256,28 +277,13 @@ output { bams: Channel { path 'bams' } counts: Channel { path 'counts' } } - -workflow NFCORE_RNASEQ { - take: - samples: Channel - aligner: String - fasta: Path - - main: - // ... - - emit: - multiqc: Value - bams: Channel - counts: Channel -} ``` -The meta-pipeline includes the core workflow from each pipeline and composes them into an entry workflow with params and outputs: +The meta-pipeline includes each pipeline and composes them into a new entry workflow with params and outputs: ```groovy -include { NFCORE_FETCHNGS } from 'nf-core/fetchngs' -include { NFCORE_RNASEQ } from 'nf-core/rnaseq' +include { workflow as NFCORE_FETCHNGS } from 'nf-core/fetchngs' +include { workflow as NFCORE_RNASEQ } from 'nf-core/rnaseq' params { input: Path @@ -289,16 +295,15 @@ params { workflow { main: // fetch FASTQ samples from NCBI SRA - ch_ids = channel.fromPath(params.input).splitCsv() - ch_samples = NFCORE_FETCHNGS( ch_ids ) + fetchngs = NFCORE_FETCHNGS( input: params.input ) // adapt fetchngs output to rnaseq input (add strandedness) - ch_samples = ch_samples.map { r -> + ch_samples = fetchngs.samples.map { r -> r + record(strandedness: params.strandedness) } // perform RNAseq analysis - rnaseq = NFCORE_RNASEQ( ch_samples, params.aligner, params.fasta ) + rnaseq = NFCORE_RNASEQ( input: ch_samples, aligner: params.aligner, fasta: params.fasta ) publish: multiqc = rnaseq.multiqc @@ -313,12 +318,10 @@ output { } ``` -Notes about the white-box approach: +Notes: - **The handoff is a channel, not a file.** A pipeline chain blocks until fetchngs finishes before rnaseq starts. Here, `ch_samples` is a live channel: rnaseq begins aligning each sample the moment fetchngs emits it. This is the dataflow composition that motivates the meta-pipeline. -- **The adapter is an operator, not a pipeline.** The strandedness gap that required a separate `fetchngs-rnaseq.nf` adapter in the chaining example collapses to a single `map` operator in the meta-pipeline. - - **Params and outputs are replicated, not inherited.** `--input` and `--strandedness` are declared in the meta-pipeline's own `params` block and passed explicitly into the core workflows. Similarly, any outputs must be declared as such in the meta-pipeline's `output` block. The included pipelines do not contribute any of their own params, entry workflows, or output blocks. **Configuration** @@ -338,14 +341,4 @@ process { } ``` -Both the meta-pipeline developer and users can override whatever they want from config. In practice, the process definitions should own the *what* (`container`, `conda`) while the meta-pipeline config should own the *how* (`cpus`, `memory`). - -**Trade-offs** - -| Concern | Chaining (black-box) | Meta-pipeline (white-box) | -| --- | --- | --- | -| Concurrency | Synchronous (each `run` completes first) | Asynchronous (rnaseq reacts to each fetchngs sample) | -| Params and outputs | Owned by each pipeline | Replicated in the meta-pipeline | -| Resource config | Per-pipeline config files | Unified meta-pipeline config | - -The most notable trade-off is the replication of params and outputs: anything the included pipelines exposed at the top level (params, published outputs) must be re-declared in the meta-pipeline. +Both the meta-pipeline developer and users can override whatever they want from config. diff --git a/adr/20260608-workflow-modules.md b/adr/20260608-workflow-modules.md new file mode 100644 index 0000000000..fe9b59088f --- /dev/null +++ b/adr/20260608-workflow-modules.md @@ -0,0 +1,199 @@ +# Workflow modules + +- Authors: Ben Sherman +- Status: draft +- Date: 2026-06-08 +- Tags: workflows, modules, registry + +## Summary + +Add the ability to include a workflow as a module from the Nextflow registry. + +## Problem Statement + +A Nextflow *module* is currently defined as a standalone process definition (with corresponding spec file). The [module system ADR](20251114-module-system.md) defines how these modules are published, distributed, and executed through the Nextflow registry. + +There is a similar need to share and re-use workflows / subworkflows. The nf-core community has curated a collection of re-usable subworkflows in the [nf-core/modules](https://github.com/nf-core/modules) repository. + +The module system should be extended to include both standalone *processes* and *workflows*. + +## Goals + +- **Workflow inclusion**: allow workflows to be included from the registry using the same include syntax as for modules. + +- **Workflow execution**: allow workflows to be executed directly via `nextflow module run`. + +- **Reuse existing infrastructure**: the module system already defines a registry API and CLI for publishing and installing modules. Treating workflows as a separate concept would require duplicating much of this infrastructure. + +## Non-goals + +- **Pipeline inclusion**: *pipelines* are distinct from *workflows* -- they specify params, publishing, and config, not just workflow logic. + +## Decision + +Extend the definition of *module* to include both standalone *processes* and standalone *workflows*. Allow workflow modules to be published, installed, included, and executed via the Nextflow registry. Store the included workflow in the including repository under `workflows///`. + +## Core Capabilities + +### Workflow modules vs process modules + +A module can refer to a *workflow module* or *process module*, depending on whether it defines a workflow or process. + +Workflow modules can be published to and queried from the Nextflow registry using the same modules API and `nextflow module` CLI. Workflow modules also have the same directory structure as process modules. + +Since workflow modules are just modules, a process module and workflow module cannot have the same name in a module namespace. + +### Module spec + +The module spec is extended as follows in order to support standalone workflows: + +- A `kind` field to distinguish between workflow modules (`kind: Workflow`) and process modules (`kind: Process`). + +- A `requires.modules` field to specify *transitive dependencies*, since a workflow can depend on other processes and workflows. + +- Certain fields (`topics`, `tools`) cannot be specified for workflow modules because they are not applicable. + +For example: + +```yaml +name: nf-core/fastq_align_star +kind: Workflow +version: 0.0.0-4e3e10e +description: Align reads to a reference genome using bowtie2 then sort with samtools +authors: + - "@JoseEspinosa" +license: MIT +requires: + nextflow: ">=24.04.0" + modules: + - nf-core/star/align@0.0.0-4e3e10e + - nf-core/samtools/sort@0.0.0-4e3e10e + - nf-core/samtools/index@0.0.0-4e3e10e + - nf-core/samtools/stats@0.0.0-4e3e10e + - nf-core/samtools/idxstats@0.0.0-4e3e10e + - nf-core/samtools/flagstat@0.0.0-4e3e10e + - nf-core/bam_sort_stats_samtools@0.0.0-4e3e10e +``` + +### Workflow inclusion and storage + +Workflow modules use the same include syntax and naming conventions as process modules: + +```groovy +// process +include { BWA_MEM } from 'nf-core/bwa/mem' + +// workflow +include { FASTQ_ALIGN_STAR } from 'nf-core/fastq_align_star' +``` + +When a workflow is included, it is vendored into the including project under `workflows///`. Included workflows should be committed to the including repository. + +Transitive dependencies (specified by `requires.modules`) should also be installed in the `modules/` and `workflows/` directories alongside the included workflow. Since modules are flattened, it is not possible for a pipeline to use two different versions of the same process or workflow. + +### Workflow execution + +Typed workflows can be executed directly by inferring the `params` and `output` blocks from the `take:` and `emit:` sections. + +For example, given the following workflow: + +```groovy +workflow RNASEQ { + take: + samples: Channel + index: Path + + main: + ch_aligned = ALIGN(samples, index) + multiqc_report = MULTIQC(ch_aligned.collect()) + + emit: + aligned: Channel = ch_aligned + multiqc_report: Path = multiqc_report +} + +record Sample { + id: String + fastq_1: Path + fastq_2: Path +} + +record AlignedSample { + id: String + bam: Path + bai: Path +} +``` + +The user can run the workflow directly as follows: + +```bash +nextflow module run rnaseq.nf \ + --samples input.csv \ + --index index.fasta +``` + +Nextflow executes the `RNASEQ` workflow as if it were wrapped in the following entry workflow: + +```groovy +params { + samples: Channel + index: Path +} + +workflow { + main: + rnaseq = RNASEQ(params.samples, params.index) + + publish: + aligned = rnaseq.aligned + multiqc_report = rnaseq.multiqc_report +} + +output { + aligned: Channel {} + multiqc_report: Path {} +} +``` + +This way, the user can run a workflow directly without having to write an entry workflow for it. + +Direct execution requires the ability to load an input channel directly from an index file (samplesheet). This can be done by loading the samplesheet data based on the file extension (CSV, JSON, YAML), casting each record to the given record type, and loading the collection as a channel. The data-loading function may be extended via plugin to support additional formats (e.g. Parquet). + +The channel input can use a generic type such as `Map` or `Record`, or a custom record type to enable further validation. In the above example, using the `Sample` type ensures that each samplesheet row is validated against the record fields and the `fastq_1` and `fastq_2` columns are treated as file paths. + +When executing a named workflow directly, output files are not published to an output directory. Instead, the workflow output printed by Nextflow simply refers to output files by their work directory path. + +## Alternatives + +### Workflows vs processes + +One alternative is to treat workflows as a separate concept from modules, restricting the definition of *module* to only include standalone processes. + +However, the broader meaning of *module* is a re-usable component, and both processes and workflows are re-usable components, so it makes more sense to extend the module system rather than introduce a parallel system for workflows. A parallel system would also require duplicating a lot of existing code (registry API, CLI, etc). + +Process modules and workflow modules can be distinguished by different specializations of the module spec (`kind: Process` vs `kind: Workflow`) and different storage locations (`modules/` vs `workflows/`). + +### Workflows vs subworkflows + +The nf-core community makes a distinction between *workflows* and *subworkflows*: + +- Subworkflows are stored in the `subworkflows/` directory and can be shared across pipelines. + +- Workflows are stored in the `workflows/` directory and are owned by a specific pipeline. + +Nextflow makes no such distinction, as there is no functional difference between a workflow and a subworkflow. Both are stored in the `workflows/` directory and both can be published and installed through the Nextflow registry. + +The Nextflow registry makes it easier for pipelines to share and reuse workflows, since the pipeline can be the source-of-truth rather than having to surrender ownership to the `nf-core/modules` repository. + +### Workflows vs pipelines + +One alternative is to treat all workflows as pipelines, restricting the definition of *module* to only include standalone processes. + +Aside from the problems mentioned under [Workflows vs processes](#workflows-vs-processes), this approach conflates two commonly-understood concepts: a *pipeline* which is an end-to-end analysis, and a *workflow* or *subworkflow* which is a component in a larger analysis. It is more congruent with conventional understanding to treat workflows (subworkflows) as a separate concept from pipelines. + +## Links + +- [nf-core terminology](https://nf-co.re/docs/usage/getting_started/terminology) +- Refines: [Module system](20251114-module-system.md) +- Related: [Remote pipeline inclusion](20260608-remote-pipeline-inclusion.md) From a7aea2fc0d0e32c62c15b902203adc8edbcca703 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 23 Jun 2026 14:17:32 -0500 Subject: [PATCH 07/13] Add version 1.1 update Signed-off-by: Ben Sherman --- adr/20260608-remote-pipeline-inclusion.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index 53642dc25f..17c544a08a 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -4,6 +4,13 @@ - Status: draft - Date: 2026-06-08 - Tags: pipelines, modules, dsl, registry +- Version: 1.1 + +## Updates + +### Version 1.1 (2026-06-22) +- **Separate remote pipelines from remote workflows**: Workflows are treated separately by the [Workflow modules ADR](./20260608-workflow-modules.md). +- **Replace core workflow distinction with pipeline inclusion**: Instead of isolating the *core workflow* of a pipeline, the include syntax is extended to support *pipeline inclusion*, in which the `params` / `workflow` / `output` trio is imported and used like a named workflow. ## Summary From 655640c0e96c7ee4a552e4eac4a2cae950191d5e Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 23 Jun 2026 15:29:14 -0500 Subject: [PATCH 08/13] Apply suggestions from internal review Signed-off-by: Ben Sherman --- adr/20260608-remote-pipeline-inclusion.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index 17c544a08a..e16d419b85 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -9,7 +9,7 @@ ## Updates ### Version 1.1 (2026-06-22) -- **Separate remote pipelines from remote workflows**: Workflows are treated separately by the [Workflow modules ADR](./20260608-workflow-modules.md). +- **Separate remote pipelines from remote workflows**: Workflows are treated separately by the [Workflow modules ADR](20260608-workflow-modules.md). - **Replace core workflow distinction with pipeline inclusion**: Instead of isolating the *core workflow* of a pipeline, the include syntax is extended to support *pipeline inclusion*, in which the `params` / `workflow` / `output` trio is imported and used like a named workflow. ## Summary @@ -20,7 +20,7 @@ Add the ability to include a remote pipeline into a *meta-pipeline*. Nextflow supports reusing process definitions via remote *module* inclusion (e.g. `include { BWA_MEM } from 'nf-core/bwa/mem'`), but there is no standard mechanism to reuse an entire *pipeline* as a building block. Users must either fork and copy code, or compose/chain multiple `nextflow run` sessions which forfeits dataflow composition. -The [module system](20251114-module-system.md) and [workflow module](20260608-workflow-modules.md) ADRs define how standalone *processes* and *workflows* should be distributed as modules through the Nextflow registry. This ADR defines how *pipelines* -- workflows with a deployment shell -- should be composed into larger *meta-pipelines*. +The [module system](20251114-module-system.md) and [workflow modules](20260608-workflow-modules.md) ADRs define how standalone *processes* and *workflows* should be distributed as modules through the Nextflow registry. This ADR defines how *pipelines* -- workflows with a deployment shell -- should be composed into larger *meta-pipelines*. ## Goals @@ -82,7 +82,7 @@ workflow { Notes: -- The pipeline must included using the `workflow` keyword and aliased to a specific name (`RNASEQ`). +- The pipeline must be included using the `workflow` keyword and aliased to a specific name (`RNASEQ`). - The `params` block becomes the `take:` section and the `output` block becomes the `emit:` section. - The workflow is called using named arguments so that defaults can be omitted. - All outputs are either a `Channel` or wrapped as `Value`, allowing them to be used in regular dataflow logic. @@ -101,7 +101,7 @@ include { workflow as NFCORE_RNASEQ } from 'nf-core/rnaseq' When a pipeline is included from the registry, it is vendored into the including project under `pipelines///`. Included pipelines are isolated -- each included pipeline has its own `modules/` directory. This way, two pipelines can use different versions of the same module without compromising reproducibility. -Included pipelines should be committed to the meta-pipeline repository. The pipeline should have a *pipeline spec* (`nextflow_spec.json`) which specifies the pipeline version, so that Nextflow can track local changes. +Included pipelines should be committed to the meta-pipeline repository. The pipeline version and checksum should be saved in a helper file (`.pipeline-info`) so that Nextflow can track local changes. ### Best practices for including pipelines @@ -211,6 +211,7 @@ Pipeline chaining can be practical for certain use cases, such as simple chains - Related: [Module system](20251114-module-system.md) - Related: [Workflow params](20250825-workflow-params.md) - Related: [Workflow outputs](20251020-workflow-outputs.md) +- Related: [Workflow modules](20260608-workflow-modules.md) ## Appendix @@ -349,3 +350,12 @@ process { ``` Both the meta-pipeline developer and users can override whatever they want from config. + +In practice, the meta-pipeline will likely need to recreate the configuration shell used by the inner pipelines: + +- Config params (`outdir`, `publish_dir_mode`, `max_cpus`, etc) +- Resource settings (`cpus`, `memory`, `time`, etc) +- Environment profiles (executors, software dependencies, test profiles) +- Reports (execution, timeline, trace) +- Manifest (name, authors, description, etc) +- Plugins From 5399bcbeb5718f044f94768552a0e31acdd97422 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 23 Jun 2026 16:12:45 -0500 Subject: [PATCH 09/13] Fix appendix example to use record channel param for rnaseq Signed-off-by: Ben Sherman --- adr/20260608-remote-pipeline-inclusion.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index e16d419b85..aa543b30fb 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -267,13 +267,12 @@ output { ```groovy // nf-core/rnaseq — main.nf params { - input: Path // samplesheet + input: Channel // samplesheet aligner: String = 'star_salmon' fasta: Path } workflow { main: - ch_samples = channel.fromPath(params.input).splitCsv() rnaseq = // ... publish: multiqc = rnaseq.multiqc @@ -328,7 +327,7 @@ output { Notes: -- **The handoff is a channel, not a file.** A pipeline chain blocks until fetchngs finishes before rnaseq starts. Here, `ch_samples` is a live channel: rnaseq begins aligning each sample the moment fetchngs emits it. This is the dataflow composition that motivates the meta-pipeline. +- **The handoff is a channel, not a file.** rnaseq declares its samplesheet input as `Channel` instead of `Path`, so that it can be executed directly from a CSV samplesheet or called by a meta-pipeline with a live channel. This new behavior is described in the [Workflow modules ADR](20260608-workflow-modules.md). It allows rnaseq to begin aligning each sample as soon as it is emitted by fetchngs, whereas a pipeline chain would block until fetchngs finished completely. - **Params and outputs are replicated, not inherited.** `--input` and `--strandedness` are declared in the meta-pipeline's own `params` block and passed explicitly into the core workflows. Similarly, any outputs must be declared as such in the meta-pipeline's `output` block. The included pipelines do not contribute any of their own params, entry workflows, or output blocks. From 691173724a2bca5428c042ce4c6baf9f0754490e Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 23 Jun 2026 18:31:02 -0500 Subject: [PATCH 10/13] Support including params/output block as record types to minimize meta-pipeline boilerplate Signed-off-by: Ben Sherman --- adr/20260608-remote-pipeline-inclusion.md | 60 +++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index aa543b30fb..03a814c6b5 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -358,3 +358,63 @@ In practice, the meta-pipeline will likely need to recreate the configuration sh - Reports (execution, timeline, trace) - Manifest (name, authors, description, etc) - Plugins + +### Reducing params/output boilerplate + +In the example above, the meta-pipeline re-declares the params and outputs from each included pipeline. This boilerplate can be avoided by importing each pipeline's `params` block and `output` block as *record types*: + +```groovy +include { + params as FetchngsParams; + workflow as NFCORE_FETCHNGS +} from 'nf-core/fetchngs' + +include { + params as RnaseqParams; + workflow as NFCORE_RNASEQ; + output as RnaseqOutput +} from 'nf-core/rnaseq' + +params { + fetchngs: FetchngsParams // input + strandedness: String = 'auto' // unique to meta-pipeline + rnaseq: RnaseqParams // input, aligner, fasta +} + +workflow { + main: + // fetch FASTQ samples from NCBI SRA + fetchngs = NFCORE_FETCHNGS( params.fetchngs ) + + // adapt fetchngs output to rnaseq input (add strandedness) + ch_samples = fetchngs.samples.map { r -> + r + record(strandedness: params.strandedness) + } + + // perform RNAseq analysis (ch_samples overrides params.rnaseq.input) + rnaseq = NFCORE_RNASEQ( params.rnaseq + record(input: ch_samples) ) + + publish: + rnaseq = rnaseq +} + +output { + rnaseq: RnaseqOutput {} +} +``` + +`RnaseqParams` is a *partial record type* -- all of its fields are nullable and defaulted fields keep their defaults. The user can provide any rnaseq param as `--rnaseq.`, the meta-pipeline can override specific params (`params.rnaseq + record(input: ch_samples)`), and the `NFCORE_RNASEQ()` call validates that all required params are present. + +`RnaseqOutput` is a record type of the rnaseq outputs which preserves their output directives (`path`, `index`). Declaring a top-level output with this type (`rnaseq: RnaseqOutput`) is equivalent to redeclaring each rnaseq output. + +This way, the developer only needs to declare one param for each included pipeline (`fetchngs: FetchngsParams`, `rnaseq: RnaseqParams`), and one output for each pipeline whose outputs should be published (`rnaseq: RnaseqOutput`). + +Notes: + +- `rnaseq.input` is always overridden by the dataflow, so a user-supplied value (`--rnaseq.input`) would be silently discarded. Nextflow can warn when a phantom input is set. + +- `rnaseq.fasta` must still be provided by the user, but the error surfaces at the `NFCORE_RNASEQ()` call rather than at launch. + +- The fetchngs outputs were not published in the base example, so they are not published here either. + +- Output record types are all-or-nothing. If the developer wants to publish only some outputs or publish them in a different way, they need to redeclare each output like normal. From 10db06bef820b59d12ac133d64d02426b59f79a3 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Thu, 25 Jun 2026 11:21:17 -0500 Subject: [PATCH 11/13] Add link to community issue [ci skip] Signed-off-by: Ben Sherman --- adr/20251017-typed-processes.md | 4 ++-- adr/20260306-record-types.md | 4 ++-- adr/20260608-remote-pipeline-inclusion.md | 1 + 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/adr/20251017-typed-processes.md b/adr/20251017-typed-processes.md index bbc9e9c24f..5e283fc217 100644 --- a/adr/20251017-typed-processes.md +++ b/adr/20251017-typed-processes.md @@ -399,6 +399,6 @@ This pattern provides the best balance of continuity with the old way and consis ## Links - [Nextflow standard types](https://nextflow.io/docs/latest/reference/stdlib-types.html) -- Community issues: #1694, #2678 +- Community issues: [#1694](https://github.com/nextflow-io/nextflow/issues/1694), [#2678](https://github.com/nextflow-io/nextflow/issues/2678) - Related nf-core discussion: https://github.com/nf-core/modules/issues/4311 -- Original implementation: #4553 +- Original implementation: [#4553](https://github.com/nextflow-io/nextflow/pull/4553) diff --git a/adr/20260306-record-types.md b/adr/20260306-record-types.md index f5248085ed..db5f5a516c 100644 --- a/adr/20260306-record-types.md +++ b/adr/20260306-record-types.md @@ -377,9 +377,9 @@ The `record()` approach works "out of the box", and it isn't much more verbose, ## Links -- Community issues: #2085, #2127 +- Community issues: [#2085](https://github.com/nextflow-io/nextflow/issues/2085), [#2127](https://github.com/nextflow-io/nextflow/issues/2127) - Related nf-core discussion: https://github.com/nf-core/modules/issues/4311 -- Original implementation: #4553 +- Original implementation: [#4553](https://github.com/nextflow-io/nextflow/pull/4553) - nf-core/fetchngs POC: https://github.com/nf-core/fetchngs/pull/309 - Inspired by: [Simple Made Easy](https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md) - Type systems: [Nominal typing](https://en.wikipedia.org/wiki/Nominal_type_system) vs [Structural typing](https://en.wikipedia.org/wiki/Structural_type_system) vs [Duck typing](https://en.wikipedia.org/wiki/Duck_typing) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-remote-pipeline-inclusion.md index 03a814c6b5..8d89deffb6 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-remote-pipeline-inclusion.md @@ -208,6 +208,7 @@ Pipeline chaining can be practical for certain use cases, such as simple chains ## Links +- Community issues: [#6474](https://github.com/nextflow-io/nextflow/issues/6474) - Related: [Module system](20251114-module-system.md) - Related: [Workflow params](20250825-workflow-params.md) - Related: [Workflow outputs](20251020-workflow-outputs.md) From 084cba064b7b66303f8af7856ac19b46fe2b2d0f Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 15 Jul 2026 14:47:30 -0500 Subject: [PATCH 12/13] Move workflow modules ADR to separate PR [ci skip] Signed-off-by: Ben Sherman --- adr/20260608-workflow-modules.md | 199 ------------------------------- 1 file changed, 199 deletions(-) delete mode 100644 adr/20260608-workflow-modules.md diff --git a/adr/20260608-workflow-modules.md b/adr/20260608-workflow-modules.md deleted file mode 100644 index fe9b59088f..0000000000 --- a/adr/20260608-workflow-modules.md +++ /dev/null @@ -1,199 +0,0 @@ -# Workflow modules - -- Authors: Ben Sherman -- Status: draft -- Date: 2026-06-08 -- Tags: workflows, modules, registry - -## Summary - -Add the ability to include a workflow as a module from the Nextflow registry. - -## Problem Statement - -A Nextflow *module* is currently defined as a standalone process definition (with corresponding spec file). The [module system ADR](20251114-module-system.md) defines how these modules are published, distributed, and executed through the Nextflow registry. - -There is a similar need to share and re-use workflows / subworkflows. The nf-core community has curated a collection of re-usable subworkflows in the [nf-core/modules](https://github.com/nf-core/modules) repository. - -The module system should be extended to include both standalone *processes* and *workflows*. - -## Goals - -- **Workflow inclusion**: allow workflows to be included from the registry using the same include syntax as for modules. - -- **Workflow execution**: allow workflows to be executed directly via `nextflow module run`. - -- **Reuse existing infrastructure**: the module system already defines a registry API and CLI for publishing and installing modules. Treating workflows as a separate concept would require duplicating much of this infrastructure. - -## Non-goals - -- **Pipeline inclusion**: *pipelines* are distinct from *workflows* -- they specify params, publishing, and config, not just workflow logic. - -## Decision - -Extend the definition of *module* to include both standalone *processes* and standalone *workflows*. Allow workflow modules to be published, installed, included, and executed via the Nextflow registry. Store the included workflow in the including repository under `workflows///`. - -## Core Capabilities - -### Workflow modules vs process modules - -A module can refer to a *workflow module* or *process module*, depending on whether it defines a workflow or process. - -Workflow modules can be published to and queried from the Nextflow registry using the same modules API and `nextflow module` CLI. Workflow modules also have the same directory structure as process modules. - -Since workflow modules are just modules, a process module and workflow module cannot have the same name in a module namespace. - -### Module spec - -The module spec is extended as follows in order to support standalone workflows: - -- A `kind` field to distinguish between workflow modules (`kind: Workflow`) and process modules (`kind: Process`). - -- A `requires.modules` field to specify *transitive dependencies*, since a workflow can depend on other processes and workflows. - -- Certain fields (`topics`, `tools`) cannot be specified for workflow modules because they are not applicable. - -For example: - -```yaml -name: nf-core/fastq_align_star -kind: Workflow -version: 0.0.0-4e3e10e -description: Align reads to a reference genome using bowtie2 then sort with samtools -authors: - - "@JoseEspinosa" -license: MIT -requires: - nextflow: ">=24.04.0" - modules: - - nf-core/star/align@0.0.0-4e3e10e - - nf-core/samtools/sort@0.0.0-4e3e10e - - nf-core/samtools/index@0.0.0-4e3e10e - - nf-core/samtools/stats@0.0.0-4e3e10e - - nf-core/samtools/idxstats@0.0.0-4e3e10e - - nf-core/samtools/flagstat@0.0.0-4e3e10e - - nf-core/bam_sort_stats_samtools@0.0.0-4e3e10e -``` - -### Workflow inclusion and storage - -Workflow modules use the same include syntax and naming conventions as process modules: - -```groovy -// process -include { BWA_MEM } from 'nf-core/bwa/mem' - -// workflow -include { FASTQ_ALIGN_STAR } from 'nf-core/fastq_align_star' -``` - -When a workflow is included, it is vendored into the including project under `workflows///`. Included workflows should be committed to the including repository. - -Transitive dependencies (specified by `requires.modules`) should also be installed in the `modules/` and `workflows/` directories alongside the included workflow. Since modules are flattened, it is not possible for a pipeline to use two different versions of the same process or workflow. - -### Workflow execution - -Typed workflows can be executed directly by inferring the `params` and `output` blocks from the `take:` and `emit:` sections. - -For example, given the following workflow: - -```groovy -workflow RNASEQ { - take: - samples: Channel - index: Path - - main: - ch_aligned = ALIGN(samples, index) - multiqc_report = MULTIQC(ch_aligned.collect()) - - emit: - aligned: Channel = ch_aligned - multiqc_report: Path = multiqc_report -} - -record Sample { - id: String - fastq_1: Path - fastq_2: Path -} - -record AlignedSample { - id: String - bam: Path - bai: Path -} -``` - -The user can run the workflow directly as follows: - -```bash -nextflow module run rnaseq.nf \ - --samples input.csv \ - --index index.fasta -``` - -Nextflow executes the `RNASEQ` workflow as if it were wrapped in the following entry workflow: - -```groovy -params { - samples: Channel - index: Path -} - -workflow { - main: - rnaseq = RNASEQ(params.samples, params.index) - - publish: - aligned = rnaseq.aligned - multiqc_report = rnaseq.multiqc_report -} - -output { - aligned: Channel {} - multiqc_report: Path {} -} -``` - -This way, the user can run a workflow directly without having to write an entry workflow for it. - -Direct execution requires the ability to load an input channel directly from an index file (samplesheet). This can be done by loading the samplesheet data based on the file extension (CSV, JSON, YAML), casting each record to the given record type, and loading the collection as a channel. The data-loading function may be extended via plugin to support additional formats (e.g. Parquet). - -The channel input can use a generic type such as `Map` or `Record`, or a custom record type to enable further validation. In the above example, using the `Sample` type ensures that each samplesheet row is validated against the record fields and the `fastq_1` and `fastq_2` columns are treated as file paths. - -When executing a named workflow directly, output files are not published to an output directory. Instead, the workflow output printed by Nextflow simply refers to output files by their work directory path. - -## Alternatives - -### Workflows vs processes - -One alternative is to treat workflows as a separate concept from modules, restricting the definition of *module* to only include standalone processes. - -However, the broader meaning of *module* is a re-usable component, and both processes and workflows are re-usable components, so it makes more sense to extend the module system rather than introduce a parallel system for workflows. A parallel system would also require duplicating a lot of existing code (registry API, CLI, etc). - -Process modules and workflow modules can be distinguished by different specializations of the module spec (`kind: Process` vs `kind: Workflow`) and different storage locations (`modules/` vs `workflows/`). - -### Workflows vs subworkflows - -The nf-core community makes a distinction between *workflows* and *subworkflows*: - -- Subworkflows are stored in the `subworkflows/` directory and can be shared across pipelines. - -- Workflows are stored in the `workflows/` directory and are owned by a specific pipeline. - -Nextflow makes no such distinction, as there is no functional difference between a workflow and a subworkflow. Both are stored in the `workflows/` directory and both can be published and installed through the Nextflow registry. - -The Nextflow registry makes it easier for pipelines to share and reuse workflows, since the pipeline can be the source-of-truth rather than having to surrender ownership to the `nf-core/modules` repository. - -### Workflows vs pipelines - -One alternative is to treat all workflows as pipelines, restricting the definition of *module* to only include standalone processes. - -Aside from the problems mentioned under [Workflows vs processes](#workflows-vs-processes), this approach conflates two commonly-understood concepts: a *pipeline* which is an end-to-end analysis, and a *workflow* or *subworkflow* which is a component in a larger analysis. It is more congruent with conventional understanding to treat workflows (subworkflows) as a separate concept from pipelines. - -## Links - -- [nf-core terminology](https://nf-co.re/docs/usage/getting_started/terminology) -- Refines: [Module system](20251114-module-system.md) -- Related: [Remote pipeline inclusion](20260608-remote-pipeline-inclusion.md) From ef692dafac906bbe6b026148af2c7451a28dc71f Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 15 Jul 2026 18:33:05 -0500 Subject: [PATCH 13/13] Reframe ADR as "pipeline composition" [ci skip] Signed-off-by: Ben Sherman --- ...on.md => 20260608-pipeline-composition.md} | 32 +++++++++++-------- 1 file changed, 19 insertions(+), 13 deletions(-) rename adr/{20260608-remote-pipeline-inclusion.md => 20260608-pipeline-composition.md} (91%) diff --git a/adr/20260608-remote-pipeline-inclusion.md b/adr/20260608-pipeline-composition.md similarity index 91% rename from adr/20260608-remote-pipeline-inclusion.md rename to adr/20260608-pipeline-composition.md index 8d89deffb6..74d92b66ff 100644 --- a/adr/20260608-remote-pipeline-inclusion.md +++ b/adr/20260608-pipeline-composition.md @@ -1,30 +1,34 @@ -# Remote pipeline inclusion +# Pipeline composition - Authors: Ben Sherman -- Status: draft +- Status: proposed - Date: 2026-06-08 - Tags: pipelines, modules, dsl, registry -- Version: 1.1 +- Version: 1.2 ## Updates +### Version 1.2 (2026-07-13) + +- **Reframe as pipeline composition**: the core feature is the ability to compose pipelines in a Nextflow-native manner. Meta-pipelines are the artifact. Remote pipelines and workflow modules are largely natural extensions of the module registry. + ### Version 1.1 (2026-06-22) - **Separate remote pipelines from remote workflows**: Workflows are treated separately by the [Workflow modules ADR](20260608-workflow-modules.md). - **Replace core workflow distinction with pipeline inclusion**: Instead of isolating the *core workflow* of a pipeline, the include syntax is extended to support *pipeline inclusion*, in which the `params` / `workflow` / `output` trio is imported and used like a named workflow. ## Summary -Add the ability to include a remote pipeline into a *meta-pipeline*. +Provide a way to compose pipelines using regular dataflow logic. ## Problem Statement -Nextflow supports reusing process definitions via remote *module* inclusion (e.g. `include { BWA_MEM } from 'nf-core/bwa/mem'`), but there is no standard mechanism to reuse an entire *pipeline* as a building block. Users must either fork and copy code, or compose/chain multiple `nextflow run` sessions which forfeits dataflow composition. +Processes and workflows can be composed into a larger workflow using dataflow logic. However, pipelines cannot be composed in the same way. The only way to call a pipeline is via `nextflow run`, which does not allow for dataflow composition. -The [module system](20251114-module-system.md) and [workflow modules](20260608-workflow-modules.md) ADRs define how standalone *processes* and *workflows* should be distributed as modules through the Nextflow registry. This ADR defines how *pipelines* -- workflows with a deployment shell -- should be composed into larger *meta-pipelines*. +This ADR defines how a pipeline can be included like a named workflow and composed with other pipelines with dataflow logic. ## Goals -- **Preserve dataflow composition**: the included pipeline participates in the meta-pipeline's dataflow graph (same session, same DAG, same work dir), enabling incremental reaction to emitted outputs. +- **Preserve dataflow composition**: the included pipeline participates in the including pipeline's dataflow graph (same session, same DAG, same work dir), enabling incremental reaction to emitted outputs. - **Preserve reproducibility**: an included pipeline should produce the exact same results as it would when executed directly. Transitive dependencies should not be silently altered to reduce duplication. @@ -38,7 +42,9 @@ The [module system](20251114-module-system.md) and [workflow modules](20260608-w ## Decision -Allow pipelines to be published and installed through the Nextflow registry, using the same namespacing conventions as modules. Store the included pipeline in the meta-pipeline repository under `pipelines///` with its own subdirectories for modules and workflows. Provide a way to include an entire pipeline (`params` block, entry workflow, `output` block) as a named workflow to facilitate workflow composition. +Provide a way to include an entire pipeline (`params` block, entry workflow, `output` block) as a named workflow to facilitate workflow composition. + +Allow pipelines to be published and installed through the Nextflow registry, using the same namespacing conventions as modules. Store the included pipeline in the including repository under `pipelines///` with its own `modules/` directory for transitive dependencies. ## Core Capabilities @@ -105,7 +111,7 @@ Included pipelines should be committed to the meta-pipeline repository. The pipe ### Best practices for including pipelines -Pipeline inclusion only captures the pipeline's main script and included modules -- it does not capture external context such as config or the `lib` directory. As a result, the pipeline should be written in a way that works when included by a meta-pipeline: +Pipeline inclusion only captures the pipeline's main script and included modules -- it does not capture external context such as config or the `lib` directory. As a result, the pipeline should be written in a way that works when included in another pipeline: 1. Pipeline parameters should be defined in the script `params` block. The config should only declare *config params* (params that only affect config settings). @@ -137,7 +143,7 @@ Alternatively, these core plugin dependencies could be specified in the pipeline ### Pipeline chaining -An alternative to a meta-pipeline is a *pipeline chain*, in which multiple Nextflow pipelines are called in sequence via `nextflow run`. +An alternative to pipeline composition is a *pipeline chain*, in which multiple Nextflow pipelines are called in sequence via `nextflow run`. For example, a fetchngs -> rnaseq pipeline chain can be implemented in a shell script: @@ -209,9 +215,9 @@ Pipeline chaining can be practical for certain use cases, such as simple chains ## Links - Community issues: [#6474](https://github.com/nextflow-io/nextflow/issues/6474) -- Related: [Module system](20251114-module-system.md) - Related: [Workflow params](20250825-workflow-params.md) - Related: [Workflow outputs](20251020-workflow-outputs.md) +- Related: [Module system](20251114-module-system.md) - Related: [Workflow modules](20260608-workflow-modules.md) ## Appendix @@ -233,12 +239,12 @@ fetchngs-rnaseq/ └── pipelines/ └── nf-core/ ├── fetchngs/ + │ ├── .pipeline-info │ ├── main.nf - │ ├── nextflow_spec.json │ └── modules/ └── rnaseq/ + ├── .pipeline-info ├── main.nf - ├── nextflow_spec.json └── modules/ ```