ADR: Pipeline composition

[bentsherman]

This PR adds an ADR for remote pipeline inclusion, aka "meta-pipelines".

It describes an approach for including remote pipelines into a meta-pipeline in a way that preserves dataflow concurrency between pipeline inputs/outputs.

It discusses alternative approaches such as pipeline chaining / nf-cascade and why they don't satisfy certain use cases (preserving dataflow concurrency).

It also walks through a basic example of fetchngs -> rnaseq.

[ewels]

Great write up, thanks for this Ben!

As you might expect, I'm most concerned about the params. You characterise it as a one-off cost which is mitigated by LLMs, however that doesn't take into account updates to included pipelines (a core functionality with included modules). The params drift with updates would be dangerous and a constant source of dev work.

I'd still love to look into how we could bulk import nested config and apply it at root level. Even if it is a separate import + apply mechanism (eg. like config profiles in a sense?). I think without it, the use of the meta pipeline functionality is substantially limited.

[pinin4fjords]

As you might expect, I'm most concerned about the params.

Agreed. Feel like we need some sort of auto-import of the params of child workflows, so e.g. they appear automatically in Platform, and I could say e.g. meta.rnaseq.pseudoaligner = 'kallisto' in the meta pipeline's nextflow.config to override.

Then some auto-assembly of docs as well.

Basically we need to standardise at the nextflow level where a bunch of the non-nextflow pieces need to live.

[adamrtalbot]

I agree with this. The added complexity is enormous.

[bentsherman]

@ewels @pinin4fjords @adamrtalbot

Pulling everyone into this thread to talk about auto-inheritance

As you might expect, I'm most concerned about the params. You characterise it as a one-off cost which is mitigated by LLMs, however that doesn't take into account updates to included pipelines (a core functionality with included modules). The params drift with updates would be dangerous and a constant source of dev work.

That's fair, but not my main point. The core problem is this -- if you want to preserve dataflow concurrency between pipelines, then you can't really just auto-import params into the meta-pipeline. You have to define which params are replaced with inter-pipeline wiring vs exposed to the top-level. That amounts to just writing the meta-workflow.

The development overhead is what it is. I suggest the AI skill just as an idea. I'm sure it could also handle updates. All of that is better than having loads of hidden behavior that makes the meta-pipeline impossible to reason about

I'd still love to look into how we could bulk import nested config and apply it at root level. Even if it is a separate import + apply mechanism (eg. like config profiles in a sense?). I think without it, the use of the meta pipeline functionality is substantially limited.

Not sure I understand this point. Most of the config is just standard boilerplate, so it doesn't make sense to auto-import it because you will just get lots of duplicate config

Unless you are talking about ext config. That will depend on whether we can move the default ext settings into the process definition

[bentsherman]

Building on what Adam said:

In a scenario where I update my workflow from v1.1 to v1.2, an update to params should be explicit in the input block, not implicit and I hope it doesn't change too much.

The nice thing about an explicit meta-pipeline definition is that when I update the included pipeline, the linter / language server will immediately pick up on any inconsistencies, because it's just regular code. I'm not sure the tooling would be able to do that if there was a lot of implicit behavior

[adamrtalbot]

Side note - I would remove MultiQC from all nf-core pipelines and put them in the metapipelines, i.e. no MultiQC repeats, but that's a matter of opinion.

FETCHNGS(ch_inputs)
RNASEQ(fetchngs.out)
MULTIQC(RNASEQ.out.qc_files)

[bentsherman]

I was wondering about that. Wasn't sure if you would want a meta-pipeline to produce one multiqc report per pipeline or just one for the whole thing

[ewels]

Having unpredictable global scope params blocks is just weird and if we were designed Nextflow today we would never include this behaviour. In other languages, globals need to be used with caution and are generally not advised.

@adamrtalbot agreed, I never said global. I would love it if the pipeline config is imported within a dedicated scope and treated as a baseline default. Then the import-ing pipeline can override anything, but doesn't need to duplicate config that isn't being changed.

Doing this would not be trivial. The only way I can think of is to do something fairly radical like rendering the config at import time and saving that to a locked config file somewhere. Or some other crazy mechanism.

[adamrtalbot]

Having unpredictable global scope params blocks is just weird and if we were designed Nextflow today we would never include this behaviour. In other languages, globals need to be used with caution and are generally not advised.

@adamrtalbot agreed, I never said global. I would love it if the pipeline config is imported within a dedicated scope and treated as a baseline default. Then the import-ing pipeline can override anything, but doesn't need to duplicate config that isn't being changed.

Doing this would not be trivial. The only way I can think of is to do something fairly radical like rendering the config at import time and saving that to a locked config file somewhere. Or some other crazy mechanism.

Config or params? In my mind they are very different concepts, I was referring to parameters here.

[adamrtalbot]

Happy to rename the ADR to "remote workflow inclusion" to align with the workflow keyword.

I agree with this. They're all workflows*, the only thing that separates a "pipeline" from a subworkflow is perception.

*except the anonymous entry workflow, which is where the sticky point about params and config comes in 😉

[ewels]

Config or params? In my mind they are very different concepts, I was referring to parameters here.

Ideally params, but might need to be config for all the ext stuff..?

Happy to rename the ADR to "remote workflow inclusion" to align with the workflow keyword.

Yeah as it stands I think this basically boils down to the functionality we already have with nf-core subworkflows, right? Which is quite far from what I think of as meta-pipelines. Still good to have and useful..

[bentsherman]

Yeah as it stands I think this basically boils down to the functionality we already have with nf-core subworkflows, right?

Can the nf-core tooling install a workflow from a pipeline repo? e.g. NFCORE_RNASEQ from nf-core/rnaseq? I think that is the main thing that this ADR adds

[bentsherman]

Version 1.1

• Added separate ADR for workflow modules based on Paolo's comment above. Keeping it in this PR for now so that we can discuss them together

• Workflows / subworkflows are just modules -- they use the same infrastructure (spec, registry API, CLI) as process modules, with some extensions for things like transitive deps

• Clarify main ADR to focus on remote pipeline inclusion via Nextflow registry -- pipelines will be treated as a separate concept from modules

• Replace the core workflow distinction with pipeline inclusion -- instead of isolating and importing only the core workflow of a pipeline, we extend the include syntax to support including an entire pipeline (the params / workflow / output trio) like a named workflow. See "Pipeline composition" in the ADR for an example

Next steps

Gonna keep exploring ways to minimize overhead for params and publishing (thread)

[bentsherman]

@pinin4fjords @adamrtalbot @ewels

Here is my latest attempt at a "best of both worlds" where the meta-pipeline is a unified Nextflow pipeline but doesn't require tons of developer overhead

Basically we allow the params block and output block to be imported like record types so that you only have to redeclare one param per pipeline, and you get all the defaults baked in

[pinin4fjords]

YES. Sweet sweet sugary goodness.

[adamrtalbot]

So params become a special type of input? Syntactic sugar over this:

take:
    RnaseqParams: Map = params
    fastqs: List[path]
    // etc

[bentsherman]

given a params block like this:

params {
    input: Channel<Sample>
    aligner: String = 'star_salmon'
    fasta: Path
}

importing it as RnaseqParams essentially gives you this:

record RnaseqParams {
    input: Channel<Sample>
    aligner: String = 'star_salmon'
    fasta: Path
}

importing the entry workflow gives you something like this:

workflow NFCORE_RNASEQ {
    take:
    params: RnaseqParams

    // ...

    emit:
    output: RnaseqOutput
}

[pditommaso]

Nice work on this — the design is coherent and the doc is refreshingly honest about its own sharp edges (the alternatives section and the best-practices caveats do a lot of the reviewer's work). Reusing include { workflow as X } instead of inventing a new keyword, per-pipeline modules/ isolation, and driving config through name-prefixed selectors are all the right calls. A few things I'd like to see resolved before this leaves draft.

1. The config gap needs real exploration, not a footnote.
"Inclusion captures the main script and modules but not config or lib" is currently a best-practices aside, but for the existing ecosystem this is the whole game — config is where behavior is actually governed and controlled (ext.args, resource labels, publish rules, profiles, plugins). The current best practices essentially ask authors to rewrite pipelines so config no longer carries logic, which almost no shipping pipeline does today. I don't think we should hand-wave this; it deserves a dedicated section that explores how much of the config shell can be captured/vendored vs. reconstructed, and what the minimum viable story is for a real pipeline. This is the make-or-break for adoption.

2. .pipeline-info vs nextflow_spec.json is inconsistent — and the choice matters.
The storage section pins version + checksum in .pipeline-info, but the appendix layout shows nextflow_spec.json per pipeline and no .pipeline-info, while Open Questions still treats the spec as undecided. These aren't interchangeable: one is an internal lockfile Nextflow manages, the other is a user-facing pipeline contract (params/outputs/requires). We should decide which is authoritative for versioning and make the examples consistent, otherwise it reads as two competing mechanisms.

3. The params→channel binding needs a general model.
The examples show input: Path becoming input: Channel<Sample> so a live channel can be passed in, but it's not clear how the mapping/binding is actually managed — it looks like it works for a few specific shapes (a samplesheet-like channel of records) rather than as a general rule. What's the binding semantics when a param is scalar vs. channel, single vs. queue, typed record vs. arbitrary? I'd want the general approach spelled out rather than inferred from one idealized case.

4. Consider a dedicated input block instead of overloading params.
Rather than overloading params to double as the composition interface (and carrying the type-conversion behavior above), it may be cleaner to introduce an explicit input definition — symmetric with output. params stays the CLI/config surface; input/output become the composition contract (take/emit). That symmetry would make the params→take / output→emit mapping explicit in the language rather than implicit, and gives the binding rules in (3) a natural home.

General suggestions

• Split out "workflow modules." The channel-typed-input behavior this ADR leans on really belongs to the workflow-modules ADR. I'd move it fully there and use that ADR to strengthen the input/output contract (the spec / meta.yml definition), so this ADR can depend on a stable interface rather than co-defining it.
• Descope params from "workflow modules." A workflow module is a module, so its params definition is already covered by the Module Parameters ADR (ADR: Module Parameters and Tool Arguments #7260) — no need to re-specify params handling here or in the workflow-modules ADR. Lean on that unified params {} model instead.
• Reframe as "pipeline composition." "Meta-pipeline" describes the artifact but not the intent. Framing the whole feature as pipeline composition makes the goal clearer and reads more naturally against "module inclusion" and "workflow modules."

Overall I'm in favor of the direction — dataflow composition over nested execution is the right choice. The above are mostly about making the contract explicit and being honest that config/environment is the hard part.

[adamrtalbot]

It looks like we're all trending towards two ADRs: workflow inclusion and pipeline composition.

[bentsherman]

Thanks @pditommaso for the reviews. I think we have converged on an approach, and I mostly just need to clarify some details in the ADRs

At this point I will move the workflow modules ADR into a separate PR, refine it based on your feedback, and begin implementation

I will also keep the workflow execution piece separate in a third PR (#7208) since it is not essential to meta-pipelines and deserves more dedicated treatment on the params -> take mapping

The syntax for pipeline composition can also be implemented independently of everything else, since a user could just copy+paste a pipeline into a meta-pipeline. But the full solution will require (1) workflow modules in the registry and (2) pipelines in the registry

[bentsherman]

Moved workflow modules ADR to #7342

Will update pipeline composition ADR soon based on latest feedback

[bentsherman]

ADR updated to focus on "pipeline composition" as the core feature. Remote pipeline inclusion is treated as a secondary goal.

Responding to Paolo's feedback in detail:

1. The config gap needs real exploration, not a footnote.

I added a blurb about which pieces of config likely need to be reconstructed. I can try to lay out a more concrete example later.

The main prerequisite is that a pipeline will need to adopt the params and output block. Otherwise the params/publishing logic is just too scattered for Nextflow to reason about it here. That migration alone should remove a lot of the burden around config.

Even ext doesn't need to be removed entirely -- the defaults just need to be moved into the process definition.

The rest of the config -- anything outside of process -- simply has to be reconstructed, because all other config settings are global. So you can't simply compose them like modules, you have to decide what you want the global settings to be and define them.

2. .pipeline-info vs nextflow_spec.json is inconsistent — and the choice matters.

Updated the appendix example to use .pipeline-info for now.

3. The params→channel binding needs a general model.

As the ADR states, this behavior is described in the workflow modules ADR. I will make sure to expand on the binding rules there. But it is the same behavior as used for executing a named workflow.

4. Consider a dedicated input block instead of overloading params.

I don't think users will want to have to define their params twice. But let's see how this evolves.

[mahesh-panchal]

What are your thoughts on this kind of structure where the container is dictated by the input.

https://github.com/mahesh-panchal/nextflow-quarto-website/blob/main/main.nf

In this example the module is the same, but one aliased module requires a container with R, while the other requires a container with Julia. The default container can't be set in the process definition and must be via the config in this case.

[adamrtalbot]

If we copy+paste the pipeline configuration into the repo on import and add an includeConfig "pipelines/nf-core/rnaseq/nextflow.config statement, this would automatically include the config from nf-core/rnaseq and make a lot of people happy?

[mahesh-panchal]

One thing that may cause issues here is then the priority of patterns and simple names potentially causing issues applying to other processes (and where those includeConfigs are placed will also determine priority), and there's also that issue where the actual process name configuration applies too all processes even if they've been aliased. It's a can of worms.

[bentsherman]

I think Adam basically covered it. You can either provide the container as a process input, or you can copy/include the config manually in the meta-pipeline (and make sure the selectors are specific enough, etc). Pick your poison

Either way, I don't think we can make config travel automatically with a pipeline/module because the config is a parallel system

[muffato]

Either way, I don't think we can make config travel automatically with a pipeline/module because the config is a parallel system

Would it be possible to restrict withName and withLabel directives to a certain workflow prefix when doing composition ?
A (sub/)workflow could ship its own nextflow.config with all the process {} config it needs for itself, possibly with very loose withName: SAMTOOLS_SORT rules.
When imported, even via include {worflow as IMPORTED_WF} ..., its nextflow.config would be considered and only applied to the subset of the entire pipeline.

[adamrtalbot]

What are your thoughts on this kind of structure where the container is dictated by the input.

https://github.com/mahesh-panchal/nextflow-quarto-website/blob/main/main.nf

In this example the module is the same, but one aliased module requires a container with R, while the other requires a container with Julia. The default container can't be set in the process definition and must be via the config in this case.

It seems like we always end up battling two conflicting things:

• workflows should be fully encapsulated so they can be imported into another workflow cleanly
• but also, I'd like it to include all the extra config and cruft not encapsulated in the workflow please

In this example, the workflow isn't self contained but reliant on a config file. So any Nextflow based solution could either magically auto import the config file in, or the author has to fully encapsulate the workflow before it can be cleanly imported.

I think this would be an effective compromise, allowing you to automatically include a config file from a repo, but it would often break things 😱

The default container can't be set in the process definition and must be via the config in this case.

It's a little unrelated to the main ADR, but I disagree with this, you could:

• write two processes with different container definitions
• use a closure in the container directive
• use a val input
• use one container that supports both for both processes

For whatever reason, you may not want to do one of these but the tradeoff is not being fully importable.

[mahesh-panchal]

It seems like we always end up battling two conflicting things:

* workflows should be fully encapsulated so they can be imported into another workflow cleanly

* but also, I'd like it to include all the extra config and cruft not encapsulated in the workflow please

But this is basically what we're allowed to do. We use what's given to us, and implement what's simplest/makes sense to us.

[adamrtalbot]

It seems like we always end up battling two conflicting things:

* workflows should be fully encapsulated so they can be imported into another workflow cleanly

* but also, I'd like it to include all the extra config and cruft not encapsulated in the workflow please

But this is basically what we're allowed to do. We use what's given to us, and implement what's simplest/makes sense to us.

Sorry, I'm not following, isn't that agreeing what I said? We want the ease of use of the config system, but the flexibility of the config system makes it much harder to include a pipeline within another pipeline?

[bentsherman]

What are your thoughts on this kind of structure where the container is dictated by the input.

@mahesh-panchal you can set the container based on a process input and supply it from a param:

process HELLO {
    container container_image

    input:
    val container_image

    // ...
}

include { HELLO as HELLO_1 } from './hello.nf'
include { HELLO as HELLO_2 } from './hello.nf'

workflow {
    HELLO_1( params.container_image_1 )
    HELLO_2( params.container_image_2 )
}