Add `nextflow modules` command to install remote modules

[edmundmiller]

New feature

include { BOWTIE_ALIGN } from "github:nf-core/modules/modules/bowtie/align@somehashabcd"

Go:

import "github.com/google/uuid"

Snakemake:

github("owner/repo", path="workflow/Snakefile", tag="v1.0.0")

Deno:

import chalk from "npm:chalk@5";

Groovy (Grapes):

@Grab(group = 'com.puravida-software.groogle', module = 'groogle-sheet', version = '3.0.0')
@GrabExclude(group = 'org.codehaus.groovy', module = '*')

Example

Usage scenario

Baked in sharing of modules and module version control, instead of tools like nf-core to manage updating and sharing them.

Suggest implementation

That's above my pay grade. I'd imagine it would work very similar to plugins, check before the pipeline starts that all the modules are there. Then store them in .nextflow/modules or somewhere.

I guess that wouldn't be too hard.

1. Find all the remote file requests
2. Check for them in .nextflow/modules
3. If the modules are not there, download them
4. Nextflow replaces the lines behind the scenes linking to .nextflow/modules/nf-core/modules/modules/bowtie/align/main.nf

Same thing as plugins and downloading pipelines from repos, I think.

[adamrtalbot]

Given how successful these have been in nf-core, I think this is worth considering.

I'd be tempted to store the cache under ${projectDir}/.nextflow/modules so that it could be checked under version control of one repo, similar to how nf-core does it. This would mean users could just pull one repo to run a pipeline and wouldn't be reliant on additional calls out to the internet (e.g. if the remote disappeared). The downside is they would have 38 duplicates of SAMTOOLS_SORT and it's more elegant to have a central storage place. Perhaps an additional command for fixing the module by copying into the project directory could be included.

[edmundmiller]

Supporting patching like nf-core would be nice as well.

[bentsherman]

I've thought about creating a module registry (similar to the plugin registry) that would encode this exact information -- repo, path, commit hash -- and would allow for a broader ecosystem of modules that don't have to meet nf-core standards. I imagined that the nextflow CLI would have a modules command that would basically work like nf-core modules, allowing you to pull modules into your pipeline.

Being able to pull the modules at runtime would be convenient, but the challenge is maintaining a local cache. You could probably cover 95% of modules just by downloading nf-core/modules, but how do you manage different revisions of the same repo being pulled? Using bare git repos (#2870) should make this idea somewhat easier.

However, due to my experience with npm, I'm hesitant to create yet another package manager with recursive dependencies and its associated horrors. The nf-core modules approach forces the developer to think through their entire module hierarchy up front. It's more tedious, but I wonder whether the convenience of loading modules at runtime will outweigh the additional complexity that it would create.

A simple middle-ground could be to just add the nextflow modules command, which works just like nf-core modules, but allows you to specify a repo+path+hash instead of an nf-core/module. You still have to maintain the module in your pipeline code, but you aren't limited to nf-core anymore. No need for a module registry, and maybe from there it would be easier to assess the extra usefulness of loading modules at runtime.

[pditommaso]

Agree with Ben. Also, adding import from Git repos would create horrible problems with non-public repositories (how to authenticate them?).

Another package manager for nextflow that was proposed for nextflow was WFPM, even tho not sure about the current status of the project

[adamrtalbot]

A simple middle-ground could be to just add the nextflow modules command, which works just like nf-core modules, but allows you to specify a repo+path+hash instead of an nf-core/module. You still have to maintain the module in your pipeline code, but you aren't limited to nf-core anymore. No need for a module registry, and maybe from there it would be easier to assess the extra usefulness of loading modules at runtime.

This is my preferred route. It's the most lightweight solution while being useful and side steps some of the problems (authentication, running offline, modifying the imported module) and we know the nf-core system works reasonably well but it would be good to extend it beyond the strict nf-core structure.

To make it more user friendly, you could perform the import based on the Nextflow files. From Edmund's first line, the following would be equivalent:

nextflow modules import BOWTIE_ALIGN 'github:nf-core/modules/modules/bowtie/align@somehashabcd'

Or...

nextflow modules import

# Where main.nf includes:
include { BOWTIE_ALIGN } from "github:nf-core/modules/modules/bowtie/align@somehashabcd"

[edmundmiller]

The nf-core modules approach forces the developer to think through their entire module hierarchy up front.

Completely agree. I think this is just moving the modules from nf-core.ymldirectly into the main.nf itself.

Subworkflows depending on other modules is another story...

(how to authenticate them?)

This may be naive, but I was imagining it would hook into providers.

[bentsherman]

I discovered today that this feature request has a long history. Notable issues include #1463 and nf-core/modules#8 .

Several approaches have been considered, including git submodules (and subtree?), the WFPM, and piggy-backing on an existing package manager like npm or conda. I was interested to see that many people were on board to use conda, but the nf-core modules approach won out over everything else because it just worked and was simpler.

I still think the best solution is to just incorporate nf-core modules into Nextflow, just wanted to add the history of this discussion for posterity.

As for this shorthand:

include { BOWTIE_ALIGN } from "github:nf-core/modules/modules/bowtie/align@somehashabcd"

I remain reluctant to implement something like this. I know that we can do it (including the authentication piece), but the question is whether we should.

Importing remote modules directly would remove the need to have a separate modules.json file to keep track of which modules were installed. But, it would open up several worm-filled cans.

You could have different versions of the same tool installed under different subworkflows.

Offline usage, and anything that requires inspecting the module dependency tree, would be more complicated. We would need another Nextflow command to parse the pipeline script and recursively fetch the remote modules, just to figure out which modules are required. I think I prefer having all of the module dependencies listed in one place, even if it requires another file.

[pditommaso]

I agree with Ben's analysis. Remote modules are not supported by design.

[bentsherman]

However we are still discussing the idea of a nextflow modules command to allow importing remote modules during development.