Repository navigation
feat: use nextflow lint -files-from stdin to avoid ARG_MAX - #12
Merged
Merged
Conversation
Replace the batching workaround with the native -files-from - option (Nextflow >= 26.04, nextflow-io/nextflow#6858). Filenames are streamed via stdin so they never hit the OS ARG_MAX limit, in a single JVM invocation regardless of file count. - hook: pipe filenames to 'nextflow lint -files-from -' instead of batching positional args across multiple JVM invocations - README: bump minimum Nextflow to v26.04+, document stdin streaming - CI: bump version matrix to 26.04.0, add a 500-file ARG_MAX test Closes nextflow-io/nextflow#6857 downstream. Generated by Claude Code
mashehu
approved these changes
Jul 2, 2026
| echo "Please install nextflow from https://www.nextflow.io/" | ||
| exit 1 | ||
| fi | ||
|
|
Contributor
There was a problem hiding this comment.
we should also check the nextflow version, I think, given we require a specific version now.
Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>
-files-from is only available in Nextflow 26.04+, so fail early with a clear message if an older version is on PATH rather than emitting a confusing 'unknown option' error. Warns (does not block) if the version cannot be parsed. Generated by Claude Code
mashehu
approved these changes
Jul 2, 2026
mashehu
left a comment
mashehu
left a comment
Contributor
There was a problem hiding this comment.
we could make a CI test for the version test, but probably not worth the bother
| # relies on to stream filenames via stdin (avoids the OS ARG_MAX limit). | ||
| required_major=26 | ||
| required_minor=4 | ||
| nf_version=$(nextflow -version 2>/dev/null | grep -oE 'version [0-9]+\.[0-9]+' | head -1 | grep -oE '[0-9]+\.[0-9]+' || true) |
Contributor
There was a problem hiding this comment.
Suggested change
| nf_version=$(nextflow -version 2>/dev/null | grep -oE 'version [0-9]+\.[0-9]+' | head -1 | grep -oE '[0-9]+\.[0-9]+' || true) | |
| nf_version=$(nextflow -version 2>/dev/null | sed -nE 's/.*version ([0-9]+\.[0-9]+).*/\1/p' || true) |
sorry, a nit-pick from all the version topics PRs in modules 😁
| nf_minor=${nf_version#*.} | ||
| # 10# forces base-10 so leading zeros (e.g. 04) are not read as octal | ||
| if (( 10#$nf_major < required_major || (10#$nf_major == required_major && 10#$nf_minor < required_minor) )); then | ||
| echo "Error: Nextflow >= ${required_major}.$(printf '%02d' "$required_minor") is required (found ${nf_version}); the -files-from option was added in ${required_major}.$(printf '%02d' "$required_minor")" |
Contributor
There was a problem hiding this comment.
Suggested change
| echo "Error: Nextflow >= ${required_major}.$(printf '%02d' "$required_minor") is required (found ${nf_version}); the -files-from option was added in ${required_major}.$(printf '%02d' "$required_minor")" | |
| echo "Error: Nextflow >= ${required_major}.$(printf '%02d' "$required_minor") is required (found ${nf_version})" |
again no need for techincal explanations (that might change in the future anyway)
mashehu
approved these changes
Jul 2, 2026
Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
nextflow lintfails withArgument list too long(E2BIG) when a large repo passes thousands of.nf/.configfiles as CLI arguments — the OSARG_MAXlimit (~2MB) is exceeded. Reported upstream in nextflow-io/nextflow#6857.The current hook works around this by batching files into groups of 200 and invoking
nextflow lintonce per batch. That spins up a fresh JVM per batch (1700 files → 9 JVM startups) and needs output filtering to hide per-batch success noise.nextflow-io/nextflow#6858 (Nextflow 26.04) added a native
-files-fromoption that reads paths from a file or stdin, keeping them offargventirely.What
printf '%s\n' "${files[@]}" | nextflow lint ${args} -files-from -. One JVM regardless of file count; no ARG_MAX ceiling; paths with spaces stay safe (newline-delimited). Exit code propagates viapipefail.25.04.6→26.04.0; add a 500-file test exercising the stdin path.🤖 Generated with Claude Code