diff --git a/.docusaurus_site/src/theme/prism-nextflow.js b/.docusaurus_site/src/theme/prism-nextflow.js index 09f335baa3..7bb425a798 100644 --- a/.docusaurus_site/src/theme/prism-nextflow.js +++ b/.docusaurus_site/src/theme/prism-nextflow.js @@ -71,7 +71,7 @@ export default function(Prism) { ], // Nextflow-specific directives and keywords 'nextflow-directive': { - pattern: /\b(?:accelerator|afterScript|arch|array|beforeScript|cache|clusterOptions|conda|container|containerOptions|cpus|debug|disk|echo|errorStrategy|executor|ext|fair|label|machineType|maxErrors|maxForks|maxRetries|memory|module|penv|pod|publishDir|queue|resourceLabels|resourceLimits|scratch|secret|spack|stageInMode|stageOutMode|storeDir|tag|time)\b/, + pattern: /\b(?:accelerator|afterScript|arch|array|beforeScript|cache|clusterOptions|conda|container|containerOptions|cpus|debug|disk|echo|errorStrategy|executor|ext|fair|hints|label|machineType|maxErrors|maxForks|maxRetries|maxSubmitAwait|memory|module|penv|pod|publishDir|queue|resourceLabels|resourceLimits|scratch|secret|spack|stageInMode|stageOutMode|storeDir|tag|time)\b/, alias: 'property' }, 'nextflow-keyword': { diff --git a/docs/reference/process.mdx b/docs/reference/process.mdx index 5a8fb24c41..94dadc62eb 100644 --- a/docs/reference/process.mdx +++ b/docs/reference/process.mdx @@ -5,7 +5,7 @@ description: Reference for all task properties, input and output qualifiers, and # Process reference -This section describes the task properties, input/output qualifiers, and directives available in [process][process-page] definitions. +[Process][process-page] definitions support the following task properties, input and output qualifiers, and directives: | Page | Description | | ---- | ----------- | diff --git a/docs/reference/process/directives.mdx b/docs/reference/process/directives.mdx index 49ca8a85df..2021af5d96 100644 --- a/docs/reference/process/directives.mdx +++ b/docs/reference/process/directives.mdx @@ -7,9 +7,9 @@ description: Reference for the directives that control the execution of process Directives are optional settings that affect the execution of the current process. They are declared at the top of a process definition, before any input, output, or script section. -By default, directives are evaluated when the process is defined. However, if the value is a dynamic string or closure, it will be evaluated separately for each task. See [Dynamic directives][dynamic-directives] for more information. +By default, directives are evaluated when the process is defined. However, if the value is a dynamic string or closure, it is evaluated separately for each task. See [Dynamic directives][dynamic-directives] for more information. -Directives can also be specified in a configuration file using the `process` config scope, which is the recommended way to configure a pipeline for a particular compute environment. See [Process configuration][config-process-scope] for more information. +You can also specify directives in a configuration file using the `process` config scope. This is the recommended way to configure a pipeline for a particular compute environment. See [Process configuration][config-process-scope] for more information. Some directives are only supported by specific executors. See [Executors][executor-page] for more information about each executor. @@ -17,7 +17,7 @@ The following directives are available: | Directive | Description | | --------- | ----------- | -| [`accelerator`][accelerator] | The number of hardware accelerators (e.g. GPUs) required by each task. | +| [`accelerator`][accelerator] | The number of hardware accelerators (for example, GPUs) required by each task. | | [`afterScript`][after-script] | A Bash snippet to execute after the task script. | | [`arch`][arch] | The CPU architecture to build the software required by each task. | | [`array`][array] | Submit tasks as job arrays, for executors that support it. | diff --git a/docs/reference/process/directives/accelerator.mdx b/docs/reference/process/directives/accelerator.mdx index d06f7811bd..336ccc70a3 100644 --- a/docs/reference/process/directives/accelerator.mdx +++ b/docs/reference/process/directives/accelerator.mdx @@ -5,7 +5,11 @@ description: Reference for the `accelerator` process directive. # `accelerator` -The `accelerator` directive defines the number of hardware accelerators (e.g. GPUs) required by each task execution. For example: +The `accelerator` directive defines the number of hardware accelerators (for example, GPUs) required by each task execution. + +## Usage + +For example: ```nextflow process hello { @@ -18,15 +22,13 @@ process hello { } ``` -The above example requests 4 GPUs of type `nvidia-tesla-k80` for each task. +This example requests 4 GPUs of type `nvidia-tesla-k80` for each task. :::note -This directive is only used by certain executors. Refer to the [Executors][executor-page] page to see which executors support this directive. +Additional options may be required to use accelerators. When using containers with GPUs, you must pass the GPU drivers through to the container. For Docker, this requires the option `--gpus all` in the `docker run` command. For Apptainer/Singularity, this requires the option `--nv`. Implementation details depend on the accelerator and container type. ::: -:::note -Additional options may be required to fully enable the use of accelerators. When using containers with GPUs, you must pass the GPU drivers through to the container. For Docker, this requires the option `--gpus all` in the `docker run` command. For Apptainer/Singularity, this requires the option `--nv`. The specific implementation details depend on the accelerator and container type being used. -::: +## Options The following options are available: @@ -34,7 +36,7 @@ The following options are available: The number of requested accelerators. -Specifying this directive with a number (e.g., `accelerator 4`) is equivalent to the `request` option (e.g., `accelerator request: 4`). +Specifying this directive with a number (for example, `accelerator 4`) is equivalent to the `request` option (for example, `accelerator request: 4`). ##### `type: String` @@ -45,6 +47,10 @@ The meaning of this option depends on the target execution platform. See the pla - [Google Cloud](https://cloud.google.com/compute/docs/gpus/) - [Kubernetes](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#clusters-containing-different-types-of-gpus) -This option is not supported for AWS Batch. You can control the accelerator type indirectly through the allowed instance types in your Compute Environment. See the [AWS Batch FAQs](https://aws.amazon.com/batch/faqs/?#GPU_Scheduling_) for more information. +This option is not supported for AWS Batch. You can control the accelerator type indirectly through the allowed instance types in your compute environment. See the [AWS Batch FAQs](https://aws.amazon.com/batch/faqs/?#GPU_Scheduling_) for more information. + +## Executor support + +This directive is only used by certain executors. See [Executors][executor-page] for the executors that support this directive. [executor-page]: ../../../executor diff --git a/docs/reference/process/directives/after-script.mdx b/docs/reference/process/directives/after-script.mdx index 47fed34869..1894df39c6 100644 --- a/docs/reference/process/directives/after-script.mdx +++ b/docs/reference/process/directives/after-script.mdx @@ -5,6 +5,10 @@ description: Reference for the `afterScript` process directive. # `afterScript` -The `afterScript` directive executes a custom (Bash) snippet immediately *after* the main process has run. This may be useful to clean up your staging area. +The `afterScript` directive executes a custom Bash snippet immediately *after* the main process has run. -When combined with the [container](./container) directive, the `afterScript` is executed outside the specified container. In other words, the `afterScript` is always executed in the host environment. +## Usage + +Use it to clean up your staging area. + +The `afterScript` is always executed in the host environment. When combined with the [container](./container) directive, the `afterScript` is executed outside the specified container. diff --git a/docs/reference/process/directives/arch.mdx b/docs/reference/process/directives/arch.mdx index c9672a6ab0..2337176d77 100644 --- a/docs/reference/process/directives/arch.mdx +++ b/docs/reference/process/directives/arch.mdx @@ -5,7 +5,11 @@ description: Reference for the `arch` process directive. # `arch` -The `arch` directive defines the CPU architecture to build the software in use by the process' task. For example: +The `arch` directive defines the CPU architecture used to build the software required by each task. + +## Usage + +For example: ```nextflow process blast { @@ -19,19 +23,20 @@ process blast { } ``` -The example above declares that the CPU generic architecture is `linux/x86_64` (X86 64 bit), and more specifically that the microarchitecture is `cascadelake` (a specific generation of Intel CPUs). +This example declares that the generic CPU architecture is `linux/x86_64` (X86 64 bit), and more specifically that the microarchitecture is `cascadelake` (a specific generation of Intel CPUs). + +This directive is used by the following Nextflow features: -This directive is currently used by the following Nextflow functionalities: +- The [spack](./spack) directive, to build microarchitecture-optimized applications. +- The [Wave][wave-page] service, to build containers for one of the generic families of CPU architectures. +- The `spack` strategy within [Wave][wave-page], to optimize the container builds for specific CPU microarchitectures. -- by the [spack](./spack) directive, to build microarchitecture-optimized applications; -- by the [Wave][wave-page] service, to build containers for one of the generic families of CPU architectures (see below); -- by the `spack` strategy within [Wave][wave-page], to optimize the container builds for specific CPU microarchitectures. +The following values are allowed, grouped by equivalent family (multiple choices are available for compatibility): -Allowed values for the `arch` directive are as follows, grouped by equivalent family (choices available for the sake of compatibility): - X86 64 bit: `linux/x86_64`, `x86_64`, `linux/amd64`, `amd64` - ARM 64 bit: `linux/aarch64`, `aarch64`, `linux/arm64`, `arm64`, `linux/arm64/v8` - ARM 64 bit, older generation: `linux/arm64/v7` -Examples of values for the architecture `target` option are `cascadelake`, `icelake`, `zen2` and `zen3`. See the [Spack documentation](https://spack.readthedocs.io/en/latest/basic_usage.html#support-for-specific-microarchitectures) for the full and up-to-date list of meaningful targets. +Examples of values for the architecture `target` option are `cascadelake`, `icelake`, `zen2`, and `zen3`. See the [Spack documentation](https://spack.readthedocs.io/en/latest/basic_usage.html#support-for-specific-microarchitectures) for the full list of targets. [wave-page]: ../../../wave diff --git a/docs/reference/process/directives/array.mdx b/docs/reference/process/directives/array.mdx index 3b238e61ae..32e0e26d48 100644 --- a/docs/reference/process/directives/array.mdx +++ b/docs/reference/process/directives/array.mdx @@ -9,9 +9,11 @@ description: Reference for the `array` process directive. The `array` directive submits tasks as *job arrays* for executors that support it. -A job array is a collection of jobs with the same resource requirements and the same script (parameterized by an index). Job arrays incur significantly less scheduling overhead compared to individual jobs, and as a result they are preferred by HPC schedulers where possible. +A job array is a collection of jobs with the same resource requirements and the same script (parameterized by an index). Job arrays incur less scheduling overhead than individual jobs and are preferred by HPC schedulers where possible. -The directive should be specified with a given array size, along with an executor that supports job arrays. For example: +## Usage + +Specify the directive with an array size, along with an executor that supports job arrays. For example: ```nextflow process hello { @@ -25,19 +27,9 @@ process hello { } ``` -Nextflow currently supports job arrays for the following executors: - -- [AWS Batch][awsbatch-executor] -- [Google Cloud Batch][google-batch-executor] -- [LSF][lsf-executor] -- [PBS][pbs-executor] -- [PBS Pro][pbspro-executor] -- [SGE][sge-executor] -- [SLURM][slurm-executor] - A process using job arrays collects tasks and submits each batch as a job array when it is ready. Any "leftover" tasks are submitted as a partial job array. -Once a job array is submitted, each "child" task is executed as an independent job. Any tasks that fail (and can be retried) are retried without interfering with the tasks that succeeded. Retried tasks are submitted individually rather than through a job array, in order to allow for the use of [dynamic resources][dynamic-task-resources]. +After a job array is submitted, each "child" task is executed as an independent job. Any tasks that fail (and can be retried) are retried without interfering with the tasks that succeeded. Retried tasks are submitted individually rather than through a job array, to allow the use of [dynamic resources][dynamic-task-resources]. The following directives must be uniform across all tasks in a process that uses job arrays, because these directives are specified once for the entire job array: @@ -52,7 +44,7 @@ The following directives must be uniform across all tasks in a process that uses - [resourceLimits](./resource-limits) - [time](./time) -For cloud-based executors like AWS Batch, or when using Fusion with any executor, the following additional directives must be uniform: +For cloud-based executors such as AWS Batch, or when using Fusion with any executor, the following additional directives must be uniform: - [container](./container) - [containerOptions](./container-options) @@ -61,6 +53,18 @@ When using Wave, the following additional directives must be uniform: - [conda](./conda) +## Executor support + +Nextflow supports job arrays for the following executors: + +- [AWS Batch][awsbatch-executor] +- [Google Cloud Batch][google-batch-executor] +- [LSF][lsf-executor] +- [PBS][pbs-executor] +- [PBS Pro][pbspro-executor] +- [SGE][sge-executor] +- [SLURM][slurm-executor] + [awsbatch-executor]: ../../../executor/aws-batch [dynamic-task-resources]: ../../../process#dynamic-task-resources [google-batch-executor]: ../../../executor/google-batch diff --git a/docs/reference/process/directives/before-script.mdx b/docs/reference/process/directives/before-script.mdx index 16f36a81b0..de6dd13c96 100644 --- a/docs/reference/process/directives/before-script.mdx +++ b/docs/reference/process/directives/before-script.mdx @@ -5,9 +5,11 @@ description: Reference for the `beforeScript` process directive. # `beforeScript` -The `beforeScript` directive executes a custom (Bash) snippet *before* the main process script is run. This may be useful to initialize the underlying cluster environment or for other custom initialization. +The `beforeScript` directive executes a custom Bash snippet *before* the main process script is run. -For example: +## Usage + +Use it to initialize the underlying cluster environment or perform other custom initialization. For example: ```nextflow process hello { @@ -20,4 +22,4 @@ process hello { } ``` -When the process is containerized (using the [container](./container) directive), the `beforeScript` is executed in the container only if the executor is *container-native* (e.g. cloud batch executors, Kubernetes). Otherwise, the `beforeScript` is executed outside the container. +When the process is containerized (using the [container](./container) directive), the `beforeScript` is executed in the container only if the executor is *container-native* (for example, cloud batch executors and Kubernetes). Otherwise, the `beforeScript` is executed outside the container. diff --git a/docs/reference/process/directives/cache.mdx b/docs/reference/process/directives/cache.mdx index 50491c0629..f399217cb3 100644 --- a/docs/reference/process/directives/cache.mdx +++ b/docs/reference/process/directives/cache.mdx @@ -7,7 +7,9 @@ description: Reference for the `cache` process directive. The `cache` directive controls whether and how task executions are cached. -By default, cached task executions are re-used when the pipeline is launched with the [resume][getstarted-resume] option. The `cache` directive can be used to disable caching for a specific process: +## Usage + +By default, cached task executions are re-used when the pipeline is launched with the [resume][getstarted-resume] option. Use the `cache` directive to disable caching for a specific process: ```nextflow process hello { @@ -19,7 +21,9 @@ process hello { See [Cache and resume][cache-resume-page] for more information. -The following options are available: +## Options + +The following values are available: ##### `false` @@ -27,7 +31,7 @@ Disable caching. ##### `true` (default) -Enable caching. Input file metadata (name, size, last updated timestamp) are included in the cache keys. +Enable caching. Input file metadata (name, size, last updated timestamp) is included in the cache keys. ##### `'deep'` @@ -35,7 +39,7 @@ Enable caching. Input file content is included in the cache keys. ##### `'lenient'` -Enable caching. Minimal input file metadata (name and size only) are included in the cache keys. +Enable caching. Minimal input file metadata (name and size only) is included in the cache keys. This strategy provides a workaround for incorrect caching invalidation observed on shared file systems due to inconsistent file timestamps. diff --git a/docs/reference/process/directives/cluster-options.mdx b/docs/reference/process/directives/cluster-options.mdx index dc6ffcdf6e..d222665067 100644 --- a/docs/reference/process/directives/cluster-options.mdx +++ b/docs/reference/process/directives/cluster-options.mdx @@ -7,6 +7,8 @@ description: Reference for the `clusterOptions` process directive. The `clusterOptions` directive specifies additional submission options for grid executors. You can use it to specify options for your cluster that are not supported directly by other process directives. +## Usage + The cluster options can be a string: ```nextflow @@ -18,7 +20,7 @@ process hello { ``` -Prior to this version, grid executors that require each option to be on a separate line in the job script would attempt to split multiple options using a variety of different conventions. Multiple options can now be specified more clearly using a string list as shown below. +Prior to this version, grid executors that require each option to be on a separate line in the job script attempted to split multiple options using different conventions. Multiple options can now be specified more clearly using a string list, as shown in the following example. The cluster options can also be a string list: @@ -31,14 +33,14 @@ process hello { } ``` -Grid executors that require one option per line will write each option to a separate line, while grid executors that allow multiple options per line will write all options to a single line, the same as with a string. This form is useful to control how the options are split across lines when it is required by the scheduler. - -:::note -This directive is only used by grid executors. Refer to the [Executors][executor-page] page to see which executors support this directive. -::: +Grid executors that require one option per line write each option to a separate line. Grid executors that allow multiple options per line write all options to a single line, the same as with a string. This form is useful to control how the options are split across lines when the scheduler requires it. :::warning -While you can use the `clusterOptions` directive to specify options that are supported as process directives (`queue`, `memory`, `time`, etc), you should not use both at the same time, as it will cause undefined behavior. Most HPC schedulers will either fail or simply ignore one or the other. +Do not use the `clusterOptions` directive to specify options that are supported as process directives, such as `queue`, `memory`, and `time`. Using both at the same time causes undefined behavior. Most HPC schedulers either fail or ignore one or the other. ::: +## Executor support + +This directive is only used by grid executors. See [Executors][executor-page] for the executors that support this directive. + [executor-page]: ../../../executor diff --git a/docs/reference/process/directives/conda.mdx b/docs/reference/process/directives/conda.mdx index 561e390aa0..80b9b51f46 100644 --- a/docs/reference/process/directives/conda.mdx +++ b/docs/reference/process/directives/conda.mdx @@ -5,7 +5,11 @@ description: Reference for the `conda` process directive. # `conda` -The `conda` directive defines the set of [Conda](https://conda.io) packages required by each task. For example: +The `conda` directive defines the set of [Conda](https://conda.io) packages required by each task. + +## Usage + +For example: ```nextflow process hello { @@ -20,8 +24,8 @@ process hello { Nextflow automatically creates an environment for each unique set of Conda packages. -The name of the desired channel for a specific package can be specified using the standard Conda notation, e.g. `bioconda::bwa=0.7.15`. Multiple packages can be specified separating them with a blank space, e.g. `bwa=0.7.15 fastqc=0.11.5`. +Specify the channel for a package using the standard Conda notation, for example, `bioconda::bwa=0.7.15`. Specify multiple packages by separating them with a space, for example, `bwa=0.7.15 fastqc=0.11.5`. -The `conda` directive can also accept a Conda environment file path or the path of an existing Conda environment. See [Conda][conda-page] for more information. +The `conda` directive also accepts a Conda environment file path or the path of an existing Conda environment. See [Conda][conda-page] for more information. [conda-page]: ../../../conda diff --git a/docs/reference/process/directives/container-options.mdx b/docs/reference/process/directives/container-options.mdx index cf73da8c90..1656f8cf6b 100644 --- a/docs/reference/process/directives/container-options.mdx +++ b/docs/reference/process/directives/container-options.mdx @@ -5,7 +5,11 @@ description: Reference for the `containerOptions` process directive. # `containerOptions` -The `containerOptions` directive specifies additional container options for the underlying container runtime (e.g. Docker, Singularity). For example: +The `containerOptions` directive specifies additional container options for the underlying container runtime (for example, Docker or Singularity). + +## Usage + +For example: ```nextflow process hello_docker { @@ -22,10 +26,10 @@ process hello_docker { } ``` -The above example provides a custom volume mount for a specific process. +This example provides a custom volume mount for a specific process. + +## Executor support -:::warning This directive is not supported by the [Kubernetes][k8s-executor] executor. -::: [k8s-executor]: ../../../executor/kubernetes diff --git a/docs/reference/process/directives/container.mdx b/docs/reference/process/directives/container.mdx index f164cba10c..15a7ff78cc 100644 --- a/docs/reference/process/directives/container.mdx +++ b/docs/reference/process/directives/container.mdx @@ -5,7 +5,11 @@ description: Reference for the `container` process directive. # `container` -The `container` directive defines the container required by each task. For example: +The `container` directive defines the container required by each task. + +## Usage + +For example: ```nextflow process hello_docker { @@ -18,10 +22,10 @@ process hello_docker { } ``` -The corresponding container runtime (e.g. Docker, Singularity) should be running on the compute nodes where tasks are executed. See [Containers][container-page] for the container runtimes supported by Nextflow. +The corresponding container runtime (for example, Docker or Singularity) must be running on the compute nodes where tasks are executed. See [Containers][container-page] for the container runtimes supported by Nextflow. :::note -This directive is ignored by [native processes][process-native] (i.e. `exec` processes). +This directive is ignored by [native processes][process-native] (that is, `exec` processes). ::: [container-page]: ../../../container diff --git a/docs/reference/process/directives/cpus.mdx b/docs/reference/process/directives/cpus.mdx index f5e1157d63..86b9515a67 100644 --- a/docs/reference/process/directives/cpus.mdx +++ b/docs/reference/process/directives/cpus.mdx @@ -5,7 +5,11 @@ description: Reference for the `cpus` process directive. # `cpus` -The `cpus` directive defines the number of CPUs required by each task execution. For example: +The `cpus` directive defines the number of CPUs required by each task execution. + +## Usage + +For example: ```nextflow process blast { @@ -18,8 +22,14 @@ process blast { } ``` -This directive is required for tasks that execute multi-process or multi-threaded commands/tools and it is meant to reserve enough CPUs when a pipeline task is executed through a cluster resource manager. +Set this directive for tasks that execute multi-process or multi-threaded commands. It reserves enough CPUs when a task is executed through a cluster resource manager. + +## See also -See also: [disk](./disk), [memory](./memory), [time](./time), [queue](./queue), [dynamic task resources][dynamic-task-resources] +- [disk](./disk) +- [memory](./memory) +- [time](./time) +- [queue](./queue) +- [Dynamic task resources][dynamic-task-resources] [dynamic-task-resources]: ../../../process#dynamic-task-resources diff --git a/docs/reference/process/directives/debug.mdx b/docs/reference/process/directives/debug.mdx index 7da376c473..e952be1e04 100644 --- a/docs/reference/process/directives/debug.mdx +++ b/docs/reference/process/directives/debug.mdx @@ -7,6 +7,8 @@ description: Reference for the `debug` process directive. The `debug` directive prints the standard output of each task to the pipeline standard output. +## Usage + For example: ```nextflow @@ -26,4 +28,4 @@ Prints: Hello ``` -Removing the `debug` directive or setting it to `false` in the above example will cause `Hello` to not be printed. +If you remove the `debug` directive or set it to `false`, `Hello` is not printed. diff --git a/docs/reference/process/directives/disk.mdx b/docs/reference/process/directives/disk.mdx index 603df3b1b1..ce20c285f6 100644 --- a/docs/reference/process/directives/disk.mdx +++ b/docs/reference/process/directives/disk.mdx @@ -5,7 +5,11 @@ description: Reference for the `disk` process directive. # `disk` -The `disk` directive defines the amount of disk storage required by each task execution. For example: +The `disk` directive defines the amount of disk storage required by each task execution. + +## Usage + +For example: ```nextflow process hello { @@ -18,7 +22,7 @@ process hello { } ``` -The following suffixes can be used to specify disk values: +Use the following suffixes to specify disk values: - `B`: Bytes - `KB`: Kilobytes @@ -28,11 +32,17 @@ The following suffixes can be used to specify disk values: See [MemoryUnit][stdlib-types-memoryunit] for more information. -:::note -The `disk` directive is only used by certain executors. Refer to the [Executors][executor-page] page to see which executors support this directive. -::: +## Executor support + +This directive is only used by certain executors. See [Executors][executor-page] for the executors that support this directive. + +## See also -See also: [cpus](./cpus), [memory](./memory), [time](./time), [queue](./queue), [dynamic task resources][dynamic-task-resources] +- [cpus](./cpus) +- [memory](./memory) +- [time](./time) +- [queue](./queue) +- [Dynamic task resources][dynamic-task-resources] [dynamic-task-resources]: ../../../process#dynamic-task-resources [executor-page]: ../../../executor diff --git a/docs/reference/process/directives/error-strategy.mdx b/docs/reference/process/directives/error-strategy.mdx index 97482613d2..d434cd41d8 100644 --- a/docs/reference/process/directives/error-strategy.mdx +++ b/docs/reference/process/directives/error-strategy.mdx @@ -9,6 +9,8 @@ The `errorStrategy` directive defines how to handle task failures. A task failure occurs when the executed script returns a non-zero exit code. By default, the pipeline run is aborted. +## Options + The following error strategies are available: ##### `'terminate'` (default) @@ -21,17 +23,7 @@ When a task fails, wait for submitted and running tasks to finish and then termi ##### `'ignore'` -When a task fails, ignore it and continue the pipeline execution. If the `workflow.failOnIgnore` config option is set to `true`, the pipeline will report an error (i.e. return a non-zero exit code) upon completion. Otherwise, the pipeline will complete successfully. - -See the [workflow][stdlib-namespaces-workflow] namespace for more information. - -##### `'retry'` - -When a task fails, retry it. - -When setting the `errorStrategy` directive to `ignore` the process doesn't stop on an error condition, it just reports a message notifying you of the error event. - -For example: +When a task fails, ignore it and continue the pipeline execution. The failure is reported as a message, but the run is not stopped. For example: ```nextflow process hello { @@ -41,9 +33,11 @@ process hello { } ``` -In this case, the workflow will complete successfully and return an exit status of 0. However, if you set `workflow.failOnIgnore = true` in your Nextflow configuration, the workflow will return a non-zero exit status and report the failed tasks as an error. +By default, the pipeline completes successfully and returns an exit status of 0. If the `workflow.failOnIgnore` config option is set to `true`, the pipeline returns a non-zero exit status and reports the failed tasks as an error upon completion. See the [workflow][stdlib-namespaces-workflow] namespace for more information. -The `retry` error strategy retries failed tasks. For example: +##### `'retry'` + +When a task fails, retry it. For example: ```nextflow process hello { @@ -53,13 +47,17 @@ process hello { } ``` -The number of times a failing process is re-executed is defined by the [maxRetries](./max-retries) and [maxErrors](./max-errors) directives. +Use the [maxRetries](./max-retries) and [maxErrors](./max-errors) directives to limit the number of retries. :::tip -More complex strategies depending on the task exit status or other parametric values can be defined using a dynamic `errorStrategy`. See [Dynamic directives][dynamic-directives] for details. +Use a dynamic `errorStrategy` to define more complex strategies based on the task exit status or other parametric values. See [Dynamic directives][dynamic-directives] for details. ::: -See also: [maxErrors](./max-errors), [maxRetries](./max-retries), [dynamic task resources][dynamic-task-resources] +## See also + +- [maxErrors](./max-errors) +- [maxRetries](./max-retries) +- [Dynamic task resources][dynamic-task-resources] [dynamic-directives]: ../../../process#dynamic-directives [dynamic-task-resources]: ../../../process#dynamic-task-resources diff --git a/docs/reference/process/directives/executor.mdx b/docs/reference/process/directives/executor.mdx index 26ea0350f3..7f0956ba31 100644 --- a/docs/reference/process/directives/executor.mdx +++ b/docs/reference/process/directives/executor.mdx @@ -5,7 +5,11 @@ description: Reference for the `executor` process directive. # `executor` -The `executor` directive defines the underlying system where tasks are executed. For example: +The `executor` directive defines the underlying system where tasks are executed. + +## Usage + +For example: ```nextflow process hello { diff --git a/docs/reference/process/directives/ext.mdx b/docs/reference/process/directives/ext.mdx index 8ac6e87925..18f15b1f98 100644 --- a/docs/reference/process/directives/ext.mdx +++ b/docs/reference/process/directives/ext.mdx @@ -5,7 +5,11 @@ description: Reference for the `ext` process directive. # `ext` -The `ext` is a generic directive for user-defined properties. For example: +The `ext` directive defines user-defined properties. + +## Usage + +For example: ```nextflow process star { @@ -22,7 +26,7 @@ process star { } ``` -In the above example, the process container version is controlled by `ext.version`, and the script supports additional command line arguments through `ext.args`. +In this example, the process container version is controlled by `ext.version`, and the script supports additional command line arguments through `ext.args`. The `ext` directive can be set in the process definition: @@ -34,7 +38,7 @@ process hello { } ``` -Or in the Nextflow configuration: +You can also set it in the Nextflow configuration: ```groovy process.ext.version = '2.5.3' diff --git a/docs/reference/process/directives/fair.mdx b/docs/reference/process/directives/fair.mdx index 779e64d564..8b809d1d19 100644 --- a/docs/reference/process/directives/fair.mdx +++ b/docs/reference/process/directives/fair.mdx @@ -7,7 +7,11 @@ description: Reference for the `fair` process directive. -The `fair` directive, when enabled, guarantees that process outputs will be emitted in the order in which they were received. For example: +When enabled, the `fair` directive guarantees that process outputs are emitted in the order in which they were received. + +## Usage + +For example: ```nextflow process hello { @@ -30,7 +34,7 @@ workflow { } ``` -The above example produces: +This example produces: ``` [1, A] diff --git a/docs/reference/process/directives/hints.mdx b/docs/reference/process/directives/hints.mdx index ec0d9366ef..4b4ce40a63 100644 --- a/docs/reference/process/directives/hints.mdx +++ b/docs/reference/process/directives/hints.mdx @@ -5,9 +5,11 @@ description: Reference for the `hints` process directive. # `hints` -The `hints` directive specifies executor-specific hints as key-value pairs. Each executor uses the hints it recognizes and ignores the rest. Hint values can be any raw value (i.e., numbers, strings, booleans, lists, and maps). +The `hints` directive specifies executor-specific hints as key-value pairs. Each executor uses the hints it recognizes and ignores the rest. Hint values can be any raw value, that is, numbers, strings, booleans, lists, and maps. -Unprefixed keys are available to **every** executor. Any executor that recognizes the key consumes it. Prefixing a key with an executor name (e.g., `awsbatch/...`) restricts the hint to that executor only. For example: +## Usage + +Unprefixed keys are available to every executor, and any executor that recognizes the key consumes it. For example: ```nextflow process hello { @@ -20,7 +22,7 @@ process hello { } ``` -To restrict a hint to a single executor, prefix the key with the executor name: +To restrict a hint to a single executor, prefix the key with the executor name, for example, `awsbatch/`: ```nextflow hints 'awsbatch/consumableResources': ['my-license': 1] @@ -28,8 +30,10 @@ hints 'awsbatch/consumableResources': ['my-license': 1] When the same hint is provided both unprefixed and with a matching executor prefix, the prefixed form takes precedence for that executor. -Calling `hints` multiple times in a process definition accumulates entries, with later calls overwriting entries for the same key. Setting `hints` via configuration (e.g., in `nextflow.config`) replaces the entire map. +Calling `hints` multiple times in a process definition accumulates entries, with later calls overwriting entries for the same key. Setting `hints` in configuration (for example, in `nextflow.config`) replaces the entire map. + +## Executor support -See [Executors][executor-page] to see which hints are recognized by each executor. +See [Executors][executor-page] for the hints recognized by each executor. [executor-page]: ../../../executor diff --git a/docs/reference/process/directives/label.mdx b/docs/reference/process/directives/label.mdx index e2393cc5f8..b726e787f8 100644 --- a/docs/reference/process/directives/label.mdx +++ b/docs/reference/process/directives/label.mdx @@ -5,7 +5,11 @@ description: Reference for the `label` process directive. # `label` -The `label` directive attaches a custom label to the process. For example: +The `label` directive attaches a custom label to the process. + +## Usage + +For example: ```nextflow process hello { @@ -22,7 +26,7 @@ A label may contain alphanumeric characters or `_`. It must start and end with a The same label can be applied to multiple processes. Multiple labels can be applied to the same process by using the `label` directive multiple times. -Process labels are used to apply shared process configuration via `withLabel` selectors. They are not recorded in execution logs, trace reports, or lineage metadata. See [Process selectors][config-process-selectors] for more information. +Process labels are used to apply shared process configuration using `withLabel` selectors. They are not recorded in execution logs, trace reports, or lineage metadata. See [Process selectors][config-process-selectors] for more information. :::note To tag individual task executions for logging and debugging, use [tag](./tag). To tag cloud computing resources for cost tracking, use [resourceLabels](./resource-labels). To attach metadata labels to output files for lineage tracking, use the `label` [output directive][workflow-output-def] in the `output` block. diff --git a/docs/reference/process/directives/machine-type.mdx b/docs/reference/process/directives/machine-type.mdx index a04be58a91..0ca20d58f8 100644 --- a/docs/reference/process/directives/machine-type.mdx +++ b/docs/reference/process/directives/machine-type.mdx @@ -5,7 +5,9 @@ description: Reference for the `machineType` process directive. # `machineType` -The `machineType` can be used to specify a predefined Google Compute Platform [machine type](https://cloud.google.com/compute/docs/machine-types) when running using the [Google Batch][google-batch-executor], or when using auto-pools with [Azure Batch][azurebatch-executor]. +The `machineType` directive specifies a predefined [machine type](https://cloud.google.com/compute/docs/machine-types), for cloud executors that support it. + +## Usage For example: @@ -20,7 +22,14 @@ process hello { } ``` -See also: [cpus](./cpus), [memory](./memory) +## Executor support + +This directive is supported by the [Google Batch][google-batch-executor] executor, and by the [Azure Batch][azurebatch-executor] executor when using auto-pools. + +## See also + +- [cpus](./cpus) +- [memory](./memory) [azurebatch-executor]: ../../../executor/azure-batch [google-batch-executor]: ../../../executor/google-batch diff --git a/docs/reference/process/directives/max-errors.mdx b/docs/reference/process/directives/max-errors.mdx index aa7781c2be..dbee522da3 100644 --- a/docs/reference/process/directives/max-errors.mdx +++ b/docs/reference/process/directives/max-errors.mdx @@ -5,7 +5,11 @@ description: Reference for the `maxErrors` process directive. # `maxErrors` -The `maxErrors` directive defines the maximum number of task failures allowed for a process when using the `retry` error strategy. For example: +The `maxErrors` directive defines the maximum number of task failures allowed for a process when using the `retry` error strategy. + +## Usage + +For example: ```nextflow process hello { @@ -19,8 +23,11 @@ process hello { } ``` -In the above example, the run will fail if the `hello` process accrues more than 5 failures across all of its task executions. +In this example, the run fails if the `hello` process accrues more than 5 failures across all of its task executions. + +By default, the number of task failures is unlimited. However, the run can still fail if an individual task exceeds the number of retries allowed by the `maxRetries` directive. -By default, there is no limit. However, the run can still fail if an individual task exceeds the number of retries allowed by the `maxRetries` directive. +## See also -See also: [errorStrategy](./error-strategy), [maxRetries](./max-retries) +- [errorStrategy](./error-strategy) +- [maxRetries](./max-retries) diff --git a/docs/reference/process/directives/max-forks.mdx b/docs/reference/process/directives/max-forks.mdx index be870a6002..6afd5fd4d6 100644 --- a/docs/reference/process/directives/max-forks.mdx +++ b/docs/reference/process/directives/max-forks.mdx @@ -5,7 +5,11 @@ description: Reference for the `maxForks` process directive. # `maxForks` -The `maxForks` directive defines the maximum number of concurrent task executions for a process. For example: +The `maxForks` directive defines the maximum number of concurrent task executions for a process. + +## Usage + +For example: ```nextflow process hello { @@ -18,6 +22,6 @@ process hello { } ``` -The above example forces the `hello` process to execute tasks sequentially. +This example forces the `hello` process to execute tasks sequentially. -By default, there is no limit. However, the number of concurrent tasks can still be limited globally by the number of CPUs (for local tasks) and the `executor.queueSize` config option. +By default, the number of concurrent task executions is unlimited. However, the number of concurrent tasks can still be limited globally by the number of CPUs (for local tasks) and the `executor.queueSize` config option. diff --git a/docs/reference/process/directives/max-retries.mdx b/docs/reference/process/directives/max-retries.mdx index b8ba91a5d4..52001ae982 100644 --- a/docs/reference/process/directives/max-retries.mdx +++ b/docs/reference/process/directives/max-retries.mdx @@ -5,7 +5,11 @@ description: Reference for the `maxRetries` process directive. # `maxRetries` -The `maxRetries` directive defines the maximum number of times a task can be retried when using the `retry` error strategy. For example: +The `maxRetries` directive defines the maximum number of times a task can be retried when using the `retry` error strategy. + +## Usage + +For example: ```nextflow process hello { @@ -19,8 +23,11 @@ process hello { } ``` -In the above example, the run will fail if any task executed by `hello` fails more than three times. +In this example, the run fails if any task executed by `hello` fails more than three times. By default, only one retry per task is allowed. However, the run can still fail if the total number of failures for the process exceeds the number allowed by the `maxErrors` directive. -See also: [errorStrategy](./error-strategy), [maxErrors](./max-errors) +## See also + +- [errorStrategy](./error-strategy) +- [maxErrors](./max-errors) diff --git a/docs/reference/process/directives/max-submit-await.mdx b/docs/reference/process/directives/max-submit-await.mdx index 8d674eb560..8d95afde2d 100644 --- a/docs/reference/process/directives/max-submit-await.mdx +++ b/docs/reference/process/directives/max-submit-await.mdx @@ -5,9 +5,11 @@ description: Reference for the `maxSubmitAwait` process directive. # `maxSubmitAwait` -The `maxSubmitAwait` directive defines how long a task can remain in submission queue without being executed. Tasks that exceed this duration in the queue will fail. +The `maxSubmitAwait` directive defines how long a task can remain in the submission queue without being executed. Tasks that exceed this duration in the queue fail. -It can be used with the `retry` error strategy to re-submit tasks to a different queue or with different resource requirements. For example: +## Usage + +Use it with the `retry` error strategy to re-submit tasks to a different queue or with different resource requirements. For example: ```nextflow process hello { @@ -23,4 +25,4 @@ process hello { } ``` -In the above example, each task is submitted to the `spot-compute` queue on the first attempt (`task.submitAttempt==1`). If a task remains in the queue for more than 10 minutes, it fails and is re-submitted to the `on-demand-compute` queue. +In this example, each task is submitted to the `spot-compute` queue on the first attempt (`task.submitAttempt==1`). If a task remains in the queue for more than 10 minutes, it fails and is re-submitted to the `on-demand-compute` queue. diff --git a/docs/reference/process/directives/memory.mdx b/docs/reference/process/directives/memory.mdx index 7a2285188e..4834f8b8e2 100644 --- a/docs/reference/process/directives/memory.mdx +++ b/docs/reference/process/directives/memory.mdx @@ -5,7 +5,11 @@ description: Reference for the `memory` process directive. # `memory` -The `memory` directive defines how much memory is required by each task execution. For example: +The `memory` directive defines how much memory is required by each task execution. + +## Usage + +For example: ```nextflow process hello { @@ -18,7 +22,7 @@ process hello { } ``` -The following suffixes can be used to specify memory values: +Use the following suffixes to specify memory values: - `B`: Bytes - `KB`: Kilobytes @@ -28,7 +32,13 @@ The following suffixes can be used to specify memory values: See [MemoryUnit][stdlib-types-memoryunit] for more information. -See also: [cpus](./cpus), [disk](./disk), [time](./time), [queue](./queue), [dynamic task resources][dynamic-task-resources] +## See also + +- [cpus](./cpus) +- [disk](./disk) +- [time](./time) +- [queue](./queue) +- [Dynamic task resources][dynamic-task-resources] [dynamic-task-resources]: ../../../process#dynamic-task-resources [stdlib-types-memoryunit]: ../../stdlib-types/memory-unit diff --git a/docs/reference/process/directives/module.mdx b/docs/reference/process/directives/module.mdx index ba861b0062..86c78c0e6f 100644 --- a/docs/reference/process/directives/module.mdx +++ b/docs/reference/process/directives/module.mdx @@ -5,7 +5,11 @@ description: Reference for the `module` process directive. # `module` -The `module` directive defines the set of [Environment Modules](http://modules.sourceforge.net/) required by each task, if supported by your compute environment. For example: +The `module` directive defines the set of [Environment Modules](http://modules.sourceforge.net/) required by each task, if supported by your compute environment. + +## Usage + +For example: ```nextflow process blast { @@ -27,6 +31,6 @@ process blast { script: """ blastp -query - """ + """ } ``` diff --git a/docs/reference/process/directives/penv.mdx b/docs/reference/process/directives/penv.mdx index 0a17cc715f..d750c61220 100644 --- a/docs/reference/process/directives/penv.mdx +++ b/docs/reference/process/directives/penv.mdx @@ -5,7 +5,11 @@ description: Reference for the `penv` process directive. # `penv` -The `penv` directive defines the parallel environment to use when submitting tasks to the [SGE][sge-executor] resource manager. For example: +The `penv` directive defines the parallel environment to use when submitting tasks to the [SGE][sge-executor] resource manager. + +## Usage + +For example: ```nextflow process blast { @@ -20,6 +24,8 @@ process blast { } ``` -Refer to your cluster documentation or your system administrator to determine whether this feature is supported in your environment. +## Executor support + +This directive is only used by the [SGE][sge-executor] executor. Refer to your cluster documentation or your system administrator to determine whether this feature is supported in your environment. [sge-executor]: ../../../executor/sge diff --git a/docs/reference/process/directives/pod.mdx b/docs/reference/process/directives/pod.mdx index 16bf571b7e..ce1432c9f3 100644 --- a/docs/reference/process/directives/pod.mdx +++ b/docs/reference/process/directives/pod.mdx @@ -7,6 +7,8 @@ description: Reference for the `pod` process directive. The `pod` directive defines pod-specific settings, such as environment variables, secrets, and config maps, when using the [Kubernetes][k8s-executor] executor. +## Usage + For example: ```nextflow @@ -20,9 +22,9 @@ process echo { } ``` -The above snippet defines an environment variable named `MESSAGE` whose value is `'hello world'`. +This example defines an environment variable named `MESSAGE` whose value is `'hello world'`. -Pod settings can be specified in Nextflow configuration: +You can also set it in the Nextflow configuration: ```groovy // single setting @@ -35,6 +37,8 @@ process.pod = [ ] ``` +## Options + The following options are available: ##### `affinity: ` @@ -49,7 +53,7 @@ Defines a pod [annotation](https://kubernetes.io/docs/concepts/overview/working- ##### `automountServiceAccountToken: true | false` -Specifies whether to [automount service account token](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#opt-out-of-api-credential-automounting) into the pod (default: `true`). +Specifies whether to [automount the service account token](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#opt-out-of-api-credential-automounting) into the pod (default: `true`). ##### `config: '/', mountPath: ''` @@ -115,7 +119,7 @@ Defines an environment variable with the given name and value. *Can be specified multiple times* -Allows creating [hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) volume and access it with the specified `mountPath` in the pod. +Creates a [hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) volume and mounts it at the specified `mountPath` in the pod. ##### `imagePullPolicy: 'IfNotPresent' | 'Always' | 'Never'` @@ -215,8 +219,8 @@ Specifies the [TTL mechanism](https://kubernetes.io/docs/concepts/workloads/cont Mounts a [Persistent volume claim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) with the given name to the given path. -The `subPath` option can be used to mount a sub-directory of the volume instead of its root. +The `subPath` option mounts a sub-directory of the volume instead of its root. -The `readOnly` option can be used to mount the volume as read-only (default: `false`) +The `readOnly` option mounts the volume as read-only (default: `false`). [k8s-executor]: ../../../executor/kubernetes diff --git a/docs/reference/process/directives/publish-dir.mdx b/docs/reference/process/directives/publish-dir.mdx index 09a3397bcf..625644c064 100644 --- a/docs/reference/process/directives/publish-dir.mdx +++ b/docs/reference/process/directives/publish-dir.mdx @@ -6,10 +6,14 @@ description: Reference for the `publishDir` process directive. # `publishDir` :::note -[Workflow outputs][workflow-output-def] can be used instead of `publishDir`. See [Migrating to workflow outputs][migrating-workflow-outputs] to learn how to migrate existing code. +You can use [workflow outputs][workflow-output-def] instead of `publishDir`. See [Migrating to workflow outputs][migrating-workflow-outputs] to learn how to migrate existing code. ::: -The `publishDir` directive publishes matching process output files to a target directory. For example: +The `publishDir` directive publishes matching process output files to a target directory. + +## Usage + +For example: ```nextflow process hello { @@ -25,13 +29,13 @@ process hello { } ``` -The above example publishes the `chunk_*` output files into the `/data/chunks` directory. +This example publishes the `chunk_*` output files into the `/data/chunks` directory. Only files that match the declaration in the `output` block are published, not all the outputs of the process. -The `publishDir` directive can be specified more than once in order to publish output files to different target directories based on different rules. +Specify the `publishDir` directive more than once to publish output files to different target directories based on different rules. -By default, files are published via *symbolic link* from the task directory to the target directory. Use the `mode` option to control this behavior: +By default, files are published as a *symbolic link* from the task directory to the target directory. Use the `mode` option to control this behavior: ```nextflow process hello { @@ -48,20 +52,22 @@ process hello { ``` :::warning -Output files are published *asynchronously* after the task execution, so they may not be immediately available in the publish directory during the pipeline run. Downstream processes should access output files through the declared process outputs, not the publish directory. +Output files are published *asynchronously* after the task execution. They may not be immediately available in the publish directory during the pipeline run. Downstream processes should access output files through the declared process outputs, not the publish directory. ::: -Available options: +## Options + +The following options are available: ##### `contentType` -*Experimental: currently only supported for S3.* +*Experimental: only supported for S3.* -Allow specifying the media content type of the published file a.k.a. [MIME type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_Types). If set to `true`, the content type is inferred from the file extension (default: `false`). +Specifies the media content type, or [MIME type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_Types), of the published file. If set to `true`, the content type is inferred from the file extension (default: `false`). ##### `enabled` -Enable or disable the publish rule depending on the boolean value specified (default: `true`). +Enables or disables the publish rule depending on the boolean value specified (default: `true`). ##### `failOnError` @@ -69,26 +75,26 @@ Enable or disable the publish rule depending on the boolean value specified (def The default value was changed from `false` to `true` -When `true` abort the execution if some file can't be published to the specified target directory or bucket for any cause (default: `true`) +When `true`, the run is aborted if a file cannot be published to the specified target directory or bucket (default: `true`). ##### `mode` The file publishing method. Can be one of the following values: - `'copy'`: Copies the output files into the publish directory. -- `'copyNoFollow'`: Copies the output files into the publish directory without following symlinks ie. copies the links themselves. +- `'copyNoFollow'`: Copies the output files into the publish directory without following symlinks, that is, the links themselves are copied. - `'link'`: Creates a hard link in the publish directory for each output file. -- `'move'`: Moves the output files into the publish directory. **Note**: this is only supposed to be used for a *terminal* process i.e. a process whose output is not consumed by any other downstream process. +- `'move'`: Moves the output files into the publish directory. Use only for a *terminal* process, that is, a process whose output is not consumed by any other downstream process. - `'rellink'`: Creates a relative symbolic link in the publish directory for each output file. - `'symlink'`: Creates an absolute symbolic link in the publish directory for each output file (default). ##### `overwrite` -When `true` any existing file in the target directory will be overridden (default: `true` during normal pipeline execution and `false` when pipeline execution is `resumed`). +When `true`, any existing file in the target directory is overridden (default: `true` during normal pipeline execution and `false` when pipeline execution is `resumed`). ##### `path` -Specifies the directory where files need to be published. **Note**: the syntax `publishDir '/some/dir'` is a shortcut for `publishDir path: '/some/dir'`. +Specifies the directory where files are published. The syntax `publishDir '/some/dir'` is a shortcut for `publishDir path: '/some/dir'`. ##### `pattern` @@ -96,21 +102,21 @@ Specifies a [glob][glob] file pattern that selects which files to publish from t ##### `saveAs` -A closure which, given the name of the file being published, returns the actual file name or a full path where the file is required to be stored. This can be used to rename or change the destination directory of the published files dynamically by using a custom strategy. Return the value `null` from the closure to *not* publish a file. This is useful when the process has multiple output files, but you want to publish only some of them. +A closure that receives the name of each file being published and returns the actual file name or a full path where the file is stored. Use it to dynamically rename published files or change their destination directory. Return `null` from the closure to *not* publish a file. This is useful when the process has multiple output files, but you want to publish only some of them. ##### `storageClass` -*Experimental: currently only supported for S3.* +*Experimental: only supported for S3.* -Allow specifying the storage class to be used for the published file. +Specifies the storage class for the published file. ##### `tags` -*Experimental: currently only supported for S3.* +*Experimental: only supported for S3.* -Allow the association of arbitrary tags with the published file e.g. `tags: [MESSAGE: 'Hello world']`. +Associates arbitrary tags with the published file, for example, `tags: [MESSAGE: 'Hello world']`. [glob]: https://docs.oracle.com/javase/tutorial/essential/io/fileOps.html#glob [migrating-workflow-outputs]: ../../../tutorials/workflow-outputs diff --git a/docs/reference/process/directives/queue.mdx b/docs/reference/process/directives/queue.mdx index a8ba5460bf..bb8f63d365 100644 --- a/docs/reference/process/directives/queue.mdx +++ b/docs/reference/process/directives/queue.mdx @@ -5,7 +5,11 @@ description: Reference for the `queue` process directive. # `queue` -The `queue` directive defines the queue to which tasks should be submitted, for executors that support queues. For example: +The `queue` directive defines the queue where tasks are submitted, for executors that support queues. + +## Usage + +For example: ```nextflow process hello { @@ -27,6 +31,8 @@ queue 'short,long,cn-el6' However, this is not generally supported by cloud executors such as AWS Batch, Azure Batch, and Google Batch. -See [Executors][executor-page] to see which executors support this directive. +## Executor support + +This directive is only used by executors that support queues. See [Executors][executor-page] for the executors that support this directive. [executor-page]: ../../../executor diff --git a/docs/reference/process/directives/resource-labels.mdx b/docs/reference/process/directives/resource-labels.mdx index 1c32e5a3d4..8cc4bcca8a 100644 --- a/docs/reference/process/directives/resource-labels.mdx +++ b/docs/reference/process/directives/resource-labels.mdx @@ -5,7 +5,11 @@ description: Reference for the `resourceLabels` process directive. # `resourceLabels` -The `resourceLabels` directive attaches custom name-value pairs to task executions, for executors that support it. For example: +The `resourceLabels` directive attaches custom name-value pairs to task executions, for executors that support it. + +## Usage + +For example: ```nextflow process hello { @@ -32,7 +36,9 @@ process { Use process selectors (`withName:` or `withLabel:`) to override resource labels for a specific process. -Resource labels are currently supported by the following executors: +## Executor support + +Resource labels are supported by the following executors: - [AWS Batch][awsbatch-executor] - [Azure Batch][azurebatch-executor] @@ -40,15 +46,16 @@ Resource labels are currently supported by the following executors: - [Kubernetes][k8s-executor] - [Seqera executor][seqera-executor] -:::note -The limits and the syntax of the corresponding executor should be taken into consideration when using resource labels. -::: +Consider the limits and syntax of the corresponding executor when using resource labels. -Resource labels in Azure are added to auto-pools, rather than jobs, in order to facilitate cost analysis. A new pool will be created for each new set of resource labels. Therefore, it is recommended to also set `azure.batch.deletePoolsOnCompletion = true` when using process-specific resource labels. +Resource labels in Azure are added to auto-pools, rather than jobs, to support cost analysis. A new pool is created for each new set of resource labels. Setting `azure.batch.deletePoolsOnCompletion = true` is recommended when using process-specific resource labels. -See also: [label](./label) (for shared process configuration), [tag](./tag) (for per-task identification) +## See also + +- [label](./label) (for shared process configuration) +- [tag](./tag) (for per-task identification) [awsbatch-executor]: ../../../executor/aws-batch [azurebatch-executor]: ../../../executor/azure-batch diff --git a/docs/reference/process/directives/resource-limits.mdx b/docs/reference/process/directives/resource-limits.mdx index be236996f0..a9b4bd37dd 100644 --- a/docs/reference/process/directives/resource-limits.mdx +++ b/docs/reference/process/directives/resource-limits.mdx @@ -9,6 +9,8 @@ description: Reference for the `resourceLimits` process directive. The `resourceLimits` directive defines environment-specific limits for task resource requests. +## Usage + Resource limits can be specified in a process: ```nextflow @@ -22,7 +24,7 @@ process hello { } ``` -Or in the Nextflow configuration: +You can also set it in the Nextflow configuration: ```nextflow process.resourceLimits = [ @@ -39,8 +41,8 @@ Resource limits can be defined for the following directives: - [memory](./memory) - [time](./time) -When a task resource request exceeds the corresponding limit, the task resources are automatically reduced to comply with these limits before the job is submitted. +When a task resource request exceeds the corresponding limit, the task resources are automatically reduced to the limit before the job is submitted. -Resource limits are a useful way to prevent tasks with [dynamic resources][dynamic-task-resources] from requesting more resources than can be provided by an executor (e.g. a task requests 32 cores but the largest node in the cluster has 24). +Resource limits are a useful way to prevent tasks with [dynamic resources][dynamic-task-resources] from requesting more resources than an executor can provide (for example, a task requests 32 cores but the largest node in the cluster has 24). [dynamic-task-resources]: ../../../process#dynamic-task-resources diff --git a/docs/reference/process/directives/scratch.mdx b/docs/reference/process/directives/scratch.mdx index 34ebd37081..9a4fb54ff6 100644 --- a/docs/reference/process/directives/scratch.mdx +++ b/docs/reference/process/directives/scratch.mdx @@ -7,6 +7,8 @@ description: Reference for the `scratch` process directive. The `scratch` directive executes each task in a temporary directory that is local to the compute node. +## Usage + This is useful when executing tasks on an executor with a shared filesystem, because it decreases the network overhead of reading and writing files. Only the files declared as process outputs are copied to the pipeline work directory. For example: @@ -25,7 +27,7 @@ process hello { } ``` -It can also be specified in the Nextflow configuration: +You can also set it in the Nextflow configuration: ```groovy process.scratch = true @@ -36,10 +38,12 @@ By default, the `scratch` directive uses the `$TMPDIR` environment variable in t Each task creates a subdirectory within the base scratch directory and automatically deletes it upon completion. :::note -Cloud-based executors enable `scratch` by default since the pipeline work directory resides in object storage. +Cloud-based executors enable `scratch` by default because the pipeline work directory resides in object storage. ::: -The following values are supported: +## Options + +The following values are available: ##### `false` @@ -51,7 +55,7 @@ Create a scratch directory in the directory defined by the `$TMPDIR` environment ##### `'$YOUR_VAR'` -Create a scratch directory in the directory defined by the given environment variable, or `$(mktemp /tmp)` if that variable is not set. The value must use single quotes, otherwise the environment variable will be evaluated in the pipeline script context. +Create a scratch directory in the directory defined by the given environment variable, or `$(mktemp /tmp)` if that variable is not set. The value must use single quotes, otherwise the environment variable is evaluated in the pipeline script context. ##### `'/my/tmp/path'` diff --git a/docs/reference/process/directives/secret.mdx b/docs/reference/process/directives/secret.mdx index b085ed2025..f208a42d77 100644 --- a/docs/reference/process/directives/secret.mdx +++ b/docs/reference/process/directives/secret.mdx @@ -5,7 +5,9 @@ description: Reference for the `secret` process directive. # `secret` -The `secret` directive allows a process to access secrets. +The `secret` directive defines the secrets required by each task. + +## Usage For example: @@ -25,8 +27,8 @@ Each secret is provided to the task as an environment variable. See [Secrets][secrets-page] for more information. -:::note -Secrets can only be used with the local or grid executors (e.g., Slurm or Grid Engine). Secrets can be used with AWS Batch and Google Batch when launched from Seqera Platform. -::: +## Executor support + +Secrets can only be used with the local or grid executors (for example, SLURM or Grid Engine). Secrets can be used with AWS Batch and Google Batch when launched from Seqera Platform. [secrets-page]: ../../../secrets diff --git a/docs/reference/process/directives/shell.mdx b/docs/reference/process/directives/shell.mdx index 72db612804..63a9310e5b 100644 --- a/docs/reference/process/directives/shell.mdx +++ b/docs/reference/process/directives/shell.mdx @@ -7,6 +7,10 @@ description: Reference for the `shell` process directive. The `shell` directive defines a custom shell command for process scripts. By default, script blocks are executed with `/bin/bash -ue`. +## Usage + +For example: + ```nextflow process hello { shell '/bin/bash', '-euo', 'pipefail' @@ -18,7 +22,7 @@ process hello { } ``` -It can also be specified in the Nextflow configuration: +You can also set it in the Nextflow configuration: ```groovy process.shell = ['/bin/bash', '-euo', 'pipefail'] diff --git a/docs/reference/process/directives/spack.mdx b/docs/reference/process/directives/spack.mdx index 8e9ead7881..e40c08b226 100644 --- a/docs/reference/process/directives/spack.mdx +++ b/docs/reference/process/directives/spack.mdx @@ -5,7 +5,11 @@ description: Reference for the `spack` process directive. # `spack` -The `spack` directive defines the set of [Spack](https://spack.io) packages required by each task. For example: +The `spack` directive defines the set of [Spack](https://spack.io) packages required by each task. + +## Usage + +For example: ```nextflow process hello { @@ -20,7 +24,7 @@ process hello { Nextflow automatically creates a Spack environment for each unique set of packages. -Multiple packages can be specified separating them with a blank space, e.g. `bwa@0.7.15 fastqc@0.11.5`. +Specify multiple packages by separating them with a space, for example, `bwa@0.7.15 fastqc@0.11.5`. The `spack` directive also accepts a Spack environment file path or the path of an existing Spack environment. See [Spack][spack-page] for more information. diff --git a/docs/reference/process/directives/stage-in-mode.mdx b/docs/reference/process/directives/stage-in-mode.mdx index c4a2455e18..fd544702ed 100644 --- a/docs/reference/process/directives/stage-in-mode.mdx +++ b/docs/reference/process/directives/stage-in-mode.mdx @@ -7,7 +7,9 @@ description: Reference for the `stageInMode` process directive. The `stageInMode` directive defines how input files are staged into the task work directory. -The following modes are supported: +## Options + +The following modes are available: ##### `'copy'` @@ -15,12 +17,12 @@ Input files are staged in the task work directory by creating a copy. ##### `'link'` -Input files are staged in the task work directory by creating a hard link for each of them. +Input files are staged in the task work directory by creating a hard link for each file. ##### `'rellink'` -Input files are staged in the task work directory by creating a symbolic link with a relative path for each of them. +Input files are staged in the task work directory by creating a symbolic link with a relative path for each file. ##### `'symlink'` -Input files are staged in the task work directory by creating a symbolic link with an absolute path for each of them (default). +Input files are staged in the task work directory by creating a symbolic link with an absolute path for each file (default). diff --git a/docs/reference/process/directives/stage-out-mode.mdx b/docs/reference/process/directives/stage-out-mode.mdx index 2eb88f1fff..7ac61b917a 100644 --- a/docs/reference/process/directives/stage-out-mode.mdx +++ b/docs/reference/process/directives/stage-out-mode.mdx @@ -7,7 +7,9 @@ description: Reference for the `stageOutMode` process directive. The `stageOutMode` directive defines how output files are staged out from the scratch directory to the task work directory. -The following modes are supported: +## Options + +The following modes are available: ##### `'copy'` @@ -17,7 +19,7 @@ Output files are copied from the scratch directory to the work directory. -Output files are copied from the scratch directory to the work directory by using the [fcp](https://github.com/Svetlitski/fcp) utility (note: it must be available in the task environment). +Output files are copied from the scratch directory to the work directory using the [fcp](https://github.com/Svetlitski/fcp) utility, which must be available in the task environment. ##### `'move'` @@ -27,10 +29,12 @@ Output files are moved from the scratch directory to the work directory. -Output files are copied from the scratch directory to the work directory by using the [rclone](https://rclone.org) utility (note: it must be available in the task environment). +Output files are copied from the scratch directory to the work directory using the [rclone](https://rclone.org) utility, which must be available in the task environment. ##### `'rsync'` -Output files are copied from the scratch directory to the work directory by using the `rsync` utility. +Output files are copied from the scratch directory to the work directory using the `rsync` utility. + +## See also -See also: [scratch](./scratch) +- [scratch](./scratch) diff --git a/docs/reference/process/directives/store-dir.mdx b/docs/reference/process/directives/store-dir.mdx index a76db4de19..3f3e911899 100644 --- a/docs/reference/process/directives/store-dir.mdx +++ b/docs/reference/process/directives/store-dir.mdx @@ -9,6 +9,8 @@ The `storeDir` directive stores task outputs in a permanent *store directory* in On subsequent runs, each task is executed only if the declared output files do not exist in the store directory. When the files are present, the task is skipped and these files are used as the task outputs. +## Usage + The following example shows how to use the `storeDir` directive to create a directory containing a BLAST database for each species specified by an input parameter: ```nextflow @@ -33,7 +35,7 @@ Caveats: - The `env`, `eval`, and `stdout` output qualifiers cannot be used with `storeDir` because they rely on helper files in the task directory. Use `path` outputs instead. -- If a process uses `storeDir` and all of its outputs are optional, the process will always be skipped, even if the store directory is empty. Avoid this issue by specifying at least one required file output. +- If a process uses `storeDir` and all of its outputs are optional, the process is always skipped, even if the store directory is empty. Avoid this issue by specifying at least one required file output. - The `storeDir` directive is not a replacement for publishing outputs. Use the [publishDir](./publish-dir) directive or [workflow outputs][workflow-output-def] instead. diff --git a/docs/reference/process/directives/tag.mdx b/docs/reference/process/directives/tag.mdx index f3aeab3ae5..6d6e2ceff1 100644 --- a/docs/reference/process/directives/tag.mdx +++ b/docs/reference/process/directives/tag.mdx @@ -5,7 +5,11 @@ description: Reference for the `tag` process directive. # `tag` -The `tag` directive defines a custom identifier for each task execution. For example: +The `tag` directive defines a custom identifier for each task execution. + +## Usage + +For example: ```nextflow process hello { @@ -26,7 +30,7 @@ workflow { } ``` -The above example logs each task with its corresponding tag: +This example logs each task with its corresponding tag: ``` [6e/28919b] Submitted process > hello (alpha) diff --git a/docs/reference/process/directives/time.mdx b/docs/reference/process/directives/time.mdx index 3f27539f5e..e4f96b05be 100644 --- a/docs/reference/process/directives/time.mdx +++ b/docs/reference/process/directives/time.mdx @@ -5,7 +5,11 @@ description: Reference for the `time` process directive. # `time` -The `time` directive defines the maximum runtime for each task. For example: +The `time` directive defines the maximum runtime for each task. + +## Usage + +For example: ```nextflow process hello { @@ -18,7 +22,7 @@ process hello { } ``` -The following suffixes can be used to specify duration values: +Use the following suffixes to specify duration values: - `ms`: milliseconds - `s`: seconds @@ -28,11 +32,17 @@ The following suffixes can be used to specify duration values: See [Duration][stdlib-types-duration] for more information. -:::note -This directive is only used by certain executors. Refer to the [Executors][executor-page] page to see which executors support this directive. -::: +## Executor support + +This directive is only used by certain executors. See [Executors][executor-page] for the executors that support this directive. + +## See also -See also: [cpus](./cpus), [disk](./disk), [memory](./memory), [queue](./queue), [dynamic task resources][dynamic-task-resources] +- [cpus](./cpus) +- [disk](./disk) +- [memory](./memory) +- [queue](./queue) +- [Dynamic task resources][dynamic-task-resources] [dynamic-task-resources]: ../../../process#dynamic-task-resources [executor-page]: ../../../executor diff --git a/docs/reference/process/inputs-outputs-typed.mdx b/docs/reference/process/inputs-outputs-typed.mdx index d9ad1f7af1..4a1f7c2dd2 100644 --- a/docs/reference/process/inputs-outputs-typed.mdx +++ b/docs/reference/process/inputs-outputs-typed.mdx @@ -11,7 +11,7 @@ description: Reference for the stage directives and output functions available i Typed processes require the `nextflow.enable.types` feature flag to be enabled in every script that uses them. ::: -### Stage directives +## Stage directives The following directives can be used in the `stage:` section of a typed process: @@ -29,9 +29,9 @@ Stages a collection of files into the task directory under the given alias. ##### `stdin( value: String )` -Stages the given value as the standard input (i.e., `stdin`) to the task script. +Stages the given value as the standard input (`stdin`) to the task script. -### Outputs +## Outputs The following functions are available in the `output:` and `topic:` sections of a typed process: @@ -71,11 +71,11 @@ Maximum number of directory levels to visit (default: no limit). ###### `optional: Boolean` -When `true`, the task will not fail if the given file is missing (default: `false`). +When `true`, the task does not fail if the given file is missing (default: `false`). ###### `type: String` -Type of paths returned, either `file`, `dir` or `any` (default: `any`, or `file` if the given file name contains a double star (`**`)). +Type of paths returned, either `file`, `dir`, or `any` (default: `any`, or `file` if the given file name contains a double star (`**`)). ##### `files( pattern: String, [options] ) -> Set` diff --git a/docs/reference/process/inputs-outputs.mdx b/docs/reference/process/inputs-outputs.mdx index 774f544b24..459e526671 100644 --- a/docs/reference/process/inputs-outputs.mdx +++ b/docs/reference/process/inputs-outputs.mdx @@ -5,11 +5,11 @@ description: Reference for the input and output qualifiers available in a legacy # Inputs and outputs (legacy) -### Inputs +## Inputs ##### `val( identifier )` -Declare a variable input. The received value can be any type, and it will be made available to the process body (i.e. `script`, `shell`, `exec`) as a variable given by `identifier`. +Declare a variable input. The received value can be any type. It is made available to the process body (that is, the `script`, `shell`, or `exec` section) as a variable given by `identifier`. ##### `file( identifier | stageName )` @@ -17,19 +17,19 @@ Declare a variable input. The received value can be any type, and it will be mad Use `path` instead. -Declare a file input. The received value can be any type, and it will be staged into the task directory. If the received value is not a file or collection of files, it is implicitly converted to a string and written to a file. +Declare a file input. The received value can be any type, and it is staged into the task directory. If the received value is not a file or collection of files, it is implicitly converted to a string and written to a file. -The argument can be an identifier or string. If an identifier, the received value will be made available to the process body as a variable. If a string, the received value will be staged into the task directory under the given alias. +The argument can be an identifier or string. If an identifier, the received value is made available to the process body as a variable. If a string, the received value is staged into the task directory under the given alias. ##### `path( identifier | stageName )` -Declare a file input. The received value should be a file or collection of files and will be staged into the task directory. +Declare a file input. The received value should be a file or collection of files, which are staged into the task directory. :::tip See [Multiple input files][process-multiple-input-files] for more information about accepting collections of files. ::: -The argument can be an identifier or string. If an identifier, the received value will be made available to the process body as a variable. If a string, the received value will be staged into the task directory under the given alias. +The argument can be an identifier or string. If an identifier, the received value is made available to the process body as a variable. If a string, the received value is staged into the task directory under the given alias. Available options: @@ -37,7 +37,7 @@ Available options: -Specify the number of expected files. Can be a number, e.g. `'1'`, or a range, e.g. `'1..*'`. If a task receives an invalid number of files for this `path` input, it will fail. +Specify the number of expected files. Can be a number, for example, `'1'`, or a range, for example, `'1..*'`. If a task receives an invalid number of files for this `path` input, the task fails. ###### `name` @@ -49,11 +49,11 @@ Alias of `name`. ##### `env( name )` -Declare an environment variable input. The received value should be a string, and it will be exported to the task environment as an environment variable given by `name`. +Declare an environment variable input. The received value should be a string, and it is exported to the task environment as an environment variable given by `name`. ##### `stdin` -Declare a `stdin` input. The received value should be a string, and it will be provided as the standard input (i.e. `stdin`) to the task script. It should be declared only once for a process. +Declare a `stdin` input. The received value should be a string, and it is provided as the standard input (`stdin`) to the task script. Declare it only once per process. ##### `tuple( arg1, arg2, ... )` @@ -61,11 +61,11 @@ Declare a tuple input. Each argument should be an input declaration such as `val The received value should be a tuple with the same number of elements as the `tuple` declaration, and each received element should be compatible with the corresponding `tuple` argument. Each tuple element is treated the same way as if it were a standalone input. -### Outputs +## Outputs ##### `val( value )` -Declare a variable output. The argument can be any value, and it can reference any output variables defined in the process body (i.e. variables declared without the `def` keyword). +Declare a variable output. The argument can be any value, and it can reference any output variables defined in the process body (that is, variables declared without the `def` keyword). ##### `file( pattern )` @@ -75,7 +75,7 @@ Use `path` instead. Declare a file output. It receives the output files from the task environment that match the given pattern. -Multiple patterns can be specified using the colon separator (`:`). The union of all files matched by each pattern will be collected. +Multiple patterns can be specified using the colon separator (`:`). The union of all files matched by each pattern is collected. ##### `path( pattern, [options] )` @@ -87,12 +87,12 @@ Available options: -Specify the number of expected files. Can be a number or a range. If a task produces an invalid number of files for this `path` output, it will fail. +Specify the number of expected files. Can be a number or a range. If a task produces an invalid number of files for this `path` output, the task fails. -If the arity is `1`, a single file will be emitted. Otherwise, a list will always be emitted, even if only one file is produced. +If the arity is `1`, a single file is emitted. Otherwise, a list is always emitted, even if only one file is produced. :::warning -If the arity is not specified, a single file or list will be emitted based on whether a single file or multiple files are produced at runtime, resulting potentially in an output channel with a mixture of files and file collections. +If the arity is not specified, a single file or list is emitted based on whether a single file or multiple files are produced at runtime. This can result in an output channel with a mixture of files and file collections. ::: ###### `followLinks` @@ -117,7 +117,7 @@ Maximum number of directory levels to visit (default: no limit). ###### `type` -Type of paths returned, either `file`, `dir` or `any` (default: `any`, or `file` if the specified file name pattern contains a double star (`**`)). +Type of paths returned, either `file`, `dir`, or `any` (default: `any`, or `file` if the specified file name pattern contains a double star (`**`)). ##### `env( name )` @@ -137,13 +137,13 @@ Declare a `stdout` output. It receives the standard output of the task script. Declare an `eval` output. It receives the standard output of the given command, which is executed in the task environment after the task script. -If the command fails, the task will also fail. +If the command fails, the task also fails. ##### `tuple( arg1, arg2, ... )` Declare a tuple output. Each argument should be an output declaration such as `val`, `path`, `env`, `stdin`, or `eval`. Each tuple element is treated the same way as if it were a standalone output. -### Generic options +## Generic options The following options are available for all process outputs: @@ -153,7 +153,7 @@ Defines the name of the output channel. ##### `optional: true | false` -When `true`, the task will not fail if the specified output is missing (default: `false`). +When `true`, the task does not fail if the specified output is missing (default: `false`). ##### `topic: ` diff --git a/docs/reference/process/task-properties.mdx b/docs/reference/process/task-properties.mdx index 3654f26e20..3a9de4819e 100644 --- a/docs/reference/process/task-properties.mdx +++ b/docs/reference/process/task-properties.mdx @@ -17,7 +17,7 @@ The current task attempt. The exit code returned by the task script. -The exit code is only available after the task has been executed (e.g., the [errorStrategy](./directives/error-strategy) directive). +The exit code is only available after the task has been executed, for example, in the [`errorStrategy`](./directives/error-strategy) directive. ##### `task.hash` @@ -41,7 +41,7 @@ The task name. The exception reported by the previous task attempt. -Since the exception is available after a failed task attempt, it can only be accessed when retrying a failed task execution, i.e., when `task.attempt` is greater than 1. +Because the exception is available only after a failed task attempt, it can only be accessed when retrying a failed task execution, that is, when `task.attempt` is greater than 1. ##### `task.previousTrace` @@ -49,7 +49,7 @@ Since the exception is available after a failed task attempt, it can only be acc The trace record associated with the previous task attempt. -Since the trace record is available after a failed task attempt, it can only be accessed when retrying a failed task execution, i.e., when `task.attempt` is greater than 1. See [Trace file][trace-report] for a list of available fields. +Because the trace record is available only after a failed task attempt, it can only be accessed when retrying a failed task execution, that is, when `task.attempt` is greater than 1. See [Trace file][trace-report] for a list of available fields. :::note The trace fields `%cpu` and `%mem` can be accessed as `pcpu` and `pmem`, respectively. @@ -66,7 +66,7 @@ The name of the process that spawned the task. The unique directory path for the task. :::note -[Directive values](./directives) for a task can be accessed via `task.`. See [Task directive values][task-directive-values] for more information. +[Directive values](./directives) for a task can be accessed as `task.`. See [Task directive values][task-directive-values] for more information. ::: [task-directive-values]: ../../process#using-task-directive-values