diff --git a/.docusaurus_site/sidebars.js b/.docusaurus_site/sidebars.js index 7effd84435..04e79669ab 100644 --- a/.docusaurus_site/sidebars.js +++ b/.docusaurus_site/sidebars.js @@ -83,7 +83,8 @@ module.exports = { "azure", "google", "kubernetes", - "fusion" + "fusion", + "seqera-filesystem" ] }, { diff --git a/docs/migrations/26-04.mdx b/docs/migrations/26-04.mdx index f8ae8390a6..404c8200c8 100644 --- a/docs/migrations/26-04.mdx +++ b/docs/migrations/26-04.mdx @@ -396,6 +396,8 @@ workflow { A Platform access token with appropriate permissions is required to download non-public datasets. It can be specified using the `TOWER_ACCESS_TOKEN` environment variable or the `tower.accessToken` config option. +See [Seqera file system][seqera-filesystem-page] for details. + ## Breaking changes - The [strict syntax parser][strict-syntax-page] is now enabled by default. The legacy parser can be enabled by setting the `NXF_SYNTAX_PARSER` environment variable to `v1`. @@ -439,6 +441,7 @@ A Platform access token with appropriate permissions is required to download non [process-hints]: ../reference/process#hints [process-typed-page]: ../process-typed [script-records]: ../script#records +[seqera-filesystem-page]: ../seqera-filesystem [static-typing-first-preview]: ./25-10#static-typing-preview [stdlib-listDirectory]: ../reference/stdlib-types#listdirectory---iterablepath [strict-syntax-page]: ../strict-syntax diff --git a/docs/reference/config.mdx b/docs/reference/config.mdx index 5beece4846..e25a9e7b59 100644 --- a/docs/reference/config.mdx +++ b/docs/reference/config.mdx @@ -1784,8 +1784,6 @@ The `seqera.executor` scope configures the Seqera scheduler service for the [Seq The following settings are available: -The following settings are available: - ###### `seqera.executor.autoLabels` When `true`, automatically adds workflow metadata labels to the session with the `nextflow.io/` prefix (default: `false`). The following labels are added: `projectName`, `userName`, `runName`, `sessionId`, `resume`, `revision`, `commitId`, `repository`, `manifestName`, `runtimeVersion`. A `seqera.io/runId` label is also added, computed as a SipHash of the session ID and run name. diff --git a/docs/seqera-filesystem.mdx b/docs/seqera-filesystem.mdx new file mode 100644 index 0000000000..d59b3183dc --- /dev/null +++ b/docs/seqera-filesystem.mdx @@ -0,0 +1,130 @@ +--- +title: Seqera file system +description: Read Seqera Platform datasets and data links in Nextflow pipelines using the seqera:// URI scheme. +--- + + + +Nextflow can read Seqera Platform datasets and data links using the `seqera` URI scheme. +Use a `seqera://` path anywhere Nextflow reads an input file. +Your pipeline then reads data registered in a Seqera Platform workspace without hard-coding the storage location or the credentials. + +The `nf-tower` plugin, bundled with Nextflow, provides the Seqera file system. +The file system is read-only. A `seqera://` path cannot be a work directory, an output directory, or a publish target. + +See [Working with files][working-with-files] and the [Path][stdlib-types-path] reference for the available file operations. + +## Authentication + +Reading a `seqera://` path requires a [Seqera Platform](https://seqera.io) access token with permission to read the target datasets or data links. See [Authentication](https://docs.seqera.io/platform-cloud/api/overview#authentication) in the Seqera Platform documentation to create one. + +Provide the token using the `TOWER_ACCESS_TOKEN` environment variable: + +```bash +export TOWER_ACCESS_TOKEN='' +``` + +Alternatively, specify the token in your configuration with the [`tower.accessToken`][config-tower-accessToken] setting, along with the plugin declaration: + +```nextflow +plugins { + id 'nf-tower' +} + +tower.accessToken = '' +``` + +Nextflow loads the `nf-tower` plugin automatically when the `TOWER_ACCESS_TOKEN` environment variable is set, or when [`tower.enabled`][config-tower-enabled] or [`fusion.enabled`][config-fusion-enabled] is `true`. Otherwise, declare the plugin explicitly. Without it, Nextflow does not register the `seqera` scheme. + +:::note +Setting `tower.enabled = true` also registers the scheme, but it turns on run monitoring with Seqera Platform. To read data without reporting your runs, declare the plugin instead. +::: + +For Seqera Platform Enterprise, set the API endpoint of your instance with the [`tower.endpoint`][config-tower-endpoint] setting or the `TOWER_API_ENDPOINT` environment variable. + +## Dataset paths + +A [dataset](https://docs.seqera.io/platform-cloud/data/datasets) is a versioned CSV or TSV file stored in a Seqera Platform workspace, typically a samplesheet. Dataset paths take the following form: + +``` +seqera:////datasets/[@] +``` + +For example, the following pipeline reads a samplesheet from the `showcase` workspace of the `seqeralabs` organization: + +```nextflow +params.dataset = 'seqera://seqeralabs/showcase/datasets/sarek_samples' + +workflow { + channel.fromPath(params.dataset) + | splitCsv(header: true) + | view +} +``` + +Omit the version to read the latest enabled version of the dataset. Append `@` to pin a specific version. A pinned path keeps a run reproducible after someone uploads a new version: + +```nextflow +params.dataset = 'seqera://seqeralabs/showcase/datasets/sarek_samples@2' +``` + +## Data link paths + + + +A [data link](https://docs.seqera.io/platform-cloud/data/data-explorer) is a cloud storage bucket or container registered in a Seqera Platform workspace, along with the credentials that grant access to it. Data link paths take the following form: + +``` +seqera:////data-links/// +``` + +The segments are: + +- ``: the cloud provider of the data link, such as `aws`, `azure`, or `google`. +- ``: the name of the data link in the workspace, as shown in **Data Explorer**. This is not the name of the underlying bucket or container. +- ``: the path of the file or directory within the data link. + +For example, the following pipeline reads a file from the `inputs` data link: + +```nextflow +workflow { + println file('seqera://acme/research/data-links/aws/inputs/data/sequences.fa').text +} +``` + +Because the data link supplies the credentials, your pipeline does not need its own cloud credentials to read the data. + +:::note +To write pipeline outputs to the storage behind a data link, use the native path of the bucket or container, such as `s3://my-bucket/results`. +::: + +## List available data + +Every level of a `seqera://` path is a directory. Use [`listDirectory()`][stdlib-listDirectory] to see the organizations, workspaces, datasets, and data links that your access token can read: + +```nextflow +workflow { + // organizations + file('seqera://').listDirectory().each { println it } + + // workspaces in an organization + file('seqera://acme').listDirectory().each { println it } + + // datasets in a workspace + file('seqera://acme/research/datasets').listDirectory().each { println it } + + // providers with data links in a workspace + file('seqera://acme/research/data-links').listDirectory().each { println it } + + // data links for a provider + file('seqera://acme/research/data-links/aws').listDirectory().each { println it } +} +``` + +[config-fusion-enabled]: ./reference/config#fusionenabled +[config-tower-accessToken]: ./reference/config#toweraccesstoken +[config-tower-enabled]: ./reference/config#towerenabled +[config-tower-endpoint]: ./reference/config#towerendpoint +[stdlib-listDirectory]: ./reference/stdlib-types#listdirectory---iterablepath +[stdlib-types-path]: ./reference/stdlib-types#path +[working-with-files]: ./working-with-files