From 3ac55287db25c3bd2851b0c459b1fe7424524bae Mon Sep 17 00:00:00 2001 From: georgiesamaha Date: Wed, 25 Jun 2025 18:34:45 +1000 Subject: [PATCH 1/8] Add persistent session info --- docs/nci_gadi.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/nci_gadi.md b/docs/nci_gadi.md index 386745888..f430e20aa 100644 --- a/docs/nci_gadi.md +++ b/docs/nci_gadi.md @@ -12,11 +12,11 @@ Please be aware that you will need to have a user account, be a member of an Gad ### Prerequisites -Before running the pipeline you will need to load Nextflow and Singularity, both of which are globally installed modules on Gadi. You can do this by running the commands below: +Before running the pipeline you will need to load Nextflow, both of which are globally installed modules on Gadi (see `/apps`). You can do this by running the commands below: ```bash module purge -module load nextflow singularity +module load nextflow ``` ### Execution command @@ -32,7 +32,7 @@ nextflow run /main.nf \ ### Cluster considerations -Please be aware that as of July 2023, NCI Gadi HPC queues **do not** have external network access. This means you will not be able to pull the workflow code base or containers if you submit your `nextflow run` command as a job on any of the standard job queues. NCI currently recommends you run your Nextflow head job either in a GNU screen or tmux session from the login node or submit it as a job to the [copyq](https://opus.nci.org.au/display/Help/Queue+Structure). See the [nf-core documentation](https://nf-co.re/docs/usage/offline) for instructions on running pipelines offline. +Please be aware that as of June 2025, NCI Gadi HPC compute nodes **do not** have external network access. This means you will not be able to pull the workflow code base or containers if you submit your `nextflow run` command as a job on any of the standard job queues. NCI currently recommends you run your Nextflow head job either in a [persistent session](https://opus.nci.org.au/spaces/Help/pages/241926895/Persistent+Sessions) from the login node or submit it as a job to the [copyq](https://opus.nci.org.au/display/Help/Queue+Structure). See the [nf-core documentation](https://nf-co.re/docs/usage/offline) for instructions on running pipelines offline. This config currently determines which Gadi queue to submit your task jobs to based on the amount of memory required. For the sake of resource and cost (service unit) efficiency, the following rules are applied by this config: From f74b747668ec745b32c452903062aed9884aca2d Mon Sep 17 00:00:00 2001 From: georgiesamaha Date: Wed, 25 Jun 2025 18:47:18 +1000 Subject: [PATCH 2/8] Update gadi docs --- docs/nci_gadi.md | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/docs/nci_gadi.md b/docs/nci_gadi.md index f430e20aa..12a7272ab 100644 --- a/docs/nci_gadi.md +++ b/docs/nci_gadi.md @@ -32,7 +32,24 @@ nextflow run /main.nf \ ### Cluster considerations -Please be aware that as of June 2025, NCI Gadi HPC compute nodes **do not** have external network access. This means you will not be able to pull the workflow code base or containers if you submit your `nextflow run` command as a job on any of the standard job queues. NCI currently recommends you run your Nextflow head job either in a [persistent session](https://opus.nci.org.au/spaces/Help/pages/241926895/Persistent+Sessions) from the login node or submit it as a job to the [copyq](https://opus.nci.org.au/display/Help/Queue+Structure). See the [nf-core documentation](https://nf-co.re/docs/usage/offline) for instructions on running pipelines offline. +#### External network access +Please be aware that as of June 2025, NCI Gadi HPC compute nodes **do not** have external network access. This means you will not be able to pull the workflow code base or containers if you submit your `nextflow run` command as a job on any of the standard job queues. NCI currently recommends you run your Nextflow head job in a [persistent session](https://opus.nci.org.au/spaces/Help/pages/241926895/Persistent+Sessions) from the login node or submit it as a job to the [copyq](https://opus.nci.org.au/display/Help/Queue+Structure). See the [nf-core documentation](https://nf-co.re/docs/usage/offline) for instructions on running pipelines offline. + +#### Downloading containers + +This config requires Nextflow to use [Singularity](https://www.nextflow.io/docs/latest/container.html#singularity) to execute processes. Before a process can be executed, any nf-core pipeline you run will download that container to a local cache. This config file specifies the download and storage location with: + +``` +cacheDir = "/scratch/params.project/$(whoami)/singularity_cache" +``` + +Additionally, the project parameter is defined using the PBS environmental variable `$PROJECT`: + +``` +params.project = System.getenv("PROJECT") +``` + +#### Gadi queues and job submission This config currently determines which Gadi queue to submit your task jobs to based on the amount of memory required. For the sake of resource and cost (service unit) efficiency, the following rules are applied by this config: @@ -56,7 +73,18 @@ The version of Nextflow installed on Gadi has been modified to make it easier to process { executor = 'pbspro' project = 'aa00' - storage = 'scratch/aa00+gdata/aa00' + storage = 'scratch/params.project' + ... +} +``` + +Similarly, you can override the default storage location to change locations or add multiple locations by manually overriding the storage definition inside the process scope. For example: + +```nextflow +process { + executor = 'pbspro' + project = System.getenv("PROJECT") + storage = 'scratch/aa11+gdata/aa11' ... } ``` From 6604a74392d6de868d02b33e559e52c156aebdc7 Mon Sep 17 00:00:00 2001 From: georgiesamaha Date: Wed, 25 Jun 2025 18:48:56 +1000 Subject: [PATCH 3/8] Update gadi conf --- conf/nci_gadi.config | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/conf/nci_gadi.config b/conf/nci_gadi.config index e726bf0e2..d7362799e 100644 --- a/conf/nci_gadi.config +++ b/conf/nci_gadi.config @@ -10,16 +10,12 @@ params { singularity { enabled = true autoMounts = true + cacheDir = "/scratch/params.project/$(whoami)/singularity_cache" } // Submit up to 300 concurrent jobs (Gadi exec max) -// pollInterval and queueStatInterval of every 5 minutes -// submitRateLimit of 20 per minute executor { queueSize = 300 - pollInterval = '5 min' - queueStatInterval = '5 min' - submitRateLimit = '20 min' } // Define process resource limits From 8de43488dba208931eb7ef19d9bf70d903572b2a Mon Sep 17 00:00:00 2001 From: georgiesamaha Date: Wed, 25 Jun 2025 18:53:48 +1000 Subject: [PATCH 4/8] Update gadi conf cacheDir --- conf/nci_gadi.config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf/nci_gadi.config b/conf/nci_gadi.config index d7362799e..3a9e421a4 100644 --- a/conf/nci_gadi.config +++ b/conf/nci_gadi.config @@ -10,7 +10,7 @@ params { singularity { enabled = true autoMounts = true - cacheDir = "/scratch/params.project/$(whoami)/singularity_cache" + cacheDir = "/scratch/params.project/singularity_cache" } // Submit up to 300 concurrent jobs (Gadi exec max) From e09360e954ecf660f8f47b761c883d2c9551d9ad Mon Sep 17 00:00:00 2001 From: georgiesamaha Date: Wed, 25 Jun 2025 18:55:47 +1000 Subject: [PATCH 5/8] Add gadi CODEOWNER --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 28bc29e7e..2a8e9d00c 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -7,3 +7,4 @@ **/unsw_katana** @jscgh **/seadragon** @jiawku **/fred_hutch** @derrik-gratz +**/nci_gadi** @georgiesamaha From 7666d32e1cebcb3725c12c6427eb7d31c112c4be Mon Sep 17 00:00:00 2001 From: georgiesamaha Date: Wed, 25 Jun 2025 19:05:25 +1000 Subject: [PATCH 6/8] Fix cache --- conf/nci_gadi.config | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/conf/nci_gadi.config b/conf/nci_gadi.config index 3a9e421a4..b7cb75894 100644 --- a/conf/nci_gadi.config +++ b/conf/nci_gadi.config @@ -1,16 +1,17 @@ // NCI Gadi nf-core configuration profile params { config_profile_description = 'NCI Gadi HPC profile provided by nf-core/configs' - config_profile_contact = 'Georgie Samaha (@georgiesamaha), Matthew Downton (@mattdton)' + config_profile_contact = 'Georgie Samaha (@georgiesamaha), Kisaru Liyanage (@kisarur)' config_profile_url = 'https://opus.nci.org.au/display/Help/Gadi+User+Guide' project = System.getenv("PROJECT") + storage = "gdata/${params.project}+scratch/${params.project}" } // Enable use of Singularity to run containers singularity { enabled = true autoMounts = true - cacheDir = "/scratch/params.project/singularity_cache" + cacheDir = "/scratch/${params.project}/singularity" } // Submit up to 300 concurrent jobs (Gadi exec max) From b4babd5c08e4b613e08e24a1e59e06ca945f4d64 Mon Sep 17 00:00:00 2001 From: georgiesamaha Date: Wed, 25 Jun 2025 19:10:19 +1000 Subject: [PATCH 7/8] Fix lint issue and typo --- docs/nci_gadi.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/nci_gadi.md b/docs/nci_gadi.md index 12a7272ab..5b2117dbc 100644 --- a/docs/nci_gadi.md +++ b/docs/nci_gadi.md @@ -33,6 +33,7 @@ nextflow run /main.nf \ ### Cluster considerations #### External network access + Please be aware that as of June 2025, NCI Gadi HPC compute nodes **do not** have external network access. This means you will not be able to pull the workflow code base or containers if you submit your `nextflow run` command as a job on any of the standard job queues. NCI currently recommends you run your Nextflow head job in a [persistent session](https://opus.nci.org.au/spaces/Help/pages/241926895/Persistent+Sessions) from the login node or submit it as a job to the [copyq](https://opus.nci.org.au/display/Help/Queue+Structure). See the [nf-core documentation](https://nf-co.re/docs/usage/offline) for instructions on running pipelines offline. #### Downloading containers @@ -40,7 +41,7 @@ Please be aware that as of June 2025, NCI Gadi HPC compute nodes **do not** have This config requires Nextflow to use [Singularity](https://www.nextflow.io/docs/latest/container.html#singularity) to execute processes. Before a process can be executed, any nf-core pipeline you run will download that container to a local cache. This config file specifies the download and storage location with: ``` -cacheDir = "/scratch/params.project/$(whoami)/singularity_cache" +cacheDir = "/scratch/${params.project}/singularity" ``` Additionally, the project parameter is defined using the PBS environmental variable `$PROJECT`: From fbacde01b0497b341fe91d5978f0c3a5bffb4946 Mon Sep 17 00:00:00 2001 From: georgiesamaha Date: Wed, 25 Jun 2025 19:13:00 +1000 Subject: [PATCH 8/8] Add updates from #872 --- conf/nci_gadi.config | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/conf/nci_gadi.config b/conf/nci_gadi.config index b7cb75894..604dd4d86 100644 --- a/conf/nci_gadi.config +++ b/conf/nci_gadi.config @@ -22,8 +22,8 @@ executor { // Define process resource limits process { executor = 'pbspro' - project = System.getenv("PROJECT") - storage = 'scratch/params.project' + project = "${params.project}" + storage = "${params.storage}" module = 'singularity' cache = 'lenient' stageInMode = 'symlink'