diff --git a/docs/source/tutorials/hemeraIn5min.rst b/docs/source/tutorials/hemeraIn5min.rst index 2c1a9b7487..becd4b78ba 100644 --- a/docs/source/tutorials/hemeraIn5min.rst +++ b/docs/source/tutorials/hemeraIn5min.rst @@ -1,13 +1,14 @@ .. _hemeraIn5min: -PIConGPU in 5 Minutes on Hemera -=============================== +PIConGPU in 5 Minutes on RoSi +============================= A guide to run, but not understand PIConGPU. -It is aimed at users of the high performance computing (HPC) cluster `"Hemera" at the HZDR `_, +It is aimed at users of the high performance computing (HPC) cluster `"RoSi" at the HZDR `_, but should be applicable to other HPC clusters with slight adjustments. -This guide needs **shell access** (probably via :command:`ssh`) and :command:`git`. +This guide needs **shell access** (probably via :command:`ssh`) and :command:`git` (preinstalled on most systems including RoSi). +The RoSi shell can also be used from a browser (currently there is no support for Firefox) at ``_ (accessible from the intranet). Consider getting familiar with the shell (*command line*, usually :command:`bash`) and git. Please also read the tutorial for your local HPC cluster. @@ -15,7 +16,8 @@ Please also read the tutorial for your local HPC cluster. resources for the command line (bash) `a tutorial `_ | `another tutorial `_ | - `scripting by examples `_ + `scripting by examples `_ | + `bash cheatsheet `_ resources for git `official tutorial `_ (also available as man page :manpage:`gittutorial(7)`) | @@ -23,21 +25,26 @@ Please also read the tutorial for your local HPC cluster. `brief introduction `_ | `cheatsheet (by github) `_ - Hemera at HZDR + RoSi at HZDR `official website `_ | - `presentation `_ + introduction workshop videos: `Part 1 `_ + `Part 2 `_ | internal links: - `wiki `_ | - `storage layout `_ - -We will use the following directories: + `wiki `_ | + `storage layout `_ | + `using jupyter notebook on RoSi `_ | + `in browser code editor for editing files on RoSi `_ -- :file:`~/src/picongpu`: source files from github -- :file:`~/fwkt_v100_picongpu.profile`: load the dependencies for your local environment +We will use the following files and directories: + +- :file:`~/src/picongpu`: source files from GitHub +- :file:`~/gpu-v100_picongpu.profile`: load the dependencies for your local environment - :file:`~/picongpu-projects`: scenarios to simulate -- :file:`/bigdata/hplsim/external/alice`: result data of the simulation runs (*scratch* storage) +- :file:`/bigdata/hplsim/scratch/`: result data of the simulation runs (*scratch* storage). + External users may have to use :file:`/bigdata/hplsim/external/` instead. Please replace them whenever appropriate. +Replace ```` with your username, you can find it by running the :command:`whoami` command on the cluster. Get the Source -------------- @@ -50,7 +57,6 @@ Use :command:`git` to obtain the source and use the current ``dev`` branch and p .. note:: If you get the error ``git: command not found`` load git by invoking ``module load git`` and try again. Attention: the example uses the ``dev`` branch instead of the latest stable release. - Due to driver changes on hemera the modules configuration of the last release might be outdated. Setup ----- @@ -60,25 +66,26 @@ You need :ref:`a lot of dependencies `. Luckily, other people already did the work and prepared a *profile* that you can use. Copy it to your home directory:: - cp ~/src/picongpu/etc/picongpu/hemera-hzdr/fwkt_v100_picongpu.profile.example ~/fwkt_v100_picongpu.profile + cp ~/src/picongpu/etc/picongpu/rosi-hzdr/gpu-v100_picongpu.profile.example ~/gpu-v100_picongpu.profile This profile determines which part of the HPC cluster (*partition*, also: *queue*) – and thereby the compute device(s) (type of CPUs/GPUs) – you will use. This particular profile will use `NVIDIA Volta V100 `_ GPUs. -You can view the full list of available profiles `on github `_ (look for :file:`NAME.profile.example`). +You can view the full list of available profiles `on GitHub `_ (look for :file:`NAME.profile.example`). +The system specific directories sometimes contain a README file with some further useful information. For this guide we will add our scratch directory location to this profile. Edit the profile file using your favorite editor. -If unsure use nano: ``nano ~/fwkt_v100_picongpu.profile`` (save with :kbd:`Control-o`, exit with :kbd:`Control-x`). +If unsure use nano: ``nano ~/gpu-v100_picongpu.profile`` (save with :kbd:`Control-o`, exit with :kbd:`Control-x`). Go to the end of the file and add a new line:: - export SCRATCH=/bigdata/hplsim/external/alice + export SCRATCH=/bigdata/hplsim/external/ -(Please replace ``alice`` with your username.) +(Please replace ```` with your username.) .. note:: This is the location where runtime data and all results will be stored. - If you're not on Hemera make sure you select the correct directory: + If you're not on RoSi make sure you select the correct directory: Consult the documentation of your HPC cluster where to save your data. **On HPC clusters this is probably not your home directory.** @@ -86,7 +93,7 @@ In the profile file you can also supply additional settings, like your email add Now activate your profile:: - source ~/fwkt_v100_picongpu.profile + source ~/gpu-v100_picongpu.profile .. warning:: You will have to repeat this command **every time** you want to use PIConGPU on a new shell, i.e. after logging in. @@ -104,7 +111,7 @@ If that works make sure that this directory actually exists by executing:: If you see output similar to this one everything worked and you can carry on:: total 0 - drwxr-xr-x 2 alice fwt 40 Nov 12 10:09 . + drwxr-xr-x 2 fwt 40 Nov 12 10:09 . drwxrwxrwt 17 root root 400 Nov 12 10:09 .. Create a Scenario @@ -113,15 +120,15 @@ Create a Scenario As an example we will use the predefined `LaserWakefield example `_. Create a directory and copy it:: - mkdir -p ~/picongpu-projects/tinkering - pic-create $PIC_EXAMPLES/LaserWakefield ~/picongpu-projects/tinkering/try01 - cd ~/picongpu-projects/tinkering/try01 + mkdir -p ~/picongpu-projects/laser-wakefield-example + pic-create $PIC_EXAMPLES/LaserWakefield ~/picongpu-projects/laser-wakefield-example/try01 + cd ~/picongpu-projects/laser-wakefield-example/try01 -Usually you would now adjust the files in the newly created directory ``~/picongpu-projects/tinkering/try01`` – for this introduction we will use the parameters as provided. +Usually you would now adjust the files in the newly created directory ``~/picongpu-projects/laser-wakefield-example/try01`` – for this introduction we will use the parameters as provided. .. note:: - The command :command:`pic-create` and the variable ``$PIC_EXAMPLES`` have been provided because you loaded the file :file:`~/fwkt_v100_picongpu.profile` in the previous step. - If this fails (printing ``pic-create: command not found``), make sure you load the PIConGPU profile by executing ``source ~/fwkt_v100_picongpu.profile``. + The command :command:`pic-create` and the variable ``$PIC_EXAMPLES`` have been provided because you loaded the file :file:`~/gpu-v100_picongpu.profile` in the previous step. + If this fails (printing ``pic-create: command not found``), make sure you load the PIConGPU profile by executing ``source ~/gpu-v100_picongpu.profile``. Compile and Run --------------- @@ -136,50 +143,52 @@ Your profile provides a helper command for that:: Now build the scenario:: # switch to the scenario directory if you haven't already - cd ~/picongpu-projects/tinkering/try01 - pic-build + cd ~/picongpu-projects/laser-wakefield-example/try01 + pic-build -j 6 This will take a while, go grab a coffee. +The ``-j 6`` behind ``pic-build`` tells the compiler to compile in parallel. +You can speed up the compile process by requesting more cores here. +However this will also require more memory for the compile process. If this fails, read the manual or ask a colleague. -After a successfull build, run (still on the compute node, still inside your scenario directory):: +After a successful build, run (still on the compute node, still inside your scenario directory):: - tbg -s bash -t $PICSRC/etc/picongpu/bash/mpiexec.tpl -c etc/picongpu/1.cfg $SCRATCH/tinkering/try01/run01 + tbg -s bash -t $PICSRC/etc/picongpu/bash/mpiexec.tpl -c etc/picongpu/1.cfg $SCRATCH/laser-wakefield-example/try01/run01 - :command:`tbg`: tool provided by PIConGPU - ``bash``: the “submit system”, e.g. use ``sbatch`` for slurm -- ``$PICSRC``: the path to your PIConGPU source code, automatically set when sourcing :file:`fwkt_v100_picongpu.profile` +- ``$PICSRC``: the path to your PIConGPU source code, automatically set when sourcing :file:`gpu-v100_picongpu.profile` - :file:`$PICSRC/etc/picongpu/bash/mpiexec.tpl`: options for the chosen submit system - :file:`etc/picongpu/1.cfg`: runtime options (number of GPUs, etc.) -- :file:`$SCRATCH/tinkering/try01/run01`: not-yet-existing destination for your result files +- :file:`$SCRATCH/laser-wakefield-example/try01/run01`: not-yet-existing destination for your result files .. note:: - Usually you would use the *workload manager* (`SLURM `_ on Hemera) to submit your jobs + Usually you would use the *workload manager* (`SLURM `_ on RoSi) to submit your jobs instead of running them interactively like we just did. You can try that with:: # go back to the login node exit hostname - # ...should now display hemera4.cluster or hemera5.cluster + # ...should now display one of rosi5, rosi4, ... # resubmit your simulation with a new directory: - tbg -s sbatch -c etc/picongpu/1.cfg -t etc/picongpu/hemera-hzdr/fwkt_v100.tpl $SCRATCH/tinkering/try01/run02 + tbg -s sbatch -c etc/picongpu/1.cfg -t etc/picongpu/rosi-hzdr/gpu-v100.tpl $SCRATCH/laser-wakefield-example/try01/run02 This will print a confirmation message (e.g. ``Submitted batch job 3769365``), but no output of PIConGPU itself will be printed. Using ``squeue -u $USER`` you can view the current status of your job. Note that we not only used a different "submit system" ``sbatch``, - but also changed the template file to :file:`etc/picongpu/hemera-hzdr/fwkt_v100.tpl`. - (This template file is directly located in your project directory.`) + but also changed the template file to :file:`etc/picongpu/rosi-hzdr/gpu-v100.tpl`. + (This template file is directly located in your project directory.) Both profile and template file are built for the same compute device, the NVIDIA Volta "V100" GPU. - Examine the Results ------------------- -Results are located at :file:`$SCRATCH/tinkering/try01/run01`. +Results are located at :file:`$SCRATCH/laser-wakefield-example/try01/run01`. To view pretty pictures from a linux workstation you can use the following process (execute on your workstation, **not the HPC cluster**):: @@ -187,15 +196,39 @@ To view pretty pictures from a linux workstation you can use the following proce mkdir -p ~/mnt/scratch # Mount the data directory using sshfs - sshfs -o default_permissions -o idmap=user -o uid=$(id -u) -o gid=$(id -g) hemera5:DATADIR ~/mnt/scratch/ + sshfs -o default_permissions -o idmap=user -o uid=$(id -u) -o gid=$(id -g) rosi5:DATADIR ~/mnt/scratch/ -Substitute DATADIR with the full path to your data (*scratch*) directory, e.g. :file:`/bigdata/hplsim/external/alice`. +Substitute DATADIR with the full path to your data (*scratch*) directory, e.g. :file:`/bigdata/hplsim/scratch/alice`. Browse the directory using a file browser/image viewer. -Check out :file:`~/mnt/scratch/tinkering/try01/run01/simOutput/pngElectronsYX/` for image files. +Check out :file:`~/mnt/scratch/laser-wakefield-example/try01/run01/simOutput/pngElectronsYX/` for image files. -Further Reading ---------------- +On RoSi, you can also use the in-browser file manager to access your files. +Go to ``_, log in, click on the Home Directory icon under Files, and navigate to :file:`/bigdata/hplsim/scratch//laser-wakefield-example/try01/run01/simOutput/pngElectronsYX/`. + +openPMD +^^^^^^^ + +The PNG output is great for having a quick look at the example results, but it is a rather inefficient way of generating and storing your simulation results. +In general, we write simulation data into ADIOS2 (``.bp``, ``.bp5``, ``.bp4`` extensions) or HDF5 (``.h5``) files with parallel I/O that follow the structure defined by the community openPMD standard. +The output can be configured using the :ref:`openPMD plugin `. +The example you just run also provides some basic openPMD output that can be now found under :file:`simOutput/openPMD/`. +You should make yourself familiar with how to read and visualize such data. +Here are some useful resources: + +- `openPMD standard `_ and the `PIC extension to the standard `_ +- `openPMD API docs `_ API for reading and writing openPMD output. + Powerful tool, its proper use requires some familiarity with the standard itself. +- `openPMD-viewer `_ Data visualization Python library built on top of the API. +- `openPMD-scipp `_ Easy visualization of mesh data (fields) using the `plopp `_ and `scipp `_ Python libraries. +- `openPMD pandas support `_ Straightforward solution for working with particle data. +- `openpmd-ls `_ Command for getting an overview over the content of an openPMD output via command line. +- `bpls `_ Powerful tool for inspecting individual ADIOS2 files via command line. + +We recommend trying out the openPMD-viewer and the openPMD-scipp tools first, followed by a quick read through the standard itself and the python part of the `first read API example `_. + +When directly using the openPMD API make sure you make yourself familiar with when you need to call ``series.flush()``. +See `Flush Chunk `_ and `Deferred Data API Contract `_ for more details. +Forgetting it or modifying the data before flush was performed is a common user mistake that can lead to very confusing results. -You now know the process of using PIConGPU. -Carry on reading the documentation to understand it. +In addition to the standard openPMD output PIConGPU offers some reduced diagnostics via :ref:`other plugins ` including the very powerful :ref:`flexible binning plugin `.