diff --git a/docs/source/campaigns.rst b/docs/source/campaigns.rst deleted file mode 100644 index 779a223e..00000000 --- a/docs/source/campaigns.rst +++ /dev/null @@ -1,23 +0,0 @@ -.. _campaigns: - -*************** -Campaigns -*************** - -The ``campaign`` parameter allows users to choose a template of default values for their ``zppy`` run. -Inheritance in configuration files still works with campaigns, so users may override values from a campaign's configuration file. - -Campaign configuration files can be found in the `zppy/zppy/templates directory `_. ``zppy`` currently supports the following campaigns: - -- cryosphere.cfg -- high_res_v1.cfg -- water_cycle.cfg - -High-resolution analysis (``campaign = "high_res_v1"``) is only appropriate for some simulations -(e.g., those with the oRRS18to6 ocean and sea-ice mesh). -It will only work if you: - -- have access to the `bigmem nodes on NERSC Cori `_ -- first load the necessary module with ``module load cmem``. - -# TODO: supported on Perlmutter? diff --git a/docs/source/contributing.rst b/docs/source/contributing.rst deleted file mode 100644 index 23f67682..00000000 --- a/docs/source/contributing.rst +++ /dev/null @@ -1,164 +0,0 @@ -********************************** -Contributing to This Documentation -********************************** - -.. highlight:: none - -Getting Started -=============== - -This documentation is created using -`Sphinx `_. Sphinx is an open-source tool -that makes it easy to create intelligent and beautiful documentation, written -by Georg Brandl and licensed under the BSD license. - -The documentation is maintained in the ``main`` branch of the GitHub repository. -You can include code and its corresponding documentation updates in a single pull request (PR). - -After merging a PR, GitHub Actions automates the documentation building process. -It pushes the HTML build to the ``gh-pages`` branch, which is hosted on GitHub Pages. - -Edit Documentation -================== - -Sphinx uses `reStructuredText `_ -as its markup language. For more information on how to write documentation -using Sphinx, you can refer to - -* `First Steps with Sphinx `_ -* `reStructuredText Primer `_ - -1. Make sure that you have conda in your path. On NERSC machines, you can load it with: :: - - $ module load python/3.9-anaconda-2021.11 - -2. Clone the repository and checkout a branch from ``main``: :: - - $ cd - $ git clone https://github.com//zppy.git - $ cd zppy - $ git checkout -b main - -3. Create and activate the conda development environment :: - - $ cd /zppy - $ conda env create -f conda/dev.yml - $ conda activate zppy_dev - -4. To modify the documentation, simply edit the files under ``/docs/source``. - -5. To see the changes you made to the documentation, rebuild the web pages :: - - $ cd /zppy/docs - $ make html - -6. View them locally in a web browser at ``file:////zppy/docs/_build/html/index.html``. - - - Sometimes the browser caches Sphinx docs, so you might need to delete your cache to view changes. - -7. Once you are satisfied with your modifications, commit and push them back to the repository: :: - - $ cd /zppy - $ # `/docs/_build` is ignored by git since it does not need to be pushed - $ git add . - $ git commit - $ git push - -8. <``OPTIONAL``> If you want to generate and view versioned docs: :: - - $ # After commiting to your branch - $ cd /zppy/docs - $ sphinx-multiversion source _build/html - $ # Check the `_build/html` folder for all generated versioned docs - $ # Open `_build/html//index.html` to view in browser - -The docs version selector dropdown is in the bottom left-hand corner - -9. Create a pull request from ``/zppy/branch-name`` to ``E3SM-Project/zppy/main``. - -Once this pull request is merged and GitHub Actions finishes building the docs, changes will be available on the -`zppy documentation page `_. - -How Documentation is Versioned ------------------------------- -The `sphinx-multiversion `_ package manages documentation versioning. - -``sphinx-multiversion`` is configured to generate versioned docs for available tags and -branches on local, ``origin`` (the likely name for your ````) and -``upstream`` (the likely name for your ````).. - -Branches or tags that don’t contain both the sphinx ``source`` directory and the ``conf.py`` file will be skipped automatically. - - - Run ``sphinx-multiversion source _build/html --dump-metadata`` to see which tags/branches matched. - -Initial setup (obsolete/for reference only) -=========================================== - -The instructions below only apply for the initial configuration of the -Sphinx documentation on the Github repository. They are documented here -for reference only. Do not follow them unless you are setting up documentation -for a new repository. (Adapted from `Sphinx documentation on GitHub -`_.) - -Create Sphinx conda environment (see above). - -Create a new git branch (gh-pages): :: - - $ git branch gh-pages - $ git checkout gh-pages - -Clear out any­thing from the main branch and start fresh :: - - $ git symbolic-ref HEAD refs/heads/gh-pages - $ rm .git/index - $ git clean -fdx - -Create documentation :: - - $ sphinx-quickstart - -accept suggested default options, except :: - - Separate source and build directories (y/N) [n]: y - -Edit Makefile and change ``BUILDIR`` :: - - BUILDDIR = docs - -Remove old build directory :: - - $ rmdir build - -Change the Sphinx theme to 'ReadTheDocs'. Edit 'source/conf.py and change :: - - html_theme = 'alabaster' - -to :: - - import sphinx_rtd_theme - html_theme = "sphinx_rtd_theme" - html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] - -Try building documentation :: - - $ make html - -Create an empty .nojekyll file to indicate to Github.com that this -is not a Jekyll static website: :: - - $ touch .nojekyll - -Create a top-level re-direction file: :: - - $ vi index.html - -with the following: :: - - - -Commit and push back to Github: :: - - $ git add . - $ git commit - $ git push origin gh-pages - diff --git a/docs/source/dependencies.dot b/docs/source/dependencies.dot new file mode 100644 index 00000000..b2e70c41 --- /dev/null +++ b/docs/source/dependencies.dot @@ -0,0 +1,104 @@ +digraph TaskDependencies { + // Graph settings + rankdir=LR // Left-to-right layout (TB for top-to-bottom) + node [shape=box, style=rounded] + + // Reviewing the dependencies in each `.py` file + // and the test cfgs, we come up with the following: + + // VERTICES (tasks) + + climo_diurnal_atm [label="climo: diurnal, atm"] + climo_month_atm [label="climo: monthly, atm"] + climo_month_lnd [label="climo: monthly, lnd"] + + ts_daily_atm [label="ts: daily, atm"] + ts_month_atm [label="ts: monthly, atm"] + ts_month_atm_glb [label="ts: monthly, atm, glb"] + ts_month_lnd [label="ts: monthly, lnd"] + ts_month_lnd_glb [label="ts: monthly, lnd, glb"] + ts_month_rof [label="ts: monthly, rof"] + + e3sm_to_cmip_month_atm [label="e3sm_to_cmip: monthly, atm"] + e3sm_to_cmip_month_lnd [label="e3sm_to_cmip: monthly, lnd"] + + tc_analysis + + e3sm_diags_diurnal [label="e3sm_diags: diurnal_cycle"] + e3sm_diags_climo_month_atm [label="e3sm_diags: lat_lon,zonal_mean_xy,\nzonal_mean_2d,polar,cosp_histogram,\nmeridional_mean_2d,annual_cycle_zonal_mean,zonal_mean_2d_stratosphere,\naerosol_aeronet,aerosol_budget"] + e3sm_diags_climo_month_lnd [label="e3sm_diags: lat_lon_land"] + e3sm_diags_streamflow [label="e3sm_diags: streamflow"] + e3sm_diags_tc [label="e3sm_diags: tc_analysis"] + e3sm_diags_ts_daily_atm [label="e3sm_diags: tropical_subseasonal,precip_pdf"] + e3sm_diags_ts_month_atm [label="e3sm_diags: enso_diags,qbo,\narea_mean_time_series,mp_partition"] + + mpas_analysis + + global_time_series_classic_atm [label="global_time_series:\nThe 5 atm classic plots"] + global_time_series_classic_ocn [label="global_time_series:\nThe 3 ocn classic plots"] + global_time_series_plots_atm [label="global_time_series:\natm plots"] + global_time_series_plots_ice [label="global_time_series:\nice plots"] // Currently unused + global_time_series_plots_lnd [label="global_time_series:\nlnd plots"] + global_time_series_plots_ocn [label="global_time_series:\nocn plots"] + + ilamb_lnd_atm [label="ilamb: lnd & atm"] + ilamb_lnd_only [label="ilamb: lnd only"] + + livvkit + + pcmdi_diags_mean_climate [label="pcmdi_diags: mean_climate"] + pcmdi_diags_variability_modes_atm [label="pcmdi_diags: variability_modes_atm"] + pcmdi_diags_variability_modes_cpl [label="pcmdi_diags: variability_modes_cpl"] + pcmdi_diags_enso [label="pcmdi_diags: enso"] + pcmdi_diags_synthetic_plots [label="pcmdi_diags: synthetic_plots"] + + + // EDGES (dependencies) + + // Defined in climo.py: + // None + + // Defined in ts.py: + // None + + // Defined in e3sm_to_cmip.py: + ts_month_atm -> e3sm_to_cmip_month_atm + ts_month_lnd -> e3sm_to_cmip_month_lnd + + // Defined in tc_analysis.py: + // None + + // Defined in e3sm_diags.py: + climo_month_atm -> e3sm_diags_climo_month_atm + climo_month_lnd -> e3sm_diags_climo_month_lnd + climo_diurnal_atm -> e3sm_diags_diurnal + tc_analysis -> e3sm_diags_tc + ts_month_atm -> e3sm_diags_ts_month_atm + ts_month_rof -> e3sm_diags_streamflow + ts_daily_atm -> e3sm_diags_ts_daily_atm + + // Defined in mpas_analysis.py: + mpas_analysis -> mpas_analysis // Later year sets depend on earlier runs + + // Defined in global_time_series.py: + ts_month_atm_glb -> {global_time_series_classic_atm, global_time_series_plots_atm} + ts_month_lnd_glb -> global_time_series_plots_lnd + mpas_analysis -> {global_time_series_classic_ocn, global_time_series_plots_ocn} + + // Defined in ilamb.py: + ts_month_atm -> ilamb_lnd_atm + ts_month_lnd -> {ilamb_lnd_atm, ilamb_lnd_only} + e3sm_to_cmip_month_atm -> ilamb_lnd_atm + e3sm_to_cmip_month_lnd -> {ilamb_lnd_atm, ilamb_lnd_only} + + // Defined in livvkit.py: + // Data sources in livvkit.py: "cmb", "smb", "racmo", "merra2", "ceres", "era5" + // climo extensions in tests/integration/template_weekly_comprehensive_v3.cfg: traave, native, racmo_gis, racmo_ais, merra2, era5 + // ts extensions in tests/integration/template_weekly_comprehensive_v3.cfg: energy, smb + climo_month_lnd -> livvkit [label="Specify mapping files"] + ts_month_lnd -> livvkit [label="Specify mapping files"] + + // Defined in pcmdi_diags.py: + {ts_month_atm, e3sm_to_cmip_month_atm} -> {pcmdi_diags_mean_climate, pcmdi_diags_variability_modes_atm, pcmdi_diags_variability_modes_cpl, pcmdi_diags_enso} -> pcmdi_diags_synthetic_plots + +} diff --git a/docs/source/dev_guide/archive/index.rst b/docs/source/dev_guide/archive/index.rst new file mode 100644 index 00000000..1101f6db --- /dev/null +++ b/docs/source/dev_guide/archive/index.rst @@ -0,0 +1,14 @@ +.. _archive: + +******* +Archive +******* + +This page collects documentation that is obsolete or refers to older +versions of ``zppy`` or the E3SM Unified environment. It is retained for +historical reference. + +.. toctree:: + :maxdepth: 1 + + initial_docs diff --git a/docs/source/dev_guide/archive/initial_docs.rst b/docs/source/dev_guide/archive/initial_docs.rst new file mode 100644 index 00000000..5758883f --- /dev/null +++ b/docs/source/dev_guide/archive/initial_docs.rst @@ -0,0 +1,72 @@ +********************************** +How to set up new documentation +********************************** + +.. warning:: + The instructions below only apply for the initial configuration of the + Sphinx documentation on the Github repository. They are documented here + for reference only. Do not follow them unless you are setting up documentation + for a new repository. (Adapted from `Sphinx documentation on GitHub + `_.) + +Create Sphinx conda environment (see above). + +Create a new git branch (gh-pages): :: + + $ git branch gh-pages + $ git checkout gh-pages + +Clear out any­thing from the main branch and start fresh :: + + $ git symbolic-ref HEAD refs/heads/gh-pages + $ rm .git/index + $ git clean -fdx + +Create documentation :: + + $ sphinx-quickstart + +accept suggested default options, except :: + + Separate source and build directories (y/N) [n]: y + +Edit Makefile and change ``BUILDDIR`` :: + + BUILDDIR = docs + +Remove old build directory :: + + $ rmdir build + +Change the Sphinx theme to 'ReadTheDocs'. Edit 'source/conf.py and change :: + + html_theme = 'alabaster' + +to :: + + import sphinx_rtd_theme + html_theme = "sphinx_rtd_theme" + html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] + +Try building documentation :: + + $ make html + +Create an empty .nojekyll file to indicate to GitHub.com that this +is not a Jekyll static website: :: + + $ touch .nojekyll + +Create a top-level re-direction file: :: + + $ vi index.html + +with the following: :: + + + +Commit and push back to GitHub: :: + + $ git add . + $ git commit + $ git push origin gh-pages \ No newline at end of file diff --git a/docs/source/dev_guide/ci.rst b/docs/source/dev_guide/ci.rst deleted file mode 100644 index 4d32a5c5..00000000 --- a/docs/source/dev_guide/ci.rst +++ /dev/null @@ -1,24 +0,0 @@ -Understanding CI -======================== - -In this guide, we'll cover: - -* Build Workflow -* Release Workflow -* GitHub Actions - -Build Workflow --------------- - -The build workflow is at https://github.com/E3SM-Project/zppy/blob/main/.github/workflows/build_workflow.yml. See comments in the file for in-depth explanations of each step. - -Release Workflow ----------------- - -The release workflow is at https://github.com/E3SM-Project/zppy/blob/main/.github/workflows/release_workflow.yml. See comments in the file for in-depth explanations of each step. - -GitHub Actions --------------- -Both of these workflows are run by GitHub Actions. See https://github.com/E3SM-Project/zppy/actions. - -When a pull request is made, the build workflow is run automatically on the pushed branch. When the pull request is merged, the build workflow is once again run, but this time on the ``main`` branch. diff --git a/docs/source/dev_guide/contributing.rst b/docs/source/dev_guide/contributing.rst new file mode 100644 index 00000000..0fd24064 --- /dev/null +++ b/docs/source/dev_guide/contributing.rst @@ -0,0 +1,73 @@ +********************************** +Contributing to This Documentation +********************************** + +.. highlight:: none + +Getting Started +=============== + +This documentation is created using Sphinx, which is an open-source tool +that makes it easy to create intelligent and beautiful documentation, written +by Georg Brandl and licensed under the BSD license. + +The documentation is maintained in the ``main`` branch of the GitHub repository. +You can include code and its corresponding documentation updates in a single pull request (PR). + +After merging a PR, GitHub Actions automates the documentation building process. +It pushes the HTML build to the ``gh-pages`` branch, which is hosted on GitHub Pages. + +Edit Documentation +================== + +Sphinx uses `reStructuredText `_ +as its markup language. For more information on how to write documentation +using Sphinx, you can refer to + +* `First Steps with Sphinx `_ +* `reStructuredText Primer `_ + +.. code-block:: bash + + # Sphinx is included in the dev environment, + # so we just need to set one up: + cd zppy + conda clean --all --y + conda env create -f conda/dev.yml -n env-name + conda activate env-name + python -m pip install . + + # Now, we can produce the documentation locally: + cd docs + make html + + # And transfer it to a web server path so we can see it online. + + # Example web server paths: + # Chrysalis: /lcrc/group/e3sm/public_html/diagnostic_output -> https://web.lcrc.anl.gov/public/e3sm/diagnostic_output + # Compy: /compyfs/www -> https://compy-dtn.pnl.gov + # Perlmutter: /global/cfs/cdirs/e3sm/www -> https://portal.nersc.gov/cfs/e3sm + + # For more machines, go to: + # https://github.com/E3SM-Project/mache/tree/main/mache/machines + # Select the cfg for the machine, and find the `[web_portal]` section. + # That will show the `base_path` and `base_url`. + + cp -r _build/ base_path/$USER/zppy_docs + # Needs to be 755 to show up on https://portal.nersc.gov/cfs/e3sm + chmod -R 755 base_path/$USER/zppy_docs + + # Optional -- if you want to generate and view versioned docs: + sphinx-multiversion source _build/html + +Pull requests should bundle together associated code changes, test changes, and documentation changes. Once you've merged your pull request and GitHub Actions finishes building the docs, changes will be visible on the +`zppy documentation page `_. + +How Documentation is Versioned +------------------------------ +The `sphinx-multiversion `_ package manages documentation versioning. + +``sphinx-multiversion`` is configured to generate versioned docs for available tags and +branches. Ones that don’t contain both the sphinx ``source`` directory and the ``conf.py`` file will be skipped automatically. + + - Run ``sphinx-multiversion source _build/html --dump-metadata`` to see which tags/branches matched. diff --git a/docs/source/dev_guide/index.rst b/docs/source/dev_guide/index.rst index af2b1a08..9ff5ade8 100644 --- a/docs/source/dev_guide/index.rst +++ b/docs/source/dev_guide/index.rst @@ -2,15 +2,19 @@ Developer Guide ############### +This guide covers everything needed to develop, test, and release ``zppy``. + .. toctree:: :maxdepth: 2 project-standards - ci - testing - release_testing - release - tutorial_testing_e3sm_unified - new_diags_set - new_glb_plot + contributing + parameters + provenance + tasks/index + test + update_expected_results + releases/index.rst new_task + new_diags_set + archive/index diff --git a/docs/source/dev_guide/new_diags_set.rst b/docs/source/dev_guide/new_diags_set.rst index c4386bcc..99a623a9 100644 --- a/docs/source/dev_guide/new_diags_set.rst +++ b/docs/source/dev_guide/new_diags_set.rst @@ -1,6 +1,6 @@ -********************************************* -Adding a new set to the E3SM Diagnostics task -********************************************* +******************************************************* +Examples: Adding a new set to the E3SM Diagnostics task +******************************************************* This guide gives a general list of things to consider when adding a new E3SM Diagnostics set. The exact code changes required will differ amongst sets. @@ -32,7 +32,7 @@ In ``zppy/e3sm_diags.py`` Diurnal Cycle: ``e3sm_diags`` had to be set up to depend on a new climatology subtask: - .. code:: + .. code-block:: python if "diurnal_cycle" in c['sets']: dependencies.append(os.path.join(scriptDir, 'climo_%s_%04d-%04d.status' % ( @@ -40,7 +40,7 @@ Diurnal Cycle: ``e3sm_diags`` had to be set up to depend on a new climatology su Streamflow: ``e3sm_diags`` had to be set up to depend on a new time series subtask: - .. code:: + .. code-block:: python if "streamflow" in c["sets"]: dependencies.append( diff --git a/docs/source/dev_guide/new_glb_plot.rst b/docs/source/dev_guide/new_glb_plot.rst deleted file mode 100644 index 4ab8fd45..00000000 --- a/docs/source/dev_guide/new_glb_plot.rst +++ /dev/null @@ -1,13 +0,0 @@ -*************************************** -Adding a new plot to Global Time Series -*************************************** - -This guide gives a general list of things to consider when adding a new -Global Time Series plot. The exact code changes required will differ amongst plots. - -Note that unlike E3SM Diags and MPAS-Analysis which are packages called by ``zppy``, Global Time Series is built into ``zppy``. - -- In `coupled_global.py `_: Add variables under "# Variables to extract". Add the code for your new plot under "Plotting functions"; the name of your function should start with ``plot_``. In ``PLOT_DICT``, add ``"function name without plot_" : function name``. -- If this plot should be added to the defaults, then complete this step: for ``plot_names`` under ``[global_time_series]`` in `default.ini `_, add the function name without plot to the string. -- Optionally, after making your edits, you can run ``pre-commit run --all-files`` to clean up your code and check for any obvious errors. -- Run the `integration tests `_ and examine the differences from the expected files. If they match what you expect, update the expected files following "Commands to run to replace outdated expected files" on the machine-specific directions. (The commands to run all the integration tests are ``pip install .`` followed by ``pytest tests/integration/test_*.py``). diff --git a/docs/source/dev_guide/new_task.rst b/docs/source/dev_guide/new_task.rst index dc160ce5..65d5a9cd 100644 --- a/docs/source/dev_guide/new_task.rst +++ b/docs/source/dev_guide/new_task.rst @@ -2,6 +2,23 @@ Adding a new task ***************** +A task should be a wrapper around an external package. + +* The ``climo`` and ``ts`` tasks use NCO. ``tc_analysis`` and ``pcmdi_diags`` also make use of NCO tools. +* The ``global_time_series`` and ``pcmdi_diags`` tasks use ``zppy-interfaces``. +* The ``tc_analysis`` uses ``TempestRemap`` and ``TempestExtremes``. +* The ``e3sm_to_cmip``, ``e3sm_diags``, ``mpas_analysis``, ``ilamb``, and ``livvkit`` use their eponymous packages. + +In general, functionality should be added via calls to external packages. There are 2 reasons for this: + +* It creates a clearer separation of concerns: ``zppy`` is the workflow manager whereas the called packages do the actual post-processing. +* The called packages often have different environment needs than ``zppy``. A package can have its own dependency list and developer environment setup instructions. Putting all the functionality in a ``zppy`` bash template makes it more difficult to customize the environment for that specific task. + +The ``zppy-interfaces`` package is meant to be a collection of tools that are too complex to just write inline in a ``zppy`` bash file but not complex enough to be worth the effort of building and maintaining a new package just for it. Currently, ``zppy-interfaces`` is home to these two tools: + +* ``global_time_series``: this functionality was initially implemented in the ``zppy`` bash template itself, but grew complex enough to require separation. +* ``pcmdi_diags``: this is "glue code" to allow for seamless integration of the PCMDI Metrics Package with ``zppy``. + The task's bash file ===================== @@ -13,25 +30,16 @@ Some key parts, however, are displayed below: .. code:: #!/bin/bash - {% include 'slurm_header.sh' %} + {% include 'inclusions/slurm_header.bash' %} + {% include 'inclusions/boilerplate.bash' %} + set -e {{ environment_commands }} + set +e - # Turn on debug output if needed - debug={{ debug }} - if [[ "${debug,,}" == "true" ]]; then - set -x - fi - - # Script dir - cd {{ scriptDir }} - - # Get jobid - id=${SLURM_JOBID} - - # Update status file - STARTTIME=$(date +%s) - echo "RUNNING ${id}" > {{ prefix }}.status + set_pkg_manager + echo "${pkg_manager} list package-this-task-calls:" + ${pkg_manager} list package-this-task-calls || true # If we can't print this, just continue on. .. code:: @@ -153,14 +161,14 @@ The main file Add the task to ``zppy/__main__.py``: - .. code:: + .. code-block:: python - from import + from task_name import task_name - .. code:: + .. code-block:: python - # tasks - (config, scriptDir) + # task_name tasks + existing_bundles = task_name(config, script_dir, existing_bundles, job_ids_file) Update defaults =============== diff --git a/docs/source/dev_guide/parameters.rst b/docs/source/dev_guide/parameters.rst new file mode 100644 index 00000000..93935e78 --- /dev/null +++ b/docs/source/dev_guide/parameters.rst @@ -0,0 +1,22 @@ +.. _parameters_for_developers: + +************************************************ +Parameter checking & inferring -- for developers +************************************************ + +There are many parameter-handling functions. + +In ``utils.py``: + +* ``get_value_from_parameter``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just use the value of ``second_choice_parameter``. If inference is turned off, raise a ``ParameterNotProvidedError``. Use this function if the backup option is another parameter (rather than a fixed value). +* ``set_value_of_parameter_if_undefined``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just set the parameter's value to the ``backup_option``. If inference is turned off, raise a ``ParameterNotProvidedError``. + +In ``e3sm_diags.py``: + +* ``check_parameter_defined``: check if parameter is in the configuration dictionary, and if not raise a ``ParameterNotProvidedError``. +* ``check_set_specific_parameter``: if any requested ``e3sm_diags`` sets require this parameter, make sure it is present. If not, raise a ``ParameterNotProvidedError``. +* ``check_parameters_for_bash``: use ``check_set_specific_parameter`` to check the existence of parameters that aren't used until the bash script. +* ``check_mvm_only_parameters_for_bash``: similar, but these are specifically parameters used for model-vs-model runs. Uses ``check_parameter_defined`` in addition to ``check_set_specific_parameter``. +* ``check_and_define_parameters``: make sure all parameters are defined, using ``utils.py get_value_from_parameter``, ``utils.py set_value_of_parameter_if_undefined``, and ``check_mvm_only_parameters_for_bash``. + +``check_parameters_for_bash`` can be run immediately for each subtask because it has very few conditions. Other checks are included in ``check_and_define_parameters`` later on in the code. \ No newline at end of file diff --git a/docs/source/dev_guide/project-standards.rst b/docs/source/dev_guide/project-standards.rst index 9596d1fc..3132648a 100644 --- a/docs/source/dev_guide/project-standards.rst +++ b/docs/source/dev_guide/project-standards.rst @@ -21,15 +21,15 @@ Guidelines for VC 3. **Rebase** with ``main`` to avoid/resolve conflicts 4. Make sure ``pre-commit`` checks pass when committing (enforced in CI/CD build) 5. Open a pull-request (PR) early for discussion -6. Once the CI/CD build passes and PR is approved, **squash and rebase** your - commits -7. Merge PR into ``main`` and **delete the branch** +6. Once the CI/CD build passes and PR is approved, the PR is ready to merge. +7. Merge PR into ``main``. For small changes, use the "Squash and merge" option. For big changes with many distinct commits, or commits contributed by different developers, use the "Create a merge commit" option. +8. Delete the branch on GitHub and in your local workspace. Things to Avoid in VC ~~~~~~~~~~~~~~~~~~~~~ 1. Don't merge in broken or commented out code 2. Don't commit onto ``main`` directly -3. Don't merge with conflicts (handle conflicts upon rebasing) +3. Don't merge with conflicts (handle conflicts by rebasing: ``git rebase upstream/main``) Source: https://gist.github.com/jbenet/ee6c9ac48068889b0912 @@ -47,6 +47,16 @@ at the commit level before submitting code reviews. Helpful ``pre-commit`` Commands ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``zppy`` uses the following pre-commit checks: + +* **trailing-whitespace** - Trims trailing whitespace from the end of each line. +* **end-of-file-fixer** - Ensures files end with a single newline character. +* **check-yaml** - Validates YAML files for correct syntax. +* **black** - Auto-formats Python code to a consistent style using the `black `_ formatter. +* **isort** - Sorts and organizes Python import statements alphabetically and by type using `isort `_. +* **flake8** - Lints Python code for style violations and common errors (PEP 8) using `flake8 `_. +* **mypy** - Runs static type checking on Python code using type annotations using `mypy `_. + Install into your cloned repo :: conda activate zppy_dev @@ -71,88 +81,13 @@ Run individual hook :: # Available hook ids: trailing-whitespace, end-of-file-fixer, check-yaml, black, isort, flake8, mypy pre-commit run -Squash and Rebase Commits -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Before you merge a support branch back into ``main``, the branch is typically -squashed down to a single* buildable commit, and then rebased on top of the main repo's ``main`` branch. - -\* *In some cases, it might be logical to have multiple squashed commits, as long as each commit passes the CI/CD build* - -Why squash and rebase commits? - -- Ensures build passes from the commit -- Cleans up Git history for easy navigation -- Makes collaboration and review process more efficient -- Makes handling conflicts from rebasing simple since there are fewer commits to fix conflicts on -- Makes ``git bisect`` easier and more effective to use. For example, it will show the exact commit that introduced a bug since the commit contains a relatively small changeset. On the otherhand, a merge commit bringing in many commits makes it much harder since it includes a large changeset. A changeset is a collection of commits that are logically grouped together. - - -How to squash and rebase commits -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Assuming that you followed :ref:`"(b) Development Environment" `: - -1. Sync ``main`` with the main repo's ``main`` :: - - git checkout main - git rebase /main - git push -f main - -2. Get the SHA of the commit OR number of commits to rebase to :: - - git log --graph --decorate --pretty=oneline --abbrev-commit - -3. Squash commits:: - - git rebase -i [SHA] - - # OR - - git rebase -i HEAD~[NUMBER OF COMMITS] - -4. Make sure your squashed commit messages are refined - -5. Rebase branch onto ``main`` :: - - git checkout - git rebase main - git push -f - -6. Force push to remote branch :: - - # You have to force push because the rebase rewrites the commit SHAs - git push -f - -Source: -https://blog.carbonfive.com/always-squash-and-rebase-your-git-commits/ - -Code Quality Assurance ----------------------- - -This project uses several tools for code formatting, linting, and type checking listed below. - -- Code Formatting: `black `__ -- Linting: `flake8 `__, `isort `__ -- Optional Type Checking: `mypy `__ - -You can run them as hooks manually/automatically when committing using ``pre-commit``, or manually through the terminal or IDE/text editor. - -Helpful Commands -~~~~~~~~~~~~~~~~ - -Run a tool - :: - - # Available tool names: black, flake8, isort, mypy - . .. _ci-cd: Continuous Integration / Continuous Delivery (CI/CD) ---------------------------------------------------- -This project uses `GitHub Actions `_ to run two CI/CD workflows. +This project uses `GitHub Actions `_ to run two CI/CD workflows. The workflows are defined `here `_. 1. CI/CD Build Workflow @@ -164,6 +99,8 @@ This project uses `GitHub Actions 2. Run test suite in a conda environment 3. Publish latest ``main`` docs (only on ``push``) + When a pull request is made, the build workflow is run automatically on the pushed branch. When the pull request is merged, the build workflow is once again run, but this time on the ``main`` branch. + 2. CI/CD Release Workflow This workflow is triggered by the Git ``publish`` event, which occurs when a new release is tagged. @@ -173,10 +110,6 @@ This project uses `GitHub Actions 1. Publish new release docs 2. Publish Anaconda package -Style Guide ------------ - -This project follows the Black code style. Please read about it more `here `__. API Documentation ----------------- diff --git a/docs/source/dev_guide/provenance.rst b/docs/source/dev_guide/provenance.rst new file mode 100644 index 00000000..cd10f27a --- /dev/null +++ b/docs/source/dev_guide/provenance.rst @@ -0,0 +1,88 @@ +.. _dev-provenance: + +Provenance (Developer Reference) +================================== + +Overview +-------- + +Every ``zppy`` run writes two provenance files for each invocation: + +.. list-table:: + :header-rows: 1 + :widths: 35 65 + + * - File + - Contents + * - ``provenance..cfg`` + - A verbatim copy of the user-supplied ``.cfg`` configuration file. + * - ``provenance..settings`` + - Key-value metadata about the run: ``case_name``, ``machine``, + ``hpc_username``, and ``diagnostics_url``. Only written when at least + one field resolves to a non-empty value. + +Both files are written to the script directory +(``/post/scripts/``) and mirrored under ``//`` when +that directory is accessible. + +The ``.settings`` file format +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One ``key = value`` pair per line, e.g.: + +.. code-block:: text + + case_name = v3.LR.historical_0051 + machine = pm-cpu + hpc_username = ac.zhang40 + diagnostics_url = https://portal.nersc.gov/cfs/e3sm/ac.zhang40/zppy_demo/v3.LR.historical_0051 + +Fields with empty or ``None`` values are omitted. + +Implementation +-------------- + +- **Python module**: ``zppy/provenance.py`` +- **Called from**: ``zppy/__main__.py`` + +``zppy/provenance.py`` exports four public helpers: + +``parse_env_case_xml(input_dir)`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Reads ``/case_scripts/env_case.xml`` and returns a dict with any +of ``case_name``, ``machine``, and ``hpc_username`` that were successfully +parsed. Degrades gracefully on a missing file, missing entries, or malformed +XML (logs a warning and returns a partial or empty dict). + +``build_diagnostics_url(www, case, machine_info)`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Constructs the public web-portal URL for the run's diagnostics output using +the machine's ``web_portal`` ``base_path`` and ``base_url`` from the +``mache`` ``MachineInfo`` config. Returns ``None`` when ``www`` is empty, +when ``www`` is not under the machine's ``base_path``, or when the machine +config lacks ``web_portal`` entries. + +``write_provenance_settings(settings_path, extras)`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Writes the ``provenance.settings`` file from a dict, skipping empty and +``None`` values. Does nothing if no usable entries remain. + +``build_provenance_extras(config_default, machine_info)`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Orchestrates the other three functions. Reads ``case_name``, ``machine``, +and ``hpc_username`` from ``env_case.xml`` under ``cfg["default"]["input"]``, +builds ``diagnostics_url`` from ``cfg["default"]["www"]`` and +``cfg["default"]["case"]``, and returns the assembled extras dict. Logs a +warning if the case name in the cfg disagrees with the value in +``env_case.xml`` (the XML value is treated as authoritative). + +Unit tests +---------- + +``tests/test_zppy_provenance.py`` covers all four public helpers, including +missing-file, partial-entry, malformed-XML, unsupported-machine, and +case-mismatch scenarios. diff --git a/docs/source/dev_guide/release.rst b/docs/source/dev_guide/release.rst deleted file mode 100644 index a6dd9ab5..00000000 --- a/docs/source/dev_guide/release.rst +++ /dev/null @@ -1,158 +0,0 @@ -How to Prepare a Release -======================== - -In this guide, we'll cover: - -* Preparing the Documentation -* Bumping the Version -* Releasing On GitHub -* Releasing The Software On Anaconda -* Creating a New Version of the Documentation - -Preparing the Documentation ---------------------------- - -This step is only required for production releases, not release candidates. - -1. Checkout a branch - - :: - - git fetch upstream main - git checkout -b versioned-docs upstream/main - -2. Edit ``docs/source/parameters.rst``: in ``https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/default.ini``, replace ``main`` with the hash of the latest commit. -3. Create a pull request to the main repo and merge it. Now, when the next version of the documentation is created (see last part of this page), it will point to the relevant defaults. Mark yourself as the assignee, and mark "Documentation" as the label. - -Bumping the Version -------------------- - -1. Checkout a branch with the name of the version. - - :: - - git fetch upstream main - # Prepend "v" to - # For release candidates, append "rc" to - git checkout -b v upstream/main - -2. Bump version using tbump. Example, bumping to v1.1.0: - - :: - - # Exclude "v" and should match the above step. - # --no-tag is required since tagging is handled in "Releasing on GitHub" - $ tbump --no-tag - - :: Bumping from 1.0.0 to 1.1.0 - => Would patch these files - - pyproject.toml: version = "1.0.0" - + pyproject.toml: version = "1.1.0" - - zppy/__init__.py:1 __version__ = "v1.0.0" - + zppy/__init__.py:1 __version__ = "v1.1.0" - - conda/meta.yaml:2 {% set version = "1.0.0" %} - + conda/meta.yaml:2 {% set version = "1.1.0" %} - - tbump.toml:5 current = "1.0.0" - + tbump.toml:5 current = "1.1.0" - => Would run these git commands - $ git add --update - $ git commit --message Bump to 1.1.0 - $ git push origin v1.1.0 - :: Looking good? (y/N) - > - -3. If you encounter ``Error: Command `git push upstream main` failed``, as in `issue 470 `_, you can run ``git push upstream `` yourself. - -4. Create a pull request to the main repo and merge it. Mark yourself as the assignee, and mark "Update version" as the label. - -.. _github-release: - -Releasing on GitHub: release candidates ---------------------------------------- - -1. Create a tag for the release candidate at https://github.com/E3SM-Project/zppy/tags. Example, bumping to v1.1.0rc1: - - :: - - $ git checkout main - $ git fetch upstream - $ git reset --hard upstream/main - $ git tag -a v1.1.0rc1 -m "v1.1.0rc1" - # Delete the branch from the tbump step. Otherwise, the push command won't work. - $ git branch -D v1.1.0rc1 - $ git push upstream v1.1.0rc1 - -Releasing on GitHub: production releases ----------------------------------------- - -1. Draft a new release `here `_. You can save this and come back to it later, if need be. -2. Set `Tag version` to ``v``, **including the "v"**. `@Target` should be ``main``. -3. Set `Release title` to ``v``, **including the "v"**. -4. Use `Describe this release` to summarize the changelog. - - * You can scroll through `zppy commits `_ for a list of changes. - * You can look at the last release to get an idea of how to format the description. - -5. Click `Publish release`. -6. CI/CD release workflow is automatically triggered. - -Releasing on conda-forge: release candidates --------------------------------------------- - -1. If you don't have a local version of the conda-forge repo, run: :: - - git clone git@github.com:conda-forge/zppy-feedstock.git - git remote add upstream git@github.com:conda-forge/zppy-feedstock.git - -2. If you don't have a fork of the conda-forge repo, on `conda-forge `_, click the "Fork" button in the upper right hand corner. Then, on your fork, click the green "Code" button, and copy the SSH path. Run: :: - - git remote add - -3. Get the sha256 of the tag you made in "Releasing on GitHub: release candidates": :: - - curl -sL https://github.com/E3SM-Project/zppy/archive/v1.1.0rc1.tar.gz | openssl sha256 - -4. Make changes on a local branch. Example, bumping to v1.1.0rc1: :: - - $ git fetch upstream dev - $ git checkout -b v1.1.0rc1 upstream/dev # You can name the branch anything you want - # In `recipe/meta.yaml`, update the version and sha256 (and the build number if needed): - {% set version = "1.1.0rc1" %} # Set to your version - sha256: ... # The sha256 from the previous step - number: 0 # build > number should always be 0 - $ git add -A - $ git commit -m "v1.1.0rc1" - $ git push v1.1.0rc1 - -5. Note that the conda-forge bot does not work for release candidates. So, make a PR manually from your fork of the feedstock to the ``dev`` branch of `conda-forge `_. Then, the package build on conda-forge will end up with the ``zppy_dev`` label. You can add the "automerge" label to have the PR automatically merge once CI checks pass. - -6. After merging, CI runs again (in a slightly different way). Then, check the https://anaconda.org/conda-forge/zppy page to view the newly updated package. Release candidates are assigned the ``zppy_dev`` label. Note that it takes about 15 minutes for the files to propagate across conda-forge's mirroring services, which must happen before you can use the files. - -Releasing on conda-forge: production releases ---------------------------------------------- - -1. Be sure to have already completed :ref:`Releasing On GitHub `. This triggers the CI/CD workflow that handles Anaconda releases. -2. Wait for a bot PR to come up automatically on conda-forge after the GitHub release. This can happen anywhere from 1 hour to 1 day later. -3. Re-render the PR (see `docs `_). -4. Merge the PR on conda-forge. -5. Check the https://anaconda.org/conda-forge/zppy page to view the newly updated package. Production releases are assigned the ``main`` label. -6. Notify the maintainers of the unified E3SM environment about the new release on the `E3SM Confluence site `_. - - * Be sure to only update the ``zppy`` version number in the correct version(s) of the E3SM Unified environment. - * This is almost certainly one of the E3SM Unified versions listed under “Next versions”. If you are uncertain of which to update, leave a comment on the page asking. - -Creating a New Version of the Documentation -------------------------------------------- - -1. Be sure to have already completed :ref:`Releasing On GitHub `. This triggers the CI/CD workflow that handles publishing documentation versions. -2. Wait until the CI/CD build is successful. You can view all workflows at `All Workflows `_. -3. Changes will be available on the `zppy documentation page `_. You can check if it really is the latest version, by going to https://e3sm-project.github.io/zppy/_build/html/main/parameters.html, then checking if the "paramter defaults" link includes the proper hash. It should match the commit before "Update defaults in docs" on https://github.com/E3SM-Project/zppy/commits/main. - -Extra Resources ---------------- - -Conda-forge: - -* https://conda-forge.org/docs/user/introduction.html#why-conda-forge -* https://conda-forge.org/docs/maintainer/infrastructure.html#admin-web-services -* https://acme-climate.atlassian.net/wiki/spaces/IPD/pages/3616735236/Releasing+E3SM+Software+on+Anaconda+conda-forge+channel diff --git a/docs/source/dev_guide/release_testing.rst b/docs/source/dev_guide/release_testing.rst deleted file mode 100644 index 7f0f5a84..00000000 --- a/docs/source/dev_guide/release_testing.rst +++ /dev/null @@ -1,150 +0,0 @@ -*************************************** -Testing directions for making a release -*************************************** - -1. Have three shells open: one on Chrysalis, one on Compy, and one on Perlmutter. Do the following steps on each machine. - -2. ``cd`` to the ``zppy`` directory. - -3. Check out a branch to test on. - - a. test dev (run before making a new zppy RC) :: - - git fetch upstream main - git checkout -b test_zppy_pre_#.#.#rc#_ upstream/main - git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main - - b. test new Unified RC :: - - git fetch upstream main - git checkout -b test_unified_#.#.#rc#_ upstream/main - git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main - - c. test final Unified :: - - git fetch upstream main - git checkout -b test_unified_#.#.#rc#_ upstream/main - git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main - -4. Set up a dev environment for E3SM Diags. This is used for the ``diags_environment_commands`` parameter. Normally, this is only used for checking ``environment_commands`` works properly. However, by modifying the test templates, it's possible to run other E3SM Diags subtasks in a different environment. This can be useful if a bug fix in E3SM Diags hasn't been incorporated into the latest Unified RC yet. :: - - # `cd` to e3sm_diags directory - $ git checkout main - $ git fetch upstream - $ git reset --hard upstream/main - $ git log # Should match https://github.com/E3SM-Project/e3sm_diags/commits/main - $ mamba clean --all - $ mamba env create -f conda-env/dev.yml -n e3sm_diags_ - $ conda activate e3sm_diags_ - $ pip install . - $ `cd` back to zppy directory - -5. Make sure you're using the latest packages. In ``tests/integration/utils.py``, do the following: - - a. test dev (run before making a new zppy RC): - - * If it's the first zppy RC (i.e., a Unified RC hasn't been made yet): Change the last line ``generate_cfgs()`` to ``generate_cfgs(unified_testing=False)``. We want to use the latest officially released E3SM Unified environment. - * Otherwise: follow the (b: test new Unified RC) directions for this step. - - - b. test new Unified RC - - * Change the last line ``generate_cfgs()`` to ``generate_cfgs(unified_testing=True)``. We **don't** want to use the latest officially released E3SM Unified environment. - * Update ``environment_commands_test`` for the machine you're currently working on. If this is not done, then ``zppy`` will use the existing value, which is likely an old E3SM Unified RC launch script that doesn't even exist anymore. This parameter should be set to source the latest E3SM Unified RC launch script, since E3SM Unified is the only environment that all ``zppy`` tasks can run from. For instance, ``ncclimo`` isn't in the ``zppy`` dev environment. - - - c. test final Unified - - * Change the last line ``generate_cfgs()`` to ``generate_cfgs(unified_testing=False)``. We want to use the latest officially released E3SM Unified environment. - - For a, b, and c, always: - - * Update ``diags_environment_commands`` for the machine you're currently working on to use the environment created in the above step. If this is not done, the ``atm_monthly_180x360_aave_environment_commands`` subtask will use an older dev environment for E3SM Diags than we want. - * Update ``UNIQUE_ID`` to be a short description of what you're testing. - -6. Set up your environment. - - a. test dev (run before making a new zppy RC): Set up a new development environment. This ensures that testing will use the latest conda changes. Note that you will need to run ``conda remove -n zppy_dev_pre_zppy_#.#.#rc# --all`` first if you have previously done this step. :: - - mamba clean --all - mamba env create -f conda/dev.yml -n zppy_dev_pre_#.#.#rc# - conda activate zppy_dev_pre_#.#.#rc# - pip install . - - b. test new Unified RC: Launch the E3SM Unified environment for the machine you're on. Change out the version numbers below. - - * Chrysalis: ``source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_#.#.#rc#_chrysalis.sh`` - * Compy: ``source /share/apps/E3SM/conda_envs/test_e3sm_unified_#.#.#rc#_compy.sh`` - * Perlmutter: ``source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_#.#.#rc#_pm-cpu.sh`` - - c. test final Unified: Launch the latest E3SM Unified environment for the machine you're on. - - * Chrysalis: ``source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh`` - * Compy: ``source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh`` - * Perlmutter: ``source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh`` - -7. Run the unit tests with ``pytest tests/test_*.py``. - - a. test dev (run before making a new zppy RC): - - * If there are any failures, fix the code (or tests). If you make any conda changes, go back to step 6a. If you otherwise change zppy source code, run ``pip install .`` and then redo step 7. If you only make changes to tests, you can immediately redo step 7. - - b. test new Unified RC: - - * If there are any failures, fix the code and go back to step 1, following the (a: test dev (run before making a new zppy RC)) directions. - - c. test final Unified: - - * There should be no failures. If there are, a patch release of E3SM Unified may be required. - - For a, b, and c: - - * If there are no failures, proceed to the next step. - -8. Run the "Commands to run before running integration tests" for the current machine. To ensure you don't encounter issues from running ``zppy`` commands simultaneously, wait for all the automatically launched jobs from one run to finish before running ``zppy`` again. - - * `Chrysalis `_ - * `Compy `_ - * `Perlmutter `_ - -9. Run the integration tests with ``pytest tests/integration/test_*.py``. Note that ``test_complete_run.py`` takes approximately 75 minutes to run on Compy. - - a. test dev (run before making a new zppy RC): - - * If there are any unexpected failures, fix the code (or tests). If you make any conda changes, go back to step 6a. If you otherwise change zppy source code, run ``pip install .`` and then go back to step 7. If you only make changes to tests, you can immediately redo step 9. - - b. test new Unified RC: - - * If there are any unexpected failures, fix the code and go back to step 1, following the (a: test dev (run before making a new zppy RC)) directions. - - c. test final Unified: - - * There should be no unexpected failures. If there are, a patch release of E3SM Unified may be required. - - For a, b, and c: - - * If there are only expected failures, then update the expected files. Use the "Commands to run to replace outdated expected files" from the links on step 8. Then repeat step 9. - * If there are no failures at all, proceed to the next step. - -10. Run ``git diff``. All of your changes should be from editing ``tests/integration/utils.py`` in step 5, and running it in step 8. - - * If this is the case, you can delete this testing branch. - * If not, you have probably made code changes to get the tests to pass. Make a pull request to merge the changes. Add the "semver: bug" label. - -11. Wrap up release testing: - - a. test dev (run before making a new zppy RC): Create the next zppy RC by following the "release candidates" directions at https://e3sm-project.github.io/zppy/_build/html/main/dev_guide/release.html. - - b. test new Unified RC: Create the next zppy release following the "production releases" directions at https://e3sm-project.github.io/zppy/_build/html/main/dev_guide/release.html. - - c. test final Unified: Run the "Commands to generate official expected results for a release" from step 8. This will create a baseline of expected files for the release. You can now safely remove old branches and environments. At https://github.com/E3SM-Project/zppy/branches, delete any branches that are no longer needed. Also, run: :: - - # Branches - $ cd - $ git branch # Look at all branch names - $ git branch -D - - # Environments - $ conda env list - # For each environment you want to delete, run: - $ conda remove -n --all diff --git a/docs/source/dev_guide/releases/index.rst b/docs/source/dev_guide/releases/index.rst new file mode 100644 index 00000000..149aa8aa --- /dev/null +++ b/docs/source/dev_guide/releases/index.rst @@ -0,0 +1,16 @@ +How to Prepare a Release +======================== + + +.. toctree:: + :maxdepth: 1 + + release_candidates + production_releases + +Extra Resources +--------------- + +- Conda-forge docs: https://conda-forge.org/docs/user/introduction.html +- Admin web services: https://conda-forge.org/docs/maintainer/infrastructure.html#admin-web-services +- E3SM Anaconda release guide: https://acme-climate.atlassian.net/wiki/spaces/IPD/pages/3616735236/Releasing+E3SM+Software+on+Anaconda+conda-forge+channel diff --git a/docs/source/dev_guide/releases/production_releases.rst b/docs/source/dev_guide/releases/production_releases.rst new file mode 100644 index 00000000..e2266865 --- /dev/null +++ b/docs/source/dev_guide/releases/production_releases.rst @@ -0,0 +1,81 @@ +.. _production_releases: + +Preparing a production release +============================== + +Step 1: Testing +--------------- + +Be sure to run the entire integration test suite before making a production release. + +Step 2: Confluence +------------------ + +This step should already have been completed during the release-candidate phase, however it is good practice to double check that the `E3SM Unified version tracking page `_ has had the next E3SM Unified version updated with the new ``zppy`` version number. + +Step 3: Prepare documentation +------------------------------------------ + +Similar to the release-candidate directions, we'll use ``v1.2.3`` as an example verison number here. + + .. code-block:: bash + + cd zppy + git status # Confirm there's no uncommitted changes + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b prepare-docs-for-1.2.3 upstream/main + + +Change the "this release's parameter defaults" link on the Parameters documentation page (``docs/source/user_guide/parameters.rst``) to refer to the latest commit's hash. See `zppy commits `_ to find the latest commit. + + .. code-block:: bash + + git add -A + conda activate env-name # Activate any zppy dev environment you have; we just need the pre-commit hooks + conda activate env-name + pre-commit run --all-files + git commit -m "Update parameter documentation pointer for v1.2.3" + git push upstream prepare-docs-for-1.2.3 + + +Create a PR, mark yourself as assignee, add the Documentation label, merge the PR and delete the GitHub branch. + +Step 4: tbump +------------- + + .. code-block:: bash + + # On Perlmutter + cd zppy + git status # Confirm there's no uncommitted changes + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b v1.2.3 upstream/main + git log --oneline | head -n 5 + # Check that the latest commits match what's on https://github.com/E3SM-Project/zppy/commits/main/ + tbump 1.2.3 --no-tag + # This creates a commit, but won't push it (because the branch isn't named `main`) + git push upstream v1.2.3 + # Create, and "Update version" label" to, and merge the PR; delete the branch on GitHub + +Step 5: Make the release on the zppy repo +----------------------------------------- + +1. Draft a new release `here `_. Click "Draft a new release". +2. Set Tag version to ``v1.2.3``, including the “v”. ``@Target`` should be ``main``. Click "Tag", then "Create new tag" and enter "v.1.2.3" +3. Set Release title to ``v1.2.3``, including the “v”. +4. Use "Describe this release" to summarize the changelog. Write two sections: "Summary of changes" (the high-level summary) & "Full list of changes" (the categorized list of commits, from reviewing the `zppy commits `_) +5. Make sure "Set as the latest release" is checked. +6. Click "Publish release". Unlike the RCs, `v1.2.3` should now appear on _both_ `Tags `_ and `Releases `_. +7. CI/CD release workflow will be automatically triggered. The docs workflow is just for the docs. Clicking "Publish release" is responsible for triggering the bot PR on conda-forge. + +Step 6: zppy-feedstock repo +--------------------------- + +1. Wait for a bot PR to come up automatically on conda-forge after the GitHub release. This can happen anywhere from 1 hour to 1 day later. Check https://github.com/conda-forge/zppy-feedstock/pulls. (Alternative: open an issue with the bot command: ``@conda-forge-admin, please update version``` and the PR will opened). +2. Complete any requirements to merge the PR. +3. Check the https://anaconda.org/conda-forge/zppy/files/manage page to view the newly updated package. Check it has the `main` label. + +Step 7: Check the docs +---------------------- + +Wait for the docs workflow to complete successfully. Then go to https://docs.e3sm.org/zppy/_build/html/main/user_guide/parameters.html, and click on "this release’s parameter defaults". That should take you the code at the hash as of the latest commit. diff --git a/docs/source/dev_guide/releases/release_candidates.rst b/docs/source/dev_guide/releases/release_candidates.rst new file mode 100644 index 00000000..0f793bd4 --- /dev/null +++ b/docs/source/dev_guide/releases/release_candidates.rst @@ -0,0 +1,106 @@ +.. _release_candidates: + +Preparing a release candidate +============================= + +Step 1: Testing +--------------- + +Be sure to run the entire integration test suite before making a release candidate. + +Step 2: Determine what the new version should be +------------------------------------------------ + +This step only needs to be done for rc1. + +Review `zppy commits `_. Identify what changes since the last production release would require a patch, minor, or major version update (as in ``vMAJOR.MINOR.PATCH``). + +The highest-level version update will be used. For example, if you found 1 commit requiring a patch update, 1 commit requiring a minor update, and 0 commits requiring a major update, then you'd want to increment the minor version. + +Step 3: Confluence +------------------ + +This step only needs to be done for rc1. + +Update the `E3SM Unified version tracking page `_ with the new ``zppy`` version number. Be sure to update the section for the next E3SM Unified environment, not the current one. + +Step 4: tbump +------------- + +In this example, ``v1.2.3rc4`` is used -- so, the major version is 1, the minor version is 2, and the patch version is 3. The release candidate number is 4. + + .. code-block:: bash + + cd zppy + git status # Make sure you don't have any uncommitted changes + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b v1.2.3rc4 upstream/main + conda activate env-name # Activate any zppy dev environment you have; we just need `tbump` + tbump 1.2.3rc4 --no-tag + # This creates a commit, but won't push it (because the branch isn't named `main`) + git diff HEAD^ HEAD | cat + # Review the change `tbump`` added + git push upstream v1.2.3rc4 + # Create, and "Update version" label" to, and merge the PR; delete the branch on GitHub + +Step 5: Tag the RC on the zppy repo +----------------------------------------- + + .. code-block:: bash + + git checkout main + git fetch upstream + git reset --hard upstream/main + git tag -a v1.2.3rc4 -m "v1.2.3rc4" # Add the tag for this RC + # Delete the branch from the tbump step. Otherwise, the push command won't work. + git branch -D v1.2.3rc4 + git push upstream v1.2.3rc4 # Push the new RC tag to GitHub + +``v1.2.3rc4`` should now appear on `Tags `__, but _not_ on `Releases `_. + +Step 6: zppy-feedstock repo +--------------------------- + +If you don't have a fork of zppy-feedstock, first go to the `zppy-feedstock_repo `__, and click the "Fork" button in the top right. + + .. code-block:: bash + + # If you don't have a clone of zppy-feedstock, first run: + # Clone using the SSH from the green "Code" button in the top right of the repo home page. + git clone git@github.com:conda-forge/zppy-feedstock.git + cd zppy-feedstock + git remote -v # See your remotes + # Optional; this lets you use upstream to refer to the main repo: + git remote add upstream git@github.com:conda-forge/zppy-feedstock.git + # Required; this allows you to push to your fork: + # Copy the SSH path from your fork's green "Code" button and run: + git remote add your-fork-name git@github.com:your-fork-name/zppy-feedstock.git + + # If you already have your clone set up, just run: + cd zppy-feedstock + + # In all cases: + curl -sL https://github.com/E3SM-Project/zppy/archive/v1.2.3rc4.tar.gz | openssl sha256 + # SHA2-256(stdin)= long hex string + git status # Check for uncommitted changes + git fetch upstream dev # Make sure you fetch the dev branch, not the main branch! + git checkout -b v1.2.3rc4 upstream/dev + emacs recipe/recipe.yaml + # Update the version and sha256 (and the build number if needed): + # version: 1.2.3.rc4 + # sha256: ... # The sha256 from a few commands earlier + # number: 0 # build >>> number should always be 0 + + # Check the diff since the last release/RC. Examples: + # - https://github.com/E3SM-Project/zppy/compare/v3.1.0...v3.2.0rc1 + # - https://github.com/E3SM-Project/zppy/compare/v1.2.3rc3...v1.2.3rc4 + # If there are changes in dependencies there (e.g., in `conda/dev.yml`, `pyproject.toml`), + # You'll want to include them in this feedstock PR too. + + git add -A + git commit -m "v1.2.3rc4" + git push your-fork-name v1.2.3rc4 + +Then, create a pull request to the ``dev`` branch. Do _not_ set the pull request to merge to ``main``, as that will create a production release! RC packages get the ``zppy_dev`` label. + +Follow any further directions given by the ``conda-forge/zppy-feedstock`` PR template. Once that PR is reviewed and merged. Once CI completes, check https://anaconda.org/conda-forge/zppy for the new package (allow ~15 minutes for mirroring). diff --git a/docs/source/dev_guide/tasks/bundle.rst b/docs/source/dev_guide/tasks/bundle.rst new file mode 100644 index 00000000..3c37fdd2 --- /dev/null +++ b/docs/source/dev_guide/tasks/bundle.rst @@ -0,0 +1,30 @@ +.. _dev-task-bundle: + +bundle (Developer Reference) +============================== + +Implementation +-------------- + +- **Python module**: ``zppy/bundle.py`` +- **Jinja2 template**: ``zppy/templates/bundle.bash`` + +The ``Bundle`` class in ``bundle.py`` collects tasks that share the same +``bundle`` name. When ``handle_bundles()`` is called, it renders a single +SLURM script that runs all bundled tasks sequentially within one job. + +A bundle is named by its ``[bundle] [[name]]`` subsection, and tasks +opt in by setting ``bundle = name``. Bundle parameters (walltime, nodes, +etc.) are taken from the first task added or from the ``[bundle]`` section. + +Dependencies +------------ + +Bundles inherit the union of all their member tasks' dependencies. The +bundle job will not start until all upstream dependencies are satisfied +for all bundled tasks. + +**Downstream:** + +- Downstream tasks that depend on bundled tasks will list the bundle's + status file as a dependency. diff --git a/docs/source/dev_guide/tasks/climo.rst b/docs/source/dev_guide/tasks/climo.rst new file mode 100644 index 00000000..6494ed01 --- /dev/null +++ b/docs/source/dev_guide/tasks/climo.rst @@ -0,0 +1,46 @@ +.. _dev-task-climo: + +climo (Developer Reference) +============================ + +Implementation +-------------- + +- **Python module**: ``zppy/climo.py`` +- **Jinja2 template**: ``zppy/templates/climo.bash`` + +The ``climo`` function in ``climo.py`` iterates over all ``[climo]`` +tasks defined in the configuration file and generates a SLURM bash script +from the ``climo.bash`` Jinja2 template for each year-range/subsection +combination. + +Key steps in ``climo.py``: + +1. Load the ``climo.bash`` template via ``initialize_template()``. +2. Call ``get_tasks(config, "climo")`` to retrieve all ``[climo]`` + subsection configurations. +3. For each task: + + a. Call ``set_mapping_file()``, ``set_grid()``, + ``set_component_and_prc_typ()`` to resolve path and component + inference. + b. Construct the output filename prefix. For EAMxx, the subsection-level + ``case`` is used as the stream ID so ``ncclimo`` can locate input + files. + c. Render the template and write the bash script. + d. Submit the script via ``submit_script()``, registering it with + any bundle if requested. + +Template variables +------------------ + +The ``climo.bash`` template receives all configuration parameters as Jinja2 +variables. Key template variables: + +- ``{{ case }}``, ``{{ input }}``, ``{{ output }}`` — from ``[default]`` +- ``{{ mapping_file }}``, ``{{ grid }}`` — resolved by ``set_mapping_file()`` + and ``set_grid()`` +- ``{{ vars }}`` — variables to process (empty = all) +- ``{{ parallel }}`` — ncclimo parallel mode (default: ``mpi``) +- ``{{ nodes }}`` — number of nodes (default: 4) +- ``{{ environment_commands }}`` — environment setup diff --git a/docs/source/dev_guide/tasks/e3sm_diags.rst b/docs/source/dev_guide/tasks/e3sm_diags.rst new file mode 100644 index 00000000..b6dd8a13 --- /dev/null +++ b/docs/source/dev_guide/tasks/e3sm_diags.rst @@ -0,0 +1,32 @@ +.. _dev-task-e3sm-diags: + +e3sm_diags (Developer Reference) +================================== + +Implementation +-------------- + +- **Python module**: ``zppy/e3sm_diags.py`` +- **Jinja2 template**: ``zppy/templates/e3sm_diags.bash`` + +``e3sm_diags.py`` is the most complex task module in ``zppy``. It handles +a wide variety of diagnostic sets, each with different parameter +requirements. + +Key functions in ``e3sm_diags.py``: + +- ``check_parameter_defined(c, param)``: raises ``ParameterNotProvidedError`` + if a required parameter is missing. +- ``check_set_specific_parameter(c, sets, param)``: if any requested + sets require this parameter, check it is present. +- ``check_parameters_for_bash(c)``: validates parameters needed for the + bash template. Runs early because it has few conditions. +- ``check_mvm_only_parameters_for_bash(c)``: validates model-vs-model + parameters. +- ``check_and_define_parameters(c)``: resolves all parameters (inference + + validation). Called later in the pipeline. + +Adding a new diagnostic set +----------------------------- + +See :doc:`../new_diags_set` for a step-by-step guide. diff --git a/docs/source/dev_guide/tasks/e3sm_to_cmip.rst b/docs/source/dev_guide/tasks/e3sm_to_cmip.rst new file mode 100644 index 00000000..32774239 --- /dev/null +++ b/docs/source/dev_guide/tasks/e3sm_to_cmip.rst @@ -0,0 +1,10 @@ +.. _dev-task-e3sm-to-cmip: + +e3sm_to_cmip (Developer Reference) +==================================== + +Implementation +-------------- + +- **Python module**: ``zppy/e3sm_to_cmip.py`` +- **Jinja2 template**: ``zppy/templates/e3sm_to_cmip.bash`` diff --git a/docs/source/dev_guide/tasks/global_time_series.rst b/docs/source/dev_guide/tasks/global_time_series.rst new file mode 100644 index 00000000..854b1fc0 --- /dev/null +++ b/docs/source/dev_guide/tasks/global_time_series.rst @@ -0,0 +1,13 @@ +.. _dev-task-global-time-series: + +global_time_series (Developer Reference) +========================================== + +Implementation +-------------- + +- **Python module**: ``zppy/global_time_series.py`` +- **Template**: ``zppy/templates/global_time_series.bash`` + + +``global_time_series`` is called through the ``zppy-interfaces`` entry-point. diff --git a/docs/source/dev_guide/tasks/ilamb.rst b/docs/source/dev_guide/tasks/ilamb.rst new file mode 100644 index 00000000..ec68bb9d --- /dev/null +++ b/docs/source/dev_guide/tasks/ilamb.rst @@ -0,0 +1,10 @@ +.. _dev-task-ilamb: + +ilamb (Developer Reference) +============================= + +Implementation +-------------- + +- **Python module**: ``zppy/ilamb.py`` +- **Jinja2 template**: ``zppy/templates/ilamb.bash`` diff --git a/docs/source/dev_guide/tasks/index.rst b/docs/source/dev_guide/tasks/index.rst new file mode 100644 index 00000000..285ed532 --- /dev/null +++ b/docs/source/dev_guide/tasks/index.rst @@ -0,0 +1,23 @@ +.. _dev-tasks: + +Tasks (Developer Reference) +============================ + +This section describes how each ``zppy`` task is implemented, what +templates and Python modules it uses, and what dependencies it has on +other tasks. + +.. toctree:: + :maxdepth: 1 + + bundle + climo + ts + e3sm_to_cmip + tc_analysis + e3sm_diags + mpas_analysis + global_time_series + ilamb + livvkit + pcmdi_diags diff --git a/docs/source/dev_guide/tasks/livvkit.rst b/docs/source/dev_guide/tasks/livvkit.rst new file mode 100644 index 00000000..394bba00 --- /dev/null +++ b/docs/source/dev_guide/tasks/livvkit.rst @@ -0,0 +1,10 @@ +.. _dev-task-livvkit: + +livvkit (Developer Reference) +============================== + +Implementation +-------------- + +- **Python module**: ``zppy/livvkit.py`` +- **Jinja2 template**: ``zppy/templates/livvkit.bash`` diff --git a/docs/source/dev_guide/tasks/mpas_analysis.rst b/docs/source/dev_guide/tasks/mpas_analysis.rst new file mode 100644 index 00000000..be7dfa74 --- /dev/null +++ b/docs/source/dev_guide/tasks/mpas_analysis.rst @@ -0,0 +1,33 @@ +.. _dev-task-mpas-analysis: + +mpas_analysis (Developer Reference) +===================================== + +Implementation +-------------- + +- **Python module**: ``zppy/mpas_analysis.py`` +- **Jinja2 template**: ``zppy/templates/mpas_analysis.bash`` + +The ``mpas_analysis`` module generates one SLURM job per combination of +year range (from ``ts_years``, ``climo_years``, ``enso_years``) and +subsection. + +Model-vs-model logic +~~~~~~~~~~~~~~~~~~~~ + +When ``reference_data_path`` is set, the run is treated as MVM. The module +locates MPAS-Analysis config files from prior runs: + +.. code-block:: text + + /post/analysis/mpas_analysis//cfg/ + mpas_analysis_.cfg + +``comparison_type`` is either ``mvo`` or ``mvm``. In ``auto`` mode: + +- If only one type is found, that type is used. +- If both are found, an error is raised and the user must set + ``reference_comparison_type`` or ``test_comparison_type`` explicitly. +- If ``reference_data_path`` points to a ``[[subsection]]``, zppy uses + that subsection's actual comparison type. diff --git a/docs/source/dev_guide/tasks/pcmdi_diags.rst b/docs/source/dev_guide/tasks/pcmdi_diags.rst new file mode 100644 index 00000000..a25a2b3e --- /dev/null +++ b/docs/source/dev_guide/tasks/pcmdi_diags.rst @@ -0,0 +1,15 @@ +.. _dev-task-pcmdi-diags: + +pcmdi_diags (Developer Reference) +=================================== + +Implementation +-------------- + +- **Python module**: ``zppy/pcmdi_diags.py`` +- **Jinja2 template**: ``zppy/templates/pcmdi_diags.bash`` + +``pcmdi_diags.py`` handles multiple diagnostic sets (``mean_climate``, +``variability_modes_cpl``, ``variability_modes_atm``, ``enso``, +``synthetic_plots``). The ``current_set`` parameter (or the subsection +name) determines which set to run. diff --git a/docs/source/dev_guide/tasks/tc_analysis.rst b/docs/source/dev_guide/tasks/tc_analysis.rst new file mode 100644 index 00000000..5782439f --- /dev/null +++ b/docs/source/dev_guide/tasks/tc_analysis.rst @@ -0,0 +1,33 @@ +.. _dev-task-tc-analysis: + +tc_analysis (Developer Reference) +=================================== + +Implementation +-------------- + +- **Python module**: ``zppy/tc_analysis.py`` +- **Jinja2 template**: ``zppy/templates/tc_analysis.bash`` + +``tc_analysis`` processes high-frequency atmospheric output (``eam.h2`` by +default) to detect and track tropical cyclones and African Easterly Waves +using TempestExtremes. Both EAM and EAMxx are supported. + +Template variables +------------------ + +In addition to the standard top-level variables: + +- ``{{ input_grid }}`` — native model grid string (e.g. ``ne30pg2``) +- ``{{ res }}`` — legacy explicit resolution override +- ``{{ tc_vars }}`` — comma-separated 7-variable sequence for TempestExtremes +- ``{{ default_case }}`` — default case ID used to construct output filenames + +Dependencies +------------ + +**Upstream:** None. + +**Downstream:** + +- ``e3sm_diags`` — required for the ``tc_analysis`` diagnostic set. diff --git a/docs/source/dev_guide/tasks/ts.rst b/docs/source/dev_guide/tasks/ts.rst new file mode 100644 index 00000000..a4a1ec62 --- /dev/null +++ b/docs/source/dev_guide/tasks/ts.rst @@ -0,0 +1,31 @@ +.. _dev-task-ts: + +ts (Developer Reference) +========================= + +Implementation +-------------- + +- **Python module**: ``zppy/ts.py`` +- **Jinja2 template**: ``zppy/templates/ts.bash`` + +The ``ts`` function iterates over all ``[ts]`` tasks and generates a SLURM +bash script from the ``ts.bash`` Jinja2 template for each +year-range/subsection combination. + +Key steps in ``ts.py``: + +1. Load the ``ts.bash`` template. +2. Call ``get_tasks(config, "ts")`` for all subtasks. +3. For each task: + + a. Resolve mapping file, grid, and component type. + b. Render the template and write the bash script. + c. Submit the script. + +Vertical remapping +~~~~~~~~~~~~~~~~~~ + +When ``vrt_remap_vars`` is non-empty, the template also calls ``ncremap`` +after ``ncclimo`` to remap specified variables from model levels to pressure +levels. Output is written to a sibling ``ts_vrt_remap/`` directory. diff --git a/docs/source/dev_guide/test.rst b/docs/source/dev_guide/test.rst new file mode 100644 index 00000000..ff57f678 --- /dev/null +++ b/docs/source/dev_guide/test.rst @@ -0,0 +1,527 @@ +.. _testing-zppy: + +************* +Testing zppy +************* + +Follow the steps below to test ``zppy``. As you do so, please produce a Markdown report summarizing your results. + +Step 1: Determine what the current expected results are +======================================================= + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + expected_results_dir=/lcrc/group/e3sm/public_html/zppy_test_resources + expected_results_records_dir=/lcrc/group/e3sm/public_html/zppy_test_resources_previous + +Compy: + +.. code-block:: bash + + expected_results_dir=/compyfs/www/zppy_test_resources + expected_results_records_dir=/compyfs/fors729/zppy_test_resources_previous + +Note that Compy doesn't give write access to ``/compyfs/www/``, so we can't add a new directory there. That's why ``zppy_test_resources_previous`` is in a separate path. + +Perlmutter: + +.. code-block:: bash + + expected_results_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources + expected_results_records_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources_previous + +Process +~~~~~~~ + +.. code-block:: bash + + ls -lt ${expected_results_dir} + +In your Markdown report, note the date the expected results were last updated. + +Step 2: Review changes since expected results were updated +========================================================== + +Now that we know the date the expected results are from, we can review what changes we'll be testing. + +Review each of the following commit logs and note commits made since the date the expected results were updated: + +* For the ``e3sm_to_cmip`` task: `e3sm_to_cmip `_ +* For the ``e3sm_diags`` task: `e3sm_diags `_ +* For the ``mpas_analysis`` task: `MPAS-Analysis `_ +* For the ``global_time_series`` and ``pcmdi_diags`` tasks: `zppy-interfaces `_ +* For ``zppy`` itself: `zppy `_ + +For the remaining tasks (``climo``, ``ts``, ``tc_analysis``, ``ilamb``, ``livvkit``), we typically just use the associated package's latest release rather than making dev environments. As such, their latest development will have no impact on our tests unless we have started using one of their newer releases. + +In your Markdown report, make a table like: + +.. code-block:: + + | Package | Changes since expected results were updated | + | --- | --- | + | [package name](link to package's commit log) | Links to all PRs merged since the expected results were updated | + ... + +Step 3: Set up environments for called packages +=============================================== + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + repo_parent_dir=~/ez/ # Or wherever you keep your repos + + start_bash_subshell() + { + bash + source ~/.bashrc # Or wherever you have your aliases, etc. defined + } + + activate_dev_env() + { + env_name=$1 + + lcrc_conda # Or however you activate conda + rm -rf build + conda clean --all --y + conda env create -f conda/dev.yml -n ${env_name} + conda activate ${env_name} + pre-commit run --all-files # Confirm this passes + python -m pip install . + } + + activate_unified_env() + { + source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh + } + +Compy: + +.. code-block:: bash + + repo_parent_dir=~/ez/ # Or wherever you keep your repos + + start_bash_subshell() + { + bash + source ~/.bash_profile # Or wherever you have your aliases, etc. defined + } + + activate_dev_env() + { + env_name=$1 + + compy_conda # Or however you activate conda + rm -rf build + conda clean --all --y + conda env create -f conda/dev.yml -n ${env_name} + conda activate ${env_name} + pre-commit run --all-files # Confirm this passes + python -m pip install . + } + + activate_unified_env() + { + source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh + } + +Perlmutter: + +.. code-block:: bash + + repo_parent_dir=~/ez/ # Or wherever you keep your repos + + start_bash_subshell() + { + bash + source ~/.bash_profile.ext # Or wherever you have your aliases, etc. defined + } + + activate_dev_env() + { + env_name=$1 + + nersc_conda # Or however you activate conda + rm -rf build + conda clean --all --y + conda env create -f conda/dev.yml -n ${env_name} + conda activate ${env_name} + pre-commit run --all-files # Confirm this passes + python -m pip install . + } + + activate_unified_env() + { + source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh + } + +Process +~~~~~~~ + +.. code-block:: bash + + set_up_repo() + { + repo_name=$1 + main_branch_name=$2 + + cd ${repo_parent_dir}/${repo_name} + git status # Check for uncommitted changes + + # If there are uncommitted changes, + # commit them so we can move cleanly to a new branch: + git add -A + git commit -m "Checkpoint" + + git fetch upstream ${main_branch_name} # This assumes you've named your remote for the main repo as "upstream" + git checkout ${main_branch_name} + git reset --hard upstream/${main_branch_name} + git log --oneline | head -n 1 + # Check that this matches the corresponding commit log: + # https://github.com/E3SM-Project/e3sm_to_cmip/commits/master + # https://github.com/E3SM-Project/e3sm_diags/commits/main + # https://github.com/MPAS-Dev/MPAS-Analysis/commits/develop/ + # https://github.com/E3SM-Project/zppy-interfaces/commits/main + + # Activate EITHER a dev environment or the Unified env: + # Dev environment -- test latest development + # Unified environment -- test latest Unified environment + activate_dev_env ${repo_name}-yyyymmdd # Use today's date + # OR: activate_unified_env ${repo_name}-yyyymmdd # Use today's date + } + + +.. code-block:: bash + + start_bash_subshell + set_up_repo e3sm_to_cmip master + exit # Exit bash subshell + + start_bash_subshell + set_up_repo e3sm_diags main + exit # Exit bash subshell + + start_bash_subshell + set_up_repo MPAS-Analysis develop + exit # Exit bash subshell + + start_bash_subshell + set_up_repo zppy-interfaces main + # Since zppy-interfaces is so integrated into `zppy`, + # run its unit tests as well: + pytest tests/unit/global_time_series/test_*.py + pytest tests/unit/pcmdi_diags/test_*.py + exit # Exit bash subshell + +Step 4: Set up zppy environment +=============================== + +.. code-block:: bash + + cd ${repo_parent_dir}/zppy + git status # Check for uncommitted changes + + # If there are uncommitted changes, + # commit them so we can move cleanly to a new branch: + git add -A + git commit -m "Checkpoint" + + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b test-zppy-yyyymmdd upstream/main # Use today's date + git log --oneline | head -n 1 + # Check that this matches the corresponding commit log: + # https://github.com/E3SM-Project/zppy/commits/main + + start_bash_subshell + # Activate EITHER a dev environment or the Unified env: + # Dev environment -- test latest development + # Unified environment -- test latest Unified environment + activate_dev_env zppy-yyyymmdd # Use today's date + # OR: activate_unified_env zppy-yyyymmdd # Use today's date + + # Note the Python version being used + # If you activated dev env: + conda list python + # If you activated unified env: + pixi list python + + # Run zppy unit tests + pytest tests/test_*.py # 44 passed in 0.87s + +Step 5: Launch zppy jobs +======================== + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + conda_setup_cmd=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh + output_dir=/lcrc/group/e3sm/ac.forsyth2 + +Compy: + +.. code-block:: bash + + conda_setup_cmd=source /qfs/people/fors729/miniforge3/etc/profile.d/conda.sh + output_dir=/compyfs/fors729 + +Perlmutter: + +.. code-block:: bash + + conda_setup_cmd=source /global/homes/f/forsyth/miniforge3/etc/profile.d/conda.sh + output_dir=/global/cfs/cdirs/e3sm/forsyth + +Process +~~~~~~~ + +First, let's edit ``tests/integration/utils.py``. + +In place of each ``{env_cmd}`` put either ``${conda_setup_cmd}; conda activate ${repo_name}-yyyymmdd`` or the command from ``activate_unified_env``. You may want to use dev environments for some tasks and the Unified environment for others. + +You can comment out ``cfg``s from ``cfgs_to_run`` to run fewer configuration files. Likewise, you can comment out tasks from ``tasks_to_run`` to run fewer tasks. + +Be sure to set the ``unique_id``; this allows us to avoid path name collisions. + +.. code-block:: python + + TEST_SPECIFICS: Dict[str, Any] = { + # This is the NCO path. + # Keep as "" to use the production-version NCO commands. + # Set to a specific path to use development-version NCO commands. + "nco_path": "", + # These are custom environment_commands for specific tasks. + # Never set these to "", because they will print the line + # `environment_commands = ""` for the corresponding task, + # thus overriding the value set higher up in the cfg. + # That is, there will be no environment set. + # (`environment_commands = ""` only redirects to Unified + # if specified under the [default] task) + "e3sm_to_cmip_environment_commands": "{env_cmd}", + "diags_environment_commands": "{env_cmd}", + "mpas_analysis_environment_commands": "{env_cmd}", + "global_time_series_environment_commands": "{env_cmd}", + "livvkit_environment_commands": "{env_cmd}", + "pcmdi_diags_environment_commands": "{env_cmd}", + # This is the environment setup for other tasks. + # Leave as "" to use the latest Unified environment. + "environment_commands": "{env_cmd}", + # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs + "cfgs_to_run": [ + "weekly_bundles", # Typically, we run on Chrysalis, Compy + "weekly_comprehensive_v2", # Typically, we run on Chrysalis, Compy + "weekly_comprehensive_v3", # Typically, we run on all 3 machines + "weekly_legacy_3.1.0_bundles", # Typically, we run on Chrysalis + "weekly_legacy_3.1.0_comprehensive_v2", # Typically, we run on Chrysalis + "weekly_legacy_3.1.0_comprehensive_v3", # Typically, we run on Chrysalis + "weekly_legacy_3.0.0_bundles", # Typically, we run on Chrysalis + "weekly_legacy_3.0.0_comprehensive_v2", # Typically, we run on Chrysalis + "weekly_legacy_3.0.0_comprehensive_v3", # Typically, we run on Chrysalis + ], + "tasks_to_run": [ + "e3sm_diags", + "mpas_analysis", + "global_time_series", + "ilamb", + "livvkit", + "pcmdi_diags", + ], + "unique_id": "test_zppy_yyyymmdd", # Use today's date + } + +.. code-block:: bash + + git diff # Check that the diff looks as you expect + python tests/integration/utils.py + # This will generate the actual test cfgs based off the templates. + + # Set up an alias for checking jobs: + alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p' + alias sq='sqa -u $USER' + + sq + # Check that you have no jobs currently queued. + # It's ok if you do, but it makes counting remaining zppy jobs easier if you don't have any existing jobs. + + # Typically run on Chrysalis, Compy, Perlmutter: + zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg + + # Typically run on Chrysalis, Compy: + zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg + + # Typically run on Chrysalis: + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg + + + sq | wc -l # This includes the header row, so subtract 1 to get the number of jobs you have running + # WAIT until that returns 1 (i.e., 0 jobs running) + + +Step 6: Launch zppy jobs -- bundles part 2 +========================================== + +This section is only relevant only if you're running the ``_bundles_`` jobs. Only run the lines relevant to the jobs you launched in step 5. + +.. code-block:: bash + + # Check on bundles status + cd ${output_dir}/zppy_weekly_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.1.0_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.0.0_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + # Now, run bundles part 2 + cd ${repo_parent_dir}/zppy + git status + # You might have changed branches while you were waiting for jobs to finish. + # Make sure you're now back on the correct branch: test-zppy-yyyymmdd + # Also confirm you're back in the correct env: zppy-yyyymmdd or the Unified env + + + zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg + sq | wc -l + # WAIT until that returns 1 (i.e., 0 jobs running) + +Step 7: Review finished returns +=============================== + +Only run the lines relevant to the jobs you launched in steps 5/6. + +.. code-block:: bash + + ### v2 ### + cd ${output_dir}/zppy_weekly_comprehensive_v2_output/${unique_id}/v2.LR.historical_0201/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/${unique_id}/v2.LR.historical_0201/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/${unique_id}/v2.LR.historical_0201/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + ### v3 ### + cd ${output_dir}/zppy_weekly_comprehensive_v3_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + ### bundles ### + cd ${output_dir}/zppy_weekly_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.0.0_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.1.0_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + +In your Markdown report, any of the output subdirectories that had non-OK statuses. + +Step 8: Run Python tests +======================== + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm + } + +Compy: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm + } + +Perlmutter: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm + } + +Process +~~~~~~~ + +.. code-block:: bash + + cd ${repo_parent_dir}/zppy + git status + # You might have changed branches while you were waiting for jobs to finish. + # Make sure you're now back on the correct branch: test-zppy-yyyymmdd + # Also confirm you're back in the correct env: zppy-yyyymmdd or the Unified env + + # This test doesn't make use of an expected results directory. + pytest tests/integration/test_last_year.py + + # These tests do make use of an expected results directory. + # That is, the expected results may need to be updated if expected behavior has changed. + pytest tests/integration/test_bash_generation.py + pytest tests/integration/test_campaign.py + pytest tests/integration/test_defaults.py + + # These tests make use of an expected results directory + # AND rely on the jobs we just ran: + # 1. The bundles test: + pytest tests/integration/test_bundles.py + # 2. The image checker test, which we'll run from a compute node: + launch_compute_node + + start_bash_subshell + # EITHER: + # Activate EITHER a dev environment or the Unified env: + conda activate zppy-yyyymmdd + # OR: the command from `activate_unified_env` + + pytest tests/integration/test_images.py + # Typically takes between 10 and 20 minutes on Chrysalis and Perlmutter. + # Typically takes closer to 50 minutes on Compy. + cat test_images_summary.md + exit # Exit bash shell + exit # Exit compute note + +In your Markdown report: + +* From the ``pytest tests/integration/test_images.py `` command-line output, copy everything after ``Captured stdout call`` to a code block labeled "Output" +* Copy the results of ``cat test_images_summary.md`` to a section labeled "Complete summary table" +* Make a new section named "Summary table -- only failing image-check tests, sorted by task". For each task that has missing and/or mismatched images, copy the relevant rows from the summary table. Skip this section if there were no failing image-check tests. +* Note any test failures from the other Python tests. +* If there were no failures at all, print "All tests pass" diff --git a/docs/source/dev_guide/testing.rst b/docs/source/dev_guide/testing.rst deleted file mode 100644 index fa9a6c46..00000000 --- a/docs/source/dev_guide/testing.rst +++ /dev/null @@ -1,34 +0,0 @@ -************ -Testing zppy -************ - -Unit tests -========== - -Run all unit tests by doing the following: - - .. code:: - - pip install . # Install your changes (`python -m pip install .` also works) - pytest tests/test_*.py # Run all unit tests - -Integration tests -================= - -Integration tests can be run on Chrysalis, Compy, or Perlmutter. -First review `machine-specific directions `_. - -Run all integration tests by doing the following: - - .. code:: - - pip install . # Install your changes (`python -m pip install .` also works) - - pytest tests/integration/test_*.py # Run all integration tests - -Automated tests -=============== - -We have a :ref:`GitHub Actions ` Continuous Integration / Continuous Delivery (CI/CD) workflow. - -The unit tests are run automatically as part of this. Integration tests must be run on a machine specified above. diff --git a/docs/source/dev_guide/testing_e3sm_unified.cfg b/docs/source/dev_guide/testing_e3sm_unified.cfg deleted file mode 100644 index ef1a4d5a..00000000 --- a/docs/source/dev_guide/testing_e3sm_unified.cfg +++ /dev/null @@ -1,93 +0,0 @@ -[default] -input = -input_subdir = archive/atm/hist -# The directory that the `post` directory is placed in. -output = -case = 20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -www = -partition = -environment_commands = "" - -[climo] -active = True -years = "51:70:20", -# Use default vars - - [[ atm_monthly_180x360_aave ]] - mapping_file = - frequency = "monthly" - - [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - input_subdir = "archive/atm/hist" - input_files = "eam.h4" - mapping_file = - vars = "PRECT" - frequency = "diurnal_8xdaily" - -[ts] -active = True -years = "51:70:10", - - [[ atm_monthly_180x360_aave ]] - input_subdir = "archive/atm/hist" - input_files = "eam.h0" - frequency = "monthly" - mapping_file = - # Use default vars - - [[ atm_daily_180x360_aave ]] - input_subdir = "archive/atm/hist" - input_files = "eam.h1" - frequency = "daily" - mapping_file = - vars = "PRECT" - - [[ atm_monthly_glb ]] - input_subdir = "archive/atm/hist" - input_files = "eam.h0" - frequency = "monthly" - mapping_file = "glb" - # Use default vars - - [[ land_monthly ]] - input_subdir = "archive/lnd/hist" - input_files = "elm.h0" - frequency = "monthly" - mapping_file = - vars = "FSH,RH2M" - -[e3sm_diags] -active = True -years = "51:70:20", -ts_num_years = 10 -ref_start_yr = 1979 -ref_final_yr = 2016 - - [[ atm_monthly_180x360_aave ]] - short_name = "20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis" - grid = "180x360_aave" - reference_data_path = "" - obs_ts = "" - dc_obs_climo = "" - climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" - climo_monthly_frequency = "diurnal_8xdaily" - -[mpas_analysis] -active = True -walltime = "24:00:00" -parallelTaskCount = 6 -ts_years = "51-70", -enso_years = "51-70", -climo_years ="51-70", -mesh = "EC30to60E2r2" -anomalyRefYear = 51 - -[global_time_series] -active = True -years = "51-70", -ts_num_years = 10 -figstr=coupled_v2rc3e -moc_file=mocTimeSeries_0051-0070.nc -experiment_name=20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -ts_years = "51-70", -climo_years ="51-70", diff --git a/docs/source/dev_guide/tutorial_testing_e3sm_unified.rst b/docs/source/dev_guide/tutorial_testing_e3sm_unified.rst deleted file mode 100644 index ab59386d..00000000 --- a/docs/source/dev_guide/tutorial_testing_e3sm_unified.rst +++ /dev/null @@ -1,372 +0,0 @@ -.. _testing_e3sm_unified: - -************************************* -Testing zppy for E3SM Unified release -************************************* - -This tutorial covers testing ``zppy`` for an E3SM Unified release. - -.. warning :: - - Examples are from testing for E3SM Unified v1.5.0. - They may be outdated because of more recent changes. - For example, ``/lcrc/group/e3sm/public_html/diagnostics/observations/Atm/`` - has replaced ``/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/``. - -Parameters -========== - -Only a few parameters in the configuration file need to change for different machines. -Be sure to change the file paths to fit your own workspaces. - -.. warning:: - - Also consider the space quotas or purge policies for the file paths you use. - You'll be storing about 2 TB of data. - -Chrysalis: - -- : /lcrc/group/e3sm/ac.forsyth2/E3SM_simulations/e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -- : /lcrc/group/e3sm/ac.forsyth2/E3SM_simulations/e3sm_unified_test_simulation/ -- : /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/E3SM/e3sm_unified_test_simulation/ -- : compute -- : source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.5.0rc8_chrysalis.sh -- : /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc -- : /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology -- : /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series -- : /lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology - -Compy: - -- : /compyfs/fors729/e3sm_unified_test_zppy/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -- : /compyfs/fors729/e3sm_unified_test_zppy/ -- : /compyfs/www/fors729/E3SM/e3sm_unified_test_simulation/ -- : slurm -- : source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.5.0rc8_compy.sh -- : /compyfs/zender/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc -- : /compyfs/e3sm_diags_data/obs_for_e3sm_diags/climatology -- : /compyfs/e3sm_diags_data/obs_for_e3sm_diags/time-series -- : /compyfs/e3sm_diags_data/obs_for_e3sm_diags/climatology - -Cori: - -- : /global/cscratch1/sd/forsyth/e3sm_unified_test_zppy/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -- : /global/cscratch1/sd/forsyth/e3sm_unified_test_zppy/ -- : /global/cfs/cdirs/e3sm/www/forsyth/E3SM/e3sm_unified_test_simulation/ -- : haswell -- : source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.5.0rc8_cori-haswell.sh -- : /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc -- : /global/cfs/cdirs/e3sm/acme_diags/obs_for_e3sm_diags/climatology -- : /global/cfs/cdirs/e3sm/acme_diags/obs_for_e3sm_diags/time-series -- : /global/cfs/cdirs/e3sm/acme_diags/obs_for_e3sm_diags/climatology - -These values don't appear in the configuration file but are still useful to specify -here: - -Chrysalis: - -- : https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/E3SM/e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ -- : /lcrc/group/e3sm/ac.forsyth2/zstash_dir -- : /lcrc/group/e3sm/ac.forsyth2/e3sm_unified_test_zstash/unified_test_cache - -Compy: - -- : https://compy-dtn.pnl.gov/fors729/E3SM/e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ -- : /compyfs/fors729/e3sm_unified_test_zstash -- : /compyfs/fors729/e3sm_unified_test_zstash/unified_test_cache - -Cori: - -- : http://portal.nersc.gov/cfs/e3sm/forsyth/E3SM/e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ -- : /global/cscratch1/sd/forsyth/e3sm_unified_test_zstash -- : /global/cscratch1/sd/forsyth/e3sm_unified_test_zstash/unified_test_cache - -Testing zppy on a small simulation -================================== - -1. Find a simulation to test on. In this example, we'll use a simulation on Chrysalis: -``/lcrc/group/e3sm/ac.golaz/E3SM_simulations/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/``. - -2. The simulation might be too large to reasonably test on. We can copy over a -subsection to a testing directory with a script like the following. In this example, -the simulation data starts at year 51. We want to have at least 20 years, so we'll -copy over years 51-79 using the pattern ``*.00[5-7]*``. This subsection turns out to be -about 2 terabytes in size. - - :: - - #!/bin/bash - - input=/lcrc/group/e3sm/ac.golaz/E3SM_simulations/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ - output=e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ - - # For E3SM Diags: About 40 minutes to run - # Copy over years xx5x - xx7x - cp ${input}archive/atm/hist/*.00[5-7]* ${output}archive/atm/hist - cp ${input}archive/cpl/hist/*.00[5-7]* ${output}archive/cpl/hist - cp ${input}archive/ice/hist/*.00[5-7]* ${output}archive/ice/hist - cp ${input}archive/lnd/hist/*.00[5-7]* ${output}archive/lnd/hist - cp ${input}archive/ocn/hist/*.00[5-7]* ${output}archive/ocn/hist - cp ${input}archive/rof/hist/*.00[5-7]* ${output}archive/rof/hist - - # For MPAS-Analysis: Less than a minute to run - cp ${input}run/mpaso_in ${output}run/mpaso_in - cp ${input}run/mpassi_in ${output}run/mpassi_in - cp ${input}run/streams.ocean ${output}run/streams.ocean - cp ${input}run/streams.seaice ${output}run/streams.seaice - cp ${input}run/*rst* ${output}run/ - - -3. Now we need to load ``zppy``. We first want to test out ``zppy`` from -the new unified environment release candidate. -We can do this by running ````. -See 3a-c below if errors are encountered in later steps. - -3a. If ``zppy`` has problems in later steps, -we'll have to make changes in the ``zppy`` code and -create a new release candidate; until the new ``zppy`` release candidate is included in the -new unified environment release candidate, -we'll have to just use the ``zppy`` development environment. -This will require running ``pip install .`` in that environment from the top level of -the repo. - -3b. It may be the case that the configuration file had an error and thus neither -``zppy`` nor any package that ``zppy`` runs actually failed. In that case, fix the -configuration file. - -3c. Keep in mind if you run ``zppy -c`` multiple times, it will only rerun tasks that -have failed. This may or may not be what we want. For example, if we change the -environment we're using, we probably want to rerun everything using the new environment. - -4. Create a configuration file ``e3sm_unified_test_simulation.cfg``. - -.. literalinclude:: testing_e3sm_unified.cfg - :language: cfg - :linenos: - -5. We can now test out ``zppy`` by running ``zppy -c e3sm_unified_test_simulation.cfg``. -If that command fails, return to step 3. - -6. If the above command completes successfully, run: - :: - - # cd into output directory - cd /post/scripts - # Check status of files that either failed or are still running. - grep -v "OK" *status - # If there is an error, return to step 3. - # If you want to remove all output run the following two steps: - rm -rf /post - rm -rf /20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis - -7. Output can be viewed at the web link corresponding to ```` in the configuration -file (i.e., ```` defined in "Parameters" above). - -Archive simulation using zstash -=============================== - -1. Create a batch script ``zstash_create.bash`` to run ``zstash``. - :: - - #!/bin/bash - - #SBATCH --job-name=zstash_create - #SBATCH --nodes=1 - #SBATCH --output=/zstash_create.o%j - #SBATCH --exclusive - #SBATCH --time=04:00:00 - - - zstash create --hpss=none --cache= /20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis - -2. Run with ``sbatch zstash_create.bash``. This example takes about 2.5 hours to run. - -Transfer simulation to other machines -===================================== -We can follow an adapted version of the steps at -https://e3sm-project.github.io/zstash/_build/html/master/best_practices.html#transfer-to-nersc-hpss -to transfer the data. - -Login to Globus: https://www.globus.org/ (using NERSC credentials) - -Transfer to Cori ----------------- - -1. On the left sidebar, choose "ENDPOINTS". - -2. Search for "NERSC DTN". Click on Green power button to activate endpoint. - -3. On the left sidebar, choose "ENDPOINTS". - -4. Search for "lcrc#dtn_bebop". Click on Green power button to activate endpoint. -Log in using LCRC credentials. - -5. Paste the path to the ``zstash`` archive (```` for Chrysalis) -in the "Path" box. - -6. Click "Transfer or Sync to..." on the right side. The screen will now be split. - -7. On the left side, choose "Select all" - -8. On the right side, put "NERSC DTN" for collection. - -9. Paste the NERSC path you want the archive copied to -(```` for Cori). This path needs to already exist. - -10. Click "Transfer & Sync Options" in the center. - -11. Choose "sync - only transfer new or changed files" -(choose "modification time is newer" in the dropdown box), -"preserve source file modification times", and "verify file integrity after transfer". - -12. For "Label This Transfer", put something like "zstash archive LCRC to NERSC". - -13. On the left side, click "Start >". This will start the transfer from LCRC to NERSC. - -Transfer to Compy ------------------ - -1. On the left sidebar, choose "ENDPOINTS". - -2. Search for "pic#compy-dtn". Click on Green power button to activate endpoint. -Log in using Compy credentials. - -3. On the left sidebar, choose "File Manager." The screen will now be split. - -4. On the left side, put "lcrc#dtn_bebop" for "Collection". - -5. On the left side, paste the path to the ``zstash`` archive -(```` for Chrysalis) in the "Path" box. - -6. On the left side, choose "Select all". - -7. On the right side, put "pic#compy-dtn" for "Collection". - -8. Paste the Compy path you want the archive copied to -(```` for Compy). -This path needs to already exist. - -9. Click "Transfer & Sync Options" in the center. - -10. Choose "sync - only transfer new or changed files" -(choose "modification time is newer" in the dropdown box), -"preserve source file modification times", and "verify file integrity after transfer". - -11. For "Label This Transfer", put something like "zstash archive LCRC to Compy". - -12. On the left side, click "Start >". This will start the transfer from LCRC to Compy. - -Check transfers were successful -=============================== - -Cori ----- -1. Create a batch script ``zstash_check.bash``: :: - - #!/bin/bash - #SBATCH --job-name=zstash_check - #SBATCH --nodes=1 - #SBATCH --output=/zstash_check.o%j - #SBATCH --exclusive - #SBATCH --time=3:00:00 - #SBATCH -q regular - #SBATCH --constraint=haswell - - - zstash check --hpss=none --cache= - -2. Run ``sbatch zstash_check.bash``. This takes about an hour to run. - -Compy ------ - -1. Create a batch script ``zstash_check.bash``: :: - - #!/bin/bash - #SBATCH --job-name=zstash_check - #SBATCH --nodes=1 - #SBATCH --output=/zstash_check.o%j - #SBATCH --exclusive - #SBATCH --time=3:00:00 - - - zstash check --hpss=none --cache= - -2. Run ``sbatch zstash_check.bash``. This takes over 3 hours to run. - -Extract the data from the archives -================================== - -Use ``zstash extract``. - -Cori ----- - -1. ``mkdir unified_test_extraction``. - -2. Create a batch script ``zstash_extract.bash``: :: - - #!/bin/bash - #SBATCH --job-name=zstash_extract - #SBATCH --nodes=1 - #SBATCH --output=/zstash_extract.o%j - #SBATCH --exclusive - #SBATCH --time=4:00:00 - #SBATCH -q regular - #SBATCH --constraint=haswell - - - cd unified_test_extraction - zstash extract --hpss=none --cache= - -3. Run ``sbatch zstash_extract.bash``. - -Compy ------ - -1. ``mkdir unified_test_extraction``. - -2. Create a batch script ``zstash_extract.bash``: :: - - #!/bin/bash - #SBATCH --job-name=zstash_extract - #SBATCH --nodes=1 - #SBATCH --output=/zstash_extract.o%j - #SBATCH --exclusive - #SBATCH --time=6:00:00 - - - cd unified_test_extraction - zstash extract --hpss=none --cache= - -3. Run ``sbatch zstash_extract.bash``. - - -Testing zppy on other machines -============================== - -``mv e3sm_unified_test_zstash/unified_test_extraction e3sm_unified_test_zppy/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis``. - -Go through steps 3-7 of "Testing zppy on a small simulation" but for -Cori (haswell) and Compy. - -Sample configuration files for Compy and Cori ---------------------------------------------- - -``/e3sm_unified_test_simulation.cfg``: -Use the configuration file from "Testing zppy on a small simulation" step 4, updating -the appropriate parameters for Compy or Cori. - -Run unit tests and integration tests -==================================== - -Run the unit tests and integration tests on Chrysalis: - -``cd`` into your clone of the ``zppy`` repo. Then: :: - - git checkout main # if not already on the `main` branch - git fetch upstream - git rebase upstream/main - # Load the appropriate environment - pytest tests/test_*.py # Run unit tests - pytest tests/integration/test_*.py # Run integration tests diff --git a/docs/source/dev_guide/update_expected_results.rst b/docs/source/dev_guide/update_expected_results.rst new file mode 100644 index 00000000..64d8f8b4 --- /dev/null +++ b/docs/source/dev_guide/update_expected_results.rst @@ -0,0 +1,116 @@ +.. _updating-expected-results: + +*************************************** +Updating expected results for the tests +*************************************** + + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + machine_name=chrysalis + repo_parent_dir=~/ez/ # Or wherever you keep your repos + expected_results_dir=/lcrc/group/e3sm/public_html/zppy_test_resources + expected_results_records_dir=/lcrc/group/e3sm/public_html/zppy_test_resources_previous + + launch_compute_node() + { + salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm + } + +Compy: + +.. code-block:: bash + + machine_name=compy + repo_parent_dir=~/ez/ # Or wherever you keep your repos + expected_results_dir=/compyfs/www/zppy_test_resources + expected_results_records_dir=/compyfs/fors729/zppy_test_resources_previous + + launch_compute_node() + { + salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm + } + +Note that Compy doesn't give write access to ``/compyfs/www/``, so we can't add a new directory there. That's why ``zppy_test_resources_previous`` is in a separate path. + +Perlmutter: + +.. code-block:: bash + + machine_name=pm-cpu + repo_parent_dir=~/ez/ # Or wherever you keep your repos + expected_results_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources + expected_results_records_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources_previous + + launch_compute_node() + { + salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm + } + + +Process +~~~~~~~ + +.. code-block:: bash + + # First, let's copy over the old expected results ############################# + cp -r ${expected_results_dir} ${expected_results_records_dir}/expected_results_until_yyyymmdd + # Use today's date + # Chrysalis -- takes between 20 and 60 minutes + + # Second, update the expected results ######################################### + # Let's update the simpler tests' results first: + cd ${repo_parent_dir}/zppy + git status + # You might have changed branches since you ran the tests. + # Make sure you're now back on the correct branch: test-zppy-yyyymmdd + # Also confirm you're back in the correct env: zppy-yyyymmdd or the Unified env + + # Make sure the update script permissions are set up + chmod 755 tests/integration/generated/update_bash_generation_expected_files_${machine_name}.sh + chmod 755 tests/integration/generated/update_campaign_expected_files_${machine_name}.sh + chmod 755 tests/integration/generated/update_defaults_expected_files_${machine_name}.sh + chmod 755 tests/integration/generated/update_weekly_expected_files_${machine_name}.sh + + # These scripts update the expected results and re-run the tests: + ./tests/integration/generated/update_bash_generation_expected_files_${machine_name}.sh + ./tests/integration/generated/update_campaign_expected_files_${machine_name}.sh + ./tests/integration/generated/update_defaults_expected_files_${machine_name}.sh + + # This script only updates the expected results + ./tests/integration/generated/update_weekly_expected_files_${machine_name}.sh + # Chrysalis -- takes about 40 minutes + # Perlmutter -- takes about 25 minutes + ls ${expected_results_dir} + # Confirm there are expected results subdirs and image lists for each cfg + + cd ${repo_parent_dir}/zppy + pytest tests/integration/test_bundles.py + launch_compute_node + + start_bash_subshell + # EITHER: + # Activate EITHER a dev environment or the Unified env: + conda activate zppy-yyyymmdd + # OR: the command from `activate_unified_env` + + pytest tests/integration/test_images.py + exit # Exit bash shell + exit # Exit compute note + +Only do this part if you're updating the Official Results (i.e., the results we'd expect from a specific ``zppy`` version): + +.. code-block:: bash + + alias this_release=unified_1.13.0 # Or whatever the release is + + # Third, let's copy the expected results into a special official results dir ## + cp -r ${expected_results_dir} ${expected_results_records_dir}/expected_results_for_${this_release} + # Compy -- takes about 1h20m + ls ${expected_results_records_dir}/expected_results_for_${this_release} + # Confirm there are expected results subdirs and image lists for each cfg diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst deleted file mode 100644 index 7cbae75a..00000000 --- a/docs/source/getting_started.rst +++ /dev/null @@ -1,298 +0,0 @@ -.. _getting-started: - -*************** -Getting started -*************** - -Activate e3sm_unified environment -================================= - -If you have an account on one of the E3SM supported machines (NERSC, Compy, Acme1, -LCRC, Cooley, Rhea), you can access ``zppy`` by activating ``e3sm_unified``, which is -a conda environment that pulls together Python and other E3SM tools such as -``e3sm_diags`` and ``zstash``. - -The paths to ``e3sm_unified`` activation scripts are machine dependent: - -**Compy** - :: - - source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh - - -**Perlmutter-CPU** - :: - - source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh - -**Anvil** - :: - - source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_anvil.sh - -**Chrysalis** - :: - - source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh - - -**Cooley** - :: - - source /lus/theta-fs0/projects/ccsm/acme/tools/e3sm-unified/load_latest_e3sm_unified_cooley.sh - - -**acme1** - :: - - source /p/user_pub/e3sm_unified/envs/load_latest_e3sm_unified_acme1.sh - - -**Andes** - :: - - source /gpfs/alpine/proj-shared/cli115/e3sm-unified/load_latest_e3sm_unified_andes.sh - - -Change ``.sh`` to ``.csh`` for ``csh`` shells. - -Note that ``e3sm_unified``'s development cycle is not in phase with ``zppy``. -Therefore the version of ``zppy`` included may not be the latest. -To install the latest stable release, refer to the following: - -.. _conda_environment: - -Installation in a Conda Environment -=================================== - -If the E3SM Unified environment doesn't serve your needs, you can alternatively -install the latest version in your own custom conda environment. - -First, activate conda or install it if it's not available. Details vary amongst machines. - -.. _conda_environment_others: - -Others/Local ------------- - -If the system doesn't come with conda pre-installed, follow these instructions: - -1. Download Mambaforge - - Linux - :: - - wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh - - MacOS x86_64 (note that ``zppy`` is not supported on MacOS, but it may be useful to contribute to the documentation on MacOS) - :: - - wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-MacOSX-x86_64.sh - -2. Install Mambaforge - - Linux - :: - - bash ./Mambaforge-Linux-x86_64.sh - - - MacOS x86_64 - :: - - bash ./Mambaforge-MacOSX-x86_64.sh - - When you see: :: - - by running conda init? [yes|no] - [no] >>> yes - - respond with ``yes`` so ``conda`` and ``mamba`` commands are available on - initializing a new bash terminal. - -3. If you are working on a machine/network that intercepts SSL communications (such as -acme1), you will get an SSL error unless you disable the SSL verification: - - :: - - conda config --set ssl_verify false - binstar config --set ssl_verify False - -4. Once conda and mamba are properly working, you can install the **(a) Latest Stable Release** or -create a **(b) Development Environment**. - -(a) Latest Stable Release -========================= - -Installation using mamba ------------------------- - -First, make sure that you're using ``bash``. :: - - bash - -You must have a conda base enviornment installed as well. -See :ref:`"Installation in a Conda Environment" ` section above for -installing conda. - -These steps should not be necessary if you installed Mambaforge as suggested -above but may be needed if you have previously installed Miniconda3 instead: :: - - conda install -y -n base mamba - conda config --add channels conda-forge - conda config --set channel_priority strict - -Create a new conda environment with ``zppy`` installed and activate it: :: - - mamba create -n zppy_env zppy - conda activate zppy_env - -Or (less recommended because of potential conflicts) you can install ``zppy`` -in an existing environment. :: - - mamba install zppy - -Updating --------- - -If you **installed into your own conda environment** (e.g., not through the -unified environment), you can update ``zppy`` by doing the following: :: - - mamba update zppy - -.. _dev-env: - -(b) Development Environment -=========================== - -Unlike the latest stable release (i.e., the user environment), the development -environment does not include ``zppy``. -Instead, the developer will ``python -m pip install -e .`` to build ``zppy`` with changes -(see step 7 below). - -Furthermore, the dev environment includes quality assurance (QA) tools such as code formatters, linters, and ``pre-commit``. -**NOTE**: These QA tools are enforced using ``pre-commit`` checks in the continuous integration/continuous delivery (CI/CD) build, so you must use the dev environment for all contributions. - -1. Follow :ref:`"Others/Local" ` section for installing conda. - -2. Clone your fork and keep it in sync with the main repo's ``main`` - - :: - - # Go to https://github.com/E3SM-Project/zppy - # Click "Fork" in the upper right hand corner. This will fork the main repo. - # Click the green "Code" button - # Choose the HTTPS or SSH option. - # (To use the SSH option, you need to have a SSH connection to GitHub set up). - # Click the clipboard icon to copy the path. - # On your command line: - git clone - git remote -v - # You should see your fork listed as `origin` - - - or if you already have a clone of your fork, rebase your fork on the main repo's ``main`` to keep it in sync: - - :: - - # Add the main repo as a remote. - # You can call it anything but "upstream" is recommended. - # We'll use `` here. - git remote add - - # Fetch all the branches of that remote into remote-tracking branches - git fetch - - # Make sure that you're on your ``main`` branch: - git checkout main - - # Rewrite your `main` branch so that any of your commits that - # aren't already in /main are replayed on top of that branch: - git rebase /main - - # Push your main branch to your GitHub fork: - # Note that should be `origin` if you cloned your fork as above. - git push -f main - - - Checkout a new branch from ``main``: - - :: - - git checkout -b /main - -3. Remove any cached conda packages. This will ensure that you always get the latest packages. - - :: - - mamba clean --all - -4. Enter the fork's clone. - - :: - - cd zppy - -5. Use conda to create a new dev environment. -(``zppy`` **is not included in this environment**). - - - Tip: Add the flag ``-n `` to customize the name of the environment - - :: - - mamba env create -f conda/dev.yml - conda activate zppy_dev - -6. Install ``pre-commit``. - - :: - - pre-commit install - -7. install ``zppy`` into the conda environment in edit mode (so code changes - are reflected in the environment as you make them) with: - - :: - - python -m pip install -e . - -8. Make the desired changes to ``zppy``. - -9. Commit changes and make sure ``pre-commit`` checks pass: - - :: - - git commit -m "commit-message" - - .. figure:: _static/pre-commit-passing.png - :alt: pre-commit Output - - ``pre-commit`` Output - - -Configuration file -================== - -The configuration files consists of sections (``[...]``) and subsections (``[[...]]``). There is -a default section at the top (``[default]``) to define some common settings, followed -by a separate section for each available task. Within each task section, you can optionally -include an arbitrary number of subsections for multiple renditions of a given -task. The name of the subsections are arbitrary. They are used to name the batch -jobs and resolve dependencies. - -Please note that the configuration file follows an inheritance model: ``[[ subsections ]]`` inherit settings -from their parent ``[section]``, which itself inherits settings from the ``[default]`` section. -Settings can be defined at arbitrary levels, with the lower level definition taking precedence: -``[[ subsection ]]`` settings can overwrite ``[section]`` settings which can overwrite ``[default]`` settings. -Many settings also take on sensible default values if they are not set. - -Running -======= - -To start the post-processing: :: - - zppy -c - -``zppy`` will parse the configuration file and then generate and submit all batch jobs. -``zppy`` can be invoked safely multiple times -- it will simply check the status of previously -submitted tasks, only submitting new or previously failed tasks. diff --git a/docs/source/index.rst b/docs/source/index.rst index 980a9921..252c453e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,7 +1,4 @@ -.. zppy documentation master file, created by - sphinx-quickstart on Fri Feb 19 18:07:14 2021. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. +.. zppy documentation master file .. _index-label: @@ -12,42 +9,36 @@ Zppy documentation What is zppy? ============= -``zppy`` (pronounced "zippee") is a post-processing toolchain for E3SM written in -Python. The goal of ``zppy`` is to speed-up the post-processing of E3SM simulations -by automating commonly performed tasks. +``zppy`` (pronounced as "zippy", or in IPA: /ˈzɪpi/) is a **workflow manager** for post-processing +E3SM simulations, written in Python. The goal of ``zppy`` is to speed up +post-processing by automating commonly performed tasks. ``zppy`` is built around a set of tasks that perform specific actions, such as generating (regridded) **time series** or **climatology** files using NCO's ``ncclimo``, or generating analysis figures with **E3SM Diags** -or **MPAS-Analysis**. ``zppy`` provides a framework to simplify the use of these -tools, especially for long simulations. - -.. - # TODO: redraw figure - The figure below illustrates schematically how zppy functions. - -``zppy`` is controlled entirely from a single user provided configuration file -(e.g., "mysimulation.cfg"). This configuration file specifies input and output -directories, as well as the list of tasks to run. Each task operates -for a specific model component and over -a specified number of years, for example generating atmosphere climatology -files every 20 and 50 years. - -``zppy`` parses the configuration file and generates batch -jobs that are submitted for execution by SLURM. Dependencies between tasks -are handled and passed to SLURM. Internally, each batch job is created by -instantiating `Jinja `_ template scripts -(usually written in bash, but other languages are supported as well). +or **MPAS-Analysis**. ``zppy`` provides a framework to simplify the use of +these tools, especially for long simulations. + +``zppy`` is controlled entirely from a single user-provided configuration file +(e.g., ``mysimulation.cfg``). This configuration file specifies input and +output directories, as well as the list of tasks to run. Each task operates +over a specified number of years—for +example, generating atmosphere climatology files every 20 and 50 years. + +``zppy`` parses the configuration file and generates batch jobs that are +submitted for execution by SLURM. Dependencies between tasks are handled and +passed to SLURM. Internally, each batch job is created by instantiating +Jinja2 template scripts (usually +written in bash, but other languages are supported as well). .. toctree:: :maxdepth: 2 - :caption: Contents: - - self - schematics - getting_started - tutorial - parameters - campaigns + :caption: User Guide + + user_guide/index + +.. toctree:: + :maxdepth: 2 + :caption: Developer Guide + dev_guide/index - contributing diff --git a/docs/source/parameters.rst b/docs/source/parameters.rst deleted file mode 100644 index 4e4999ab..00000000 --- a/docs/source/parameters.rst +++ /dev/null @@ -1,149 +0,0 @@ -.. _parameters: - -*************** -Parameters -*************** - -Parameters make use of inheritance. Parameters set in ``[default]`` can -be overridden by parameters set in a ``[section]``, which can themselves -be overridden by parameters set in a ``[[subsection]``. - -.. warning:: - Note that some parameters will be overriden by defaults if you define them too high up in the inheritance hierarchy. - -See `this release's parameter defaults `_ -on GitHub for a complete list of parameters and their default values. -You can also view the most up-to-date, -`unreleased parameter defaults `_. - -Deprecated parameters -===================== - -The following parameters no longer perform any function: - :: - - "e3sm_to_cmip_environment_commands" - "ts_fmt" - "scratch" - "atmosphere_only" - "plot_names" - -Parameter checking & inferring -- for users -=========================================== - -There are two types of inferences, each with their own parameter in ``default.ini``: - -* ``infer_path_parameters``: infer paths that are not explicitly provided in the configuraiton file. Default is ``True``. -* ``infer_section_parameters``: infer subtask dependency names that are not explicitly provided in the configuration file. Default is ``True``. - -**Section inferences** - -For the ``climo``, ``ts``, ``e3sm_to_cmip``, and ``e3sm_diags`` tasks: - -* If ``subsection`` (the name of the subtask) is undefined, just use the value of ``grid``. - -For the ``e3sm_to_cmip`` task: - -* If ``ts_subsection`` (the name of the ``ts`` subtask that this ``e3sm_to_cmip`` subtask is dependent on) is undefined, assume it had the same name as this ``e3sm_to_cmip`` subtask. - -For the ``ilamb`` task: - -* If ``ts_land_subsection`` (the name of the ``ts`` land-specific subtask that this ``ilamb`` task is dependent on), assume it is ``land_monthly``. -* If ``e3sm_to_cmip_land_subsection`` (the name of the ``e3sm_to_cmip`` land-specific subtask that this ``ilamb`` task is dependent on), again assume it is ``land_monthly``. -* If we are not doing a ``land_only`` run and ``ts_atm_subsection`` (the name of the ``ts`` atm-specific subtask that this ``ilamb`` task is dependent on), assume it is ``atm_monthly_180x360_aave``. -* If we are not doing a ``land_only`` run and ``e3sm_to_cmip_atm_subsection`` (the name of the ``e3sm_to_cmip`` atm-specific subtask that this ``ilamb`` task is dependent on), again assume it is ``atm_monthly_180x360_aave``. - - -For the ``livvkit`` task: - -* A grid-native climatology must be generated, assumed to be ``land_monthly_climo_native`` -* For each reanalysis comparison to be performed, a corresponding ``land_monthly_climo_GRID`` where ``GRID`` corresponds to the reanalysis grid name -* Available reanalysis comparisons are - * racmo_gis, racmo_ais - * merra2 - * era5 - * ceres (default cmip6 grid) - -**Path inferences** - -For the ``e3sm_diags`` task: - -* If ``reference_data_path`` (the path to the reference data) is undefined, assume it is the ``diagnostics_base_path`` from Mache plus ``/observations/Atm/climatology/``. (So, it is important to change this for model-vs-model runs). - - -For the ``mpas_analysis`` task: - -* ``reference_data_path`` and ``test_data_path`` are optional and are only used for model-vs-model comparisons. - If provided, ``zppy`` uses them to locate the MPAS-Analysis config files from a *previous* MPAS-Analysis run and passes those through to MPAS-Analysis as ``controlRunConfigFile`` (reference) and ``mainRunConfigFile`` (test). - - The comparison type of the *current* MPAS-Analysis run is inferred implicitly: - if ``reference_data_path`` is set, the run is treated as model-vs-model (``mvm``); - otherwise it is treated as model-vs-observations (``mvo``). Users normally do - not need to set a comparison type for the current run. - - .. note:: - These parameter names are intentionally consistent with the terminology used by ``e3sm_diags`` for model-vs-model runs: in both cases, ``reference_data_path`` identifies the *reference simulation's zppy-generated outputs*. - - The practical difference is what each downstream tool consumes: - ``e3sm_diags`` needs ``reference_data_path`` to be the specific directory containing the reference climatology files (typically under the reference run's ``post/.../clim`` tree), whereas ``mpas_analysis`` needs to find the reference MPAS-Analysis config file. - For MPAS-Analysis, ``zppy`` resolves the config file when ``reference_data_path`` points to the prior run's zppy output directory (the one containing ``post/``). - - ``reference_data_path`` is intended to point to the prior run's zppy output directory (the one containing ``post/``). ``zppy`` will then use: - ``/post/analysis/mpas_analysis//cfg/mpas_analysis_.cfg`` - where ```` is ``mvo`` or ``mvm``. - - For referenced prior runs, ``reference_comparison_type`` and - ``test_comparison_type`` can be set to ``"auto"``, ``"mvo"``, or ``"mvm"``. - The default is ``"auto"``. In auto mode: - - * if the path points to ``[[subsection]]``, zppy uses the referenced subsection's actual comparison type - * if the path points to an external zppy output directory, zppy looks for the matching cfg under ``mvo`` and ``mvm`` - * if both exist for the same identifier, zppy raises an error and the user should set ``reference_comparison_type`` or ``test_comparison_type`` explicitly - - When ``reference_data_path`` is set to a non-subsection path, ``reference_case`` is required so the MVM output directory can include the reference case name. If ``reference_data_path`` is set to ``[[subsection]]``, ``reference_case`` is inferred to be the same as the current ``case``. - - -**MPAS-Analysis model-vs-model year ranges** - -MPAS-Analysis comparisons are configured by year ranges, similar to other ``zppy`` tasks. -For model-vs-model comparisons, ``zppy`` supports separate year ranges for the test and reference runs: - -* ``ts_years``, ``climo_years``, ``enso_years`` define the test run year ranges. - If ``test_data_path`` references a prior ``[mpas_analysis]`` subsection using - ``[[subsection]]``, and these values are not provided, zppy uses that - subsection's year ranges instead. -* ``ref_ts_years``, ``ref_climo_years``, ``ref_enso_years`` optionally override the reference run year ranges. - -If a ``ref_*_years`` parameter is not provided, it defaults to the corresponding test year ranges. -If ``reference_data_path`` references a prior ``[mpas_analysis]`` subsection using ``[[subsection]]``, -the defaults come from that subsection's year ranges instead. -If a ``ref_*_years`` parameter contains a single range and multiple test ranges are requested, the single reference range is used for each test range. - - -For the ``ilamb`` task: - -* If ``ilamb_obs`` (the path to observation data for ``ilamb``) is undefined, assume it is the ``diagnostics_base_path`` from Mache plus ``/ilamb_data``. - -**Required parameters, by e3sm_diags set** - -See `Confluence `_ for tables of which sets require which parameters. - -Parameter checking & inferring -- for developers -================================================ - -There are many parameter-handling functions. - -In ``utils.py``: - -* ``get_value_from_parameter``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just use the value of ``second_choice_parameter``. If inferenceis turned off, raise a ``ParameterNotProvidedError``. Use this function if the backup option -* ``set_value_of_parameter_if_undefined``: check if parameter is in the configuration dictionary. If not, if inferenceis turned on (the default), then just set the parameter's value to the ``backup_option``. If inferenceis turned off, raise a ``ParameterNotProvidedError``. - -In ``e3sm_diags.py``: - -* ``check_parameter_defined``: check if parameter is in the configuration dictionary, and if not raise a ``ParameterNotProvidedError``. -* ``check_set_specific_parameter``: if any requested ``e3sm_diags`` sets require this parameter, make sure it is present. If not, raise a ``ParameterNotProvidedError``. -* ``check_parameters_for_bash``: use ``check_set_specific_parameter`` to check the existence of parameters that aren't used until the bash script. -* ``check_mvm_only_parameters_for_bash``: similar, but these are specifically parameters used for model-vs-model runs. Uses ``check_parameter_defined`` in addition to ``check_set_specific_parameter``. -* ``check_and_define_parameters``: make sure all parameters are defined, using ``utils.py get_value_from_parameter``, ``utils.py set_value_of_parameter_if_undefined``, and ``check_mvm_only_parameters_for_bash``. - -``check_parameters_for_bash`` can be run immediately for each subtask because it has very few conditions. Other checks are included in ``check_and_define_parameters`` later on in the code. diff --git a/docs/source/user_guide/campaigns.rst b/docs/source/user_guide/campaigns.rst new file mode 100644 index 00000000..a2cbcda6 --- /dev/null +++ b/docs/source/user_guide/campaigns.rst @@ -0,0 +1,25 @@ +.. _campaigns: + +*************** +Campaigns +*************** + +The ``campaign`` parameter allows users to choose a template of default values for their ``zppy`` run. +Inheritance in configuration files still works with campaigns, so users may override values from a campaign's configuration file. + +Campaign configuration files can be found in the `zppy/defaults directory `_. ``zppy`` currently supports the following campaigns: + +- cryosphere.cfg +- high_res_v1.cfg +- water_cycle.cfg + + +.. warning:: + High-resolution analysis (``campaign = "high_res_v1"``) is only appropriate for some simulations + (e.g., those with the oRRS18to6 ocean and sea-ice mesh). + It will only work if you: + + - have access to the `bigmem nodes on NERSC Cori `_ + - first load the necessary module with ``module load cmem``. + + Perlmutter support has not yet been implemented. This is documented in https://github.com/E3SM-Project/zppy/issues/355 diff --git a/docs/source/user_guide/components/atmosphere.rst b/docs/source/user_guide/components/atmosphere.rst new file mode 100644 index 00000000..4b1c114a --- /dev/null +++ b/docs/source/user_guide/components/atmosphere.rst @@ -0,0 +1,140 @@ +.. _component-atmosphere: + +Atmosphere — EAM and EAMxx +=========================== + +E3SM has two atmosphere models: + +- **EAM** (E3SM Atmosphere Model) — the standard atmosphere model used in + most E3SM configurations. +- **EAMxx** (E3SM Atmosphere Model, next generation) — uses a different + vertical grid (L128) and a different file format. + +Relevant zppy tasks +------------------- + +The following ``zppy`` tasks are commonly used for atmosphere post-processing: + +- :doc:`../tasks/climo` — generate atmosphere climatology files +- :doc:`../tasks/ts` — generate atmosphere time-series files +- :doc:`../tasks/e3sm_to_cmip` — convert to CMIP6 format +- :doc:`../tasks/e3sm_diags` — atmospheric diagnostics (most sets) +- :doc:`../tasks/tc_analysis` — tropical cyclone analysis +- :doc:`../tasks/global_time_series` — global atmospheric time series plots +- :doc:`../tasks/pcmdi_diags` — atmospheric performance metrics + +EAM configuration +----------------- + +For EAM (standard resolution), typical history streams are: + +- ``eam.h0`` — monthly averages (most common for diagnostics) +- ``eam.h1`` — daily averages +- ``eam.h2`` — high-frequency output (used for TC analysis) + +Typical ``[climo]`` and ``[ts]`` subsection configuration for EAM: + +.. code-block:: cfg + + [climo] + active = True + + [[atm_monthly_180x360_aave]] + input_files = eam.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + grid = 180x360_aave + years = 1:100:20 + + [ts] + active = True + + [[atm_monthly_180x360_aave]] + input_files = eam.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + grid = 180x360_aave + years = 1:100:10 + + [[atm_daily_180x360_aave]] + input_files = eam.h1 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + grid = 180x360_aave + frequency = daily + years = 1:100:10 + + [[atm_monthly_glb]] + input_files = eam.h0 + input_subdir = archive/atm/hist + years = 1:100:10 + +EAMxx configuration +------------------- + +EAMxx uses a 128-level vertical grid (L128) and stores output differently +from EAM. Key differences when configuring ``zppy`` for EAMxx: + +- Set ``input_component = eamxx`` in ``[climo]`` and ``[ts]`` sections. +- The ``vrt_in_file`` parameter may be needed for vertical remapping. + When ``input_component = eamxx``, ``zppy`` defaults to a + ``vert_L128.nc`` file under ``diagnostics_base_path``. +- Vertical remapping (``vrt_remap_vars``) is often needed to produce + pressure-level data for diagnostics. + +Example EAMxx configuration: + +.. code-block:: cfg + + [ts] + active = True + + [[eamxx_monthly_180x360_aave]] + input_component = eamxx + input_files = eamxx.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.nc + grid = 180x360_aave + vrt_remap_vars = T,Q,U,V,Z3,OMEGA + years = 1:50:10 + +Mapping files +------------- + +Mapping files remap EAM/EAMxx output from the native spectral-element +grid (e.g., ``ne30pg2``) to a regular latitude-longitude grid +(e.g., ``180x360_aave``). These files are typically pre-computed and +available on supported machines under the ``diagnostics_base_path``. + +Machine-specific mapping file locations: + +- **Chrysalis/Anvil (LCRC)**: typically under + ``/home/ac.zender/data/maps/`` +- **Perlmutter (NERSC)**: typically under + ``/global/homes/z/zender/data/maps/`` +- **Compy**: typically under ``/compyfs/zender/maps/`` + +Diurnal cycle +------------- + +For diurnal cycle diagnostics in ``e3sm_diags``, high-frequency +(8× daily) climatologies are needed. Configure an additional ``[climo]`` +subsection: + +.. code-block:: cfg + + [climo] + [[atm_monthly_diurnal_8xdaily_180x360_aave]] + input_files = eam.h3 + input_subdir = archive/atm/hist + frequency = diurnal_8xdaily + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.nc + grid = 180x360_aave + years = 1:100:20 + + [e3sm_diags] + [[atm_monthly_180x360_aave]] + sets = diurnal_cycle,... + climo_diurnal_subsection = atm_monthly_diurnal_8xdaily_180x360_aave + climo_diurnal_frequency = diurnal_8xdaily + dc_obs_climo = /path/to/obs/climatology diff --git a/docs/source/user_guide/components/index.rst b/docs/source/user_guide/components/index.rst new file mode 100644 index 00000000..c390a954 --- /dev/null +++ b/docs/source/user_guide/components/index.rst @@ -0,0 +1,20 @@ +.. _user-components: + +Model Components +================ + +``zppy`` post-processes output from all E3SM model components. This section +provides component-specific guidance for developers and users working with +each component's output. + +.. toctree:: + :maxdepth: 1 + + atmosphere + land + rivers + ocean + seaice + +`E3SM component documentation `_ provides +further details on each component. diff --git a/docs/source/user_guide/components/land.rst b/docs/source/user_guide/components/land.rst new file mode 100644 index 00000000..71d091a9 --- /dev/null +++ b/docs/source/user_guide/components/land.rst @@ -0,0 +1,107 @@ +.. _component-land: + +Land — ELM +=========== + +**ELM** (E3SM Land Model) is E3SM's land component, based on CLM5. It +simulates vegetation, soil, hydrology, and carbon/nitrogen cycles. + +Relevant zppy tasks +------------------- + +- :doc:`../tasks/ts` — generate land time-series files +- :doc:`../tasks/e3sm_to_cmip` — convert land output to CMIP6 format +- :doc:`../tasks/e3sm_diags` — land diagnostics (e.g., ``lat_lon`` set with land variables) +- :doc:`../tasks/ilamb` — land benchmarking against observational datasets +- :doc:`../tasks/livvkit` — ice sheet validation (for cryosphere configurations) +- :doc:`../tasks/global_time_series` — land global time series (via ``plots_lnd``) + +ELM time-series configuration +------------------------------ + +ELM output is typically in monthly ``elm.h0`` files. A typical ``[ts]`` +configuration for land: + +.. code-block:: cfg + + [ts] + active = True + + [[land_monthly]] + input_files = elm.h0 + input_subdir = archive/lnd/hist + input_component = elm + mapping_file = /path/to/map_r05_to_cmip6_180x360_aave.nc + grid = 180x360_aave + years = 1:100:10 + +The ``input_component = elm`` setting tells ``zppy`` that this is ELM +output. This is required for correct processing by ``e3sm_to_cmip`` and +``ilamb``. + +CMIP conversion for ILAMB +-------------------------- + +ILAMB requires CMIP6-format data. The ``e3sm_to_cmip`` task must be run +first. Typical configuration: + +.. code-block:: cfg + + [e3sm_to_cmip] + active = True + + [[land_monthly]] + input_component = elm + ts_land_subsection = land_monthly + + [ilamb] + active = True + land_only = False + e3sm_to_cmip_land_subsection = land_monthly + +The ``ilamb`` task will depend on both the land and atmosphere +``e3sm_to_cmip`` subtasks (unless ``land_only = True``). With +``infer_section_parameters = True`` (default), subsection names for land +tasks default to ``land_monthly`` and atmosphere tasks default to +``atm_monthly_180x360_aave``. + +Ice sheet validation with LIVVkit +---------------------------------- + +For cryosphere-enabled simulations, the ``livvkit`` task validates the ice +sheet component against reanalysis datasets. LIVVkit requires grid-native +and regridded climatology files from the ``[climo]`` task: + +.. code-block:: cfg + + [climo] + [[land_monthly_climo_native]] + input_files = elm.h0 + input_subdir = archive/lnd/hist + input_component = elm + years = 1:100:20 + + [[land_monthly_climo_era5]] + input_files = elm.h0 + input_subdir = archive/lnd/hist + input_component = elm + mapping_file = /path/to/map_r05_to_era5.nc + grid = era5 + years = 1:100:20 + + [livvkit] + active = True + climo_subsections = land_monthly_climo_native, land_monthly_climo_era5 + icesheets = gis,ais + sets = cmb,smb,energy_era5 + +Notes for ELM developers +-------------------------- + +- The ``input_component = elm`` parameter is used internally by ``zppy`` + to set the correct processing type (``prc_typ``) for ``ncclimo``. +- ELM variables to process can be set via the ``vars`` parameter. Leaving + ``vars = ""`` processes all variables, which can be slow. Consider + specifying only the variables needed for your diagnostics. +- The land component mapping file maps from ELM's finite-volume grid (e.g., + ``r05``, ``ne30pg2``) to a regular lat-lon grid. diff --git a/docs/source/user_guide/components/ocean.rst b/docs/source/user_guide/components/ocean.rst new file mode 100644 index 00000000..e7f1af36 --- /dev/null +++ b/docs/source/user_guide/components/ocean.rst @@ -0,0 +1,90 @@ +.. _component-ocean: + +Ocean — MPAS-Ocean +=================== + +**MPAS-Ocean** is E3SM's ocean model, based on the Model for Prediction +Across Scales (MPAS) framework. It uses an unstructured mesh that allows +variable horizontal resolution. + +Relevant zppy tasks +------------------- + +- :doc:`../tasks/mpas_analysis` — ocean and sea-ice analysis (primary task) +- :doc:`../tasks/global_time_series` — ocean global time series plots + +MPAS-Analysis for ocean +------------------------ + +MPAS-Analysis is the primary tool for MPAS-Ocean diagnostics. It operates +on MPAS-Ocean's native unstructured mesh. The ``[mpas_analysis]`` task in +``zppy`` calls MPAS-Analysis automatically. + +Typical ``[mpas_analysis]`` configuration: + +.. code-block:: cfg + + [mpas_analysis] + active = True + walltime = 06:00:00 + shortTermArchive = True + ts_years = 1:100:50 + climo_years = 1:100:50 + enso_years = 1:100:50 + generate = all, no_landIceCavities, no_BGC, no_icebergs, no_min, no_max + +Key MPAS-Ocean-specific parameters: + +- ``mpaso_nml``: name of the MPAS-Ocean namelist file (default: ``mpaso_in``) +- ``stream_ocn``: name of the MPAS-Ocean streams file (default: ``streams.ocean``) +- ``PostMOC``: whether to post-process MOC (meridional overturning + circulation) data + +Model-vs-model ocean comparisons +--------------------------------- + +MPAS-Analysis supports comparing two model simulations. In ``zppy``, this +is configured via ``reference_data_path``: + +.. code-block:: cfg + + [mpas_analysis] + + [[main_run]] + ts_years = 1:100:50 + climo_years = 1:100:50 + + [[mvm_run]] + ts_years = 1:100:50 + climo_years = 1:100:50 + reference_data_path = [[main_run]] + # or: reference_data_path = /path/to/prior/zppy/output + +See the :doc:`../tasks/mpas_analysis` page and the :ref:`tutorial ` +for a complete model-vs-model example. + +Global time series ocean plots +------------------------------- + +The ``[global_time_series]`` task can produce ocean time series plots such +as: + +- ``change_ohc`` — ocean heat content change +- ``max_moc`` — maximum meridional overturning circulation +- ``change_sea_level`` — global mean sea level change + +These require access to MPAS-Ocean output. The ``input_subdir`` in +``[global_time_series]`` should be set to ``archive/ocn/hist`` (the default). + +Notes for MPAS-Ocean developers +--------------------------------- + +- The ``generate`` parameter in ``[mpas_analysis]`` controls which + MPAS-Analysis tasks to run. Use ``no_*`` filters to exclude irrelevant + analyses for your simulation (e.g., ``no_BGC`` if biogeochemistry is not + enabled, ``no_landIceCavities`` if land ice cavities are not used). +- The ``environment_commands`` for ``[mpas_analysis]`` must be consistent + across all year-range runs. Mixing environments can cause MPAS-Analysis to + fail. +- MPAS-Analysis caches intermediate results (``cache = True`` by default). + Set ``purge = True`` to clear previous output and start fresh. diff --git a/docs/source/user_guide/components/rivers.rst b/docs/source/user_guide/components/rivers.rst new file mode 100644 index 00000000..ea30ac0d --- /dev/null +++ b/docs/source/user_guide/components/rivers.rst @@ -0,0 +1,69 @@ +.. _component-rivers: + +Rivers — MOSART +================ + +**MOSART** (Model for Scale Adaptive River Transport) is E3SM's river +routing component. It routes runoff from ELM through river networks to +the ocean. + +Relevant zppy tasks +------------------- + +- :doc:`../tasks/ts` — generate river time-series files +- :doc:`../tasks/e3sm_diags` — streamflow diagnostics (``streamflow`` set) + +MOSART time-series configuration +--------------------------------- + +MOSART output is in monthly ``mosart.h0`` files. Configuration example: + +.. code-block:: cfg + + [ts] + active = True + + [[rof_monthly]] + input_files = mosart.h0 + input_subdir = archive/rof/hist + input_component = mosart + years = 1:100:10 + +E3SM Diags streamflow set +-------------------------- + +The ``streamflow`` set in ``e3sm_diags`` compares MOSART-simulated +streamflow against stream gauge observations. It requires: + +1. A river time-series subtask in ``[ts]`` +2. The ``streamflow_obs_ts`` parameter pointing to observational gauge data + +Example configuration: + +.. code-block:: cfg + + [ts] + [[rof_monthly]] + input_files = mosart.h0 + input_subdir = archive/rof/hist + input_component = mosart + years = 1:100:10 + + [e3sm_diags] + [[streamflow_diags]] + sets = streamflow + ts_subsection = rof_monthly + streamflow_obs_ts = /path/to/obs/streamflow + # For model-vs-model: + # run_type = model_vs_model + # reference_data_path_ts_rof = /path/to/reference/post/atm/... + # gauges_path = /path/to/gauges.nc + +Notes for MOSART developers +---------------------------- + +- The ``input_component = mosart`` parameter tells ``zppy`` how to + handle MOSART output files. +- River routing output can differ between E3SM configurations, so + the ``input_files`` stream name (``mosart.h0``) should be verified + for your specific simulation. diff --git a/docs/source/user_guide/components/seaice.rst b/docs/source/user_guide/components/seaice.rst new file mode 100644 index 00000000..f92d499b --- /dev/null +++ b/docs/source/user_guide/components/seaice.rst @@ -0,0 +1,54 @@ +.. _component-seaice: + +Sea Ice — MPAS-seaice +====================== + +**MPAS-seaice** is E3SM's sea-ice model, also based on the MPAS framework +and running on the same unstructured mesh as MPAS-Ocean. + +Relevant zppy tasks +------------------- + +- :doc:`../tasks/mpas_analysis` — primary task for sea-ice diagnostics +- :doc:`../tasks/global_time_series` — sea-ice global time series plots + +MPAS-Analysis for sea ice +-------------------------- + +MPAS-Analysis includes a suite of sea-ice diagnostics. These run +automatically alongside ocean diagnostics when ``[mpas_analysis]`` is +active. Sea-ice analyses can be controlled via the ``generate`` parameter: + +.. code-block:: cfg + + [mpas_analysis] + active = True + # To disable specific sea-ice analyses: + generate = all, no_icebergs, no_landIceCavities + +Key MPAS-seaice-specific parameters: + +- ``mpassi_nml``: name of the MPAS sea-ice namelist file (default: + ``mpassi_in``) +- ``stream_ice``: name of the MPAS sea-ice streams file (default: + ``streams.seaice``) + +Global time series sea-ice plots +--------------------------------- + +The ``[global_time_series]`` task can produce sea-ice time series plots when +``make_viewer = True``. Use the ``plots_ice`` parameter to specify which +sea-ice variables to include. + +Notes for MPAS-seaice developers +--------------------------------- + +- Sea-ice and ocean analyses share the same ``[mpas_analysis]`` section. + They are differentiated internally by MPAS-Analysis based on the + ``generate`` list. +- ``icebergs`` and ``landIceCavities`` are disabled by default in the + standard ``generate`` list because most E3SM configurations do not include + these features. Enable them if your simulation does. +- The ``anomalyRefYear`` parameter sets the reference year for anomaly + plots in MPAS-Analysis. For historical simulations, this is typically + set to the first year of the observational record. diff --git a/docs/source/user_guide/dependencies.rst b/docs/source/user_guide/dependencies.rst new file mode 100644 index 00000000..4b665adc --- /dev/null +++ b/docs/source/user_guide/dependencies.rst @@ -0,0 +1,11 @@ +.. _dependency_graph: + +************ +Dependencies +************ + +Below defines a dependency graph for ``zppy`` tasks. It can be visualized with graph-drawing tools like GraphViz or ``pydot``. + +.. literalinclude:: ../dependencies.dot + :language: dot + :linenos: diff --git a/docs/source/user_guide/getting_started.rst b/docs/source/user_guide/getting_started.rst new file mode 100644 index 00000000..8019606f --- /dev/null +++ b/docs/source/user_guide/getting_started.rst @@ -0,0 +1,139 @@ +.. _getting-started: + +*************** +Getting started +*************** + +Activate e3sm_unified environment +================================= + +If you have an account on one of the E3SM supported machines, you can access ``zppy`` by activating ``e3sm_unified``, which is +a conda environment that pulls together Python and other E3SM tools such as +``e3sm_diags`` and ``zstash``. + +The paths to ``e3sm_unified`` activation scripts are machine dependent. As of E3SM Unified 1.13.0, the supported machines and their corresponding activation scripts are: + +**Andes** + :: + + source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_andes.sh + +**Aurora** + :: + + source /lus/flare/projects/E3SMinput/soft/e3sm-unified/load_latest_e3sm_unified_aurora.sh + +**Chrysalis** + :: + + source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh + +**Compy** + :: + + source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh + +**Dane** + :: + + source /usr/workspace/e3sm/apps/e3sm-unified/load_latest_e3sm_unified_dane.sh + +**Frontier** + :: + + source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_frontier.sh + +**Perlmutter (login or CPU nodes)** + :: + + source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh + +**ALCF Polaris** + :: + + source /lus/grand/projects/E3SMinput/soft/e3sm-unified/load_latest_e3sm_unified_polaris.sh + + +Change ``.sh`` to ``.csh`` for ``csh`` shells. + +E3SM Unified and zppy versions +============================== + +``zppy`` development is largely synced with ``e3sm_unified``. The last several releases have been: + +* E3SM Unified 1.13.0: ``zppy 3.2.0``, ``zppy-interfaces 0.2.1`` +* E3SM Unified 1.12.0: ``zppy 3.1.0``, ``zppy-interfaces 0.2.0`` +* E3SM Unified 1.11.1: ``zppy 3.1.0``, ``zppy-interfaces 0.1.2`` +* E3SM Unified 1.11.0: ``zppy 3.0.0``, ``zppy-interfaces 0.1.1`` (new auxiliary package) +* E3SM Unified 1.10.0: ``zppy 2.5.0`` + +To use ``zppy`` features/fixes not yet in a production release, you can use a development environment. + +Nevertheless, it is possible that the version of ``zppy`` included in ``e3sm_unified`` may not be the latest. To install the latest development environment, refer to the following section. + +Setting up a development environment +==================================== + +.. code-block:: bash + + # Get the code ######################################################################## + + # Set up your fork: + # Go to https://github.com/E3SM-Project/zppy + # Click the green "Code" button + # Choose the SSH option and paste it here: + git clone git@github.com:E3SM-Project/zppy.git + cd zppy + git remote -v # You should see the main repo listed as `origin` + + # A couple optional steps: + git remote add upstream git@github.com:E3SM-Project/zppy.git # Use the name "upstream" instead of origin + git remote add your-fork-name git@github.com:your-fork-name/zppy.git # Use your fork, if you have one + + # To use the latest code: + git fetch upstream + git checkout -b main upstream/main + # To use code from a specific branch: + git checkout -b that-branch-name upstream/that-branch-name + + # Set up the environment ############################################################## + # First, make sure you have conda activated. Then: + rm -rf build # Sometimes an existing `build` directory can cause problems. + conda clean --all --y # This makes sure conda will pick up the latest information. + conda env create -f conda/dev.yml -n env-name + conda activate env-name + pre-commit run --all-files # This is only necessary if you've made changes + python -m pip install . # Install the code into your development environment (env-name) + +Note: if you'd like to contribute to ``zppy`` rather than just using the latest code, please refer to the Developer Guide instead. + + +Configuration file +================== + +The configuration files consists of sections (``[...]``) and subsections (``[[...]]``). There is +a default section at the top (``[default]``) to define some common settings, followed +by a separate section for each available task. Within each task section, you can optionally +include an arbitrary number of subsections for multiple renditions of a given +task. The name of the subsections are arbitrary. They are used to name the batch +jobs and resolve dependencies. + +Please note that the configuration file follows an inheritance model: ``[[ subsections ]]`` inherit settings +from their parent ``[section]``, which itself inherits settings from the ``[default]`` section. +Settings can be defined at arbitrary levels, with the lower level definition taking precedence: +``[[ subsection ]]`` settings can overwrite ``[section]`` settings which can overwrite ``[default]`` settings. +Many settings also take on sensible default values if they are not set. + +Running +======= + +To start the post-processing: :: + + zppy -c + +``zppy`` will parse the configuration file and then generate and submit all batch jobs. +For the most part, ``zppy`` can be invoked safely multiple times -- it will simply check the status of previously +submitted tasks, only submitting new or previously failed tasks. + +.. warning:: + Some tasks cannot be safely rerun. Notably, ``mpas_analysis`` has historically needed to be rerun from scratch if a job failed. This is documented in https://github.com/E3SM-Project/zppy/issues/764. diff --git a/docs/source/user_guide/index.rst b/docs/source/user_guide/index.rst new file mode 100644 index 00000000..62e30b8f --- /dev/null +++ b/docs/source/user_guide/index.rst @@ -0,0 +1,20 @@ +.. _user-guide: + +********** +User Guide +********** + +This guide covers everything a researcher needs to use ``zppy`` to +post-process E3SM simulations. + +.. toctree:: + :maxdepth: 2 + + getting_started + tutorial + schematics + campaigns + parameters + dependencies + tasks/index + components/index diff --git a/docs/source/user_guide/parameters.rst b/docs/source/user_guide/parameters.rst new file mode 100644 index 00000000..ee44b65a --- /dev/null +++ b/docs/source/user_guide/parameters.rst @@ -0,0 +1,409 @@ +.. _parameters: + +*************** +Parameters +*************** + +Parameters make use of inheritance. Parameters set in ``[default]`` can +be overridden by parameters set in a ``[section]``, which can themselves +be overridden by parameters set in a ``[[subsection]]``. + +.. warning:: + Note that some parameters will be overridden by defaults if you define them too high up in the inheritance hierarchy. + +See `this release's parameter defaults `_ +on GitHub for a complete list of parameters and their default values. +You can also view the most up-to-date, +`unreleased parameter defaults `_. + +Parameters for individual tasks +=============================== + +For per-task parameter tables, see the :ref:`Tasks ` section. + +.. _parameters-top-level: + +Parameters at the top-level +=========================== + +These parameters appear in the top-level ``[default]`` section of ``zppy/defaults/default.ini``. They can be overridden in individual tasks and subtasks. + +There are 38 (non-deprecated) parameters that can be defined at the top level. + +**Input specifics** + +There are 2 input-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``case`` + - **Yes** + - *(none)* + - The case name of the simulation. + * - ``input`` + - **Yes** + - *(none)* + - The top-level directory of the simulation output to post-process. + +**Output specifics** + +There are 6 output-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``output`` + - **Yes** + - *(none)* + - Where the post-processing results (``post/`` directory) should go. + * - ``www`` + - **Yes** + - *(none)* + - Where the post-processing visuals should go (to be viewed online). + * - ``campaign`` + - No + - ``"none"`` + - Specify which campaign you are running. Campaigns can be found in ``zppy/defaults``. Possible campaigns include ``cryosphere`` and ``water_cycle``. + * - ``debug`` + - No + - ``False`` + - Set to True to have ``zppy`` produce more verbose output and retain temporary workdirs. This is helpful for debugging. + * - ``dry_run`` + - No + - ``False`` + - This should be set to True if you don't want the batch jobs to be submitted. I.e., you only want to see what *would* be submitted. + * - ``fail_on_dependency_skip`` + - No + - ``False`` + - If set to False, zppy will launch other jobs, if possible. + +**Machine specifics** + +There are 8 machine-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``account`` + - No + - ``""`` + - SLURM account to charge. + * - ``constraint`` + - No + - ``""`` + - The constraint of the machine to run on. + * - ``nodes`` + - No + - ``1`` + - Number of compute nodes. + * - ``parallel`` + - No + - ``"mpi"`` + - The ``parallel`` option passed to ``ncclimo``. + * - ``partition`` + - No + - ``""`` + - SLURM partition to submit the job to. + * - ``qos`` + - No + - ``"regular"`` + - Quality of service + * - ``reservation`` + - No + - ``""`` + - If you have access to a node reservation, specify it here. + * - ``walltime`` + - No + - ``"02:00:00"`` + - Maximum wall time for the SLURM job. + +**Environment specifics** + +There are 3 environment-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``environment_commands`` + - No + - ``""`` + - Shell commands to set up the software environment before running the + task (e.g., ``source /path/to/e3sm_unified.sh``). + * - ``environment_commands_secondary`` + - No + - ``""`` + - Shell commands to set up the *secondary* software environment before running the task (e.g., ``source /path/to/e3sm_unified.sh``). This is currently only used for the ``pcmdi_diags`` task. + * - ``nco_path`` + - No + - ``""`` + - Keep as the empty string to use production-version NCO commands. Set to the development-version path to use that instead. + +**Inference specifics** + +There are 2 inference-specific parameters. See "Parameter checking & inferring" below for more info. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``infer_path_parameters`` + - No + - ``True`` + - If true, infer file paths not given. + * - ``infer_section_parameters`` + - No + - ``True`` + - If true, infer dependencies not given. + +**Dependency specifics** + +There are 7 dependency-specific parameters. See :ref:`Dependencies ` for more info on which tasks depend on which. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``ts_atm_grid`` + - No + - ``"180x360_aave"`` + - Grid name of the atmosphere ``[ts]`` subtask. + * - ``ts_atm_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` atmosphere subtask (for multi-component runs). + * - ``ts_grid`` + - No + - ``"180x360_aave"`` + - Grid name of the ``[ts]`` subtask to depend on. + * - ``ts_land_grid`` + - No + - ``"180x360_aave"`` + - Grid name of the land ``[ts]`` subtask. + * - ``ts_land_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` land subtask (for multi-component runs). + * - ``ts_num_years`` + - No + - ``5`` + - Years increment from the dependent ``[ts]`` task. + * - ``ts_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` subtask to depend on. If empty and + ``infer_section_parameters`` is ``True``, inferred from the + subsection name. + +**Task specifics** + +There are 10 task-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``bundle`` + - No + - ``""`` + - (Set on individual tasks) Name of the bundle to include this task + in. + * - ``frequency`` + - No + - ``"monthly"`` + - The frequency of the data. Options include "monthly", "diurnal_8xdaily", "daily". + * - ``grid`` + - No + - ``""`` + - Name of the output grid (e.g., ``180x360_aave``). Used for naming + output directories. + * - ``input_files`` + - No + - ``"eam.h0"`` + - Pattern matching the input history files (e.g., ``eam.h0``, + ``eam.h1``). + * - ``input_subdir`` + - No + - ``"archive/atm/hist"`` + - Subdirectory under ``input``/``case`` containing the data files. + * - ``mapping_file`` + - No + - ``""`` + - Path to the mapping (regridding) file. Leave empty for no regridding. + * - ``plugins`` + - No + - ``""`` + - External zppy plugin modules + * - ``vars`` + - No + - ``"FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,PSL"`` + - Variables to process. An empty string processes all variables. + * - ``years`` + - No + - ``[""]`` + - Year ranges to process. Format: ``"start:end:increment"`` (e.g., + ``"1:100:20"``). + +**Deprecated Parameters** + +There is 1 deprecated parameter. Specifying it will have no effect. + +.. code-block:: text + + ncclimo_cmd + +Parameter checking & inferring +============================== + +There are two types of inferences, each with their own parameter in ``default.ini``: + +* ``infer_path_parameters``: infer paths that are not explicitly provided in the configuration file. Default is ``True``. +* ``infer_section_parameters``: infer subtask dependency names that are not explicitly provided in the configuration file. Default is ``True``. + +**Section inferences** + +For the ``climo``, ``ts``, ``e3sm_to_cmip``, and ``e3sm_diags`` tasks: + +* If ``subsection`` (the name of the subtask) is undefined, just use the value of ``grid``. + +For the ``e3sm_to_cmip`` task: + +* If ``ts_subsection`` (the name of the ``ts`` subtask that this ``e3sm_to_cmip`` subtask is dependent on) is undefined, assume it had the same name as this ``e3sm_to_cmip`` subtask. + +For the ``ilamb`` task: + +* If ``ts_land_subsection`` (the name of the ``ts`` land-specific subtask that this ``ilamb`` task is dependent on), assume it is ``land_monthly``. +* If ``e3sm_to_cmip_land_subsection`` (the name of the ``e3sm_to_cmip`` land-specific subtask that this ``ilamb`` task is dependent on), again assume it is ``land_monthly``. +* If we are not doing a ``land_only`` run and ``ts_atm_subsection`` (the name of the ``ts`` atm-specific subtask that this ``ilamb`` task is dependent on), assume it is ``atm_monthly_180x360_aave``. +* If we are not doing a ``land_only`` run and ``e3sm_to_cmip_atm_subsection`` (the name of the ``e3sm_to_cmip`` atm-specific subtask that this ``ilamb`` task is dependent on), again assume it is ``atm_monthly_180x360_aave``. + + +For the ``livvkit`` task: + +* A grid-native climatology must be generated, assumed to be ``land_monthly_climo_native`` +* For each reanalysis comparison to be performed, a corresponding ``land_monthly_climo_GRID`` where ``GRID`` corresponds to the reanalysis grid name +* Available reanalysis comparisons are + * racmo_gis, racmo_ais + * merra2 + * era5 + * ceres (default cmip6 grid) + +**Path inferences** + +For the ``e3sm_diags`` task: + +* If ``reference_data_path`` (the path to the reference data) is undefined, assume it is the ``diagnostics_base_path`` from Mache plus ``/observations/Atm/climatology/``. (So, it is important to change this for model-vs-model runs). + + +For the ``mpas_analysis`` task: + +* ``reference_data_path`` and ``test_data_path`` are optional and are only used for model-vs-model comparisons. + If provided, ``zppy`` uses them to locate the MPAS-Analysis config files from a *previous* MPAS-Analysis run and passes those through to MPAS-Analysis as ``controlRunConfigFile`` (reference) and ``mainRunConfigFile`` (test). + + The comparison type of the *current* MPAS-Analysis run is inferred implicitly: + if ``reference_data_path`` is set, the run is treated as model-vs-model (``mvm``); + otherwise it is treated as model-vs-observations (``mvo``). Users normally do + not need to set a comparison type for the current run. + + .. note:: + + These parameter names are intentionally consistent with the terminology used by ``e3sm_diags`` for model-vs-model runs: in both cases, ``reference_data_path`` identifies the *reference simulation's zppy-generated outputs*. + + The practical difference is what each downstream tool consumes: + ``e3sm_diags`` needs ``reference_data_path`` to be the specific directory containing the reference climatology files (typically under the reference run's ``post/.../clim`` tree), whereas ``mpas_analysis`` needs to find the reference MPAS-Analysis config file. + For MPAS-Analysis, ``zppy`` resolves the config file when ``reference_data_path`` points to the prior run's zppy output directory (the one containing ``post/``). + + ``reference_data_path`` is intended to point to the prior run's zppy output directory (the one containing ``post/``). ``zppy`` will then use: + ``/post/analysis/mpas_analysis//cfg/mpas_analysis_.cfg`` + where ```` is ``mvo`` or ``mvm``. + + For referenced prior runs, ``reference_comparison_type`` and + ``test_comparison_type`` can be set to ``"auto"``, ``"mvo"``, or ``"mvm"``. + The default is ``"auto"``. In auto mode: + + * if the path points to ``[[subsection]]``, zppy uses the referenced subsection's actual comparison type + * if the path points to an external zppy output directory, zppy looks for the matching cfg under ``mvo`` and ``mvm`` + * if both exist for the same identifier, zppy raises an error and the user should set ``reference_comparison_type`` or ``test_comparison_type`` explicitly + + When ``reference_data_path`` is set to a non-subsection path, ``reference_case`` is required so the MVM output directory can include the reference case name. If ``reference_data_path`` is set to ``[[subsection]]``, ``reference_case`` is inferred to be the same as the current ``case``. + + +**MPAS-Analysis model-vs-model year ranges** + +MPAS-Analysis comparisons are configured by year ranges, similar to other ``zppy`` tasks. +For model-vs-model comparisons, ``zppy`` supports separate year ranges for the test and reference runs: + +* ``ts_years``, ``climo_years``, ``enso_years`` define the test run year ranges. + If ``test_data_path`` references a prior ``[mpas_analysis]`` subsection using + ``[[subsection]]``, and these values are not provided, zppy uses that + subsection's year ranges instead. +* ``ref_ts_years``, ``ref_climo_years``, ``ref_enso_years`` optionally override the reference run year ranges. + +If a ``ref_*_years`` parameter is not provided, it defaults to the corresponding test year ranges. +If ``reference_data_path`` references a prior ``[mpas_analysis]`` subsection using ``[[subsection]]``, +the defaults come from that subsection's year ranges instead. +If a ``ref_*_years`` parameter contains a single range and multiple test ranges are requested, the single reference range is used for each test range. + + +For the ``ilamb`` task: + +* If ``ilamb_obs`` (the path to observation data for ``ilamb``) is undefined, assume it is the ``diagnostics_base_path`` from Mache plus ``/ilamb_data``. + +Deprecated parameters +===================== + +The following ``zppy`` parameters no longer perform any function. + +These are no longer defined in ``zppy/defaults/default.ini``: + +.. code-block:: text + + e3sm_to_cmip_environment_commands + ts_fmt + scratch + atmosphere_only + plot_names + vars_exclude + +These are still defined in ``zppy/defaults/default.ini``, but have no effect: + +.. code-block:: text + ncclimo_cmd + nrows + ncols diff --git a/docs/source/schematics.rst b/docs/source/user_guide/schematics.rst similarity index 97% rename from docs/source/schematics.rst rename to docs/source/user_guide/schematics.rst index 3a0a5301..560c4b5d 100644 --- a/docs/source/schematics.rst +++ b/docs/source/user_guide/schematics.rst @@ -38,7 +38,7 @@ The plots for years 51-100 have similar requirements – however the graph calls attention to the fact that MPAS-Analysis for years 51-100 itself depends on MPAS-Analysis for years 1-50. -.. image:: figures/zppy_jobs.png +.. image:: ../figures/zppy_jobs.png :scale: 80% Figure 1. Illustration of how zppy uses sections of the configuration file and @@ -47,7 +47,7 @@ climo stands for climatology and ts stands for time series. Note that the configuration file is abridged – see the :ref:`zppy tutorial ` for more complete examples. -.. image:: figures/zppy_dependencies.png +.. image:: ../figures/zppy_dependencies.png :scale: 80% Figure 2. Two possible job dependency graphs. diff --git a/docs/source/user_guide/tasks/bundle.rst b/docs/source/user_guide/tasks/bundle.rst new file mode 100644 index 00000000..bc419854 --- /dev/null +++ b/docs/source/user_guide/tasks/bundle.rst @@ -0,0 +1,59 @@ +.. _task-bundle: + +bundle — Job Bundling +====================== + +The ``bundle`` task groups multiple ``zppy`` tasks into a single SLURM job +script. This is useful for running many lightweight tasks together on a +single node, avoiding scheduler overhead and resource waste. + +When a task specifies ``bundle = ""`` in its configuration, it will +be included in the named bundle. The bundle itself is defined in a +``[bundle]`` section. + +.. note:: + Bundle parameters (e.g., ``walltime``, ``nodes``) are taken from the + *first task* added to the bundle, or from the ``[bundle]`` section + itself. The ``active`` parameter in ``[bundle]`` always overrides the + ``[default]`` value and defaults to ``True``. + +Configuration example +--------------------- + +.. code-block:: cfg + + [bundle] + [[long_jobs]] + walltime = 08:00:00 + nodes = 1 + + [climo] + [[atm_monthly_180x360_aave]] + bundle = long_jobs + years = 1:100:20 + + [ts] + [[atm_monthly_180x360_aave]] + bundle = long_jobs + years = 1:100:10 + +Parameters +---------- + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``True`` + - Whether the bundle is active. Overrides the ``[default]`` value. + * - ``bundle`` + - No + - ``""`` + - (Set on individual tasks) Name of the bundle to include this task + in. diff --git a/docs/source/user_guide/tasks/climo.rst b/docs/source/user_guide/tasks/climo.rst new file mode 100644 index 00000000..d52fa42c --- /dev/null +++ b/docs/source/user_guide/tasks/climo.rst @@ -0,0 +1,102 @@ +.. _task-climo: + +climo — Climatology Files +========================= + +The ``climo`` task generates climatology files from E3SM simulation output +using NCO's ``ncclimo``. It produces time-averaged files (monthly and/or +seasonal means) over a specified set of years, optionally regridding the +output to a target grid. + +Climatology files produced by ``climo`` are required inputs for the +:doc:`e3sm_diags` task. + +Configuration example +--------------------- + +.. code-block:: cfg + + [climo] + active = True + nodes = 4 + years = 1:100:20 + + [[atm_monthly_180x360_aave]] + input_files = eam.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.nc + grid = 180x360_aave + vars = FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TS,TREFHT,CLDTOT + +Parameters +---------- + +These 3 parameters are specific to the ``climo`` task: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``exclude`` + - No + - ``False`` + - Set to ``True`` to exclude this subtask from running (subsection-level). + * - ``climo_jobs`` + - No + - ``0`` + - Number of simultaneous ``ncclimo`` jobs. ``0`` uses ``ncclimo``'s + default. + * - ``input_component`` + - No + - ``""`` + - Model component that generated the input files (e.g., ``eam``, + ``eamxx``, ``elm``, ``mosart``). Used to set processing type + internally. + + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These 3 parameters have ``climo``-specific defaults, which means even if they are set at the top level (``[default]``) section, these default values will be used instead. Therefore, to specify a custom value, these parameters must be defined inside ``[climo]``: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``parallel`` + - No + - ``"mpi"`` + - The ``parallel`` option passed to ``ncclimo``. + Overrides the ``[default]`` value (which is ``""``). + * - ``nodes`` + - No + - ``4`` + - Number of compute nodes. Overrides the ``[default]`` value (``1``). + * - ``vars`` + - No + - ``""`` + - Variables to process. An empty string processes *all* variables (no + ``-v`` flag is passed to ``ncclimo``). Overrides the ``[default]`` + value. + +For other top-level parameters, see :ref:`top-level parameters `. + +Dependencies +------------ + +**Upstream (what climo depends on):** + +- None + +**Downstream (what depends on climo):** + +- :doc:`e3sm_diags` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly diurnal-atm climo: diurnal_cycle +- :doc:`livvkit` — Monthly-lnd climo diff --git a/docs/source/user_guide/tasks/e3sm_diags.rst b/docs/source/user_guide/tasks/e3sm_diags.rst new file mode 100644 index 00000000..cf5ba265 --- /dev/null +++ b/docs/source/user_guide/tasks/e3sm_diags.rst @@ -0,0 +1,484 @@ +.. _task-e3sm-diags: + +e3sm_diags — E3SM Diagnostics +============================== + +The ``e3sm_diags`` task runs `E3SM Diagnostics +`_ to produce a wide variety of +atmospheric, land, and river diagnostic plots. It depends on the :doc:`climo` +and/or :doc:`ts` tasks for its input files. + +For a complete list of available E3SM Diags parameters, see the +`E3SM Diags documentation +`_. + +For a table of which diagnostic sets require which parameters, see +`Confluence +`_. + +Diagnostic sets +--------------- + +The ``sets`` parameter controls which diagnostic sets are run. All 20 +available sets are: + +``aerosol_aeronet``, ``aerosol_budget``, ``annual_cycle_zonal_mean``, +``area_mean_time_series``, ``cosp_histogram``, ``diurnal_cycle``, +``enso_diags``, ``lat_lon``, ``lat_lon_land``, ``meridional_mean_2d``, ``mp_partition``, +``polar``, ``precip_pdf``, ``qbo``, ``streamflow``, ``tc_analysis``, +``tropical_subseasonal``, ``zonal_mean_2d``, ``zonal_mean_2d_stratosphere``, +``zonal_mean_xy`` + +Parameters +---------- + +These 35 parameters are specific to the ``e3sm_diags`` task. + +General parameters +~~~~~~~~~~~~~~~~~~ + +There are 12 general parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``backend`` + - No + - ``"mpl"`` + - Plotting backend. + * - ``cfg`` + - No + - ``""`` + - Path to an additional E3SM Diags configuration file. + * - ``keep_mvm_case_name_in_fig`` + - No + - ``True`` + - Include case name in model-vs-model output paths. + * - ``multiprocessing`` + - No + - ``True`` + - Use multiprocessing. + * - ``num_workers`` + - No + - ``24`` + - Number of worker processes. + * - ``output_format`` + - No + - ``["png"]`` + - Output plot formats. + * - ``output_format_subplot`` + - No + - ``[]`` + - Output formats for subplots. + * - ``run_type`` + - No + - ``"model_vs_obs"`` + - Comparison type: ``"model_vs_obs"`` or ``"model_vs_model"``. + * - ``sets`` + - No + - ``"lat_lon,zonal_mean_xy,..."`` + - List of diagnostic sets to run. + * - ``short_name`` + - No + - ``""`` + - Short name used as test name and label. + * - ``swap_test_ref`` + - No + - ``False`` + - Swap test and reference in model-vs-model runs. + * - ``tag`` + - No + - ``"model_vs_obs"`` + - Label for the results directory. + +Set-specific parameters +~~~~~~~~~~~~~~~~~~~~~~~ + +There are 23 set-specific parameters. + +**Overview by set** + +We will review useful parameters by set. To do this, we will group them by dependency. For more info on dependencies, see :ref:`Dependencies `. + +.. list-table:: + :header-rows: 1 + :stub-columns: 1 + :widths: 20 80 + + * - Group + - Sets + * - ``climo_month_atm`` + - ``lat_lon``, ``zonal_mean_xy``, ``zonal_mean_2d``, ``polar``, + ``cosp_histogram``, ``meridional_mean_2d``, + ``annual_cycle_zonal_mean``, ``zonal_mean_2d_stratosphere``, + ``aerosol_aeronet``, ``aerosol_budget`` + * - ``climo_month_lnd`` + - ``lat_lon_land`` + * - ``diurnal`` + - ``diurnal_cycle`` + * - ``ts_month_atm`` + - ``enso_diags``, ``qbo``, ``area_mean_time_series``, ``mp_partition`` + * - ``ts_daily_atm`` + - ``tropical_subseasonal``, ``precip_pdf`` + * - ``streamflow`` + - ``streamflow`` + * - ``tc`` + - ``tc_analysis`` + +Cell values: **Always** = required for both run types; +**MvO** = ``model_vs_obs`` only; **MvM** = ``model_vs_model`` only; +**—** = not used. Values marked with \* differ across sets within the group +(see notes below the table). + +.. list-table:: + :header-rows: 1 + :stub-columns: 1 + :widths: 30 14 14 10 16 14 12 10 + + * - Parameter + - ``climo_month_atm`` + - ``climo_month_lnd`` + - ``diurnal`` + - ``ts_month_atm`` + - ``ts_daily_atm`` + - ``streamflow`` + - ``tc`` + * - ``climo_diurnal_frequency`` + - — + - — + - Always + - — + - — + - — + - — + * - ``climo_diurnal_subsection`` + - — + - — + - Always + - — + - — + - — + - — + * - ``climo_subsection`` [#subsection]_ + - Always + - Always + - — + - — + - — + - — + - — + * - ``dc_obs_climo`` [#dc_obs]_ + - — + - — + - MvO + - — + - — + - — + - — + * - ``diff_title`` + - — + - — + - — + - MvM + - MvM + - MvM + - MvM + * - ``gauges_path`` [#inferred]_ + - — + - — + - — + - — + - — + - MvM + - — + * - ``obs_ts`` [#inferred]_ + - — + - — + - — + - MvO + - MvO + - — + - — + * - ``ref_final_yr`` \* + - — + - — + - — + - Always \* + - MvM + - MvM + - MvM + * - ``ref_name`` + - — + - — + - — + - MvM + - MvM + - MvM + - MvM + * - ``ref_start_yr`` \* + - — + - — + - — + - Always \* + - MvM + - MvM + - MvM + * - ``reference_data_path`` [#refpath]_ + - Always + - Always + - Always + - — + - — + - — + - — + * - ``reference_data_path_climo_diurnal`` [#inferred]_ + - — + - — + - MvM + - — + - — + - — + - — + * - ``reference_data_path_tc`` [#inferred]_ + - — + - — + - — + - — + - — + - — + - MvM + * - ``reference_data_path_ts`` [#inferred]_ + - — + - — + - — + - MvM + - — + - — + - — + * - ``reference_data_path_ts_daily`` [#inferred]_ + - — + - — + - — + - — + - MvM + - — + - — + * - ``reference_data_path_ts_rof`` [#inferred]_ + - — + - — + - — + - — + - — + - MvM + - — + * - ``short_ref_name`` + - — + - — + - — + - MvM + - MvM + - MvM + - MvM + * - ``streamflow_obs_ts`` [#inferred]_ + - — + - — + - — + - — + - — + - MvO + - — + * - ``tc_obs`` [#inferred]_ + - — + - — + - — + - — + - — + - — + - MvO + * - ``ts_daily_subsection`` [#subsection]_ + - — + - — + - — + - — + - Always + - — + - — + * - ``ts_num_years_ref`` + - — + - — + - — + - MvM + - MvM + - MvM + - — + * - ``ts_subsection`` [#subsection]_ + - — + - — + - — + - Always + - — + - — + - — + +.. [#subsection] Falls back to ``sub`` → ``grid`` if not set. +.. [#refpath] Inferred from ``diagnostics_base_path`` if not set. In MvM, + also used as a base path from which ``reference_data_path_climo_diurnal`` + and ``reference_data_path_ts`` are derived. +.. [#dc_obs] Inferred from ``reference_data_path`` if not set (MvO only). +.. [#inferred] Inferred from ``diagnostics_base_path`` (or a sibling path) + if not set. + +Note on \* cells (``ts_month_atm``): within ``ts_month_atm`` the four sets do not behave uniformly for ``ref_start_yr`` and ``ref_final_yr``: + +.. list-table:: + :header-rows: 1 + :widths: 30 20 20 20 20 + + * - Parameter + - ``enso_diags`` + - ``qbo`` + - ``area_mean_time_series`` + - ``mp_partition`` + * - ``ref_final_yr`` + - MvM + - Always + - — + - MvM + * - ``ref_start_yr`` + - Always + - Always + - — + - MvM + + +**Parameter list** + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``climo_diurnal_frequency`` + - No + - ``""`` + - Frequency name for the diurnal cycle ``[climo]`` subtask. + * - ``climo_diurnal_subsection`` + - No + - ``""`` + - ``[climo]`` subtask for diurnal cycle data. + * - ``climo_subsection`` + - No + - ``""`` + - Name of the ``[climo]`` subtask to depend on. + * - ``dc_obs_climo`` + - No + - ``""`` + - Path to observation climatology for the ``diurnal_cycle`` set. + * - ``diff_title`` + - No + - ``"Model - Observations"`` + - Title for difference plots. + * - ``gauges_path`` + - No + - ``""`` + - Path to stream gauge data for ``streamflow`` model-vs-model runs. + * - ``obs_ts`` + - No + - ``""`` + - Path to observation time-series data for ``enso_diags``, ``qbo``, and + ``area_mean_time_series``. + * - ``ref_final_yr`` + - No* + - ``""`` + - End year of reference data. Required for ``qbo`` and certain + model-vs-model sets. + * - ``ref_name`` + - No* + - ``""`` + - Reference dataset name. + **Required for** ``run_type="model_vs_model"``. + * - ``ref_start_yr`` + - No* + - ``""`` + - Start year of reference data. Required for ``enso_diags``, ``qbo``. + * - ``reference_data_path`` + - No* + - ``""`` + - Path to reference climatology data. + **Required for** ``run_type="model_vs_model"``. + * - ``reference_data_path_climo_diurnal`` + - No + - ``""`` + - Reference climatology path for ``diurnal_cycle`` MVM runs. + * - ``ref_years`` + - No + - ``[""]`` + - Year ranges for reference data in model-vs-model runs. + * - ``reference_data_path_tc`` + - No + - ``""`` + - Reference data path for ``tc_analysis`` MVM runs. + * - ``reference_data_path_ts`` + - No + - ``""`` + - Reference time-series path for ``enso_diags``, ``qbo``, ``area_mean_time_series``. + * - ``reference_data_path_ts_daily`` + - No + - ``""`` + - Reference daily time-series path for ``tropical_subseasonal``. + * - ``reference_data_path_ts_rof`` + - No + - ``""`` + - Reference river time-series path for ``streamflow`` MVM runs. + * - ``short_ref_name`` + - No* + - ``""`` + - Short reference name. + **Required for** ``run_type="model_vs_model"``. + * - ``streamflow_obs_ts`` + - No + - ``""`` + - Path to observation data for ``streamflow`` set. + * - ``tc_obs`` + - No + - ``""`` + - Path to observation data for ``tc_analysis`` set. + * - ``ts_daily_subsection`` + - No + - ``""`` + - ``[ts]`` subtask for daily data (required for + ``tropical_subseasonal``). + * - ``ts_num_years_ref`` + - No + - ``5`` + - Year increment for reference time-series data. + * - ``ts_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` subtask to depend on. + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For top-level parameters, see :ref:`top-level parameters `. + +Dependencies +------------ + +**Upstream (what e3sm_diags depends on):** + +- :doc:`climo` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly diurnal-atm climo: diurnal_cycle +- :doc:`ts` — Monthly-atm ts: enso_diags, qbo, area_mean_time_series, mp_partition. Monthly-rof ts: streamflow. Daily-atm ts: tropical_subseasonal, precip_pdf. +- :doc:`tc_analysis` — Required for tc_analysis set. + +**Downstream (what depends on e3sm_diags):** + +- None. diff --git a/docs/source/user_guide/tasks/e3sm_to_cmip.rst b/docs/source/user_guide/tasks/e3sm_to_cmip.rst new file mode 100644 index 00000000..06b96d31 --- /dev/null +++ b/docs/source/user_guide/tasks/e3sm_to_cmip.rst @@ -0,0 +1,67 @@ +.. _task-e3sm-to-cmip: + +e3sm_to_cmip — CMIP6 Conversion +================================= + +The ``e3sm_to_cmip`` task converts E3SM output to CMIP6-compliant format +using the ``e3sm_to_cmip`` package. It depends on the :doc:`ts` task to +provide input time-series files. Its output is consumed by :doc:`ilamb` +and :doc:`pcmdi_diags`. + +Parameters +---------- + +These 6 parameters are specific to the ``e3sm_to_cmip`` task. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``cmip_metadata`` + - No + - ``"inclusions/e3sm_to_cmip/default_metadata.json"`` + - Path to the CMIP metadata JSON file. + * - ``cmip_plevdata`` + - No + - ``""`` + - Path to pressure-level data NetCDF file. Defaults to a file under + ``diagnostics_base_path`` when empty. + * - ``cmip_vars`` + - No + - ``""`` + - Variables to convert to CMIP format. Leave empty to convert all + supported variables. + * - ``input_component`` + - No + - ``""`` + - Model component (e.g., ``eam``, ``eamxx``, ``elm``). + * - ``interp_vars`` + - No + - ``"U,V,T,Q,RELHUM,OMEGA,Z3"`` + - Variables to interpolate to pressure levels during conversion. + * - ``vrt_in_file`` + - No + - ``""`` + - Source vertical coordinate file for ``ncremap``. Required for + EAMxx (L128 grid). + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For top-level parameters, see :ref:`top-level parameters `. + +Dependencies +------------ + +**Upstream (what e3sm_to_cmip depends on):** + +- :doc:`ts` — Monthly-atm ts: monthly-atm e3sm_to_cmip. Monthly-lnd ts: monthly-lnd e3sm_to_cmip. + +**Downstream (what depends on e3sm_to_cmip):** + +- :doc:`ilamb` — Monthly-lnd e3sm_to_cmip: required. Monthly-atm e3sm_to_cmip: optional. +- :doc:`pcmdi_diags` — Monthly-atm e3sm_to_cmip: mean_climate, variability_modes_atm,variability_modes_cpl, enso diff --git a/docs/source/user_guide/tasks/global_time_series.rst b/docs/source/user_guide/tasks/global_time_series.rst new file mode 100644 index 00000000..6652dd9b --- /dev/null +++ b/docs/source/user_guide/tasks/global_time_series.rst @@ -0,0 +1,181 @@ +.. _task-global-time-series: + +global_time_series — Global Time Series Plots +============================================= + +The ``global_time_series`` task generates global time series plots of +key climate metrics. It can optionally produce a Viewer page. It depends on +the :doc:`ts` task (for atmospheric and land global-mean time series) +and on :doc:`mpas_analysis` when ocean plots are requested. + +Parameters +---------- + +These 14 (non-deprecated) parameters are specific to the ``global_time_series`` task. + +How to plot +~~~~~~~~~~~ + +There are 4 parameters that define how to plot: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``color`` + - No + - ``"Blue"`` + - Color used for the plots. + * - ``experiment_name`` + - No + - ``""`` + - Label displayed on the plots. + * - ``figstr`` + - No + - ``""`` + - Prefix for output figure file names. + * - ``make_viewer`` + - No + - ``False`` + - Set to ``True`` to construct a Viewer page with extended plots. If ``False``, a classic PDF will be made and/or individual PDFs will be made for additional variables specified in the ``plot_`` parameters. + + +What to plot +~~~~~~~~~~~~ + +There are 6 parameters that define what to plot: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``plots_original`` + - No + - ``"net_toa_flux_restom,..."`` + - Names of the standard ("classic") plots to generate. + Available plots and their variable requirements: + + - ``net_toa_flux_restom`` — requires ``RESTOM`` + - ``net_atm_energy_imbalance`` — requires ``RESTOM``, ``RESSURF`` + - ``global_surface_air_temperature`` — requires ``TREFHT`` + - ``toa_radiation`` — requires ``FSNTOA``, ``FLUT`` + - ``net_atm_water_imbalance`` — requires ``PRECC``, ``PRECL``, ``QFLX`` + - ``change_ohc`` — requires ocean data + - ``max_moc`` — requires ocean MOC data + - ``change_sea_level`` — requires ocean data + + Remove the three ocean plots if you don't have ocean data. + * - ``plots_atm`` + - No + - ``""`` + - Extra atmosphere plots. These should be + a subset of the variables from ``ts`` global subtasks. + * - ``plots_ice`` + - No + - ``""`` + - Extra sea-ice plots. + * - ``plots_lnd`` + - No + - ``""`` + - Extra land plots. Set to ``"all"`` to + include every variable in the built-in land CSV file. + * - ``plots_ocn`` + - No + - ``""`` + - Extra ocean plots. + * - ``regions`` + - No + - ``"glb,n,s"`` + - Regions to plot: ``glb`` (global), ``n`` (northern hemisphere), + ``s`` (southern hemisphere). + +Ocean plot requirements +~~~~~~~~~~~~~~~~~~~~~~~ + +There are 4 parameters that are used if ocn plots are included (``plots_ocn`` or any of the 3 ocn plots of ``plots_original``): + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``climo_years`` + - No + - ``[""]`` + - Year ranges for climatology data (if needed). + * - ``ts_years`` + - No + - ``[""]`` + - Year ranges for the time-series sub-runs to depend on. + * - ``moc_file`` + - No + - ``""`` + - Path to the MOC file (for the ``max_moc`` plot). + * - ``mpas_analysis_subsections`` + - No + - ``[""]`` + - Names of ``[mpas_analysis]`` subtasks to depend on. Leave empty if + no subsections are defined. + +Deprecated Parameters +~~~~~~~~~~~~~~~~~~~~~ + +There are 2 deprecated parameters. Specifying either will have no effect. + +.. code-block:: text + + nrows + ncols + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This parameter has a ``global_time_series``-specific default, which means even +if it is set at the top level (``[default]``) section, this default value will +be used instead. Therefore, to specify a custom value, this parameter must be +defined inside ``[global_time_series]``: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``input_subdir`` + - No + - ``"archive/ocn/hist"`` + - Subdirectory with ocean data. Overrides the ``[default]`` value + (``archive/atm/hist``). + +For other top-level parameters, see :ref:`top-level parameters `. + +Deprecated parameters +~~~~~~~~~~~~~~~~~~~~~ + +The following parameters are still in ``default.ini`` but are deprecated and +have no effect: ``ncols``, ``nrows``. + +Dependencies +------------ + +**Upstream (what global_time_series depends on):** + +- :doc:`ts` — Monthly-atm-glb ts: the 5 classic atm plots, plots for specific atm variables. Monthly-lnd-glb ts: plots for specific lnd variables. +- :doc:`mpas_analysis` — the 3 classic ocn plots, plots for specific ocn variables. + +**Downstream (what depends on global_time_series):** + +- None. \ No newline at end of file diff --git a/docs/source/user_guide/tasks/ilamb.rst b/docs/source/user_guide/tasks/ilamb.rst new file mode 100644 index 00000000..3751285d --- /dev/null +++ b/docs/source/user_guide/tasks/ilamb.rst @@ -0,0 +1,76 @@ +.. _task-ilamb: + +ilamb — Land Benchmarking +========================== + +The ``ilamb`` task runs `ILAMB `_ (International +Land Model Benchmarking) to evaluate E3SM's land component against +observational benchmarks. It depends on the :doc:`e3sm_to_cmip` task to +provide CMIP-format land and atmosphere time-series files. + +Parameters +---------- + +These 5 parameters are specific to the ``ilamb`` task. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``cfg`` + - No + - ``"inclusions/ilamb/ilamb.cfg"`` + - Path to the ILAMB configuration file. + * - ``e3sm_to_cmip_atm_subsection`` + - No + - ``""`` + - Name of the ``[e3sm_to_cmip]`` atmosphere subtask to depend on. + Inferred as ``atm_monthly_180x360_aave`` if not specified. + * - ``e3sm_to_cmip_land_subsection`` + - No + - ``""`` + - Name of the ``[e3sm_to_cmip]`` land subtask to depend on. Inferred + as ``land_monthly`` if not specified. + * - ``ilamb_obs`` + - No + - ``""`` + - Path to ILAMB observational data. Defaults to + ``diagnostics_base_path/ilamb_data`` if empty. + * - ``land_only`` + - No + - ``False`` + - Set to ``True`` for land-only runs (skips atmosphere dependencies). + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For top-level parameters, see :ref:`top-level parameters `. + +Dependencies +------------ + +**Upstream (what ilamb depends on):** + +- :doc:`ts` -- Monthly-lnd ts: required. Monthly-atm ts: optional. +- :doc:`e3sm_to_cmip` — Monthly-lnd e3sm_to_cmip: required. Monthly-atm e3sm_to_cmip: optional. + +**Downstream (what depends on ilamb):** + +- None. + +Dependency inference +~~~~~~~~~~~~~~~~~~~~ + +When ``infer_section_parameters = True`` (the default), ``zppy`` will +automatically determine the following if not explicitly set: + +- ``ts_land_subsection`` defaults to ``land_monthly`` +- ``e3sm_to_cmip_land_subsection`` defaults to ``land_monthly`` +- ``ts_atm_subsection`` defaults to ``atm_monthly_180x360_aave`` (for + non-land-only runs) +- ``e3sm_to_cmip_atm_subsection`` defaults to ``atm_monthly_180x360_aave`` + (for non-land-only runs) diff --git a/docs/source/user_guide/tasks/index.rst b/docs/source/user_guide/tasks/index.rst new file mode 100644 index 00000000..9dc8dd63 --- /dev/null +++ b/docs/source/user_guide/tasks/index.rst @@ -0,0 +1,62 @@ +.. _user-tasks: + +***** +Tasks +***** + +``zppy`` is organized around **tasks**. Each task corresponds to a specific +post-processing operation. Tasks are defined as sections (``[task_name]``) in +the configuration file and may contain subsections (``[[subsection_name]]``) +for multiple independent runs of the same task. + +**Available tasks:** + +Listed for reference (bundle jobs are submitted after task jobs are generated in ``__main__.py``): + +.. list-table:: + :header-rows: 1 + :widths: 25 75 + + * - Task + - Description + * - :doc:`bundle` + - Bundle multiple tasks into a single SLURM job + * - :doc:`climo` + - Generate climatology files using NCO's ``ncclimo`` + * - :doc:`ts` + - Generate time-series files using NCO's ``ncclimo`` + * - :doc:`e3sm_to_cmip` + - Convert E3SM output to CMIP6 format using ``e3sm_to_cmip`` + * - :doc:`tc_analysis` + - Tropical cyclone analysis + * - :doc:`e3sm_diags` + - Run E3SM Diagnostics (atmosphere, land, river diagnostics, etc.) + * - :doc:`mpas_analysis` + - Run MPAS-Analysis (ocean and sea-ice diagnostics) + * - :doc:`global_time_series` + - Generate global time series plots + * - :doc:`ilamb` + - Run ILAMB land benchmarking + * - :doc:`livvkit` + - Run LIVVkit ice sheet validation + * - :doc:`pcmdi_diags` + - Run PCMDI metrics diagnostics + + +**Common (default) parameters** shared by all tasks are documented on the +:doc:`../parameters` page. + +.. toctree:: + :hidden: + + bundle + climo + ts + e3sm_to_cmip + tc_analysis + e3sm_diags + mpas_analysis + global_time_series + ilamb + livvkit + pcmdi_diags diff --git a/docs/source/user_guide/tasks/livvkit.rst b/docs/source/user_guide/tasks/livvkit.rst new file mode 100644 index 00000000..6ba8a0fe --- /dev/null +++ b/docs/source/user_guide/tasks/livvkit.rst @@ -0,0 +1,74 @@ +.. _task-livvkit: + +livvkit — Ice Sheet Validation +================================ + +The ``livvkit`` task runs `LIVVkit +`_ (Land Ice Verification and Validation +Toolkit) for ice sheet model evaluation. It compares climatology files +generated by the :doc:`climo` task against reanalysis datasets. + +Parameters +---------- + +These 4 parameters are specific to the ``livvkit`` task. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``climo_subsections`` + - No + - ``[""]`` + - List of ``[climo]`` subtasks to depend on. + * - ``cfg`` + - No + - ``"inclusions/livvkit/livvkit_r05.jinja"`` + - Path to the LIVVkit configuration template. + * - ``icesheets`` + - No + - ``"gis,ais"`` + - Icesheets to analyze: ``gis`` (Greenland), ``ais`` (Antarctica). + * - ``sets`` + - No + - ``"cmb,smb,energy_racmo,energy_era5,energy_merra2,energy_ceres"`` + - Validation sets to perform. + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For top-level parameters, see :ref:`top-level parameters `. + +Dependencies +------------ + +**Upstream (what livvkit depends on):** + +- :doc:`climo` — Monthly-lnd climo +- :doc:`ts` -- Monthly-lnd ts. + +**Downstream (what depends on livvkit):** + +- None. + +Climo dependency requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``livvkit`` task requires specific ``[climo]`` subtasks to be defined: + +- A grid-native climatology must be generated, assumed to be named + ``land_monthly_climo_native``. +- For each reanalysis comparison, a corresponding + ``land_monthly_climo_GRID`` subtask is needed, where ``GRID`` is the + reanalysis grid name. + +Available reanalysis comparisons and their grid names: + +- ``racmo_gis``, ``racmo_ais`` +- ``merra2`` +- ``era5`` +- ``ceres`` (uses the default CMIP6 grid) diff --git a/docs/source/user_guide/tasks/mpas_analysis.rst b/docs/source/user_guide/tasks/mpas_analysis.rst new file mode 100644 index 00000000..1dd1bdc5 --- /dev/null +++ b/docs/source/user_guide/tasks/mpas_analysis.rst @@ -0,0 +1,233 @@ +.. _task-mpas-analysis: + +mpas_analysis — Ocean and Sea-Ice Analysis +========================================== + +The ``mpas_analysis`` task runs `MPAS-Analysis +`_ to produce ocean and sea-ice +diagnostics. It supports both model-vs-observations (``mvo``) and +model-vs-model (``mvm``) comparison modes. The comparison type is inferred +automatically: if ``reference_data_path`` is set, the run is treated as MVM; +otherwise it is treated as MVO. + +.. note:: + ``environment_commands`` should be the same for all related + ``[mpas_analysis]`` runs. Mixing environments across year ranges can cause + MPAS-Analysis to fail. + +Parameters +---------- + +These 25 parameters are specific to the ``mpas_analysis`` task. + +MPAS-Analysis configuration parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**General parameters** + +There are 8 general parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``cache`` + - No + - ``True`` + - Whether to cache MPAS-Analysis intermediate results. + * - ``reference_comparison_type`` + - No + - ``"auto"`` + - Comparison type for the referenced prior run: ``"auto"``, ``"mvo"``, + or ``"mvm"``. + * - ``test_comparison_type`` + - No + - ``"auto"`` + - Comparison type for the test run's referenced prior run. + * - ``reference_case`` + - No* + - ``""`` + - Case name of the reference run. \*Required when + ``reference_data_path`` points to a non-subsection path. + * - ``generate`` + - No + - ``['all', 'no_landIceCavities', 'no_BGC', 'no_icebergs', 'no_min', 'no_max', 'no_sose', 'no_waves', 'no_eke', 'no_climatologyMapAntarcticMelt', 'no_regionalTSDiagrams', 'no_timeSeriesAntarcticMelt', 'no_timeSeriesOceanRegions', 'no_climatologyMapSose', 'no_woceTransects', 'no_soseTransects', 'no_geojsonTransects', 'no_oceanRegionalProfiles', 'no_hovmollerOceanRegions', 'no_oceanConservation']`` + - List of MPAS-Analysis analyses to generate. + * - ``PostMOC`` + - No + - ``False`` + - Whether to post-process MOC data. + * - ``purge`` + - No + - ``False`` + - Whether to purge previous MPAS-Analysis output before running. + * - ``shortTermArchive`` + - No + - ``True`` + - Whether the input uses the short-term archive directory structure. + +**Year parameters** + +There are 7 year parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``anomalyRefYear`` + - No + - ``1`` + - Reference year for anomaly calculations. + * - ``ts_years`` + - No + - ``[""]`` + - Year ranges for time-series sub-runs. + * - ``climo_years`` + - No + - ``[""]`` + - Year ranges for climatology sub-runs. + * - ``enso_years`` + - No + - ``[""]`` + - Year ranges for ENSO sub-runs. + * - ``ref_ts_years`` + - No + - ``[""]`` + - Year ranges for reference time-series sub-runs. Defaults to + ``ts_years`` if empty. + * - ``ref_climo_years`` + - No + - ``[""]`` + - Year ranges for reference climatology sub-runs. + * - ``ref_enso_years`` + - No + - ``[""]`` + - Year ranges for reference ENSO sub-runs. + +**Data path parameters** + +There are 6 data path parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``reference_data_path`` + - No + - ``""`` + - For MVM runs: path to a prior zppy output directory (the one + containing ``post/``) to use as the reference run. Setting this + triggers MVM mode. + * - ``test_data_path`` + - No + - ``""`` + - For MVM runs: path to the test run's prior zppy output directory (if + different from the current run). + * - ``mpaso_nml`` + - No + - ``"mpaso_in"`` + - Name of the MPAS-Ocean namelist file. + * - ``mpassi_nml`` + - No + - ``"mpassi_in"`` + - Name of the MPAS sea-ice namelist file. + * - ``stream_ice`` + - No + - ``"streams.seaice"`` + - Name of the MPAS sea-ice streams file. + * - ``stream_ocn`` + - No + - ``"streams.ocean"`` + - Name of the MPAS ocean streams file. + +**Computational parameters** + +There are 4 computational parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``mapMpiTasks`` + - No + - ``6`` + - Number of MPI tasks for mapping. + * - ``ncclimoThreads`` + - No + - ``12`` + - Number of threads for ``ncclimo`` in MPAS-Analysis. + * - ``ncclimoParallelMode`` + - No + - ``"bck"`` + - Parallel mode for ``ncclimo`` in MPAS-Analysis. + * - ``parallelTaskCount`` + - No + - ``12`` + - Number of parallel analysis tasks in MPAS-Analysis. + + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This parameter has an ``mpas_analysis``-specific default, which means even if +it is set at the top level (``[default]``) section, this default value will be +used instead. Therefore, to specify a custom value, this parameter must be +defined inside ``[mpas_analysis]``: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``walltime`` + - No + - ``"06:00:00"`` + - Maximum wall time. Overrides the ``[default]`` value (``02:00:00``). + +For other top-level parameters, see :ref:`top-level parameters `. + +Year range behavior for MVM runs +--------------------------------- + +For model-vs-model comparisons, ``zppy`` supports separate year ranges for +the test and reference runs: + +- ``ts_years``, ``climo_years``, ``enso_years``: define the **test** run + year ranges. +- ``ref_ts_years``, ``ref_climo_years``, ``ref_enso_years``: optionally + override the **reference** run year ranges (defaults to test year ranges + if not set). +- If ``reference_data_path`` points to a prior ``[[subsection]]`` in the + same cfg, the reference year ranges default to that subsection's values. +- If a single reference year range is provided and multiple test ranges are + requested, the single reference range is used for each test range. + +Dependencies +------------ + +**Upstream (what mpas_analysis depends on):** + +- None. Note that later year sets will depend on previously completed ones. + +**Downstream (what depends on mpas_analysis):** + +- :doc:`global_time_series` — the 3 classic ocn plots, plots for specific ocn variables. diff --git a/docs/source/user_guide/tasks/pcmdi_diags.rst b/docs/source/user_guide/tasks/pcmdi_diags.rst new file mode 100644 index 00000000..d7d9b87a --- /dev/null +++ b/docs/source/user_guide/tasks/pcmdi_diags.rst @@ -0,0 +1,496 @@ +.. _task-pcmdi-diags: + +pcmdi_diags — PCMDI Metrics +============================ + +The ``pcmdi_diags`` task runs `PCMDI Metrics Package (PMP) +`_ diagnostics to compute climate +model performance metrics. It depends on the :doc:`e3sm_to_cmip` task for +CMIP-format input data. + +Diagnostic sets +--------------- + +The ``current_set`` parameter selects which PMP diagnostic to run: + +- ``mean_climate`` — climatological mean metrics +- ``variability_modes_cpl`` — coupled variability modes (PDO, NPGO, AMO) +- ``variability_modes_atm`` — atmospheric variability modes (NAM, NAO, PNA, etc.) +- ``enso`` — ENSO metrics (not currently enabled by default) +- ``synthetic_plots`` — composite viewer page combining outputs from other sets + +Parameters +---------- + +These 71 parameters are specific to the ``pcmdi_diags`` task. + +Task-level parameters (all sets) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 6 task-level (all sets) parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``figure_format`` + - No + - ``"png"`` + - Output figure format. + * - ``run_type`` + - No + - ``"model_vs_obs"`` + - ``"model_vs_obs"`` or ``"model_vs_model"``. + * - ``model_name`` + - No + - ``"e3sm.historical.v3-LR.0051"`` + - Model name (required for model-vs-observations runs). + * - ``model_tableID`` + - No + - ``"Amon"`` + - CMIP table ID (required for model-vs-observations runs). + * - ``model_name_ref`` + - No + - ``"ERA5"`` + - Reference model name (required for model-vs-model runs). + * - ``model_tableID_ref`` + - No + - ``"Amon"`` + - Reference CMIP table ID (required for model-vs-model runs). + +Task-level parameters (all sets except synthetic_plots) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 14 task-level (all sets except synthetic_plots) parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``e3sm_to_cmip_atm_subsection`` + - No + - ``""`` + - Name of the ``[e3sm_to_cmip]`` atmosphere subtask to depend on. + * - ``generate_sftlf`` + - No + - ``True`` + - Process the land/sea mask within PCMDI. + * - ``mov_plot_obs`` + - No + - ``True`` + - Generate variability-modes observation plots. + * - ``mov_plot_model`` + - No + - ``True`` + - Generate variability-modes model plots. + * - ``mov_nc_out_obs`` + - No + - ``True`` + - Write variability-modes observation NetCDF output. + * - ``mov_nc_out_model`` + - No + - ``True`` + - Write variability-modes model NetCDF output. + * - ``multiprocessing`` + - No + - ``True`` + - Use multiprocessing. + * - ``num_workers`` + - No + - ``24`` + - Number of worker processes. + * - ``obs_ts`` + - No + - ``""`` + - Path to observation time-series data. + * - ``pcmdi_debug`` + - No + - ``False`` + - Enable debug mode in PCMDI diagnostics. + * - ``save_test_clims`` + - No + - ``True`` + - Save derived test climatology data. + * - ``reference_data_path`` + - No + - ``""`` + - Path to reference model data for model-vs-model runs. + * - ``reference_data_path_ts`` + - No + - ``""`` + - Path to reference model time-series data for model-vs-model runs. + * - ``ts_years`` + - No + - ``[""]`` + - Year ranges for test model data. + +Per-subtask shared parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 5 per-subtask shared parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``current_set`` + - No + - ``""`` + - Diagnostic set to run. + * - ``obs_sets`` + - No + - ``"default"`` + - Observational dataset aliases to use. + * - ``ref_final_yr`` + - No + - ``""`` + - End year for reference data. + * - ``ref_start_yr`` + - No + - ``""`` + - Start year for reference data. + * - ``ref_years`` + - No + - ``[""]`` + - Year ranges for reference data. + +Mean_climate parameters +~~~~~~~~~~~~~~~~~~~~~~~ + +There are 7 mean_climate parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``cmip_vars`` + - No + - ``"pr,prw,psl,..."`` + - CMIP variables to include for mean-climate diagnostics. + * - ``target_grid`` + - No + - ``"1x1"`` + - Target regridding grid. + * - ``target_grid_string`` + - No + - ``"1px1p"`` + - Descriptor string for the target grid. + * - ``regrid_tool`` + - No + - ``"esmf"`` + - Regridding tool for atmosphere fields. + * - ``regrid_tool_ocn`` + - No + - ``"esmf"`` + - Regridding tool for ocean fields. + * - ``regrid_method`` + - No + - ``"regrid2"`` + - Regridding method for atmosphere fields. + * - ``regrid_method_ocn`` + - No + - ``"conservative"`` + - Regridding method for ocean fields. + +Mean_climate & synthetic_plots parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 2 mean_climate & synthetic_plots parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``clim_vars`` + - No + - ``"pr,prw,psl,..."`` + - Variables for mean-climate and synthetic-plot diagnostics. + * - ``clim_regions`` + - No + - ``"global,ocean,land"`` + - Regions used for mean-climate metrics. + +Variability_modes parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 9 variability_modes parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``CBF`` + - No + - ``True`` + - Use Common Base Function approach. + * - ``ConvEOF`` + - No + - ``True`` + - Compute conventional EOF. + * - ``eofn_mod_max`` + - No + - ``3`` + - Maximum number of EOF modes for model data. + * - ``EofScaling`` + - No + - ``False`` + - Apply EOF scaling. + * - ``landmask`` + - No + - ``False`` + - Apply land-mask handling in variability diagnostics. + * - ``ModUnitsAdjust`` + - No + - ``""`` + - Unit-adjustment keywords for model data. + * - ``ObsUnitsAdjust`` + - No + - ``""`` + - Unit-adjustment keywords for observation data. + * - ``RmDomainMean`` + - No + - ``True`` + - Remove domain mean before variability calculations. + * - ``seasons`` + - No + - ``"monthly"`` + - Seasons/frequency bins to analyze. + +Variability_modes & synthetic_plots parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 4 variability_modes & synthetic_plots parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``mova_vars`` + - No + - ``"psl"`` + - Variables for ``variability_modes_atm``. + * - ``mova_modes`` + - No + - ``"NAM,NAO,PNA,NPO,SAM,PSA1,PSA2"`` + - Modes for ``variability_modes_atm``. + * - ``movc_vars`` + - No + - ``"ts"`` + - Variables for ``variability_modes_cpl``. + * - ``movc_modes`` + - No + - ``"PDO,NPGO,AMO"`` + - Modes for ``variability_modes_cpl``. + +ENSO parameters +~~~~~~~~~~~~~~~ + +There is 1 ENSO parameter: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``enso_groups`` + - No + - ``"ENSO_perf,ENSO_proc,ENSO_tel"`` + - ENSO metric groups to run. + +ENSO & synthetic_plots parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There is 1 ENSO & synthetic_plots parameter: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``enso_vars`` + - No + - ``"psl,pr,prsn,ts,tas,tauu,tauv,hflx,hfss,rlds,rsds,rlus,rlut,rsdt"`` + - Variables used by ENSO diagnostics. + +Synthetic_plots parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 22 synthetic_plots parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``clim_viewer`` + - No + - ``True`` + - Include mean-climate panels in the synthetic viewer. + * - ``clim_years`` + - No + - ``""`` + - Year range used for mean-climate synthetic panels. + * - ``mova_viewer`` + - No + - ``True`` + - Include atmospheric variability-mode panels in the synthetic viewer. + * - ``mova_years`` + - No + - ``""`` + - Year range used for atmospheric variability synthetic panels. + * - ``movc_viewer`` + - No + - ``True`` + - Include coupled variability-mode panels in the synthetic viewer. + * - ``movc_years`` + - No + - ``""`` + - Year range used for coupled variability synthetic panels. + * - ``enso_viewer`` + - No + - ``False`` + - Include ENSO panels in the synthetic viewer. + * - ``enso_years`` + - No + - ``""`` + - Year range used for ENSO synthetic panels. + * - ``save_all_data`` + - No + - ``True`` + - Save all intermediate synthetic-diagnostic data. + * - ``reference_alias`` + - No + - ``"inclusions/pcmdi_diags/reference_alias.json"`` + - Observation alias file. + * - ``regions_specs`` + - No + - ``"inclusions/pcmdi_diags/regions_specs.json"`` + - Region-specification file. + * - ``pcmdi_version`` + - No + - ``"v3.8.2"`` + - Version tag for the zppy-pcmdi workflow. + * - ``pcmdi_viewer_template`` + - No + - ``"pcmdi_data/viewer"`` + - Template directory for synthetic-viewer generation. + * - ``pcmdi_webtitle`` + - No + - ``"E3SM-PMP-Diagnostics"`` + - Title used in generated PCMDI web output. + * - ``synthetic_metrics_list`` + - No + - ``"inclusions/pcmdi_diags/synthetic_metrics_list.json"`` + - Metrics-list file used to assemble synthetic plots. + * - ``synthetic_sets`` + - No + - ``"portrait,parcoord"`` + - Synthetic plot types to generate. + * - ``cmip_clim_dir`` + - No + - ``""`` + - Directory containing CMIP mean-climate metrics. + * - ``cmip_enso_dir`` + - No + - ``""`` + - Directory containing CMIP ENSO metrics. + * - ``cmip_movs_dir`` + - No + - ``""`` + - Directory containing CMIP variability-mode metrics. + * - ``cmip_clim_set`` + - No + - ``"cmip6.historical.v20250707"`` + - CMIP mean-climate metrics set ID. + * - ``cmip_enso_set`` + - No + - ``"cmip6.historical.v20210620"`` + - CMIP ENSO metrics set ID. + * - ``cmip_movs_set`` + - No + - ``"cmip6.historical.v20220825"`` + - CMIP variability-mode metrics set ID. + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These 3 parameters have ``pcmdi_diags``-specific defaults, which means even if +they are set at the top level (``[default]``) section, these default values +will be used instead. Therefore, to specify custom values, these parameters +must be defined inside ``[pcmdi_diags]``: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``grid`` + - No + - ``"180x360_aave"`` + - Model data grid after remapping. Overrides the ``[default]`` value + (``""``). + * - ``ts_num_years`` + - No + - ``5`` + - Year increment for test model data. Overrides the + ``[default]`` value (``5``), which in this case is actually the same value. + * - ``frequency`` + - No + - ``"mo"`` + - Data frequency for variability-mode calculations. Overrides the + ``[default]`` value (``"monthly"``). + +For other top-level parameters, see :ref:`top-level parameters `. + +Dependencies +------------ + +**Upstream (what pcmdi_diags depends on):** + +- :doc:`ts` — Monthly-atm ts: mean_climate, variability_modes_atm,variability_modes_cpl, enso +- :doc:`e3sm_to_cmip` — Monthly-atm e3sm_to_cmip: mean_climate, variability_modes_atm,variability_modes_cpl, enso +- Note that the synthetic_plots set depends on the other sets. + +**Downstream (what depends on pcmdi_diags):** + +- None. diff --git a/docs/source/user_guide/tasks/tc_analysis.rst b/docs/source/user_guide/tasks/tc_analysis.rst new file mode 100644 index 00000000..9883c3e5 --- /dev/null +++ b/docs/source/user_guide/tasks/tc_analysis.rst @@ -0,0 +1,106 @@ +.. _task-tc-analysis: + +tc_analysis — Tropical Cyclone Analysis +======================================== + +The ``tc_analysis`` task performs tropical cyclone (TC) and African Easterly +Wave (AEW) analysis on E3SM output. It uses high-frequency atmospheric data +(typically ``eam.h2``) to detect and track TCs and AEWs using TempestExtremes. + +Both EAM and EAMxx output are supported. EAMxx requires the ``input_grid`` +parameter to be set because its output sets ``topography_file="NONE"``, +preventing automatic grid inference. + +Configuration example +--------------------- + +EAM: + +.. code-block:: cfg + + [tc_analysis] + active = True + input_grid = ne30pg2 + years = 1:100:10 + +EAMxx: + +.. code-block:: cfg + + [tc_analysis] + active = True + input_grid = ne30pg2 + tc_vars = SeaLevelPressure,T_mid_at_200hPa,T_mid_at_500hPa,U_at_model_bot,V_at_model_bot,U_at_850hPa,V_at_850hPa + years = 1:100:10 + +Parameters +---------- + +These 3 parameters are specific to the ``tc_analysis`` task: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``input_grid`` + - No + - ``""`` + - Native model grid of the input files, e.g. ``ne30pg2``, ``ne120pg2``, + ``ne30np4``. When set, the resolution and pg2 flag are derived from + this value automatically. Required for EAMxx. Accepted formats: + ``neXpg2`` or ``neXnp4`` (e.g. ``ne30pg2``, ``ne120np4``). + * - ``res`` + - No + - ``""`` + - Explicit ``--res`` value for TempestExtremes (EAM only). Used when + ``input_grid`` is empty. If both are empty, ``zppy`` attempts to infer + ``res`` from the ``topography_file`` global attribute of the input + files (EAM only). + * - ``tc_vars`` + - No + - ``"PSL,T200,T500,UBOT,VBOT,U850,V850"`` + - Ordered list of exactly 7 comma-separated variable names in the + sequence required by the TempestExtremes workflow: + ``SLP, T@200hPa, T@500hPa, U@model_bottom, V@model_bottom, U@850hPa, + V@850hPa``. The default covers EAM output. EAMxx users should set this + to the corresponding EAMxx variable names, e.g. + ``SeaLevelPressure,T_mid_at_200hPa,T_mid_at_500hPa,U_at_model_bot,V_at_model_bot,U_at_850hPa,V_at_850hPa``. + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This parameter has a ``tc_analysis``-specific default, which means even if it +is set at the top level (``[default]``) section, the default value will be +used instead. Therefore, to specify a custom value, this parameter must be +defined inside ``[tc_analysis]``: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``input_files`` + - No + - ``"eam.h2"`` + - Input history file pattern. Overrides the ``[default]`` value + (``eam.h0``). + +For other top-level parameters, see :ref:`top-level parameters `. + +Dependencies +------------ + +**Upstream (what tc_analysis depends on):** + +- None. + +**Downstream (what depends on tc_analysis):** + +- :doc:`e3sm_diags` — Required for tc_analysis set. diff --git a/docs/source/user_guide/tasks/ts.rst b/docs/source/user_guide/tasks/ts.rst new file mode 100644 index 00000000..ab44e478 --- /dev/null +++ b/docs/source/user_guide/tasks/ts.rst @@ -0,0 +1,115 @@ +.. _task-ts: + +ts — Time Series Files +====================== + +The ``ts`` task generates per-variable time-series files from E3SM +simulation output using NCO's ``ncclimo``. It produces one file per variable +covering a specified year range. Time-series files produced by ``ts`` are +required inputs for the :doc:`e3sm_diags`, :doc:`global_time_series`, +:doc:`e3sm_to_cmip`, :doc:`ilamb`, :doc:`livvkit`, and :doc:`pcmdi_diags` tasks. + +Configuration example +--------------------- + +.. code-block:: cfg + + [ts] + active = True + years = 1:100:10 + + [[atm_monthly_180x360_aave]] + input_files = eam.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.nc + grid = 180x360_aave + vars = FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TS,TREFHT + + [[land_monthly]] + input_files = elm.h0 + input_subdir = archive/lnd/hist + input_component = elm + mapping_file = /path/to/map_r05_to_cmip6_180x360_aave.nc + grid = 180x360_aave + +Parameters +---------- + +These 9 parameters are specific to the ``ts`` task. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``area_nm`` + - No + - ``"area"`` + - Name of the area variable in the input files. + * - ``dpf`` + - No + - ``30`` + - Days per file (for daily or sub-daily data). + * - ``extra_vars`` + - No + - ``""`` + - Additional variables to include alongside the primary ``vars`` list. + * - ``job_nbr`` + - No + - ``0`` + - Number of simultaneous ``ncclimo`` jobs. ``0`` uses ``ncclimo``'s + default. + * - ``tpd`` + - No + - ``1`` + - Time-steps per day (for sub-daily data, e.g., ``8`` for 3-hourly). + * - ``input_component`` + - No + - ``""`` + - Model component that generated the input files (e.g., ``eam``, + ``eamxx``, ``elm``, ``mosart``). Used to set processing type + internally. + * - ``vrt_remap_vars`` + - No + - ``""`` + - Variables to additionally regrid from model levels to pressure levels + after generating time-series files. Output is written to a sibling + ``ts_vrt_remap/`` directory. Empty string disables vertical remapping. + * - ``vrt_remap_file`` + - No + - ``""`` + - Path to the target vertical grid file (pressure levels) for + ``ncremap``. Defaults to a standard file under + ``diagnostics_base_path`` when empty. + * - ``vrt_in_file`` + - No + - ``""`` + - Path to the source vertical coordinate file for ``ncremap``. Required + for EAMxx (L128 grid). Defaults to a standard file under + ``diagnostics_base_path`` when ``input_component`` is ``eamxx`` and + this is left empty. + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For top-level parameters, see :ref:`top-level parameters `. + + +Dependencies +------------ + +**Upstream (what ts depends on):** + +- None. + +**Downstream (what depends on ts):** + +- :doc:`e3sm_to_cmip` — Monthly-atm ts: monthly-atm e3sm_to_cmip. Monthly-lnd ts: monthly-lnd e3sm_to_cmip. +- :doc:`e3sm_diags` — Monthly-atm ts: enso_diags, qbo, area_mean_time_series, mp_partition. Monthly-rof ts: streamflow. Daily-atm ts: tropical_subseasonal, precip_pdf. +- :doc:`global_time_series` — Monthly-atm-glb ts: the 5 classic atm plots, plots for specific atm variables. Monthly-lnd-glb ts: plots for specific lnd variables. +- :doc:`ilamb` -- Monthly-lnd ts: required. Monthly-atm ts: optional. +- :doc:`livvkit` -- Monthly-lnd ts. +- :doc:`pcmdi_diags` — Monthly-atm ts: mean_climate, variability_modes_atm,variability_modes_cpl, enso diff --git a/docs/source/tutorial.rst b/docs/source/user_guide/tutorial.rst similarity index 70% rename from docs/source/tutorial.rst rename to docs/source/user_guide/tutorial.rst index e4144f24..bf8a97c1 100644 --- a/docs/source/tutorial.rst +++ b/docs/source/user_guide/tutorial.rst @@ -18,7 +18,7 @@ Let's say we want to post-process 100 years of an existing simulation. Copy and paste the following into ``post.mysimulation.cfg`` -.. literalinclude:: post.mysimulation.cfg +.. literalinclude:: ../post.mysimulation.cfg :language: cfg :linenos: @@ -38,11 +38,10 @@ Once the ``[ts]`` subsection ``[[ atm_monthly_glb ]]`` and the ``[mpas_analysis] section finish, ``[global_time_series]`` will run. Post-processing results will be located in ``output`` and ``www``. Some machines have -a web server. ``www`` should be pointed to that so that E3SM Diags, MPAS-Analysis, and -the global time series plots will be visible online. +a web server. ``www`` should be pointed to that so that the plotting tasks (here: ``e3sm_diags``, ``mpas_analysis``, ``global_time_series``, ``livvkit``) will be visible online. Because we have specified ``campaign = "water_cycle"``, some parameters will -be automatically set. ``zppy/templates/water_cycle.cfg`` specifies what +be automatically set. ``zppy/defaults/water_cycle.cfg`` specifies what ``[e3sm_diags] > sets``, and ``[mpas_analysis] > generate`` should be for the water cycle campaign. Users may specify their own values for any of these parameters, @@ -54,7 +53,7 @@ Example 2 This is another example of a configuration file, this time using a RRM simulation. -.. literalinclude:: post.rrm_simulation.cfg +.. literalinclude:: ../post.rrm_simulation.cfg :language: cfg :linenos: @@ -79,22 +78,31 @@ makes it an ``mvm`` run, otherwise it is an ``mvo`` run. If a referenced prior run could resolve to either ``mvo`` or ``mvm``, use ``reference_comparison_type`` or ``test_comparison_type`` to disambiguate. -.. literalinclude:: post.mpas_analysis_model_vs_model.cfg +.. literalinclude:: ../post.mpas_analysis_model_vs_model.cfg :language: cfg :linenos: Debugging failures ================== - .. code:: - - $ cd /post/scripts - $ grep -v "OK" *status # See what failed - # Review `.o` files corresponding to failed `.status` files. - # If an error is obvious, make a fix in the bash file and rerun: - $ sbatch .bash - # If the error is not obvious, do the following: - $ emacs .bash - # In this file, set `debug = True`. This will provide more information. - # Note: another option is to set `debug = True` in your `cfg` and rerun `zppy`. - $ sbatch .bash +.. code-block:: bash + + grep "output =" your_zppy_config.cfg # Easy way to remember your output directory + cd your_output_dir/post/scripts + grep -v "OK" *status # See what failed + + # Say failing_task.status is showing a non-OK status, then: + ls failing_task* # See everything associated with it + + # Say its job ID was 123456, then: + emacs failing_task.o123456 # Review the output + emacs failing_task.settings # Review how each of its parameters was set/configured. + + # If an error is obvious, make a fix in the bash file and rerun: + sbatch failing_task.bash + + # If the error is not obvious, do the following: + emacs failing_task.bash + # In this file, set `debug = True`. This will provide more information. + # Note: another option is to set `debug = True` in your `cfg` and rerun `zppy -c`. + sbatch failing_task.bash diff --git a/zppy/defaults/default.ini b/zppy/defaults/default.ini index 948fadbb..05800b22 100755 --- a/zppy/defaults/default.ini +++ b/zppy/defaults/default.ini @@ -541,6 +541,7 @@ e3sm_to_cmip_atm_subsection = string(default="") # Flag to process the land/sea mask within pcmdi generate_sftlf = string(default=True) # model data grid after remapping +# NOTE: always overrides value in [default] grid = string(default="180x360_aave") mov_plot_obs = string(default=True) mov_plot_model = string(default=True) @@ -562,6 +563,7 @@ save_test_clims = string(default=True) reference_data_path = string(default="") reference_data_path_ts = string(default="") # The years increment for test model data +# NOTE: always overrides value in [default] ts_num_years = integer(default=5) # The years range for test model data ts_years = string_list(default=list("")) @@ -619,6 +621,7 @@ ConvEOF = string(default=True) eofn_mod_max = integer(default=3) EofScaling = string(default=False) # Frequency of the model data +# NOTE: always overrides value in [default] frequency = string(default="mo") landmask = string(default=False) # Keywords for unit conversion in pcmdi (model)