Revamp docs to be more comprehensive

[Copilot]

Summary

Objectives:

• Revamp zppy documentation to be more organized and comprehensive.
• Include a user guide and dev guide.
• Explain parameters and dependencies in more detail.

Resolves:

• Add descriptions for all parameters #78
• Create mid-level overview #311
• In docs, emphasize packages for new tasks #326

Select one: This pull request is...

• a bug fix: increment the patch version
• a small improvement: increment the minor version
• a new feature: increment the minor version
• an incompatible (non-backwards compatible) API change: increment the major version

Small Change

(It's technically a fairly large change, but it's only impacting docs)

• To merge, I will use "Squash and merge". That is, this change should be a single commit.
• Logic: I have visually inspected the entire pull request myself.
• Pre-commit checks: All the pre-commits checks have passed.

[forsyth2]

Instructions for generating docs:

git fetch upstream
git checkout -b copilot/revamp-zppy-docs upstream/copilot/revamp-zppy-docs
nersc_conda # Activate conda
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr839-docs
python -m pip install .  

cd docs
make html
cp -r _build/ /global/cfs/cdirs/e3sm/www/forsyth/zppy_docs
chmod -R 755 /global/cfs/cdirs/e3sm/www/forsyth/zppy_docs

Go to: https://portal.nersc.gov/cfs/e3sm/forsyth/zppy_docs/html/

[forsyth2]

Added commits for manual revisions and to add a dependencies graph.

Remaining action items:

• Update the dependencies in the task pages to match what we have in the dependencies graph.
• Optional: actually render the dependencies graph (need to install graphviz) and include it in the docs. It may be too busy to be worthwhile though.
• Revise the tests section. This will serve as a good base for continuing the implementation of Add main branch testing automation #774. That is, we'll have the most up-to-date testing process documented, so in theory it will be easier to convert to an automated script.
• Re-render the docs and fix any rst syntax errors I've made in my additions

[Copilot]

Pull request overview

This PR reorganizes the Sphinx documentation into a structured User Guide
and Developer Guide, significantly expanding task- and component-specific
documentation and updating terminology (e.g., “post-processing toolchain” →
“workflow manager”).

Changes:

• Restructured docs into user_guide/ and dev_guide/ with new indices,
  moved/removed legacy pages, and added archived/obsolete content.
• Added per-task User Guide pages with parameter tables and dependency notes,
  plus per-component guidance pages.
• Added Developer Guide content for task implementation references, testing
  workflows, and release procedures.

Reviewed changes

Copilot reviewed 63 out of 63 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
docs/source/user_guide/tutorial.rst	Updates tutorial includes/paths and expands troubleshooting guidance
docs/source/user_guide/tasks/ts.rst	Adds user-facing ts task documentation + parameter tables
docs/source/user_guide/tasks/tc_analysis.rst	Adds user-facing tc_analysis task documentation
docs/source/user_guide/tasks/pcmdi_diags.rst	Adds user-facing pcmdi_diags task documentation
docs/source/user_guide/tasks/mpas_analysis.rst	Adds user-facing mpas_analysis task documentation
docs/source/user_guide/tasks/livvkit.rst	Adds user-facing livvkit task documentation
docs/source/user_guide/tasks/index.rst	Adds User Guide tasks landing page and toctree
docs/source/user_guide/tasks/ilamb.rst	Adds user-facing ilamb task documentation
docs/source/user_guide/tasks/global_time_series.rst	Adds user-facing global_time_series task documentation
docs/source/user_guide/tasks/e3sm_to_cmip.rst	Adds user-facing e3sm_to_cmip task documentation
docs/source/user_guide/tasks/e3sm_diags.rst	Adds user-facing e3sm_diags task documentation
docs/source/user_guide/tasks/climo.rst	Adds user-facing climo task documentation
docs/source/user_guide/tasks/bundle.rst	Adds user-facing bundle task documentation
docs/source/user_guide/schematics.rst	Fixes schematic image paths for new layout
docs/source/user_guide/parameters.rst	Updates parameter defaults links + reorganizes parameter inference content
docs/source/user_guide/index.rst	Adds new User Guide top-level index/toctree
docs/source/user_guide/getting_started.rst	Adds new User Guide “Getting started”
docs/source/user_guide/dependencies.rst	Adds dependencies page embedding DOT graph
docs/source/user_guide/components/seaice.rst	Adds MPAS-seaice component guidance
docs/source/user_guide/components/rivers.rst	Adds MOSART component guidance
docs/source/user_guide/components/ocean.rst	Adds MPAS-Ocean component guidance
docs/source/user_guide/components/land.rst	Adds ELM component guidance
docs/source/user_guide/components/index.rst	Adds components landing page and toctree
docs/source/user_guide/components/atmosphere.rst	Adds EAM/EAMxx component guidance
docs/source/user_guide/campaigns.rst	Adds User Guide campaign documentation
docs/source/index.rst	Reworks docs homepage and splits toctrees into User/Developer guides
docs/source/getting_started.rst	Removes old top-level getting started page (moved under User Guide)
docs/source/dev_guide/update_expected_results.rst	Adds expected-results update procedure documentation
docs/source/dev_guide/tutorial_testing_e3sm_unified.rst	Removes obsolete unified testing tutorial
docs/source/dev_guide/testing.rst	Removes old testing page (replaced by expanded workflows)
docs/source/dev_guide/testing_e3sm_unified.cfg	Removes embedded testing cfg (obsolete/moved)
docs/source/dev_guide/test.rst	Adds expanded, step-by-step integration testing workflow
docs/source/dev_guide/tasks/ts.rst	Adds developer reference for ts task implementation
docs/source/dev_guide/tasks/tc_analysis.rst	Adds developer reference for tc_analysis implementation
docs/source/dev_guide/tasks/pcmdi_diags.rst	Adds developer reference for pcmdi_diags implementation
docs/source/dev_guide/tasks/mpas_analysis.rst	Adds developer reference for mpas_analysis implementation
docs/source/dev_guide/tasks/livvkit.rst	Adds developer reference for livvkit implementation
docs/source/dev_guide/tasks/index.rst	Adds Developer Guide task reference index + dependency overview
docs/source/dev_guide/tasks/ilamb.rst	Adds developer reference for ilamb implementation
docs/source/dev_guide/tasks/global_time_series.rst	Adds developer reference for global_time_series entrypoint
docs/source/dev_guide/tasks/e3sm_to_cmip.rst	Adds developer reference for e3sm_to_cmip implementation
docs/source/dev_guide/tasks/e3sm_diags.rst	Adds developer reference for e3sm_diags implementation
docs/source/dev_guide/tasks/climo.rst	Adds developer reference for climo implementation
docs/source/dev_guide/tasks/bundle.rst	Adds developer reference for bundle implementation
docs/source/dev_guide/releases/release_candidates.rst	Adds release-candidate workflow documentation
docs/source/dev_guide/releases/production_releases.rst	Adds production release workflow documentation
docs/source/dev_guide/releases/index.rst	Adds releases section index
docs/source/dev_guide/release.rst	Removes older release guide (replaced by new releases section)
docs/source/dev_guide/release_testing.rst	Removes older release testing directions (replaced)
docs/source/dev_guide/project-standards.rst	Updates VC and pre-commit/CI documentation
docs/source/dev_guide/parameters.rst	Moves developer-specific parameter inference docs into Dev Guide
docs/source/dev_guide/new_task.rst	Updates new-task instructions to match current template/entrypoint patterns
docs/source/dev_guide/new_glb_plot.rst	Removes obsolete global time series plot guide
docs/source/dev_guide/new_diags_set.rst	Minor retitling of E3SM Diags “new set” guide
docs/source/dev_guide/index.rst	Rebuilds Dev Guide index/toctree to match new structure
docs/source/dev_guide/contributing.rst	Moves/updates docs contribution instructions
docs/source/dev_guide/ci.rst	Removes old CI page (content merged into standards/other docs)
docs/source/dev_guide/archive/initial_docs.rst	Adds archived initial Sphinx setup notes
docs/source/dev_guide/archive/index.rst	Adds archive index for obsolete content
docs/source/dev_guide/archive/deprecated_parameters.rst	Adds archived deprecated-parameter list
docs/source/dependencies.dot	Adds DOT graph describing task dependencies
docs/source/contributing.rst	Removes old top-level docs contributing page (moved under Dev Guide)
docs/source/campaigns.rst	Removes old top-level campaigns page (moved under User Guide)

Comments suppressed due to low confidence (4)

docs/source/user_guide/tutorial.rst:41

• In the list of plotting tasks, there is a missing comma between e3sm_diags and mpas_analysis, which makes the list hard to read.
  docs/source/user_guide/tutorial.rst:46
• The tutorial points to zppy/templates/water_cycle.cfg, but campaign config files live under zppy/defaults (as also described in the Campaigns docs). This path will mislead users.
  docs/source/user_guide/tutorial.rst:92
• The literal block is introduced with an indented :: and uses a grep pattern (output=) that won't match typical cfg lines like output = .... The tab-indented lines will also render inconsistently in Sphinx.
  docs/source/user_guide/tutorial.rst:108
• The final rerun command uses a different placeholder (<failed job>.bash) than the rest of the example (failing_task.bash), which is confusing and likely a copy/paste leftover.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[forsyth2]

Remaining action items:

• Implement suggestions from Copilot reviewer.
• Re-render the docs and re-review them.

Independent of this PR: use the testing docs as a most-up-to-date guide for automating the testing process (#774)

[forsyth2]

@copilot I've updated these files to be more how I want the parameter documentation to look:

docs/source/user_guide/parameters.rst
docs/source/user_guide/tasks/climo.rst
docs/source/user_guide/tasks/ts.rst
docs/source/user_guide/tasks/e3sm_to_cmip.rst
docs/source/user_guide/tasks/tc_analysis.rst

Can you please update the following 6 files (only these files) to have them match the structure of the files listed above? The parameters should match up with what's in zppy/defaults/defaults.ini.

docs/source/user_guide/tasks/e3sm_diags.rst
docs/source/user_guide/tasks/mpas_analysis.rst
docs/source/user_guide/tasks/global_time_series.rst
docs/source/user_guide/tasks/ilamb.rst
docs/source/user_guide/tasks/livvkit.rst
docs/source/user_guide/tasks/pcmdi_diags.rst

Notably:

• Top-level parameters should only be listed if they are overridden in zppy/defaults/defaults.ini.
• There should be a count of parameters included for each task.
• Parameters should be listed in the order they appear in zppy/defaults/defaults.ini. If there are a large number of parameters, they should be put into logical groups/categories.

[forsyth2]

I rebased off main and added a commit to include docs on PRs that have merged since I started this documentation update: #828 & #836

[Copilot]

Pull request overview

Copilot reviewed 65 out of 65 changed files in this pull request and generated 4 comments.

[forsyth2]

cd docs
make html
cp -r _build/ /global/cfs/cdirs/e3sm/www/forsyth/zppy_docs_20260626_try2
chmod -R 755 /global/cfs/cdirs/e3sm/www/forsyth/zppy_docs_20260626_try2

Latest results here

[forsyth2]

I've been reading the rendered docs as I've added commits. I also just did a quick visual inspection of the latest rendering from today. I think this is ready to merge.

[forsyth2]

The docs workflow has completed and the official docs are now updated: https://docs.e3sm.org/zppy/_build/html/main/index.html