Add main branch testing automation#774
Draft
forsyth2 wants to merge 32 commits into
Draft
Conversation
forsyth2
added
Testing
Collaborator
Author
|
Current status: made it to the end of phase 1 (launched jobs), but the MPAS-Analysis jobs didn't have sufficient wall times allocated. (Note: because of #764, it is usually better to rerun from scratch in this case). Action items:
|
forsyth2
force-pushed
the
automate-main-branch-testing
branch
from
January 26, 2026 20:44
f33b563 to
ed8e6fc
Compare
Collaborator
Author
|
Using the latest commit (88cdcb2), I've made it to phase 2. It turns out I needed to update |
Collaborator
Author
|
I've run through phase 3 now. There are a number of issues to resolve before fully switching from the manual testing workflow to this one. Immediate KeyError on later zppy runsI cannot get Phase 2 to work in an automated manner. I can launch Status file errorsI'm not sure how these could possibly be real failures (i.e., not caused by this branch), considering the last change to To be certain though, I will have to do the usual workflow and see if the same thing happens or not. MPAS Analysis errorscd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260126_run4/v2.LR.historical_0201/post/scripts
cat mpas_analysis_ts_1980-1984_climo_1980-1984.o1135588gives: OutputI had extended the time limit (here) because it was hitting the time limit, but it turns out the time limit isn't the problem at all. The job stays active even after the segmentation fault, but doesn't actually do anything to recover from it. Notes:
PCMDI Diags errorscd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts
cat pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.o1135637gives: OutputRelevant lines, though I'm not sure why they aren't working:
Notes:
|
Collaborator
Author
|
After running the standard manual weekly test, I've confirmed the That means this branch for automated testing has two major issues to resolve:
|
Collaborator
Author
|
#803 has some helper functions that can at least partially automate the process until this more complete automation is merged. |
7 tasks
forsyth2
force-pushed
the
automate-main-branch-testing
branch
from
June 9, 2026 22:28
b35347d to
fa6f2bb
Compare
Collaborator
Author
|
I gave Claude the updated testing documentation of #839 (specifically this file), the helper functions of #803, and I then had it update I've rebased this PR off the latest Remaining action items:
|
Collaborator
Author
Results from 06/11-06/12 runI've completed a run of this test script. A few notes:
e3sm_diags
Failing image checks:
Debugging: grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260611_run5/v2.LR.historical_0201/post/scripts/e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.o1234499 | wc -l
# 2021
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260611_run5/v2.LR.historical_0201/post/scripts/e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.o1234499 | tail -n 7
# 35122: raise IOError(
# 35125:OSError: No files found for target variable FLUT or derived variables ([('rlut',), ('LW_flux_up_at_model_top',), ('olr',)]) in /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260611_run5/v2.LR.historical_0201/post/atm/180x360_aave/ts/daily/2yr.
# 35126:2026-06-12 00:47:41,055 [ERROR]: core_parameter.py(_run_diag:418) >> Error in e3sm_diags.driver.tropical_subseasonal_driver
# 35139: raise IOError(
# 35142:OSError: No files found for target variable U850 or derived variables ([('U_at_850hPa',)]) in /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260611_run5/v2.LR.historical_0201/post/atm/180x360_aave/ts/daily/2yr.
# 35144:slurmstepd: error: *** STEP 1234499.0 ON chr-0232 CANCELLED AT 2026-06-12T05:21:59 DUE TO TIME LIMIT ***
# 35145:slurmstepd: error: *** JOB 1234499 ON chr-0232 CANCELLED AT 2026-06-12T05:21:59 DUE TO TIME LIMIT ***
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260611_run5/v2.LR.historical_0201/post/scripts/e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1234561 | wc -l
# 2387
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260611_run5/v2.LR.historical_0201/post/scripts/e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1234561 | tail -n 7
# 40511: raise IOError(
# 40514:OSError: The dataset file has no matching source variables for LWCF06_FRAC and could not be derived using [('LWCF06_FRAC',), ('LWCF06', 'LWCF'), ('FLSU06', 'FLSUCLR06', 'FLUT', 'FLUTC'), ('olr_band06', 'olr_clr_band06', 'olr', 'olr_clr')].
# 40516:2026-06-12 02:25:17,122 [ERROR]: core_parameter.py(_run_diag:418) >> Error in e3sm_diags.driver.lat_lon_driver
# 40527: raise IOError(
# 40530:OSError: The dataset file has no matching source variables for LWCF06_FRAC and could not be derived using [('LWCF06_FRAC',), ('LWCF06', 'LWCF'), ('FLSU06', 'FLSUCLR06', 'FLUT', 'FLUTC'), ('olr_band06', 'olr_clr_band06', 'olr', 'olr_clr')].
# 41094:slurmstepd: error: *** STEP 1234561.0 ON chr-0159 CANCELLED AT 2026-06-12T06:42:29 DUE TO TIME LIMIT ***
# 41095:slurmstepd: error: *** JOB 1234561 ON chr-0159 CANCELLED AT 2026-06-12T06:42:29 DUE TO TIME LIMIT ***mpas_analysis
That matches up with the 3 Failing image checks:
The 3 missing-images rows make sense based on the 3 E3SMv2 The 2 mismatched-images rows look to be actual changes based on recent PRs (i.e., we should probably update the expected results) Debugging: grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260611_run5/v2.LR.historical_0201/post/scripts/mpas_analysis_ts_1980-1984_climo_1980-1984.o1234421
# 71:srun: error: chr-0232: task 0: Segmentation fault
# 73:slurmstepd: error: *** STEP 1234421.0 ON chr-0232 CANCELLED AT 2026-06-11T14:44:50 DUE TO TIME LIMIT ***
# 74:slurmstepd: error: *** JOB 1234421 ON chr-0232 CANCELLED AT 2026-06-11T14:44:50 DUE TO TIME LIMIT ***
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260611_run5/v2.LR.historical_0201/post/scripts/mpas_analysis_ts_1980-1984_climo_1980-1984.o1234501
# 80:srun: error: chr-0232: task 0: Segmentation fault
# 82:slurmstepd: error: *** STEP 1234501.0 ON chr-0232 CANCELLED AT 2026-06-11T16:09:50 DUE TO TIME LIMIT ***
# 83:slurmstepd: error: *** JOB 1234501 ON chr-0232 CANCELLED AT 2026-06-11T16:09:50 DUE TO TIME LIMIT ***
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260611_run5/v2.LR.historical_0201/post/scripts/mpas_analysis_ts_1980-1984_climo_1980-1984.o1234564
# 71:srun: error: chr-0232: task 0: Segmentation fault
# 73:slurmstepd: error: *** STEP 1234564.0 ON chr-0232 CANCELLED AT 2026-06-11T16:59:50 DUE TO TIME LIMIT ***
# 74:slurmstepd: error: *** JOB 1234564 ON chr-0232 CANCELLED AT 2026-06-11T16:59:50 DUE TO TIME LIMIT ***So, we're still encountering the segfault issue we were seeing when we first tried automating the script ~6 months ago. global_time_series
That matches up with the 3 Failing image checks:
It makes sense the images are missing, since the 3 E3SMv2 pcmdi_diagsFailing image checks:
These look to be actual changes based on recent PRs (i.e., we should probably update the expected results) |
Collaborator
Author
|
Hi @xylar -- I'm having a lot of difficulty debugging a segfault in This segfault only ever occurs when I try to automate the integration testing process. I don't think I've ever seen it when I've run the test manually. That leads me to believe it's an environment setup issue. Manual testsI documented the manual testing process I used throughout the Spring Unified testing period on the new docs of #839 (latest rendering here). Initial attempt at automationThis segfault first showed up when I first started this automation PR in January. (See above comment's "MPAS Analysis errors" section). Latest attempt at automationThis segfault hasn't gone away despite making the test script match my manual process as much as possible. I even have it using bash subshells now for more environment isolation. Output from automated test scriptcd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260615_run6/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:RUNNING 1236065
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 1236066
# The first job says RUNNING, but that's because it hit the time limit before it could update the status file.
tail mpas_analysis_ts_1980-1984_climo_1980-1984.o1236065gives: So, the segfault is still occurring on my latest automation attempt. It's also important to note that the segfault occurs ~10-15 minutes before the time limit hits, so the script is just hanging after the segfault. Output from manual sbatch runsbatch mpas_analysis_ts_1980-1984_climo_1980-1984.bash
tail -f mpas_analysis_ts_1980-1984_climo_1980-1984.o1236067
cat mpas_analysis_ts_1980-1984_climo_1980-1984.status
# OKThis works! Output from manual zppy run# Remove status file so zppy will rerun this job:
rm mpas_analysis_ts_1980-1984_climo_1980-1984.status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy/
lcrc_conda
conda activate env_the_script_used
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT NAME
# ac.forsy e3sm 1 compute 1236069 PD Priority 0:00 30:00 mpas_analysis_ts_1980-1984_climo_1980-1984
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260615_run6/v2.LR.historical_0201/post/scripts
tail -f mpas_analysis_ts_1980-1984_climo_1980-1984.o1236069
cat mpas_analysis_ts_1980-1984_climo_1980-1984.status
# OKThis works! |
Collaborator
|
@forsyth2, I'm on vacation for 2 weeks. I can take a look when I get back on the 29th. |
Collaborator
Author
|
Ok thanks @xylar! |
Collaborator
|
At a glance, the information you posted doesn't give me any useful clues about what's wrong. I'd have to reproduce the problem and debug myself. |
Contributor
There was a problem hiding this comment.
⚠️ Not ready to approve
The legacy 3.1.0 bundles template contains invalid/misapplied expansion lines and the automation script currently masks failures (status/pytest), which can cause false “success” runs.
Pull request overview
This PR adds a repeatable, phase-based automation workflow under tests/main_branch_testing/ to orchestrate the existing “main branch” integration testing process for zppy and related repos on supported HPC machines, and updates the weekly integration-test templates/utilities to support task/dependency activation via expansion flags.
Changes:
- Add a configurable
run_integration_test.bashdriver (with a companionzppy_test.cfgand README) to automate environment setup, cfg generation, SLURM submission, and validation phases. - Extend
tests/integration/utils.pywith more granular “active_*” expansion flags to model dependencies between tasks. - Update the 9 weekly integration template cfgs to use the new expansion flags (and add missing environment settings where needed).
File summaries
| File | Description |
|---|---|
| tests/main_branch_testing/zppy_test.cfg | Adds a sourced bash-style config for the automation script. |
| tests/main_branch_testing/run_integration_test.bash | Adds a multi-phase automation script for running the weekly + legacy integration workflow. |
| tests/main_branch_testing/README.md | Documents how to use/configure the automation script and phases. |
| tests/integration/utils.py | Adds finer-grained dependency activation expansions used by weekly templates. |
| tests/integration/template_weekly_bundles.cfg | Updates subsection activation to use new dependency expansion flags. |
| tests/integration/template_weekly_comprehensive_v2.cfg | Updates subsection activation to use new dependency expansion flags. |
| tests/integration/template_weekly_comprehensive_v3.cfg | Updates subsection activation to use new dependency expansion flags (incl. livvkit-specific deps). |
| tests/integration/template_weekly_legacy_3.1.0_bundles.cfg | Updates activation flags for legacy 3.1.0 bundle workflow templates. |
| tests/integration/template_weekly_legacy_3.1.0_comprehensive_v2.cfg | Updates activation flags and mpas_analysis env settings for legacy 3.1.0 v2 comprehensive template. |
| tests/integration/template_weekly_legacy_3.1.0_comprehensive_v3.cfg | Updates activation flags and mpas_analysis env settings for legacy 3.1.0 v3 comprehensive template. |
| tests/integration/template_weekly_legacy_3.0.0_bundles.cfg | Updates activation flags for legacy 3.0.0 bundle workflow templates. |
| tests/integration/template_weekly_legacy_3.0.0_comprehensive_v2.cfg | Updates activation flags for legacy 3.0.0 v2 comprehensive template. |
| tests/integration/template_weekly_legacy_3.0.0_comprehensive_v3.cfg | Updates activation flags for legacy 3.0.0 v3 comprehensive template. |
Copilot's findings
- Files reviewed: 13/13 changed files
- Comments generated: 6
Note
Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Collaborator
Author
Just to confirm for sure, I did a manual run and indeed it completed successfully, so it really does only happen when trying to run the automated script. Manual RunFollowing the testing documentation here: # Step 3: Set up environments for called packages
bash
source ~/.bashrc
cd ~/ez/MPAS-Analysis
git status
# On branch test_mpas_20260615_run4
# nothing to commit, working tree clean
git fetch upstream develop # This assumes you've named your remote for the main repo as "upstream"
git checkout develop
git reset --hard upstream/develop
git log --oneline | head -n 1
# 22956fd31 Merge pull request #1150 from MPAS-Dev/dependabot/github_actions/prefix-dev/setup-pixi-0.9.6
# Good, matches https://github.com/MPAS-Dev/MPAS-Analysis/commits/develop/
lcrc_conda # Function to activate conda
rm -rf build
conda clean --all --y
conda create --name mpas-analysis-develop-20260616 --file dev-spec.txt --yes
conda activate mpas-analysis-develop-20260616
python -m pip install .
exit # Exit bash subshell
# Step 4: Set up zppy environment
cd ~/ez/zppy
git status
git fetch upstream automate-main-branch-testing
git checkout -b zppy-automate-main-branch-testing-20260616 upstream/automate-main-branch-testing
git log --oneline | head -n 1
# 69f8212d Address comments
# Good, matches https://github.com/E3SM-Project/zppy/pull/774/commits
lcrc_conda # Function to activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-automate-main-branch-testing-20260616
conda activate zppy-automate-main-branch-testing-20260616
pre-commit run --all-files # Confirm this passes
conda list python # 3.14.6
# Step 5: Launch zppy jobs
emacs tests/integration/utils.pyTEST_SPECIFICS: Dict[str, Any] = {
# This is the NCO path.
# Keep as "" to use the production-version NCO commands.
# Set to a specific path to use development-version NCO commands.
"nco_path": "",
# These are custom environment_commands for specific tasks.
# Never set these to "", because they will print the line
# `environment_commands = ""` for the corresponding task,
# thus overriding the value set higher up in the cfg.
# That is, there will be no environment set.
# (`environment_commands = ""` only redirects to Unified
# if specified under the [default] task)
"e3sm_to_cmip_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
"diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
"mpas_analysis_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate mpas-analysis-develop-20260616",
"global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
"livvkit_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
"pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# This is the environment setup for other tasks.
# Leave as "" to use the latest Unified environment.
"environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
"cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.1.0_bundles",
"weekly_legacy_3.1.0_comprehensive_v2",
# "weekly_legacy_3.1.0_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
],
"tasks_to_run": [
# "e3sm_diags",
"mpas_analysis",
# "global_time_series",
# "ilamb",
# "livvkit",
# "pcmdi_diags",
],
"unique_id": "test_mpas_analysis_manual",
}git diff # Check that the diff looks as you expect
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_mpas_analysis_manual
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# mpas_analysis_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate mpas-analysis-develop-20260616
# global_time_series_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# livvkit_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# pcmdi_diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
sq
python -m pip install .
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT NAME
# ac.forsy e3sm 1 compute 1236588 PD Priority 0:00 30:00 mpas_analysis_ts_1980-1984_climo_1980-1984
# ac.forsy e3sm 1 compute 1236589 PD Dependency 0:00 30:00 mpas_analysis_ts_1980-1990_climo_1985-199
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/test_mpas_analysis_manual/v2.LR.historical_0201/post/scripts
tail -f mpas_analysis_ts_1980-1984_climo_1980-1984.o1236588
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT NAME
# ac.forsy e3sm 1 compute 1236589 PD Dependency 0:00 30:00 mpas_analysis_ts_1980-1990_climo_1985-1990
# ac.forsy e3sm 1 compute 1236588 CG None 14:30 30:00 mpas_analysis_ts_1980-1984_climo_1980-1984
# It completes successfully, well before hitting the walltime!
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT NAME
# ac.forsy e3sm 1 compute 1236589 R None 0:26 30:00 mpas_analysis_ts_1980-1990_climo_1985-1990
cat mpas_analysis_ts_1980-1984_climo_1980-1984.o1236588 |
Collaborator
|
@forsyth2, sorry I haven't had time to look at this. I'm not sure if I'll get to it before I travel again on Monday. MPAS-Analysis doesn't use esmpy, so there shouldn't be any need to install it. |
Collaborator
Author
|
@xylar That's ok, thanks though!
Indeed, trying to add it causes a |
Collaborator
|
@forsyth2, when you're back, could you provide me with a smaller reproducer of the issue? I'm not that familiar with this workflow. |
Collaborator
Author
Recap of what's going wrongI've updated the official Trying to automate it, as in this PR, keeps causing a segfault in MPAS-Analysis. This has been a persistent issue, first appearing when I first attempted this test automation in January. After many many debugging iterations with multiple AI chat interfaces, I'm at a complete loss as to why this happens. I suspect it must have something to do with the environment setup in the script versus when I do it manually. Another theory is that maybe the automated script is launching things so much faster than a human would manually that it's causing issues. Steps to reproduce
@xylar I've tried to trim down the full test script to the bare minimum to reproduce the issue here. I apologize that is is still rather long. I didn't have much luck with Claude in trying to make it any more concise. Instructions# Edit paths/parameters in these instructions and the bash/cfg provided below to match your workspace.
# I recommend creating scratch copies of the repos since the script changes branches.
# Alternatively, just make sure you don't have uncommitted changes in either the MPAS-Analysis or zppy repos.
# (In any case, the function ensure_test_branch in the script should check this.)
# Use scratch copies of the repos to avoid interference:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/MPAS-Analysis && git status
# Good -- nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Good -- nothing to commit, working tree clean
squeue -u $USER # Good -- No jobs queued (Not required, just makes it easier to track jobs)
WORK_DIR=~/ez/zppy_main_branch_tests/debug_mpas2 # Whatever you want; this is just what I used most recently
cd ${WORK_DIR}
# Copy the bash/cfg provided below into this dir
head -n 18 debug_mpas_analysis.bash # Check the date stamp and run number are what you expect:
# DATE_STAMP="20260721"
# RUN_NUMBER=1
screen
cd ${WORK_DIR}
head -n 18 debug_mpas_analysis.bash # Good, matches previous look
time ./debug_mpas_analysis.bash 2>&1 | tee debug_mpas_20260721_run1.log
# Ctrl-A D to detach from screen
tail -f debug_mpas_20260721_run1.log
# Takes about ~25-30 minutes, so you might not even want to bother with using `screen`
# Segfault usually appears 5-15 minutes after the mpas-analysis job launches (well before the time limit hits)Bash file
#!/bin/bash
# zppy Integration Test Automation Script
#
# Usage: Run `./debug_mpas_analysis.bash` (outside of the zppy repo)
# This script is a reduced version of `run_integration_test.bash`
set -e # Exit on error
set -u # Exit on undefined variable
# Change accordingly when running this debugging script #######################
# Function to activate conda:
CONDA_ACTIVATION_CMD="lcrc_conda" # Defined in my .bashrc
CONDA_PROFILE="$HOME/miniforge3/etc/profile.d/conda.sh"
# Dir where you have a zppy repo & a mpas-analysis repo:
EZ_DIR=/lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/
DATE_STAMP="20260721"
RUN_NUMBER=1
# Constants ###################################################################
TAG="${DATE_STAMP}_run${RUN_NUMBER}"
UNIQUE_ID="zppy_main_branch_test_${TAG}"
SCRIPT_RUN_DIR="$PWD"
CFG_PATH="${SCRIPT_RUN_DIR}/debug_mpas_analysis.cfg"
UNIFIED_ENV_CMD="source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
MPAS_ANALYSIS_DIR="$EZ_DIR/MPAS-Analysis"
MPAS_BASE_BRANCH="develop"
MPAS_ENV="test-mpas-${MPAS_BASE_BRANCH}-${TAG}"
ZPPY_DIR="$EZ_DIR/zppy"
ZPPY_BASE_BRANCH="main"
ZPPY_ENV="test-zppy-${ZPPY_BASE_BRANCH}-${TAG}"
# Colors for output
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Functions ###################################################################
log() {
echo -e "${BLUE}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $*"
}
# Checkout test branch, creating it from upstream/<base> if it doesn't exist.
# Stashes/commits any in-progress work first.
ensure_test_branch() {
local test_branch="$1"
local base_branch="$2"
local current_branch
current_branch=$(git rev-parse --abbrev-ref HEAD 2>/dev/null)
if [ "$current_branch" = "$test_branch" ]; then
log "Already on branch '$test_branch'"
return 0
fi
log "Saving current work before switching branches..."
git add -A
git commit -m "Auto-save before test" --no-verify || true
if git show-ref --verify --quiet "refs/heads/$test_branch"; then
log "Checking out existing branch '$test_branch'..."
git checkout "$test_branch"
else
log "Creating new branch '$test_branch' from upstream/${base_branch}..."
git fetch upstream "${base_branch}"
git checkout -b "$test_branch" "upstream/${base_branch}"
fi
log "On branch '$test_branch'"
}
# Activate conda and (optionally) a named environment.
activate_env() {
local env_name="${1:-}"
set +u
# shellcheck disable=SC1090
source ~/.bashrc
$CONDA_ACTIVATION_CMD # Machine-specific conda init
if [ -n "$env_name" ]; then
conda activate "$env_name"
log "Installing/updating package in '$env_name'..."
python -m pip install .
fi
set -u
}
# `main` ######################################################################
log "TAG: $TAG"
log "Unique ID: $UNIQUE_ID"
log "Setting up MPAS-Analysis..."
(
cd "$MPAS_ANALYSIS_DIR"
ensure_test_branch "test_mpas_${TAG}" "$MPAS_BASE_BRANCH"
log "Latest MPAS-Analysis commit (should match https://github.com/MPAS-Dev/MPAS-Analysis/commits/${MPAS_BASE_BRANCH}):"
git log -1 --oneline
activate_env # Ensure conda itself is available
rm -rf build
conda clean --all --yes
conda create --name "$MPAS_ENV" --file dev-spec.txt --yes
activate_env "$MPAS_ENV"
)
log "Setting up zppy..."
(
cd "$ZPPY_DIR"
ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH"
log "Latest zppy commit (should match https://github.com/E3SM-Project/zppy/commits/${ZPPY_BASE_BRANCH}):"
git log -1 --oneline
activate_env # Ensure conda itself is available
rm -rf build
conda clean --all --yes
conda env create -f "conda/dev.yml" -n "$ZPPY_ENV"
activate_env "$ZPPY_ENV"
)
cd "$ZPPY_DIR"
activate_env "$ZPPY_ENV"
ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH"
log "Submitting SLURM jobs for: $CFG_PATH"
zppy -c "$CFG_PATH"
job_count=$(squeue -u "${USER}" | wc -l)
job_count=$((job_count - 1))
log "Submitted jobs. Current queue depth: $job_count"
log "Once MPAS-Analysis job runs for ~10 minutes, run:"
log "cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/${UNIQUE_ID}/v2.LR.historical_0201/post/scripts/"
log 'grep -v "OK" *status'
log "There should be no matches. If there are, look at the matching .o file."Cfg
|
Collaborator
|
@forsyth2, thanks! That's concise enough for me to be able to work with it. I'll keep you posted. |
Collaborator
Author
|
Thanks Xylar!! |
Collaborator
|
@forsyth2, I adapted the script to my own paths and ran it. The whole deal is in: Creating the conda environments took a long time but everything ran without a hitch for me. The second zppy job is still finishing up but there's no sign of trouble. I will try a few more runs since you indicated that it might be an intermittent problem. |
Collaborator
|
@forsyth2, I have run my version of the reproducer 3 times without reproducing the problem. Do you have further suggestions on what I can do to see it? |
Collaborator
|
One question, do you run the debug bash script itself on a login or a compute node? I was running on a login node. |
Collaborator
Author
|
@xylar Thanks for debugging!
Hmm, this is confusing me even more. It actually hasn't been intermittent for me. It always works manually and always fails when I try to automate it.
I run from a login node. I typically always launch I'll take a look at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Objectives:
Issue resolution:
Select one: This pull request is...
Small Change