From be8b3673c53b78d5cff960f73ebc0a849214e8fa Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sat, 27 Jun 2026 12:02:48 -0700 Subject: [PATCH 01/10] Add checkpoint/resume design doc (PoC-validated) Design for checkpointing mpi-sppy runs (serial, multi-rank, and cylinders) so a killed or interrupted job can resume where it left off. Approach: reconstruct the scaffolding (MPI comms, RMA windows, persistent solvers, Pyomo models) via the normal startup path, then restore the algorithmic state (iteration counter, W, nonant values, rho, the incumbent / best xhat, and stateful-extension internals). Pickling the whole object graph with dill is not viable -- the core objects hold live MPI communicators, RMA windows, and persistent solver handles. Doc-only; no library code yet. Covers: why dill-everything fails, the state inventory (with a reconstructed-vs-restored and bit-reproducible-vs-carried- forward breakdown), the determinism contract (primal trajectory bit-identical; bounds/incumbent valid but carried forward, not reproducible), the required core changes, a proposed file layout, and a 6-phase rollout. Validated by a serial + multi-rank + cylinders proof of concept on farmer. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/checkpointing_design.md | 422 ++++++++++++++++++++++++++++ 1 file changed, 422 insertions(+) create mode 100644 doc/designs/checkpointing_design.md diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md new file mode 100644 index 000000000..2e4d8acc5 --- /dev/null +++ b/doc/designs/checkpointing_design.md @@ -0,0 +1,422 @@ +# Checkpoint / Resume for mpi-sppy — Design + +Status: **draft** (PoC validated; not yet implemented in the library). +Scope: checkpoint a running mpi-sppy job so a killed/interrupted run can resume +where it left off. Must work on multiple MPI ranks and for cylinder +(hub-and-spoke) runs. + +--- + +## 1. Goals and non-goals + +**Goals** + +- Resume a Progressive Hedging (PH) run — serial, multi-rank, or full cylinders + (hub + spokes) — after a crash or a deliberate stop. +- Preserve the **best feasible solution found so far (the best xhat), not just + the best bound.** +- Survive a hard kill (`kill -9`, node failure, walltime), not only a clean + shutdown. +- Add no measurable overhead when checkpointing is off, and modest, tunable + overhead when on. + +**Non-goals (initially)** + +- Resuming across a *different* rank count or scenario-to-rank distribution + (cross-geometry remap). The first cut requires identical geometry and refuses + a mismatch with a clear error. +- Bit-identical reproduction of *bounds* (see §7 — bounds are async and not + reproducible; only the primal trajectory is). +- APH. A C++ APH is expected to replace the Python `opt/aph.py`; PH-family only. + +--- + +## 2. Why "pickle every object with dill" does not work + +The obvious idea — `dill.dump()` the whole opt/hub object graph and reload — is +not viable, because the core objects are built around **live, non-serializable +OS/MPI/solver handles**, not data: + +- **MPI communicators** — `SPBase.mpicomm`, `SPBase.comms` (`spbase.py`), and the + `fullcomm`/`strata_comm`/`cylinder_comm` on the spcomm + (`cylinders/spcommunicator.py`). Handles into the running MPI runtime; + meaningless once the process exits. +- **MPI RMA windows and `MPI.Alloc_mem` buffers** — `SPWindow` + (`cylinders/spwindow.py`) and the `FieldArray` send/receive buffers. Kernel + shared-memory regions. +- **Persistent solver handles** — `s._solver_plugin` for + gurobi/cplex/xpress persistent interfaces (`spopt.py`). A C handle + license + session. + +The moment you write a custom `__getstate__`/`__reduce__` to drop and rebuild +those, you have conceded the "just pickle it" dream and are doing +reconstruct-scaffolding + restore-state anyway — only with the checkpoint format +welded to the live object graph (fragile across versions). Better to do that +split explicitly. + +dill *is* useful for the leaf data (Pyomo scenario models, closures), and the +repo already uses it that way — see §4. + +--- + +## 3. Approach: reconstruct the scaffolding, restore the state + +A checkpoint is **not** a snapshot of the object graph. It is the **algorithmic +state** needed to continue. On resume: + +1. **Reconstruct the scaffolding** via the normal startup path — comms, RMA + windows, persistent solvers, and the Pyomo scenario models (from + `scenario_creator`, or from the existing scenario-pickle path). This is + exactly what a fresh run already does. +2. **Restore the state** into that fresh scaffolding: iteration counter, dual + weights, nonant values, rho, the incumbent (best xhat), aggregated bounds, + and any stateful-extension internals. + +This is the design the library *already leans toward* (it pickles scenario +models alone, never the comms/solvers) — we are extending it from "warm-start +iteration 0" to "resume at iteration k". + +--- + +## 4. Building blocks already in the repo + +- **Scenario-model pickling (dill):** `utils/pickle_bundle.py` + (`dill_pickle`/`dill_unpickle`) and `generic/scenario_io.py` pickle each + scenario Pyomo model *alone*. Iter-0 solution + bounds can be baked into + `pickle_metadata` and warm-started via `--iter0-from-pickle`. Precedent for + persisting algorithmic state, and a ready path for reconstructing expensive / + nondeterministic scenario models. +- **W / xbar persistence:** `utils/w_utils/wxbarwriter.py` (writes in + `post_everything`) / `wxbarreader.py` (reads in `pre_iter0`) round-trip the + dual weights `W` and the consensus `xbar` as CSV, as extensions. Covers part + of the restore set; does **not** cover rho, the iteration counter, bounds, the + incumbent, or RNG. +- **Incumbent-to-disk (spokes):** `cylinders/spoke.py` + `_maybe_write_incumbent_on_improvement` (`--incumbent-on-improvement-filename-prefix`) + already writes the first-stage solution on each improvement — a reference for + serializing the incumbent. + +--- + +## 5. State inventory + +The key question for each piece: **reconstructed** (rebuilt by startup, no save +needed) vs **restored** (must be checkpointed), and if restored, is it +**bit-reproducible** on resume or only **carried forward** as a valid +best-so-far value? + +### 5.1 Hub PH primal state — *restored, bit-reproducible* + +The hub's primal trajectory is pure synchronous PH and is **independent of the +spokes** (lagrangian only contributes an outer bound; xhatshuffle only an +incumbent; neither perturbs the iterate). Minimum set, per local scenario: + +| State | Where it lives | Note | +|---|---|---| +| `_PHIter` | `phbase` | iteration counter | +| `W[ndn_i]` (accumulated duals) | `s._mpisppy_model.W` | **must** restore — `Update_W` accumulates, so it is not recomputable | +| nonant values | `s._mpisppy_data.nonant_indices` vardata `_value` | restoring these lets `Compute_Xbar` reproduce `xbar`; `xbar` itself need not be saved | +| nonant **fixedness** | same vardata `.fixed` (+ the fixed value) | **must** restore — variable-fixing/forcing extensions (`fixer`, `slammer`) leave nonants fixed; a rebuilt model has them free. See §5.4 | +| `rho[ndn_i]` | `s._mpisppy_model.rho` | restore (rho-updaters mutate it) | +| smoothing `z/p/beta` | `s._mpisppy_model` | only if `--smoothing` | + +Helpers already exist: `_populate_W_cache` / `W_from_flat_list` (`phbase.py`), +`_save_nonants` / `_restore_nonants` (`spopt.py`). Note `_save_nonants` already +captures fixedness (`fixedness_cache`) — the checkpoint must gather it too (the +PoC captured only `_value`). + +**Restore point:** between `Iter0()` and `iterk_loop()`. In serial this can be a +driver call; in cylinders the hub runs `ph_main` internally, so restore happens +in the **`post_iter0_after_sync`** extension hook (fires at the end of `Iter0`, +`phbase.py:1079`). Iteration 1 of the resumed loop then reproduces what would +have been iteration k+1 — verified bit-identical (§6). + +### 5.2 Hub bounds + incumbent objective — *restored, carried forward* + +- `spcomm.BestInnerBound`, `spcomm.BestOuterBound` — aggregated bounds. +- `opt.best_bound_obj_val`, `opt.best_solution_obj_val`. + +These are products of the **async** spoke interaction; their *timing* is not +reproducible, so they are carried forward (restored as best-so-far). They remain +valid: a restored looser bound is simply improved again by the fresh spokes; a +restored incumbent objective is never regressed because +`update_best_solution_if_improving` (`spbase.py:578`) only accepts improvements. + +Note: in a cylinders run the hub's `opt.best_solution_obj_val` is often `None` — +the inner bound arrives as a scalar via `receive_innerbounds` +(`spcommunicator.py:1019`) and lands in `spcomm.BestInnerBound`. So the +incumbent **objective** is on the hub, but the incumbent **solution** is not (see +5.3). + +### 5.3 Spoke incumbent — *the best xhat solution* — *restored, carried forward* + +**The best xhat solution VALUES live on the xhat spoke, not the hub.** The +xhatshuffle spoke holds them in `spoke.opt.best_solution_cache` (a `ComponentMap` +over all vars) and `spoke.best_inner_bound`; `InnerBoundSpoke.finalize()` +(`spoke.py:293`) loads them back. The hub keeps only the bound scalar. + +Therefore **"keep the best xhat" requires checkpointing the spoke incumbent.** +Checkpointing the hub alone preserves the incumbent *number* but can lose the +*solution* if the crash is late and a short resume does not re-find it. + +Serialize the `ComponentMap` by **variable name** (`{var.name: value}`) — the +`Var` objects are tied to one process's model and cannot cross a restart; rebuild +the map by name lookup on the reconstructed model. + +### 5.4 Stateful extensions — *restored, bit-reproducible (if saved)* + +Several extensions hold trajectory-driving state and **must** be checkpointed or +resume diverges: + +- rho updaters (`mult_rho_updater`, `norm_rho_updater`, `grad_rho`) — multipliers + / previous-gradient history. +- convergers — convergence history. +- **variable-fixing/forcing extensions** — `fixer.py` and `slammer.py` both + decide to pin nonants and then **skip what they have already pinned**, so their + tracking *is* the trajectory. They illustrate two complications: + - **Where the tracking lives differs.** `slammer` keeps `self._slammed` + (`(ndn,i) → value`, sticky) on the **extension object** — covered directly by + the contract below. `fixer` keeps its per-variable convergence counters on the + **scenario model** (`s._mpisppy_data.conv_iter_count`), *not* on `self` — so a + contract that only serializes `self.*` would miss them. The contract must let + an extension serialize its model-attached state too. + - **The tracker and the actual variable state must be restored together and + agree.** Both extensions call `.fix()` / set a value on real nonant vardata. + On resume the rebuilt model has those vars free. If the tracker is restored + ("already fixed/slammed X") but X's actual `.fixed`/value is not (§5.1), they + **disagree**: the extension skips X as done while the solver re-optimizes X + freely — drift, and X is never re-pinned. This is *worse* than not tracking. + So variable fixedness (§5.1) and the extension's records are one unit: + checkpoint and restore them together. + +The `Extension` base class has **no serialization hook today**. We add a +`checkpoint_state()` / `restore_state()` contract (§9, item 3) that can serialize +both `self.*` state and the extension's model-attached state +(`s._mpisppy_data.*`). The same contract serves the hub's extensions and an +xhatter's extensions, because both run through the same `extobject` machinery +(`MultiExtension`). + +**Answer to "do fixing extensions keep tracking across resume?"** Yes — that is +the intent: their records are restored, so they continue to keep track (and +`slammer`'s slams stay sticky, `fixer` does not re-count from zero). But only if +both points above hold; the variable fixedness (§5.1) and model-attached counters +must be restored alongside the extension object, consistently. + +### 5.5 RNG and spoke cursor — *partially restored* + +- xhatshuffle seeds its stream to a fixed `42` and samples **once** + (`xhatshufflelooper_bounder.py:88,94`) — the shuffle is deterministic, so no + RNG state needs saving. +- The `ScenarioCycler` cursor and `xh_iter` are **local variables inside + `main()`** — unreachable from outside. Exact spoke-cursor resume needs those + hoisted onto `self` (Phase 5). Without it, the spoke restarts its cursor; for + correctness this only changes *which* scenario it tries next, not the + preserved best (which is restored from 5.3). +- lagrangian / lagranger spokes use **no RNG**; their bound is deterministic + given the hub's W. State to carry: `_PHIter`, `trivial_bound`, last `bound`, + received `localWs`. + +### 5.6 Geometry / cfg fingerprint — *checkpoint metadata* + +Each per-rank file records `{n_proc, rank, local scenario list}` (and, for the +header, a cfg hash). Resume verifies the current layout matches and **refuses a +mismatch with a clear error** (validated — see §6). + +--- + +## 6. PoC evidence (what is already validated) + +A throwaway PoC (serial + multi-rank + cylinders, farmer, gurobi_persistent) +confirmed the design: + +- **Serial:** resume-from-iter-6 reproduced a full 12-iteration run with + `max|diff| = 0.000e+00` for W, nonants, and rho. Persistent solver survives the + rebuild (Iter0 re-creates + `set_instance`). +- **Multi-rank:** `-np 3` (1 scenario/rank) and uneven `-np 2` (2+1) both resume + bit-identical on every rank. Per-rank rank-tagged files, barrier + atomic + temp-then-rename write. Geometry mismatch (resume `-np 3` ckpt under `-np 2`) + fails with a clear error. +- **Cylinders (PH hub + lagrangian + xhatshuffle):** hub primal resumes + **bit-identical** inside `WheelSpinner` (windows, spokes, async). The best xhat + *solution* (carried on the spoke) is preserved exactly across a crash; + `BestInnerBound` carried exactly; `BestOuterBound` **differed** run-to-run + (async timing) but stayed valid. + +These also surfaced the findings folded into §5 and §9. + +--- + +## 7. Determinism contract (what resume guarantees) + +- **Primal trajectory (W, nonants, rho, xbar): bit-identical.** This is the + correctness guarantee — the resumed run produces the same iterates as an + uninterrupted run. +- **Bounds and incumbent: valid and best-so-far, but NOT bit-reproducible.** They + come from async, timing-dependent spoke interaction. Resume never reports a + *worse* best-so-far than the checkpoint, but it may report a different + (equally valid) bound than the original run would have at the same iteration. + +This distinction must be stated in user docs so a tightened/loosened bound after +resume is not mistaken for a bug. + +--- + +## 8. Configuration and semantics + +Checkpointing is **opt-in** and adds nothing when off. + +- **`--checkpoint-every k`** — write a checkpoint every `k` PH iterations. + - `k` unset or `0` ⇒ checkpointing is **disabled**: the `Checkpointer` + extension is not attached, so there is zero overhead and no files. + - `k ≥ 1` ⇒ checkpoint every `k` iterations **and always at the end of + iteration 0**, regardless of `k`. Iteration 0 establishes the resume + baseline — solvers are created, the trivial bound is computed, the initial + (`W = 0`) solve is done — so a crash anywhere in the `iterk` loop has a valid + resume point even before the first periodic checkpoint, and the iter-0 state + composes with the existing `--iter0-from-pickle` warm-start. +- **`--checkpoint-dir `** — where per-rank checkpoint files and the + manifest are written (see §10). +- **`--resume-from `** (or `--resume`, auto-selecting the latest *complete* + checkpoint from the manifest) — rebuild the wheel and restore that checkpoint. + Resume requires identical geometry (§5.6); a mismatch is refused with a clear + error. + +Hard-kill coverage is the user's tradeoff: small `k` is safest, larger `k` cuts +I/O. Combined with last-*k* retention (§9, item 7), a kill *during* a write still +leaves a usable earlier checkpoint. + +### 8.1 Bundles + +mpi-sppy has **only proper bundles** now — loose bundling (`bundles_per_rank`) +was removed in 2026 (`spbase.py`; see `doc/src/properbundles.rst`). A proper +bundle consumes whole second-stage tree nodes and is a **first-class +subproblem**: it appears in `local_scenarios` with its own `nonant_indices`, and +PH has no separate `local_subproblems` for it. + +Consequently checkpointing **applies to bundled runs directly and uniformly** — +the same code that gathers/restores `W`, nonants, and `rho` by iterating +`local_scenarios` + `nonant_indices` covers a bundle exactly as it covers a plain +scenario. This holds whether bundles are built in memory +(`--scenarios-per-bundle`) or pickled and re-read +(`--pickle-bundles-dir` / `--unpickle-bundles-dir`); both produce proper bundles +and checkpoint identically. Pickling is an independent optimization (skip the +bundle rebuild on resume), not a requirement for checkpointing. + +One cleanup: `_restore_nonants` still carries a 2019 comment that it "will not +work on bundles" (`spopt.py`). That predates proper bundles and refers to the +removed loose mechanism; it should be re-verified and refreshed when bundle +checkpointing is validated (Phase 2). + +--- + +## 9. Core changes required + +These are the touch-points an implementation needs beyond the PoC's +extension/subclass hacks: + +1. **Global iteration counter / resume offset.** `iterk_loop` hardcodes + `for _PHIter in range(1, max+1)` (`phbase.py:1156`), so a resumed run renumbers + from 1 and its checkpoints collide with the pre-crash ones. Add a resume + offset so checkpoint numbering is the global iteration, and so termination + honors the original `max_iterations`. +2. **One-line xhatter write hook.** The xhatter `main()` loop calls no + per-iteration extension hook. Add a single `self.opt.extobject.enditer()` (or a + dedicated checkpoint hook) inside the loop. Then **one `Checkpointer` extension + serves hub and xhatter uniformly** — restore at `post_iter0` / + `post_iter0_after_sync`, write at `enditer` — instead of bespoke spoke + subclasses. (Spoke restore already has a home: `pre_iter0`/`post_iter0` fire + once in `xhat_prep`, `xhatbase.py:36,49`.) +3. **Extension `checkpoint_state` / `restore_state` contract** on the `Extension` + base (no-ops by default; implemented by rho updaters, `fixer`, `slammer`, + convergers). Must serialize both `self.*` state and the extension's + model-attached state (`s._mpisppy_data.*`, e.g. `fixer`'s `conv_iter_count`), + and be restored consistently with the variable fixedness in §5.1 (see §5.4). + The `Checkpointer` aggregates the other extensions' dicts into the per-rank + file. +4. **Full-var snapshot around any xhat/incumbent evaluation.** Evaluating an xhat + fixes the first stage and re-solves; `_restore_nonants` restores *only* + nonants, leaving second-stage recourse vars in the eval state. The checkpointed + primal state (W/nonants/rho) is unaffected (it is gathered from nonants/params), + but any *full-objective* read taken right after an eval is wrong. Snapshot and + restore all vars around an eval, or evaluate on a copy. +5. **Geometry / cfg fingerprint** (§5.6) with a clear refusal on mismatch. +6. **Hub↔spoke snapshot coordination.** A spoke writes its *latest* incumbent and + does not know the hub's iteration number, so a naive set of files is not a + globally-consistent snapshot at a known iteration. The hub's checkpoint barrier + should trigger spoke snapshots (e.g. a checkpoint Field broadcast), so a + resume restores a coherent cross-cylinder state. +7. **Atomic, per-rank, barriered writes** (already in the PoC): each rank writes + only its local state to a rank-tagged file via temp-then-rename, inside a + barrier, so a hard kill never yields a half-written or partial-across-ranks + checkpoint. Retain the last *k* checkpoints (configurable) so a kill *during* + a checkpoint still leaves a usable earlier one. + +--- + +## 10. File layout (proposed) + +``` +/ + manifest.json # cfg hash, n_proc, cylinder map, latest complete iter + iter_/ + hub_rank_.pkl # hub primal + extension state (+ bounds/incumbent obj) + spoke__rank_.pkl# spoke incumbent / bound state +``` + +`manifest.json` names the latest *complete* checkpoint (all ranks + cylinders +flushed); resume reads that. Use plain `pickle` for the numeric state; `dill` +only where models/closures must be serialized. + +--- + +## 11. Phased rollout + +Each phase is a review-sized PR that is green on its own and adds user-visible +value. New tests are wired into `run_coverage.bash` **and** +`test_pr_and_main.yml` in the same commit. + +- **Phase 1 — Serial hub checkpoint/resume.** `Checkpointer` extension + (write at `enditer`, restore at `post_iter0_after_sync`); global iteration + counter / resume offset; geometry+cfg fingerprint; atomic per-rank writes; + CLI flags `--checkpoint-dir`, `--checkpoint-every k` (0/unset ⇒ off; else every + `k` iters **and always after iter 0** — §8), `--resume-from`/`--resume`. Restore + W/nonants/rho. Test: serial farmer kill+resume bit-identical. +- **Phase 2 — Multi-rank + bundles.** Barriers, rank-tagged files, last-*k* + retention. Validate checkpointing with **proper bundles** (bundle = first-class + subproblem, §8.1) and refresh the stale `_restore_nonants` bundle comment. + Test: `mpiexec` farmer kill+resume bit-identical on every rank, incl. uneven + distribution and a `--scenarios-per-bundle` run; mismatch refusal. +- **Phase 3 — Extension state contract.** `checkpoint_state`/`restore_state` on + `Extension` (covering `self.*` and model-attached state); restore nonant + fixedness in the primal set (§5.1); implement the contract for rho updaters, + `fixer`, and `slammer`. Test: PH + norm-rho-updater resumes bit-identical; PH + + `fixer` resumes with the *same* variables fixed (and the same in-progress + counters); PH + `slammer` resumes with slams intact. +- **Phase 4 — Cylinders / spokes.** One-line xhatter write hook; unified + `Checkpointer` on spoke opts; checkpoint the **spoke incumbent (best xhat + solution)**; hub↔spoke snapshot coordination. Test: farmer cylinders + (hub+lagrangian+xhatshuffle) crash+resume — hub primal bit-identical, best xhat + preserved. +- **Phase 5 — Exact spoke continuity (optional).** Hoist `ScenarioCycler`/`xh_iter` + onto `self`; checkpoint cursor (+ RNG getstate if a stream becomes stateful). +- **Phase 6 — Broader coverage.** lagranger, FWPH, subgradient spokes; + scenario-pickle reconstruction path for expensive `scenario_creator`s. + +--- + +## 12. Open questions / risks + +Resolved (see §8): the `--checkpoint-every k` semantics (0/unset ⇒ off; otherwise +every `k` iters and always after iter 0) and bundles (only proper bundles exist; +each is a first-class subproblem, so checkpointing applies uniformly — no special +handling needed). + +Still open: + +- **Coordinated spoke snapshots** (§9, item 6) add a sync point; need to confirm + it does not stall fast spokes or deadlock with `got_kill_signal`. +- **variable_probability / surrogate vars.** The masked-W / prob-0 design must be + re-checked under restore. +- **Cross-geometry resume** is explicitly deferred; revisit if HPC users need to + resume on a different node count. +``` From 5a79b5a711505468d31ac8932eb69ce04b3a8de5 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Sat, 27 Jun 2026 15:17:01 -0700 Subject: [PATCH 02/10] doc: fix receive_innerbounds line reference (1019 -> 1010) The method definition is at spcommunicator.py:1010, not 1019. Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/checkpointing_design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md index 2e4d8acc5..03f338b56 100644 --- a/doc/designs/checkpointing_design.md +++ b/doc/designs/checkpointing_design.md @@ -144,7 +144,7 @@ restored incumbent objective is never regressed because Note: in a cylinders run the hub's `opt.best_solution_obj_val` is often `None` — the inner bound arrives as a scalar via `receive_innerbounds` -(`spcommunicator.py:1019`) and lands in `spcomm.BestInnerBound`. So the +(`spcommunicator.py:1010`) and lands in `spcomm.BestInnerBound`. So the incumbent **objective** is on the hub, but the incumbent **solution** is not (see 5.3). From 29409436ffbc62b2b961f21209015a1cbc182caa Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Wed, 1 Jul 2026 16:06:36 -0700 Subject: [PATCH 03/10] doc: lean into async per-spoke checkpoints (review feedback) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adopt bknueven's review suggestions on #777: - Drop the hub-triggered snapshot barrier. Each spoke checkpoints its own incumbent asynchronously on improvement (reusing _maybe_write_incumbent_on_improvement); the determinism contract already makes bounds/incumbent carried-forward best-so-far, so a globally-consistent cross-cylinder snapshot at iteration k is unnecessary — and this removes the stall/deadlock risk against got_kill_signal. (§5.3, §9 item 6, §11 Phase 4) - Single published generation instead of last-k retention: the manifest flip is the atomic commit point, so one committed generation is enough for kill-safety; older generations are optional history. (§8, §9 item 7, §10) - File layout: hub/ keeps iteration-tagged generations; spokes/ keeps latest-wins per-spoke files overwritten async on improvement. (§10) - Clarify the variable_probability note: restore must reproduce the zero-prob W-masking invariant (surrogate fixed-at-0), not just reload raw W. (§12) Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/designs/checkpointing_design.md | 116 +++++++++++++++++++--------- 1 file changed, 80 insertions(+), 36 deletions(-) diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md index 03f338b56..ad4d49ce6 100644 --- a/doc/designs/checkpointing_design.md +++ b/doc/designs/checkpointing_design.md @@ -159,6 +159,12 @@ Therefore **"keep the best xhat" requires checkpointing the spoke incumbent.** Checkpointing the hub alone preserves the incumbent *number* but can lose the *solution* if the crash is late and a short resume does not re-find it. +The spoke already holds this cache, so it checkpoints it **itself, on its own +schedule** — on each incumbent improvement, independent of the hub's checkpoint +iteration (§9, item 6). No hub↔spoke coordination is needed: the incumbent is a +carried-forward best-so-far value, not part of the bit-reproducible primal +trajectory (§7), so it does not have to align to a global iteration `k`. + Serialize the `ComponentMap` by **variable name** (`{var.name: value}`) — the `Var` objects are tied to one process's model and cannot cross a restart; rebuild the map by name lookup on the reconstructed model. @@ -282,8 +288,9 @@ Checkpointing is **opt-in** and adds nothing when off. error. Hard-kill coverage is the user's tradeoff: small `k` is safest, larger `k` cuts -I/O. Combined with last-*k* retention (§9, item 7), a kill *during* a write still -leaves a usable earlier checkpoint. +I/O. Each checkpoint is published atomically (§9, item 7; §10), so a kill *during* +a write leaves the previous complete checkpoint intact and referenced — never a +half-written one. ### 8.1 Bundles @@ -340,16 +347,32 @@ extension/subclass hacks: but any *full-objective* read taken right after an eval is wrong. Snapshot and restore all vars around an eval, or evaluate on a copy. 5. **Geometry / cfg fingerprint** (§5.6) with a clear refusal on mismatch. -6. **Hub↔spoke snapshot coordination.** A spoke writes its *latest* incumbent and - does not know the hub's iteration number, so a naive set of files is not a - globally-consistent snapshot at a known iteration. The hub's checkpoint barrier - should trigger spoke snapshots (e.g. a checkpoint Field broadcast), so a - resume restores a coherent cross-cylinder state. -7. **Atomic, per-rank, barriered writes** (already in the PoC): each rank writes - only its local state to a rank-tagged file via temp-then-rename, inside a - barrier, so a hard kill never yields a half-written or partial-across-ranks - checkpoint. Retain the last *k* checkpoints (configurable) so a kill *during* - a checkpoint still leaves a usable earlier one. +6. **Async per-spoke incumbent checkpoints — no hub↔spoke coordination.** Each + spoke serializes its *own* best incumbent (the best xhat solution values, §5.3) + and bound on its own schedule — whenever its incumbent improves, reusing the + trigger already in `_maybe_write_incumbent_on_improvement` — to its own + rank-tagged file with the same atomic write (item 7). Spokes are **not** + synchronized to the hub's checkpoint iteration. The determinism contract (§7) + makes bounds and the incumbent best-so-far, not bit-reproducible, so a + globally-consistent "snapshot at iteration `k`" across cylinders is + unnecessary: on resume the hub restores its bit-identical primal state (§5.1) + while each spoke independently reloads its latest incumbent/bound, all of which + are accepted only if improving (`update_best_solution_if_improving`, + `spbase.py:578`). Leaning into the existing asynchrony this way avoids a + hub-triggered snapshot barrier and its stall/deadlock risk against + `got_kill_signal`. +7. **Atomic writes with a single published generation** (per-rank temp-then-rename + is already in the PoC). Each rank writes only its local state to a rank-tagged + temp file and renames it into place; the hub's set of per-rank files is then + published as one checkpoint by atomically rewriting `manifest.json` (itself + temp-then-rename) to point at the new complete generation (§10). Because that + manifest flip is the single commit point, **one committed generation is + enough**: a kill before the flip leaves the previous complete checkpoint intact + and still referenced; a kill after it leaves the new one. The prior generation + can therefore be deleted as soon as the new manifest is in place — there is no + correctness need to retain the last *k*. Keeping a few older generations is an + optional convenience (roll back further, inspect history), not a kill-safety + requirement. --- @@ -357,15 +380,24 @@ extension/subclass hacks: ``` / - manifest.json # cfg hash, n_proc, cylinder map, latest complete iter - iter_/ - hub_rank_.pkl # hub primal + extension state (+ bounds/incumbent obj) - spoke__rank_.pkl# spoke incumbent / bound state + manifest.json # cfg hash, n_proc, cylinder map, latest complete hub iter + hub/ + iter_/ + hub_rank_.pkl # hub primal + extension state (+ bounds/incumbent obj) + spokes/ + spoke__rank_.pkl # each spoke's latest incumbent/bound, overwritten + # asynchronously on improvement (§9, item 6) ``` -`manifest.json` names the latest *complete* checkpoint (all ranks + cylinders -flushed); resume reads that. Use plain `pickle` for the numeric state; `dill` -only where models/closures must be serialized. +The hub writes iteration-tagged generations under `hub/`; each spoke keeps a +single latest-wins file under `spokes/` that it overwrites (atomically) whenever +its incumbent improves — the two are deliberately *not* aligned to a common +iteration (§9, item 6). `manifest.json` is the single commit point: it names the +latest *complete* hub generation (all hub ranks flushed), and publishing a new +one means atomically renaming a freshly written `manifest.json` into place (§9, +item 7). Resume reads whatever the manifest currently names, plus whatever each +spoke last committed under `spokes/`. Use plain `pickle` for the numeric state; +`dill` only where models/closures must be serialized. --- @@ -381,11 +413,12 @@ value. New tests are wired into `run_coverage.bash` **and** CLI flags `--checkpoint-dir`, `--checkpoint-every k` (0/unset ⇒ off; else every `k` iters **and always after iter 0** — §8), `--resume-from`/`--resume`. Restore W/nonants/rho. Test: serial farmer kill+resume bit-identical. -- **Phase 2 — Multi-rank + bundles.** Barriers, rank-tagged files, last-*k* - retention. Validate checkpointing with **proper bundles** (bundle = first-class - subproblem, §8.1) and refresh the stale `_restore_nonants` bundle comment. - Test: `mpiexec` farmer kill+resume bit-identical on every rank, incl. uneven - distribution and a `--scenarios-per-bundle` run; mismatch refusal. +- **Phase 2 — Multi-rank + bundles.** Barriers, rank-tagged files, single-generation + atomic publish via the manifest (§9, item 7). Validate checkpointing with + **proper bundles** (bundle = first-class subproblem, §8.1) and refresh the stale + `_restore_nonants` bundle comment. Test: `mpiexec` farmer kill+resume + bit-identical on every rank, incl. uneven distribution and a + `--scenarios-per-bundle` run; mismatch refusal. - **Phase 3 — Extension state contract.** `checkpoint_state`/`restore_state` on `Extension` (covering `self.*` and model-attached state); restore nonant fixedness in the primal set (§5.1); implement the contract for rho updaters, @@ -393,10 +426,10 @@ value. New tests are wired into `run_coverage.bash` **and** `fixer` resumes with the *same* variables fixed (and the same in-progress counters); PH + `slammer` resumes with slams intact. - **Phase 4 — Cylinders / spokes.** One-line xhatter write hook; unified - `Checkpointer` on spoke opts; checkpoint the **spoke incumbent (best xhat - solution)**; hub↔spoke snapshot coordination. Test: farmer cylinders - (hub+lagrangian+xhatshuffle) crash+resume — hub primal bit-identical, best xhat - preserved. + `Checkpointer` on spoke opts; each spoke checkpoints its own **incumbent (best + xhat solution)** asynchronously on improvement — no hub↔spoke coordination (§9, + item 6). Test: farmer cylinders (hub+lagrangian+xhatshuffle) crash+resume — hub + primal bit-identical, best xhat preserved. - **Phase 5 — Exact spoke continuity (optional).** Hoist `ScenarioCycler`/`xh_iter` onto `self`; checkpoint cursor (+ RNG getstate if a stream becomes stateful). - **Phase 6 — Broader coverage.** lagranger, FWPH, subgradient spokes; @@ -406,17 +439,28 @@ value. New tests are wired into `run_coverage.bash` **and** ## 12. Open questions / risks -Resolved (see §8): the `--checkpoint-every k` semantics (0/unset ⇒ off; otherwise -every `k` iters and always after iter 0) and bundles (only proper bundles exist; -each is a first-class subproblem, so checkpointing applies uniformly — no special -handling needed). +Resolved: + +- **`--checkpoint-every k` semantics** (§8): 0/unset ⇒ off; otherwise every `k` + iters and always after iter 0. +- **Bundles** (§8.1): only proper bundles exist; each is a first-class subproblem, + so checkpointing applies uniformly — no special handling needed. +- **Spoke snapshot coordination** (§9, item 6): resolved by *not* coordinating. + Spokes checkpoint their own incumbents asynchronously on improvement; the + determinism contract (§7) makes this sufficient and removes the sync point that + would otherwise risk stalling fast spokes or deadlocking with `got_kill_signal`. +- **Checkpoint retention** (§9, item 7): a single manifest-published generation is + enough for kill-safety; last-*k* retention is optional history, not required. Still open: -- **Coordinated spoke snapshots** (§9, item 6) add a sync point; need to confirm - it does not stall fast spokes or deadlock with `got_kill_signal`. -- **variable_probability / surrogate vars.** The masked-W / prob-0 design must be - re-checked under restore. +- **variable_probability / surrogate vars.** With variable probabilities mpi-sppy + masks `W` (but not prox) for zero-probability nonants and assumes each such + nonant's surrogate var is fixed at 0. Restore must **reproduce that invariant, + not just reload the raw `W` array**: confirm the zero-probability mask is + re-applied and the surrogate vars come back fixed-at-0 (§5.1 fixedness), so a + masked component is never treated as live on resume. Needs verification; among + the spokes only `xhatxbar` supports variable_probability today. - **Cross-geometry resume** is explicitly deferred; revisit if HPC users need to resume on a different node count. ``` From 895d906bb93d62945b375894f08731b5edcda318 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Fri, 10 Jul 2026 14:17:30 -0700 Subject: [PATCH 04/10] doc(checkpointing): pivot to dill-the-scenario-model backend for planned stop/resume Re-center the checkpoint/resume design on the concrete use case: a multi-day run that intentionally stops at end of day and resumes the next morning, with large MIP scenarios, in the same environment, checkpointing about twice as often as it resumes. At that (infrequent) checkpoint cadence the per-write overhead of dilling the whole scenario model is negligible, and same-day resume makes dill's version fragility irrelevant -- so dilling the mid-run scenario models becomes the recommended backend. It captures W/rho/nonants/fixedness, the second-stage MIP warm start (via the existing warmstart_subproblems / solution_available path), the linearized-prox cuts + ProxApproxManager, and model-attached extension state (fixer's counter) in one consistent shot, and avoids re-running an expensive scenario_creator on resume. Keep the leaf-rebuild path as the low-cost backend (tiny, fast, version-robust checkpoints) for small/cheap-creator runs or frequent kill-safety writes; select via --checkpoint-backend {dill-model, leaf}. Reconstructing the scaffolding (comms/windows/solvers) and restoring the non-model state (iteration counter, hub bounds/incumbent, spoke best-xhat by name, extension- object state, cursor/RNG) is common to both backends. Soften the determinism contract for MIPs (correct warm-started continuation + preserved incumbent, not bit-reproducible), shift the primary trigger to end-of-run/on-signal, and flag the one load-bearing unvalidated assumption -- that a mid-run MIP model round-trips through dill -- as Phase 1's first job. Co-Authored-By: Claude Opus 4.8 --- doc/designs/checkpointing_design.md | 786 ++++++++++++++++------------ 1 file changed, 447 insertions(+), 339 deletions(-) diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md index ad4d49ce6..6ffb40454 100644 --- a/doc/designs/checkpointing_design.md +++ b/doc/designs/checkpointing_design.md @@ -1,80 +1,146 @@ # Checkpoint / Resume for mpi-sppy — Design -Status: **draft** (PoC validated; not yet implemented in the library). -Scope: checkpoint a running mpi-sppy job so a killed/interrupted run can resume -where it left off. Must work on multiple MPI ranks and for cylinder -(hub-and-spoke) runs. +Status: **draft** (framework PoC validated; dill-reload backend designed, not yet +validated). Scope: checkpoint a running mpi-sppy job so it can be stopped and +resumed later. Must work on multiple MPI ranks and for cylinder (hub-and-spoke) +runs. --- ## 1. Goals and non-goals +**Primary use case.** A long (multi-day) run that is **intentionally stopped and +resumed** on a schedule — e.g. a three-day study that ends each day and picks up +the next morning on the same cluster. Checkpoints are **infrequent** (a small +number over the whole run; roughly twice as many writes as resumes) and resumes +are planned, not crash-driven. The scenarios are **large MIPs**. + +That regime drives the design decisions below (dill the scenario models, restore +a MIP warm start), and it is worth stating up front because a *different* use +case — frequent checkpoints purely for hard-kill safety — would push toward a +lighter, leaf-data-only checkpoint (still supported; §4, §11 Phase 6). + **Goals** - Resume a Progressive Hedging (PH) run — serial, multi-rank, or full cylinders - (hub + spokes) — after a crash or a deliberate stop. -- Preserve the **best feasible solution found so far (the best xhat), not just - the best bound.** -- Survive a hard kill (`kill -9`, node failure, walltime), not only a clean - shutdown. -- Add no measurable overhead when checkpointing is off, and modest, tunable - overhead when on. + (hub + spokes) — after a planned stop (and, as a bonus, after a crash). +- **Continue the optimization as if it had not stopped**, warm-started from the + last iterate, without losing the **best feasible solution found so far (the best + xhat), not just the best bound.** +- Survive a hard kill (`kill -9`, node failure, walltime) as a secondary benefit, + via atomic publication (§9). +- Add no measurable overhead when checkpointing is off, and — because checkpoints + are infrequent here — tolerate a heavier per-checkpoint cost in exchange for a + complete, warm-startable restore. **Non-goals (initially)** - Resuming across a *different* rank count or scenario-to-rank distribution - (cross-geometry remap). The first cut requires identical geometry and refuses - a mismatch with a clear error. -- Bit-identical reproduction of *bounds* (see §7 — bounds are async and not - reproducible; only the primal trajectory is). + (cross-geometry remap). The first cut requires identical geometry and refuses a + mismatch with a clear error. +- **Bit-identical reproduction for MIPs.** Multi-threaded MIP solves are not + deterministic and admit multiple optima, so a resumed MIP run is *not* + bit-reproducible against a hypothetical uninterrupted run. The guarantee is + correct, warm-started continuation with the incumbent preserved (§7). (A + deterministic LP/QP solve *can* be bit-identical under the leaf-rebuild backend + — that is what the PoC showed — but it is not the target here.) +- Bit-identical reproduction of *bounds* (§7 — bounds are async and not + reproducible; carried forward as best-so-far). +- Robustness across a library/env upgrade between stop and resume. The use case + resumes in the **same environment** the next day, so a dill-based checkpoint + (welded to the current Pyomo/mpi-sppy/model code) is acceptable. Cross-version + resume is out of scope. - APH. A C++ APH is expected to replace the Python `opt/aph.py`; PH-family only. --- -## 2. Why "pickle every object with dill" does not work +## 2. What to serialize, and what not to + +There are two very different "just dill it" ideas, and they get opposite answers. + +### 2.1 Do NOT dill the opt/hub object graph -The obvious idea — `dill.dump()` the whole opt/hub object graph and reload — is -not viable, because the core objects are built around **live, non-serializable -OS/MPI/solver handles**, not data: +`dill.dump()` of the whole opt/hub object and reload is not viable — the core +objects are built around **live, non-serializable OS/MPI/solver handles**, not +data: - **MPI communicators** — `SPBase.mpicomm`, `SPBase.comms` (`spbase.py`), and the `fullcomm`/`strata_comm`/`cylinder_comm` on the spcomm - (`cylinders/spcommunicator.py`). Handles into the running MPI runtime; - meaningless once the process exits. + (`cylinders/spcommunicator.py`). Handles into a running MPI runtime; meaningless + once the process exits. - **MPI RMA windows and `MPI.Alloc_mem` buffers** — `SPWindow` (`cylinders/spwindow.py`) and the `FieldArray` send/receive buffers. Kernel shared-memory regions. -- **Persistent solver handles** — `s._solver_plugin` for - gurobi/cplex/xpress persistent interfaces (`spopt.py`). A C handle + license - session. - -The moment you write a custom `__getstate__`/`__reduce__` to drop and rebuild -those, you have conceded the "just pickle it" dream and are doing -reconstruct-scaffolding + restore-state anyway — only with the checkpoint format -welded to the live object graph (fragile across versions). Better to do that -split explicitly. - -dill *is* useful for the leaf data (Pyomo scenario models, closures), and the -repo already uses it that way — see §4. +- **Persistent solver handles** — `s._solver_plugin` for gurobi/cplex/xpress + persistent interfaces (`spopt.py`). A C handle + license session. + +A resumed run launches a **fresh process** (new MPI job) anyway, so these must be +**reconstructed via normal startup** regardless of how anything else is restored. +That is not negotiable and not something a checkpoint can carry. + +### 2.2 DO dill the scenario models (the recommended backend here) + +The narrower idea — dill each **scenario Pyomo model** — is not only viable, it is +the right backend for this use case. The repo already dills *clean* scenario +models for the scenario-pickle path (§4); the only extension is to dill them +**mid-run**. The single non-serializable attribute on a scenario model is +`s._solver_plugin`, which we drop before writing and rebuild with `set_instance` +on resume (a dance we already do — the reconstruct step needs it regardless); +`_mpisppy_data` is a `pyo.Block` on the model with **no** back-references to the +comms or opt object. + +Dilling the mid-run model captures, in one shot and mutually consistent, +everything that lives *on* the scenario model: + +- the dual weights `W`, `rho`, `xbars` (on `s._mpisppy_model`); +- nonant values **and fixedness** (from variable-fixing/forcing extensions); +- **second-stage (recourse) variable values → a MIP warm start** (§5.2); +- the proximal-approximation `xsqvar` and its accumulated `xsqvar_cuts`, plus the + `ProxApproxManager` bookkeeping on `_mpisppy_data` (§5.3) — the linearized prox + is likely in use for large MIPs (a MIQP prox is often intractable), and dill + brings the cuts back for free instead of replaying them; +- model-attached extension state, e.g. `fixer`'s per-variable `conv_iter_count` + on `s._mpisppy_data`. + +The costs that argued against this backend elsewhere — **version fragility** (dill +serializes by class/closure reference) and **per-checkpoint overhead** (full +models are large) — are both moot for this use case: the resume is same-environment +and the checkpoints are infrequent, so a heavy write paid a handful of times over +three days is negligible. And it *avoids re-running an expensive `scenario_creator`* +on every resume, which for large models is itself a real saving. + +The alternative — rebuild each model via `scenario_creator` and overlay the state +as leaf data (arrays/name→value maps) — is retained as the **low-cost backend**: +its checkpoints are tiny and fast (a handful of `O(first-stage)` arrays, no model +structure) and version-robust (plain numbers, not pickled classes). That makes it +the right choice for small scenarios, cheap creators, or *frequent* kill-safety +checkpoints where a per-write model dump would hurt — the mirror image of this +use case. It is also what the PoC validated (§6, §11 Phase 6). The two backends +share the same framework and manifest; only the scenario-model restore step +differs. --- ## 3. Approach: reconstruct the scaffolding, restore the state -A checkpoint is **not** a snapshot of the object graph. It is the **algorithmic -state** needed to continue. On resume: +A checkpoint is **not** a snapshot of the object graph. On resume: 1. **Reconstruct the scaffolding** via the normal startup path — comms, RMA - windows, persistent solvers, and the Pyomo scenario models (from - `scenario_creator`, or from the existing scenario-pickle path). This is - exactly what a fresh run already does. -2. **Restore the state** into that fresh scaffolding: iteration counter, dual - weights, nonant values, rho, the incumbent (best xhat), aggregated bounds, - and any stateful-extension internals. - -This is the design the library *already leans toward* (it pickles scenario -models alone, never the comms/solvers) — we are extending it from "warm-start -iteration 0" to "resume at iteration k". + windows, and persistent solvers. This is exactly what a fresh run already does. +2. **Restore the scenario-model state** via the chosen backend: + - **dill-reload (recommended here):** load each rank's dilled mid-run scenario + models; rebuild `_solver_plugin` (`set_instance`) and mark + `solution_available` so the first solve warm-starts (§5.2). Because the + reloaded model already carries the spliced W/prox objective and the prox + cuts, startup must **skip** re-attaching them (§9, item 2). + - **leaf-rebuild (alternative):** rebuild each model via `scenario_creator`, + then overlay W / rho / nonant values+fixedness / prox `cut_values` from the + checkpoint. +3. **Restore the non-model state** — the pieces that do *not* live on a scenario + model and so are never captured by dilling models: the global iteration + counter, hub bounds/incumbent objective, the spoke incumbent (best xhat), the + internal state of extension *objects*, and cursor/RNG. These are always small + leaf data (§5.4–5.6). --- @@ -82,188 +148,227 @@ iteration 0" to "resume at iteration k". - **Scenario-model pickling (dill):** `utils/pickle_bundle.py` (`dill_pickle`/`dill_unpickle`) and `generic/scenario_io.py` pickle each - scenario Pyomo model *alone*. Iter-0 solution + bounds can be baked into - `pickle_metadata` and warm-started via `--iter0-from-pickle`. Precedent for - persisting algorithmic state, and a ready path for reconstructing expensive / - nondeterministic scenario models. + scenario Pyomo model *alone*, driven by `--pickle-scenarios-dir` / + `--unpickle-scenarios-dir` (with `iter0_before_pickle` baking an iter-0 solve + into the pickle). The dill-reload backend (§2.2) is exactly this path + **generalized from iter-0 to iter-k** — pickle the model as it stands at the + checkpoint, not only after iter 0. +- **Warm-start plumbing:** `spopt.py` already supports warm-starting subproblem + solves — the `warmstart_subproblems` option plus `WarmstartStatus.PRIOR_SOLUTION` + use a warm start when `s._mpisppy_data.solution_available` is set + (`spopt.py:301-305`). Restoring the model's variable values and setting + `solution_available=True` feeds the restored MIP solution straight into this + path — no new solver code. - **W / xbar persistence:** `utils/w_utils/wxbarwriter.py` (writes in - `post_everything`) / `wxbarreader.py` (reads in `pre_iter0`) round-trip the - dual weights `W` and the consensus `xbar` as CSV, as extensions. Covers part - of the restore set; does **not** cover rho, the iteration counter, bounds, the - incumbent, or RNG. + `post_everything`) / `wxbarreader.py` (reads in `pre_iter0`) round-trip `W` and + `xbar` as CSV. A precedent for the leaf-rebuild backend; unnecessary under + dill-reload (W/xbar ride in the model). - **Incumbent-to-disk (spokes):** `cylinders/spoke.py` - `_maybe_write_incumbent_on_improvement` (`--incumbent-on-improvement-filename-prefix`) - already writes the first-stage solution on each improvement — a reference for - serializing the incumbent. + `_maybe_write_incumbent_on_improvement` + (`--incumbent-on-improvement-filename-prefix`) already writes the first-stage + solution on each improvement — the reference for serializing the incumbent. --- ## 5. State inventory -The key question for each piece: **reconstructed** (rebuilt by startup, no save -needed) vs **restored** (must be checkpointed), and if restored, is it -**bit-reproducible** on resume or only **carried forward** as a valid -best-so-far value? +For each piece: is it **reconstructed** (rebuilt by startup, no save), **carried +in the dilled model** (dill-reload backend), or **restored as non-model leaf +data** (always)? And if restored, is it **carried forward** as a valid +best-so-far value or (LP/QP only) potentially bit-reproducible? -### 5.1 Hub PH primal state — *restored, bit-reproducible* +### 5.1 Hub PH primal state — *in the dilled model* -The hub's primal trajectory is pure synchronous PH and is **independent of the -spokes** (lagrangian only contributes an outer bound; xhatshuffle only an -incumbent; neither perturbs the iterate). Minimum set, per local scenario: +The hub's primal trajectory is pure synchronous PH, independent of the spokes +(lagrangian only contributes an outer bound; xhatshuffle only an incumbent). +Per local scenario, all of the following live **on the scenario model** and are +therefore captured by dilling it: | State | Where it lives | Note | |---|---|---| -| `_PHIter` | `phbase` | iteration counter | -| `W[ndn_i]` (accumulated duals) | `s._mpisppy_model.W` | **must** restore — `Update_W` accumulates, so it is not recomputable | -| nonant values | `s._mpisppy_data.nonant_indices` vardata `_value` | restoring these lets `Compute_Xbar` reproduce `xbar`; `xbar` itself need not be saved | -| nonant **fixedness** | same vardata `.fixed` (+ the fixed value) | **must** restore — variable-fixing/forcing extensions (`fixer`, `slammer`) leave nonants fixed; a rebuilt model has them free. See §5.4 | -| `rho[ndn_i]` | `s._mpisppy_model.rho` | restore (rho-updaters mutate it) | +| `W[ndn_i]` (accumulated duals) | `s._mpisppy_model.W` | `Update_W` accumulates — not recomputable; must be preserved | +| nonant values | nonant vardata `_value` | drive `Compute_Xbar`; `xbar` itself need not be separately saved | +| nonant **fixedness** (+ fixed value) | nonant vardata `.fixed` | `fixer`/`slammer` leave nonants fixed; must survive (§5.5) | +| `rho[ndn_i]` | `s._mpisppy_model.rho` | rho-updaters mutate it | +| `xbars[ndn_i]` | `s._mpisppy_model.xbars` | consensus target | | smoothing `z/p/beta` | `s._mpisppy_model` | only if `--smoothing` | -Helpers already exist: `_populate_W_cache` / `W_from_flat_list` (`phbase.py`), -`_save_nonants` / `_restore_nonants` (`spopt.py`). Note `_save_nonants` already -captures fixedness (`fixedness_cache`) — the checkpoint must gather it too (the -PoC captured only `_value`). - -**Restore point:** between `Iter0()` and `iterk_loop()`. In serial this can be a -driver call; in cylinders the hub runs `ph_main` internally, so restore happens -in the **`post_iter0_after_sync`** extension hook (fires at the end of `Iter0`, -`phbase.py:1079`). Iteration 1 of the resumed loop then reproduces what would -have been iteration k+1 — verified bit-identical (§6). - -### 5.2 Hub bounds + incumbent objective — *restored, carried forward* - -- `spcomm.BestInnerBound`, `spcomm.BestOuterBound` — aggregated bounds. -- `opt.best_bound_obj_val`, `opt.best_solution_obj_val`. - -These are products of the **async** spoke interaction; their *timing* is not -reproducible, so they are carried forward (restored as best-so-far). They remain -valid: a restored looser bound is simply improved again by the fresh spokes; a -restored incumbent objective is never regressed because -`update_best_solution_if_improving` (`spbase.py:578`) only accepts improvements. - -Note: in a cylinders run the hub's `opt.best_solution_obj_val` is often `None` — -the inner bound arrives as a scalar via `receive_innerbounds` -(`spcommunicator.py:1010`) and lands in `spcomm.BestInnerBound`. So the -incumbent **objective** is on the hub, but the incumbent **solution** is not (see -5.3). - -### 5.3 Spoke incumbent — *the best xhat solution* — *restored, carried forward* - -**The best xhat solution VALUES live on the xhat spoke, not the hub.** The -xhatshuffle spoke holds them in `spoke.opt.best_solution_cache` (a `ComponentMap` -over all vars) and `spoke.best_inner_bound`; `InnerBoundSpoke.finalize()` -(`spoke.py:293`) loads them back. The hub keeps only the bound scalar. - -Therefore **"keep the best xhat" requires checkpointing the spoke incumbent.** -Checkpointing the hub alone preserves the incumbent *number* but can lose the -*solution* if the crash is late and a short resume does not re-find it. - -The spoke already holds this cache, so it checkpoints it **itself, on its own -schedule** — on each incumbent improvement, independent of the hub's checkpoint -iteration (§9, item 6). No hub↔spoke coordination is needed: the incumbent is a -carried-forward best-so-far value, not part of the bit-reproducible primal -trajectory (§7), so it does not have to align to a global iteration `k`. - -Serialize the `ComponentMap` by **variable name** (`{var.name: value}`) — the -`Var` objects are tied to one process's model and cannot cross a restart; rebuild -the map by name lookup on the reconstructed model. - -### 5.4 Stateful extensions — *restored, bit-reproducible (if saved)* - -Several extensions hold trajectory-driving state and **must** be checkpointed or -resume diverges: - -- rho updaters (`mult_rho_updater`, `norm_rho_updater`, `grad_rho`) — multipliers - / previous-gradient history. -- convergers — convergence history. -- **variable-fixing/forcing extensions** — `fixer.py` and `slammer.py` both - decide to pin nonants and then **skip what they have already pinned**, so their - tracking *is* the trajectory. They illustrate two complications: - - **Where the tracking lives differs.** `slammer` keeps `self._slammed` - (`(ndn,i) → value`, sticky) on the **extension object** — covered directly by - the contract below. `fixer` keeps its per-variable convergence counters on the - **scenario model** (`s._mpisppy_data.conv_iter_count`), *not* on `self` — so a - contract that only serializes `self.*` would miss them. The contract must let - an extension serialize its model-attached state too. - - **The tracker and the actual variable state must be restored together and - agree.** Both extensions call `.fix()` / set a value on real nonant vardata. - On resume the rebuilt model has those vars free. If the tracker is restored - ("already fixed/slammed X") but X's actual `.fixed`/value is not (§5.1), they - **disagree**: the extension skips X as done while the solver re-optimizes X - freely — drift, and X is never re-pinned. This is *worse* than not tracking. - So variable fixedness (§5.1) and the extension's records are one unit: - checkpoint and restore them together. - -The `Extension` base class has **no serialization hook today**. We add a -`checkpoint_state()` / `restore_state()` contract (§9, item 3) that can serialize -both `self.*` state and the extension's model-attached state -(`s._mpisppy_data.*`). The same contract serves the hub's extensions and an -xhatter's extensions, because both run through the same `extobject` machinery -(`MultiExtension`). - -**Answer to "do fixing extensions keep tracking across resume?"** Yes — that is -the intent: their records are restored, so they continue to keep track (and -`slammer`'s slams stay sticky, `fixer` does not re-count from zero). But only if -both points above hold; the variable fixedness (§5.1) and model-attached counters -must be restored alongside the extension object, consistently. - -### 5.5 RNG and spoke cursor — *partially restored* +Under **leaf-rebuild**, this set is instead gathered/restored explicitly — helpers +already exist: `_populate_W_cache`/`W_from_flat_list` (`phbase.py`), +`_save_nonants`/`_restore_nonants` (`spopt.py`, which already captures fixedness in +`fixedness_cache`). + +**Restore point:** after the scaffolding exists and models are loaded, before the +`iterk` loop. In serial this can be a driver call; in cylinders the hub runs +`ph_main` internally, so restore happens in the **`post_iter0_after_sync`** +extension hook (end of `Iter0`, `phbase.py:1079`). + +### 5.2 Recourse variable values — *warm start, in the dilled model* + +The second-stage (recourse) variable values are the bulk of a large scenario and +are **not** part of the algorithmic primal state (only the nonants and params +are). Their value is as a **MIP warm start**: restoring them and setting +`s._mpisppy_data.solution_available = True` makes the first resumed subproblem +solve start from the last iterate's solution via the existing +`warmstart_subproblems` path (§4). For large MIPs this can save substantial +branch-and-bound time on the first solve after each resume. + +Because they ride in the dilled model, they cost nothing extra here. Under the +leaf-rebuild backend they would be an **optional** all-var snapshot (off by +default): a per-checkpoint O(scenario) cost that only pays off for MIP/simplex +warm starts and is pure overhead for barrier solves — a bad trade when +checkpoints are frequent, which is why it is opt-in there. + +**Caveat (both backends):** an xhat/incumbent evaluation fixes the first stage and +re-solves, leaving recourse vars in the *eval* state; `_restore_nonants` restores +only nonants. So the model must be checkpointed (dilled) at a point where its +recourse values reflect the true last subproblem solve, not a mid-eval state — +i.e. snapshot before an eval corrupts them, or evaluate on a copy (§9, item 4). + +### 5.3 Proximal-approximation cuts (`--linearize-proximal-terms`) — *in the dilled model* + +When the linearized prox is on, `attach_PH_to_objective` builds, per scenario +(`phbase.py:892-894`): + +- `s._mpisppy_model.xsqvar` — the epigraph var for `x²`; +- `s._mpisppy_model.xsqvar_cuts` — a `Constraint` that accumulates one linear cut + per visited x-location (`prox_approx.py:232,278,291`); +- `s._mpisppy_data.xsqvar_prox_approx[ndn_i]` — a `ProxApproxManager` whose + bookkeeping (`cut_index`, the sorted `cut_values` array; `prox_approx.py:46-47`) + decides when a new cut is redundant. + +The cut *constraints* live on the model; the manager's bookkeeping lives on +`_mpisppy_data` (a Block on the model). **Dilling the model captures both, and +keeps them consistent** (the manager's references and the constraint set come back +in lockstep) — no replay, no re-binding. + +Under **leaf-rebuild**, neither survives (`attach_PH_to_objective` rebuilds +`xsqvar_cuts` empty). Each cut is fully determined by its x-location (continuous: +`xsqvar ≥ 2v·x − v²`; discrete: integer-keyed), so the checkpoint stores only the +per-nonant `cut_values` arrays and **replays** `add_cut` into the fresh model on +restore. Skipping this leaves resume correct (cuts regenerate lazily via +`check_tol_add_cut`) but coarser initially — fine for MIPs (not bit-reproducible +anyway), relevant only if an LP/QP run wants bit-identity. + +### 5.4 Hub bounds + incumbent, and the spoke incumbent — *non-model leaf data, carried forward* + +None of this lives on a hub scenario model, so it is restored as leaf data under +**both** backends: + +- `spcomm.BestInnerBound`, `spcomm.BestOuterBound`; `opt.best_bound_obj_val`, + `opt.best_solution_obj_val`. Products of **async** spoke interaction — their + timing is not reproducible, so they are carried forward as best-so-far. They + stay valid: a restored looser bound is improved again; a restored incumbent + objective is never regressed because `update_best_solution_if_improving` + (`spbase.py:578`) only accepts improvements. In cylinders the hub's + `best_solution_obj_val` is often `None` — the inner bound arrives as a scalar via + `receive_innerbounds` (`spcommunicator.py:1010`) into `spcomm.BestInnerBound`. +- **The best xhat SOLUTION values live on the xhat spoke**, in + `spoke.opt.best_solution_cache` (a `ComponentMap` over all vars) + + `spoke.best_inner_bound`; `InnerBoundSpoke.finalize()` (`spoke.py:293`) loads + them back. So **"keep the best xhat" requires checkpointing the spoke + incumbent**, not just hub bounds. The spoke checkpoints its own cache **on its + own schedule** — on each improvement, reusing + `_maybe_write_incumbent_on_improvement`, independent of the hub checkpoint (§9, + item 6). Serialize the `ComponentMap` **by variable name** (`{var.name: value}`) + and rebuild by name lookup on the reconstructed model. + +### 5.5 Stateful extensions — *split: object state is leaf data, model state rides in the dill* + +Several extensions hold trajectory-driving state and **must** be restored or resume +diverges: + +- rho updaters (`mult_rho_updater`, `norm_rho_updater`, `grad_rho`), convergers — + multiplier / gradient / convergence history, kept on the **extension object**. +- variable-fixing/forcing extensions — `fixer.py` and `slammer.py` pin nonants and + then **skip what they already pinned**, so their tracking *is* the trajectory. + They span both storage locations: `slammer._slammed` is on the **extension + object**, while `fixer`'s per-variable `conv_iter_count` is on the **scenario + model** (`s._mpisppy_data`). + +Consequences: + +- **Model-attached tracker state (`fixer`) rides in the dilled model** for free — + consistent with the nonant fixedness it pairs with (§5.1). Under leaf-rebuild it + must be gathered explicitly. +- **Extension-object state is never on a model**, so it needs a serialization + contract regardless of backend. The `Extension` base has none today; add + `checkpoint_state()` / `restore_state()` (no-ops by default; implemented by rho + updaters, `fixer`, `slammer`, convergers), aggregated by the `Checkpointer` + (§9, item 3). The same contract serves hub and xhatter extensions + (`MultiExtension`). +- **The tracker and the actual variable state must agree.** Restoring "already + fixed X" without X's real `.fixed`/value (§5.1) makes the extension skip X while + the solver frees it — worse than no tracking. dill-reload gives this for free + (both come back together); leaf-rebuild must restore fixedness and the tracker as + one unit. + +### 5.6 RNG and spoke cursor — *non-model leaf data, partially restored* - xhatshuffle seeds its stream to a fixed `42` and samples **once** - (`xhatshufflelooper_bounder.py:88,94`) — the shuffle is deterministic, so no - RNG state needs saving. -- The `ScenarioCycler` cursor and `xh_iter` are **local variables inside - `main()`** — unreachable from outside. Exact spoke-cursor resume needs those - hoisted onto `self` (Phase 5). Without it, the spoke restarts its cursor; for - correctness this only changes *which* scenario it tries next, not the - preserved best (which is restored from 5.3). -- lagrangian / lagranger spokes use **no RNG**; their bound is deterministic - given the hub's W. State to carry: `_PHIter`, `trivial_bound`, last `bound`, - received `localWs`. - -### 5.6 Geometry / cfg fingerprint — *checkpoint metadata* - -Each per-rank file records `{n_proc, rank, local scenario list}` (and, for the -header, a cfg hash). Resume verifies the current layout matches and **refuses a -mismatch with a clear error** (validated — see §6). + (`xhatshufflelooper_bounder.py:88,94`) — deterministic, no RNG state to save. +- The `ScenarioCycler` cursor and `xh_iter` are **local variables inside `main()`** + — unreachable. Exact spoke-cursor resume needs them hoisted onto `self` (Phase + 5). Without it the spoke restarts its cursor; this only changes *which* scenario + it tries next, not the preserved best (restored from §5.4). +- lagrangian / lagranger spokes use **no RNG**; their bound is deterministic given + the hub's `W`. State to carry: `_PHIter`, `trivial_bound`, last `bound`, received + `localWs`. + +### 5.7 Geometry / cfg fingerprint — *checkpoint metadata* + +Each per-rank file records `{n_proc, rank, local scenario list}` and a cfg hash. +Resume verifies the current layout matches and **refuses a mismatch with a clear +error** (validated — §6). --- -## 6. PoC evidence (what is already validated) +## 6. PoC evidence (what is validated, and what is not) -A throwaway PoC (serial + multi-rank + cylinders, farmer, gurobi_persistent) -confirmed the design: +A throwaway PoC (serial + multi-rank + cylinders, farmer LP, gurobi_persistent) +validated the **framework and the leaf-rebuild backend**: - **Serial:** resume-from-iter-6 reproduced a full 12-iteration run with - `max|diff| = 0.000e+00` for W, nonants, and rho. Persistent solver survives the - rebuild (Iter0 re-creates + `set_instance`). -- **Multi-rank:** `-np 3` (1 scenario/rank) and uneven `-np 2` (2+1) both resume - bit-identical on every rank. Per-rank rank-tagged files, barrier + atomic - temp-then-rename write. Geometry mismatch (resume `-np 3` ckpt under `-np 2`) - fails with a clear error. + `max|diff| = 0.000e+00` for W, nonants, rho (bit-identical — LP, deterministic + solver). Persistent solver survives the rebuild (Iter0 re-creates + + `set_instance`). +- **Multi-rank:** `-np 3` (1 scenario/rank) and uneven `-np 2` (2+1) resume + bit-identical on every rank; per-rank rank-tagged files, barrier + atomic + temp-then-rename write. Geometry mismatch fails with a clear error. - **Cylinders (PH hub + lagrangian + xhatshuffle):** hub primal resumes - **bit-identical** inside `WheelSpinner` (windows, spokes, async). The best xhat - *solution* (carried on the spoke) is preserved exactly across a crash; - `BestInnerBound` carried exactly; `BestOuterBound` **differed** run-to-run - (async timing) but stayed valid. + bit-identical inside `WheelSpinner`; the best xhat *solution* (on the spoke) is + preserved exactly; `BestInnerBound` carried exactly; `BestOuterBound` differed + run-to-run (async) but stayed valid. + +**Not yet validated (the primary backend for this use case):** + +- Dilling and reloading a **mid-run** scenario model — spliced W/prox objective, + `LinearExpression` cut bodies, mutable params, post-`set_instance` — round-trips + cleanly. The repo only dills *clean* models today. +- The **MIP** path end-to-end: warm start via `solution_available`, incumbent + preserved across a stop/resume, run continues correctly (bit-identity is *not* + expected — §7). -These also surfaced the findings folded into §5 and §9. +These are the first things Phase 1 must prove (a small MIP, e.g. `sizes`). --- ## 7. Determinism contract (what resume guarantees) -- **Primal trajectory (W, nonants, rho, xbar): bit-identical.** This is the - correctness guarantee — the resumed run produces the same iterates as an - uninterrupted run. -- **Bounds and incumbent: valid and best-so-far, but NOT bit-reproducible.** They - come from async, timing-dependent spoke interaction. Resume never reports a - *worse* best-so-far than the checkpoint, but it may report a different - (equally valid) bound than the original run would have at the same iteration. - -This distinction must be stated in user docs so a tightened/loosened bound after -resume is not mistaken for a bug. +- **For the target MIP use case:** resume **continues the optimization correctly + and warm-started**, and **never loses or regresses the best xhat**. It is *not* + bit-reproducible — multi-threaded MIP solves are nondeterministic and admit + multiple optima, so the resumed iterates may differ from a hypothetical + uninterrupted run. That is expected, not a bug. +- **Bounds and incumbent:** valid and best-so-far, not bit-reproducible (async, + timing-dependent). Resume never reports a *worse* best-so-far than the + checkpoint. +- **Leaf-rebuild on a deterministic LP/QP solver:** the primal trajectory (W, + nonants, rho, xbar) *can* be bit-identical — this is what the PoC showed — but it + is a bonus, not the target guarantee. + +State this in user docs so a differing (but valid) trajectory or bound after +resuming a MIP is not mistaken for a bug. --- @@ -271,108 +376,104 @@ resume is not mistaken for a bug. Checkpointing is **opt-in** and adds nothing when off. -- **`--checkpoint-every k`** — write a checkpoint every `k` PH iterations. - - `k` unset or `0` ⇒ checkpointing is **disabled**: the `Checkpointer` - extension is not attached, so there is zero overhead and no files. - - `k ≥ 1` ⇒ checkpoint every `k` iterations **and always at the end of - iteration 0**, regardless of `k`. Iteration 0 establishes the resume - baseline — solvers are created, the trivial bound is computed, the initial - (`W = 0`) solve is done — so a crash anywhere in the `iterk` loop has a valid - resume point even before the first periodic checkpoint, and the iter-0 state - composes with the existing `--iter0-from-pickle` warm-start. -- **`--checkpoint-dir `** — where per-rank checkpoint files and the - manifest are written (see §10). +- **Trigger — end-of-run / on-signal (primary).** The use case stops on a + schedule, so the natural trigger is "checkpoint and exit cleanly at the end of + the run or on a signal" — after a fixed number of iterations + (`--max-iterations`), on a wall-clock budget, or on `SIGTERM`/`SIGUSR1`. This + writes one complete, resumable checkpoint per planned stop. +- **`--checkpoint-every k` (optional insurance).** Also checkpoint every `k` PH + iterations for unplanned-crash coverage. `k` unset/`0` ⇒ disabled (the + `Checkpointer` extension is not attached; zero overhead, no files). `k ≥ 1` ⇒ + every `k` iterations **and always at the end of iteration 0** (which establishes + the resume baseline — solvers created, trivial bound computed, initial `W = 0` + solve done — and composes with `--iter0-from-pickle`). Given infrequent planned + stops, most runs will leave this off or large. +- **`--checkpoint-dir `** — where per-rank files and the manifest are written + (§10). +- **`--checkpoint-backend {dill-model, leaf}`** — how scenario-model state is + restored (§2.2). Default `dill-model` (captures the warm start + cuts, dodges an + expensive `scenario_creator` re-run). `leaf` is the **low-cost** option — tiny, + fast, version-robust checkpoints — for small/cheap-creator runs or frequent + kill-safety writes. - **`--resume-from `** (or `--resume`, auto-selecting the latest *complete* - checkpoint from the manifest) — rebuild the wheel and restore that checkpoint. - Resume requires identical geometry (§5.6); a mismatch is refused with a clear - error. + checkpoint from the manifest) — reconstruct the wheel and restore. Resume + requires identical geometry (§5.7); a mismatch is refused with a clear error. -Hard-kill coverage is the user's tradeoff: small `k` is safest, larger `k` cuts -I/O. Each checkpoint is published atomically (§9, item 7; §10), so a kill *during* -a write leaves the previous complete checkpoint intact and referenced — never a +Each checkpoint is published atomically (§9, item 7; §10), so a kill *during* a +write leaves the previous complete checkpoint intact and referenced — never a half-written one. ### 8.1 Bundles -mpi-sppy has **only proper bundles** now — loose bundling (`bundles_per_rank`) -was removed in 2026 (`spbase.py`; see `doc/src/properbundles.rst`). A proper -bundle consumes whole second-stage tree nodes and is a **first-class +mpi-sppy has **only proper bundles** now — loose bundling was removed in 2026 +(`spbase.py`; `doc/src/properbundles.rst`). A proper bundle is a **first-class subproblem**: it appears in `local_scenarios` with its own `nonant_indices`, and -PH has no separate `local_subproblems` for it. - -Consequently checkpointing **applies to bundled runs directly and uniformly** — -the same code that gathers/restores `W`, nonants, and `rho` by iterating -`local_scenarios` + `nonant_indices` covers a bundle exactly as it covers a plain -scenario. This holds whether bundles are built in memory -(`--scenarios-per-bundle`) or pickled and re-read -(`--pickle-bundles-dir` / `--unpickle-bundles-dir`); both produce proper bundles -and checkpoint identically. Pickling is an independent optimization (skip the -bundle rebuild on resume), not a requirement for checkpointing. - -One cleanup: `_restore_nonants` still carries a 2019 comment that it "will not -work on bundles" (`spopt.py`). That predates proper bundles and refers to the -removed loose mechanism; it should be re-verified and refreshed when bundle -checkpointing is validated (Phase 2). +is itself a Pyomo model. So checkpointing **applies uniformly** — dilling +`local_scenarios` dills bundles exactly as it dills plain scenarios, and the +leaf-rebuild path iterates `nonant_indices` identically. Holds whether bundles are +in memory (`--scenarios-per-bundle`) or pickled (`--pickle-bundles-dir` / +`--unpickle-bundles-dir`). + +One cleanup: `_restore_nonants` still carries a 2019 comment that it "will not work +on bundles" (`spopt.py`). That predates proper bundles and refers to the removed +loose mechanism; re-verify and refresh it when bundle checkpointing is validated +(Phase 2). --- ## 9. Core changes required -These are the touch-points an implementation needs beyond the PoC's -extension/subclass hacks: +Touch-points an implementation needs beyond the PoC's extension/subclass hacks: 1. **Global iteration counter / resume offset.** `iterk_loop` hardcodes `for _PHIter in range(1, max+1)` (`phbase.py:1156`), so a resumed run renumbers - from 1 and its checkpoints collide with the pre-crash ones. Add a resume - offset so checkpoint numbering is the global iteration, and so termination - honors the original `max_iterations`. -2. **One-line xhatter write hook.** The xhatter `main()` loop calls no - per-iteration extension hook. Add a single `self.opt.extobject.enditer()` (or a - dedicated checkpoint hook) inside the loop. Then **one `Checkpointer` extension - serves hub and xhatter uniformly** — restore at `post_iter0` / - `post_iter0_after_sync`, write at `enditer` — instead of bespoke spoke - subclasses. (Spoke restore already has a home: `pre_iter0`/`post_iter0` fire - once in `xhat_prep`, `xhatbase.py:36,49`.) -3. **Extension `checkpoint_state` / `restore_state` contract** on the `Extension` - base (no-ops by default; implemented by rho updaters, `fixer`, `slammer`, - convergers). Must serialize both `self.*` state and the extension's - model-attached state (`s._mpisppy_data.*`, e.g. `fixer`'s `conv_iter_count`), - and be restored consistently with the variable fixedness in §5.1 (see §5.4). - The `Checkpointer` aggregates the other extensions' dicts into the per-rank - file. -4. **Full-var snapshot around any xhat/incumbent evaluation.** Evaluating an xhat - fixes the first stage and re-solves; `_restore_nonants` restores *only* - nonants, leaving second-stage recourse vars in the eval state. The checkpointed - primal state (W/nonants/rho) is unaffected (it is gathered from nonants/params), - but any *full-objective* read taken right after an eval is wrong. Snapshot and - restore all vars around an eval, or evaluate on a copy. -5. **Geometry / cfg fingerprint** (§5.6) with a clear refusal on mismatch. + from 1 and its checkpoints collide with the pre-crash ones. Add a resume offset + so checkpoint numbering is the global iteration and termination honors the + original `max_iterations`. +2. **A reload-model resume branch.** When restoring via dill-reload, startup must + reconstruct comms/windows/solvers but **skip `attach_PH_to_objective`** — the + reloaded model already carries the spliced W/prox objective and the prox cuts — + then strip/rebuild `_solver_plugin` (`set_instance`) and set + `solution_available` for the warm start (§5.2). This is a distinct branch from + the leaf-rebuild "build fresh, overlay values" path; the `Checkpointer` picks + the branch from `--checkpoint-backend`. +3. **Extension `checkpoint_state` / `restore_state` contract** on `Extension` + (no-ops by default; implemented by rho updaters, `fixer`, `slammer`, + convergers). Covers **extension-object** state under both backends; + model-attached state (`fixer`'s `conv_iter_count`) rides in the dill under + dill-reload but must be gathered explicitly under leaf-rebuild (§5.5). The + `Checkpointer` aggregates the dicts into the per-rank file. +4. **Clean-point model snapshot (xhat/incumbent eval).** Evaluating an xhat fixes + the first stage and re-solves, corrupting recourse vars (§5.2). The model must + be dilled (or its values gathered) when recourse values reflect the true last + solve — snapshot before an eval, or evaluate on a copy. +5. **Geometry / cfg fingerprint** (§5.7) with a clear refusal on mismatch. 6. **Async per-spoke incumbent checkpoints — no hub↔spoke coordination.** Each - spoke serializes its *own* best incumbent (the best xhat solution values, §5.3) - and bound on its own schedule — whenever its incumbent improves, reusing the - trigger already in `_maybe_write_incumbent_on_improvement` — to its own - rank-tagged file with the same atomic write (item 7). Spokes are **not** - synchronized to the hub's checkpoint iteration. The determinism contract (§7) - makes bounds and the incumbent best-so-far, not bit-reproducible, so a - globally-consistent "snapshot at iteration `k`" across cylinders is - unnecessary: on resume the hub restores its bit-identical primal state (§5.1) - while each spoke independently reloads its latest incumbent/bound, all of which - are accepted only if improving (`update_best_solution_if_improving`, - `spbase.py:578`). Leaning into the existing asynchrony this way avoids a - hub-triggered snapshot barrier and its stall/deadlock risk against - `got_kill_signal`. -7. **Atomic writes with a single published generation** (per-rank temp-then-rename - is already in the PoC). Each rank writes only its local state to a rank-tagged - temp file and renames it into place; the hub's set of per-rank files is then - published as one checkpoint by atomically rewriting `manifest.json` (itself - temp-then-rename) to point at the new complete generation (§10). Because that - manifest flip is the single commit point, **one committed generation is - enough**: a kill before the flip leaves the previous complete checkpoint intact - and still referenced; a kill after it leaves the new one. The prior generation - can therefore be deleted as soon as the new manifest is in place — there is no - correctness need to retain the last *k*. Keeping a few older generations is an - optional convenience (roll back further, inspect history), not a kill-safety - requirement. + spoke serializes its *own* best incumbent (the best xhat solution values, §5.4) + and bound whenever its incumbent improves — reusing + `_maybe_write_incumbent_on_improvement` — to its own rank-tagged file with the + same atomic write (item 7). Spokes are **not** synchronized to the hub's + checkpoint iteration: the determinism contract (§7) makes bounds/incumbent + best-so-far, not bit-reproducible, so a globally-consistent "snapshot at + iteration `k`" across cylinders is unnecessary. On resume the hub restores its + primal state while each spoke reloads its latest incumbent/bound, all accepted + only if improving (`update_best_solution_if_improving`, `spbase.py:578`). This + also avoids a hub-triggered snapshot barrier and its stall/deadlock risk. +7. **Atomic writes with a single published generation.** Each rank writes only its + local state (dilled models + leaf non-model data) to rank-tagged temp files and + renames them into place; the set of per-rank files is then published as one + checkpoint by atomically rewriting `manifest.json` (itself temp-then-rename) to + point at the new complete generation (§10). That flip is the single commit + point, so **one committed generation is enough**: a kill before it keeps the + previous checkpoint, a kill after it keeps the new one. The prior generation can + be deleted once the manifest is in place. Keeping a few older generations is + optional convenience, not a kill-safety requirement. +8. **A `Checkpointer` extension** (write at `enditer` / on trigger, restore at + `post_iter0_after_sync`). For spokes, the xhatter `main()` loop calls no + per-iteration extension hook — add a single `self.opt.extobject.enditer()` (or a + dedicated checkpoint hook) inside it so **one `Checkpointer` serves hub and + xhatter uniformly** (restore already has a home: `pre_iter0`/`post_iter0` fire + once in `xhat_prep`, `xhatbase.py:36,49`). --- @@ -380,24 +481,24 @@ extension/subclass hacks: ``` / - manifest.json # cfg hash, n_proc, cylinder map, latest complete hub iter + manifest.json # cfg hash, n_proc, backend, cylinder map, latest complete hub generation hub/ - iter_/ - hub_rank_.pkl # hub primal + extension state (+ bounds/incumbent obj) + gen_/ # NNNN = global PH iteration at the checkpoint + hub_rank_.pkl # non-model leaf state: iter counter, bounds, extension-object state + hub_rank__scen_.dill # dilled scenario model(s) for this rank (dill-model backend) spokes/ - spoke__rank_.pkl # each spoke's latest incumbent/bound, overwritten - # asynchronously on improvement (§9, item 6) + spoke__rank_.pkl # each spoke's latest incumbent (best xhat, by name) + bound, + # overwritten asynchronously on improvement (§9, item 6) ``` The hub writes iteration-tagged generations under `hub/`; each spoke keeps a -single latest-wins file under `spokes/` that it overwrites (atomically) whenever -its incumbent improves — the two are deliberately *not* aligned to a common -iteration (§9, item 6). `manifest.json` is the single commit point: it names the -latest *complete* hub generation (all hub ranks flushed), and publishing a new -one means atomically renaming a freshly written `manifest.json` into place (§9, -item 7). Resume reads whatever the manifest currently names, plus whatever each -spoke last committed under `spokes/`. Use plain `pickle` for the numeric state; -`dill` only where models/closures must be serialized. +single latest-wins file under `spokes/` that it overwrites atomically on +improvement — the two are deliberately *not* aligned (§9, item 6). +`manifest.json` is the single commit point: it names the latest *complete* hub +generation and records the backend so resume loads the right way. Under the `leaf` +backend the `.dill` model files are replaced by numeric arrays inside the +`hub_rank_*.pkl`. Use plain `pickle` for the numeric/leaf state; `dill` for the +scenario models. --- @@ -407,60 +508,67 @@ Each phase is a review-sized PR that is green on its own and adds user-visible value. New tests are wired into `run_coverage.bash` **and** `test_pr_and_main.yml` in the same commit. -- **Phase 1 — Serial hub checkpoint/resume.** `Checkpointer` extension - (write at `enditer`, restore at `post_iter0_after_sync`); global iteration - counter / resume offset; geometry+cfg fingerprint; atomic per-rank writes; - CLI flags `--checkpoint-dir`, `--checkpoint-every k` (0/unset ⇒ off; else every - `k` iters **and always after iter 0** — §8), `--resume-from`/`--resume`. Restore - W/nonants/rho. Test: serial farmer kill+resume bit-identical. +- **Phase 1 — Serial hub checkpoint/resume, dill-model backend.** `Checkpointer` + extension; global iteration counter / resume offset; reload-model resume branch + (skip `attach_PH_to_objective`, rebuild `_solver_plugin`, set warm start); + geometry+cfg fingerprint; atomic per-rank writes + manifest; end-of-run/on-signal + trigger (+ optional `--checkpoint-every k`); CLI flags `--checkpoint-dir`, + `--checkpoint-backend`, `--resume-from`/`--resume`. Test: serial **MIP** (e.g. + `sizes`) stop+resume — run continues correctly, incumbent preserved, warm start + taken (mid-run model dill round-trip proven, §6). - **Phase 2 — Multi-rank + bundles.** Barriers, rank-tagged files, single-generation - atomic publish via the manifest (§9, item 7). Validate checkpointing with - **proper bundles** (bundle = first-class subproblem, §8.1) and refresh the stale - `_restore_nonants` bundle comment. Test: `mpiexec` farmer kill+resume - bit-identical on every rank, incl. uneven distribution and a - `--scenarios-per-bundle` run; mismatch refusal. -- **Phase 3 — Extension state contract.** `checkpoint_state`/`restore_state` on - `Extension` (covering `self.*` and model-attached state); restore nonant - fixedness in the primal set (§5.1); implement the contract for rho updaters, - `fixer`, and `slammer`. Test: PH + norm-rho-updater resumes bit-identical; PH + - `fixer` resumes with the *same* variables fixed (and the same in-progress - counters); PH + `slammer` resumes with slams intact. + atomic publish. Validate with **proper bundles** (§8.1) and refresh the stale + `_restore_nonants` comment. Test: `mpiexec` MIP stop+resume on every rank, incl. + uneven distribution and `--scenarios-per-bundle`; mismatch refusal. +- **Phase 3 — Extension-object state contract.** `checkpoint_state`/`restore_state` + on `Extension`; implement for rho updaters, `fixer`, `slammer`, convergers. + (Model-attached `fixer` counter and nonant fixedness ride in the dill.) Test: PH + + norm-rho-updater, PH + `fixer`, PH + `slammer` each resume with state intact + and consistent with variable fixedness. - **Phase 4 — Cylinders / spokes.** One-line xhatter write hook; unified - `Checkpointer` on spoke opts; each spoke checkpoints its own **incumbent (best - xhat solution)** asynchronously on improvement — no hub↔spoke coordination (§9, - item 6). Test: farmer cylinders (hub+lagrangian+xhatshuffle) crash+resume — hub - primal bit-identical, best xhat preserved. + `Checkpointer` on spoke opts; each spoke checkpoints its own **best xhat** (by + name) asynchronously on improvement — no hub↔spoke coordination (§9, item 6). + Test: farmer/`sizes` cylinders (hub+lagrangian+xhatshuffle) stop+resume — run + continues, best xhat preserved. - **Phase 5 — Exact spoke continuity (optional).** Hoist `ScenarioCycler`/`xh_iter` - onto `self`; checkpoint cursor (+ RNG getstate if a stream becomes stateful). -- **Phase 6 — Broader coverage.** lagranger, FWPH, subgradient spokes; - scenario-pickle reconstruction path for expensive `scenario_creator`s. + onto `self`; checkpoint the cursor (+ RNG getstate if a stream becomes stateful). +- **Phase 6 — Leaf-rebuild backend + broader coverage.** The lighter, + version-robust `--checkpoint-backend leaf` path (rebuild via `scenario_creator`, + overlay W/rho/nonants/fixedness, replay prox `cut_values`, optional all-var warm + start); this is what the PoC prototyped. Plus lagranger, FWPH, subgradient + spokes. --- ## 12. Open questions / risks -Resolved: - -- **`--checkpoint-every k` semantics** (§8): 0/unset ⇒ off; otherwise every `k` - iters and always after iter 0. -- **Bundles** (§8.1): only proper bundles exist; each is a first-class subproblem, - so checkpointing applies uniformly — no special handling needed. +Resolved (given the §1 use case): + +- **Backend choice.** dill the scenario models (§2.2): overhead is negligible at a + few checkpoints, version robustness is unneeded (same-environment resume next + day), and it captures the warm start + prox cuts + model-attached state for free + while avoiding an expensive `scenario_creator` re-run. +- **Warm start.** Worthwhile for MIPs (branch-and-bound benefits), free via the + dilled model, fed through the existing `warmstart_subproblems` / + `solution_available` path. +- **Checkpoint retention** (§9, item 7): a single manifest-published generation + suffices; older generations are optional history. - **Spoke snapshot coordination** (§9, item 6): resolved by *not* coordinating. - Spokes checkpoint their own incumbents asynchronously on improvement; the - determinism contract (§7) makes this sufficient and removes the sync point that - would otherwise risk stalling fast spokes or deadlocking with `got_kill_signal`. -- **Checkpoint retention** (§9, item 7): a single manifest-published generation is - enough for kill-safety; last-*k* retention is optional history, not required. Still open: -- **variable_probability / surrogate vars.** With variable probabilities mpi-sppy - masks `W` (but not prox) for zero-probability nonants and assumes each such - nonant's surrogate var is fixed at 0. Restore must **reproduce that invariant, - not just reload the raw `W` array**: confirm the zero-probability mask is - re-applied and the surrogate vars come back fixed-at-0 (§5.1 fixedness), so a - masked component is never treated as live on resume. Needs verification; among - the spokes only `xhatxbar` supports variable_probability today. +- **Mid-run MIP model dill round-trip** (§6) — the load-bearing unvalidated + assumption. Must be proven first in Phase 1 (spliced objective, `LinearExpression` + cut bodies, mutable params, post-`set_instance`). +- **Disk footprint.** Dilled large MIP models × scenarios/rank × a few generations + can be large; the single-generation policy (§9, item 7) keeps only one live, but + document the peak (two generations during a publish). +- **variable_probability / surrogate vars.** mpi-sppy masks `W` (not prox) for + zero-probability nonants and assumes each surrogate var is fixed at 0. Restore + must **reproduce that invariant**: dill-reload preserves the mask and fixedness + automatically, but verify; leaf-rebuild must re-apply the mask and re-fix the + surrogate at 0 (§5.1 fixedness), not just reload raw `W`. Among the spokes only + `xhatxbar` supports variable_probability today. - **Cross-geometry resume** is explicitly deferred; revisit if HPC users need to resume on a different node count. ``` From be6b9685d190961215628e90e0c50ebadb4c4b2a Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Fri, 10 Jul 2026 14:50:26 -0700 Subject: [PATCH 05/10] doc(checkpointing): record MIP dill-reload PoC results; resolve mid-run-dill risk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the MIP dill-reload PoC (sizes SIZES3, gurobi_persistent single-thread) into the design: - §6: the dill-reload backend is now validated on a MIP. A mid-run scenario model -- including the linearized-prox case (845 xsqvar_cuts + 65 ProxApproxManagers) -- round-trips through dill in-process and cross-process, re-solving to identical objective + decision vars; a serial stop->reload-> continue is bit-identical to an uninterrupted run under the deterministic single-thread solve. Note what remains for later phases (multi-rank, cylinders, incumbent carry, warm-start speedup, scale footprint). - §9 item 2: the reload branch must skip attach_Ws_and_prox as well as attach_PH_to_objective, and refresh saved_objectives[sname] (Eobjective reads those handles; they dangle to the discarded fresh model after a swap). The swap targets local_scenarios (there is no local_subproblems). - §12: move the mid-run-dill round-trip from "Still open" to "Resolved". Also reword the leaf-rebuild backend in §2.2 from "is retained as" to "remains available as" the low-cost backend. Co-Authored-By: Claude Opus 4.8 --- doc/designs/checkpointing_design.md | 72 +++++++++++++++++++---------- 1 file changed, 48 insertions(+), 24 deletions(-) diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md index 6ffb40454..c5430131c 100644 --- a/doc/designs/checkpointing_design.md +++ b/doc/designs/checkpointing_design.md @@ -1,9 +1,9 @@ # Checkpoint / Resume for mpi-sppy — Design -Status: **draft** (framework PoC validated; dill-reload backend designed, not yet -validated). Scope: checkpoint a running mpi-sppy job so it can be stopped and -resumed later. Must work on multiple MPI ranks and for cylinder (hub-and-spoke) -runs. +Status: **draft** (framework and dill-reload backend both PoC-validated — the +latter on a serial MIP; multi-rank and cylinders pending). Scope: checkpoint a +running mpi-sppy job so it can be stopped and resumed later. Must work on multiple +MPI ranks and for cylinder (hub-and-spoke) runs. --- @@ -110,7 +110,8 @@ three days is negligible. And it *avoids re-running an expensive `scenario_creat on every resume, which for large models is itself a real saving. The alternative — rebuild each model via `scenario_creator` and overlay the state -as leaf data (arrays/name→value maps) — is retained as the **low-cost backend**: +as leaf data (arrays/name→value maps) — remains available as the **low-cost +backend**: its checkpoints are tiny and fast (a handful of `O(first-stage)` arrays, no model structure) and version-robust (plain numbers, not pickled classes). That makes it the right choice for small scenarios, cheap creators, or *frequent* kill-safety @@ -340,16 +341,32 @@ validated the **framework and the leaf-rebuild backend**: preserved exactly; `BestInnerBound` carried exactly; `BestOuterBound` differed run-to-run (async) but stayed valid. -**Not yet validated (the primary backend for this use case):** - -- Dilling and reloading a **mid-run** scenario model — spliced W/prox objective, - `LinearExpression` cut bodies, mutable params, post-`set_instance` — round-trips - cleanly. The repo only dills *clean* models today. -- The **MIP** path end-to-end: warm start via `solution_available`, incumbent - preserved across a stop/resume, run continues correctly (bit-identity is *not* - expected — §7). - -These are the first things Phase 1 must prove (a small MIP, e.g. `sizes`). +A second PoC then validated the **dill-reload backend on a MIP** (`sizes` SIZES3, +`gurobi_persistent`, single-thread `Threads=1`/`Seed=1`/`MIPGap=0` for a +deterministic solve — the §7 validation crutch): + +- **Mid-run model round-trip.** After a few PH iterations, a scenario model was + stripped of `_solver_plugin`, dilled, and reloaded **both in-process and in a + fresh process**; a new solver was attached with `set_instance` and the + subproblem re-solved. The reloaded model reproduced the original solve's + objective and **every decision variable exactly** — including the hardest case, + **linearized prox** (176 KB carrying **845 `xsqvar_cuts` + 65 + `ProxApproxManager`s** on `_mpisppy_data`), which came back structurally + identical and self-consistent. The only difference was the x² epigraph auxiliary + `xsqvar` wobbling ~1.5e-6 at solver feasibility tolerance (immaterial; MIQP was + exact). This is the load-bearing assumption — that a mid-run MIP model, cuts and + all, survives dill — and it **holds**. +- **Stop → reload → continue, bit-identical.** Stopping PH at iteration 3, dilling + the mid-run models, then rebuilding the scaffolding and continuing through the + reload branch reproduced an uninterrupted 6-iteration run with + `max|dW| = max|d nonant| = 0.0` — for **both** quadratic and linearized prox. + Under the deterministic single-thread solve this is exact bit-identity, the + strong "nothing was lost" check. + +Still to prove in later phases (this PoC was serial and focused on the model +round-trip + continuation): the dill-reload backend under **multi-rank** and +**cylinders**; carrying the **incumbent** across a dill-reload stop; a measured +warm-start speedup; and the disk/time footprint at true model scale. --- @@ -431,12 +448,18 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: so checkpoint numbering is the global iteration and termination honors the original `max_iterations`. 2. **A reload-model resume branch.** When restoring via dill-reload, startup must - reconstruct comms/windows/solvers but **skip `attach_PH_to_objective`** — the - reloaded model already carries the spliced W/prox objective and the prox cuts — - then strip/rebuild `_solver_plugin` (`set_instance`) and set - `solution_available` for the warm start (§5.2). This is a distinct branch from - the leaf-rebuild "build fresh, overlay values" path; the `Checkpointer` picks - the branch from `--checkpoint-backend`. + reconstruct comms/windows/solvers but **skip both `attach_Ws_and_prox` and + `attach_PH_to_objective`** — the reloaded model already carries the W/rho/xbars + params, the spliced objective, and the prox cuts, so re-running either would + duplicate components or double the terms. Then strip/rebuild `_solver_plugin` + (`set_instance`) and set `solution_available` for the warm start (§5.2). Two + details the PoC surfaced: **refresh `saved_objectives[sname]`** for each + reloaded model — `Eobjective` reads those objective handles and they otherwise + dangle to the discarded fresh model — and note the reload targets + `local_scenarios` only (there is no `local_subproblems`; the solve path + iterates `local_scenarios`). This is a distinct branch from the leaf-rebuild + "build fresh, overlay values" path; the `Checkpointer` picks the branch from + `--checkpoint-backend`. 3. **Extension `checkpoint_state` / `restore_state` contract** on `Extension` (no-ops by default; implemented by rho updaters, `fixer`, `slammer`, convergers). Covers **extension-object** state under both backends; @@ -554,12 +577,13 @@ Resolved (given the §1 use case): - **Checkpoint retention** (§9, item 7): a single manifest-published generation suffices; older generations are optional history. - **Spoke snapshot coordination** (§9, item 6): resolved by *not* coordinating. +- **Mid-run MIP model dill round-trip** — was the load-bearing unvalidated + assumption; **validated by the MIP dill-reload PoC** (§6), including the + linearized-prox cuts, in-process and cross-process, with serial stop→reload→ + continue bit-identical under a deterministic solver. Still open: -- **Mid-run MIP model dill round-trip** (§6) — the load-bearing unvalidated - assumption. Must be proven first in Phase 1 (spliced objective, `LinearExpression` - cut bodies, mutable params, post-`set_instance`). - **Disk footprint.** Dilled large MIP models × scenarios/rank × a few generations can be large; the single-generation policy (§9, item 7) keeps only one live, but document the peak (two generations during a publish). From 3bcc02929293a67842b2223a1a2b7fe5ba4894e9 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Fri, 10 Jul 2026 15:34:04 -0700 Subject: [PATCH 06/10] doc(checkpointing): address Copilot review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Correct the local_subproblems claim (§9 item 2): a plain PH has none, but the generic file-based path the dill-reload backend builds on sets sp.local_subproblems = sp.local_scenarios (scenario_io.py) and CGBase.solve_loop iterates it, so the reload branch must refresh that alias where it exists or it dangles to the discarded model. - Reconcile backend naming: the CLI flag is now --checkpoint-backend dill-reload (was dill-model) so the flag and the narrative use one name; "reload" is the restore step. - Replace volatile file:line references with file + symbol (e.g. "warmstart_subproblems in spopt.py" instead of "spopt.py:301-305") so they do not drift as the code moves. Co-Authored-By: Claude Opus 4.8 --- doc/designs/checkpointing_design.md | 45 +++++++++++++++-------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md index c5430131c..f8d920b75 100644 --- a/doc/designs/checkpointing_design.md +++ b/doc/designs/checkpointing_design.md @@ -157,7 +157,7 @@ A checkpoint is **not** a snapshot of the object graph. On resume: - **Warm-start plumbing:** `spopt.py` already supports warm-starting subproblem solves — the `warmstart_subproblems` option plus `WarmstartStatus.PRIOR_SOLUTION` use a warm start when `s._mpisppy_data.solution_available` is set - (`spopt.py:301-305`). Restoring the model's variable values and setting + (`warmstart_subproblems` in `spopt.py`). Restoring the model's variable values and setting `solution_available=True` feeds the restored MIP solution straight into this path — no new solver code. - **W / xbar persistence:** `utils/w_utils/wxbarwriter.py` (writes in @@ -202,7 +202,7 @@ already exist: `_populate_W_cache`/`W_from_flat_list` (`phbase.py`), **Restore point:** after the scaffolding exists and models are loaded, before the `iterk` loop. In serial this can be a driver call; in cylinders the hub runs `ph_main` internally, so restore happens in the **`post_iter0_after_sync`** -extension hook (end of `Iter0`, `phbase.py:1079`). +extension hook (`post_iter0_after_sync`, end of `Iter0` in `phbase.py`). ### 5.2 Recourse variable values — *warm start, in the dilled model* @@ -229,13 +229,13 @@ i.e. snapshot before an eval corrupts them, or evaluate on a copy (§9, item 4). ### 5.3 Proximal-approximation cuts (`--linearize-proximal-terms`) — *in the dilled model* When the linearized prox is on, `attach_PH_to_objective` builds, per scenario -(`phbase.py:892-894`): +(in `attach_PH_to_objective`, `phbase.py`): - `s._mpisppy_model.xsqvar` — the epigraph var for `x²`; - `s._mpisppy_model.xsqvar_cuts` — a `Constraint` that accumulates one linear cut - per visited x-location (`prox_approx.py:232,278,291`); + per visited x-location (`add_cut` in `prox_approx.py`); - `s._mpisppy_data.xsqvar_prox_approx[ndn_i]` — a `ProxApproxManager` whose - bookkeeping (`cut_index`, the sorted `cut_values` array; `prox_approx.py:46-47`) + bookkeeping (`cut_index`, the sorted `cut_values` array; `ProxApproxManager` in `prox_approx.py`) decides when a new cut is redundant. The cut *constraints* live on the model; the manager's bookkeeping lives on @@ -261,12 +261,12 @@ None of this lives on a hub scenario model, so it is restored as leaf data under timing is not reproducible, so they are carried forward as best-so-far. They stay valid: a restored looser bound is improved again; a restored incumbent objective is never regressed because `update_best_solution_if_improving` - (`spbase.py:578`) only accepts improvements. In cylinders the hub's + (`spbase.py`) only accepts improvements. In cylinders the hub's `best_solution_obj_val` is often `None` — the inner bound arrives as a scalar via - `receive_innerbounds` (`spcommunicator.py:1010`) into `spcomm.BestInnerBound`. + `receive_innerbounds` (`spcommunicator.py`) into `spcomm.BestInnerBound`. - **The best xhat SOLUTION values live on the xhat spoke**, in `spoke.opt.best_solution_cache` (a `ComponentMap` over all vars) + - `spoke.best_inner_bound`; `InnerBoundSpoke.finalize()` (`spoke.py:293`) loads + `spoke.best_inner_bound`; `InnerBoundSpoke.finalize()` (`spoke.py`) loads them back. So **"keep the best xhat" requires checkpointing the spoke incumbent**, not just hub bounds. The spoke checkpoints its own cache **on its own schedule** — on each improvement, reusing @@ -307,7 +307,7 @@ Consequences: ### 5.6 RNG and spoke cursor — *non-model leaf data, partially restored* - xhatshuffle seeds its stream to a fixed `42` and samples **once** - (`xhatshufflelooper_bounder.py:88,94`) — deterministic, no RNG state to save. + (`main()` in `xhatshufflelooper_bounder.py`) — deterministic, no RNG state to save. - The `ScenarioCycler` cursor and `xh_iter` are **local variables inside `main()`** — unreachable. Exact spoke-cursor resume needs them hoisted onto `self` (Phase 5). Without it the spoke restarts its cursor; this only changes *which* scenario @@ -407,8 +407,8 @@ Checkpointing is **opt-in** and adds nothing when off. stops, most runs will leave this off or large. - **`--checkpoint-dir `** — where per-rank files and the manifest are written (§10). -- **`--checkpoint-backend {dill-model, leaf}`** — how scenario-model state is - restored (§2.2). Default `dill-model` (captures the warm start + cuts, dodges an +- **`--checkpoint-backend {dill-reload, leaf}`** — how scenario-model state is + restored (§2.2). Default `dill-reload` (captures the warm start + cuts, dodges an expensive `scenario_creator` re-run). `leaf` is the **low-cost** option — tiny, fast, version-robust checkpoints — for small/cheap-creator runs or frequent kill-safety writes. @@ -443,7 +443,7 @@ loose mechanism; re-verify and refresh it when bundle checkpointing is validated Touch-points an implementation needs beyond the PoC's extension/subclass hacks: 1. **Global iteration counter / resume offset.** `iterk_loop` hardcodes - `for _PHIter in range(1, max+1)` (`phbase.py:1156`), so a resumed run renumbers + `for _PHIter in range(1, max+1)` (in `iterk_loop`, `phbase.py`), so a resumed run renumbers from 1 and its checkpoints collide with the pre-crash ones. Add a resume offset so checkpoint numbering is the global iteration and termination honors the original `max_iterations`. @@ -455,11 +455,14 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: (`set_instance`) and set `solution_available` for the warm start (§5.2). Two details the PoC surfaced: **refresh `saved_objectives[sname]`** for each reloaded model — `Eobjective` reads those objective handles and they otherwise - dangle to the discarded fresh model — and note the reload targets - `local_scenarios` only (there is no `local_subproblems`; the solve path - iterates `local_scenarios`). This is a distinct branch from the leaf-rebuild - "build fresh, overlay values" path; the `Checkpointer` picks the branch from - `--checkpoint-backend`. + dangle to the discarded fresh model — and swap the reloaded model into + `local_scenarios` (which `SPOpt.solve_loop` iterates). A plain PH keeps no + `local_subproblems`, but the **generic file-based path** the dill-reload backend + builds on (`scenario_io.py` sets `sp.local_subproblems = sp.local_scenarios`) + maintains that alias, and `CGBase.solve_loop` iterates it — so where the alias + exists the reload branch must refresh it too, or it dangles to the discarded + model. This is a distinct branch from the leaf-rebuild "build fresh, overlay + values" path; the `Checkpointer` picks the branch from `--checkpoint-backend`. 3. **Extension `checkpoint_state` / `restore_state` contract** on `Extension` (no-ops by default; implemented by rho updaters, `fixer`, `slammer`, convergers). Covers **extension-object** state under both backends; @@ -480,7 +483,7 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: best-so-far, not bit-reproducible, so a globally-consistent "snapshot at iteration `k`" across cylinders is unnecessary. On resume the hub restores its primal state while each spoke reloads its latest incumbent/bound, all accepted - only if improving (`update_best_solution_if_improving`, `spbase.py:578`). This + only if improving (`update_best_solution_if_improving` in `spbase.py`). This also avoids a hub-triggered snapshot barrier and its stall/deadlock risk. 7. **Atomic writes with a single published generation.** Each rank writes only its local state (dilled models + leaf non-model data) to rank-tagged temp files and @@ -496,7 +499,7 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: per-iteration extension hook — add a single `self.opt.extobject.enditer()` (or a dedicated checkpoint hook) inside it so **one `Checkpointer` serves hub and xhatter uniformly** (restore already has a home: `pre_iter0`/`post_iter0` fire - once in `xhat_prep`, `xhatbase.py:36,49`). + once in `xhat_prep` in `xhatbase.py`). --- @@ -508,7 +511,7 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: hub/ gen_/ # NNNN = global PH iteration at the checkpoint hub_rank_.pkl # non-model leaf state: iter counter, bounds, extension-object state - hub_rank__scen_.dill # dilled scenario model(s) for this rank (dill-model backend) + hub_rank__scen_.dill # dilled scenario model(s) for this rank (dill-reload backend) spokes/ spoke__rank_.pkl # each spoke's latest incumbent (best xhat, by name) + bound, # overwritten asynchronously on improvement (§9, item 6) @@ -531,7 +534,7 @@ Each phase is a review-sized PR that is green on its own and adds user-visible value. New tests are wired into `run_coverage.bash` **and** `test_pr_and_main.yml` in the same commit. -- **Phase 1 — Serial hub checkpoint/resume, dill-model backend.** `Checkpointer` +- **Phase 1 — Serial hub checkpoint/resume, dill-reload backend.** `Checkpointer` extension; global iteration counter / resume offset; reload-model resume branch (skip `attach_PH_to_objective`, rebuild `_solver_plugin`, set warm start); geometry+cfg fingerprint; atomic per-rank writes + manifest; end-of-run/on-signal From 2ac5ee16a633479ce8557f502569d9ad9bcc77c9 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Mon, 20 Jul 2026 18:15:45 -0700 Subject: [PATCH 07/10] doc(checkpointing): add time-based and at-termination checkpoint triggers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extend §8 from the single `--checkpoint-every k` cadence to three composable triggers, enabled by `--checkpoint-dir`: - `--checkpoint-at-termination` (default on) — terminal checkpoint fired from the hub's existing `post_everything` hook when the run terminates for any internal reason (convergence, `--max-iterations`, cylinder convergence, `--time-limit`); the planned-stop use case pairs it with `--time-limit`. Not driven by external OS signals (recorded as a non-goal in §1). - `--checkpoint-every-seconds S` — wall-clock cadence checked at `enditer` via `allreduce_or(now - last_checkpoint >= S)`, mirroring the existing `time_limit` collective check so ranks decide together (no barrier deadlock). - `--checkpoint-every-iterations K` — the former `--checkpoint-every k`, renamed for symmetry. Also update §9 item 8 (two write points: periodic at `enditer`, terminal at `post_everything`, plus the scenario_denouement caveat) and §11 Phase 1 flag list. Co-Authored-By: Claude Opus 4.8 --- doc/designs/checkpointing_design.md | 92 +++++++++++++++++++++-------- 1 file changed, 69 insertions(+), 23 deletions(-) diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md index f8d920b75..63b542d78 100644 --- a/doc/designs/checkpointing_design.md +++ b/doc/designs/checkpointing_design.md @@ -51,6 +51,12 @@ lighter, leaf-data-only checkpoint (still supported; §4, §11 Phase 6). (welded to the current Pyomo/mpi-sppy/model code) is acceptable. Cross-version resume is out of scope. - APH. A C++ APH is expected to replace the Python `opt/aph.py`; PH-family only. +- **Catching external OS signals** (SIGTERM/SIGUSR1 from a scheduler, Ctrl-C) to + trigger a checkpoint. The terminal checkpoint (§8) fires on the run's *own* + termination — including hitting `--time-limit` — which covers the planned-stop + use case. A scheduler that hard-kills mid-solve is covered only insofar as the + previous checkpoint is preserved by atomic publication (§9), not by catching the + signal and checkpointing in response. --- @@ -391,22 +397,47 @@ resuming a MIP is not mistaken for a bug. ## 8. Configuration and semantics -Checkpointing is **opt-in** and adds nothing when off. - -- **Trigger — end-of-run / on-signal (primary).** The use case stops on a - schedule, so the natural trigger is "checkpoint and exit cleanly at the end of - the run or on a signal" — after a fixed number of iterations - (`--max-iterations`), on a wall-clock budget, or on `SIGTERM`/`SIGUSR1`. This - writes one complete, resumable checkpoint per planned stop. -- **`--checkpoint-every k` (optional insurance).** Also checkpoint every `k` PH - iterations for unplanned-crash coverage. `k` unset/`0` ⇒ disabled (the - `Checkpointer` extension is not attached; zero overhead, no files). `k ≥ 1` ⇒ - every `k` iterations **and always at the end of iteration 0** (which establishes +Checkpointing is **opt-in** and adds nothing when off. It is enabled by +`--checkpoint-dir`; with a directory set, the triggers below decide *when* a +checkpoint is written. They **compose** — whichever fires writes a checkpoint, all +sharing the same atomic publish (§9). With no `--checkpoint-dir` the `Checkpointer` +extension is not attached at all — zero overhead, no files. + +**Triggers** + +- **`--checkpoint-at-termination` (terminal checkpoint; default on).** Write one + complete, resumable checkpoint when the run terminates for *any* internal reason + — convergence, `--max-iterations`, cylinder convergence, or hitting + `--time-limit`. This is the primary trigger for the planned-stop use case: set + `--time-limit` to the daily budget and the run stops itself and checkpoints, + ready to resume the next morning. Nearly free — it fires in the hub's existing + `post_everything` hook (`phbase.py`), which runs once after the PH loop + regardless of *why* it exited, capturing the state of the last completed solve. + Turn it off (`--checkpoint-at-termination=False`) for a run that only wants + periodic insurance. It is *not* driven by external OS signals (a non-goal, §1); + the run's own termination — including `--time-limit` — is the trigger. +- **`--checkpoint-every-seconds S` (optional insurance).** Also checkpoint roughly + every `S` wall-clock seconds, for crash coverage during a long run. Checked at + each `enditer` as `allreduce_or(now − last_checkpoint ≥ S)` — the same + collective-decision pattern the existing `--time-limit` termination uses + (`phbase.py`), so every rank agrees and none writes at the barrier while others + sail past (a deadlock). Because it is tested only at iteration boundaries, the + guarantee is "the first boundary at least `S` seconds after the previous + checkpoint"; for large MIPs one solve can exceed `S`, which is expected — a + checkpoint cannot be taken mid-solve. Distinct from `--time-limit`, which *stops* + the run: this keeps it running and snapshots. +- **`--checkpoint-every-iterations K` (optional insurance).** Also checkpoint every + `K` PH iterations, **and always at the end of iteration 0** (which establishes the resume baseline — solvers created, trivial bound computed, initial `W = 0` - solve done — and composes with `--iter0-from-pickle`). Given infrequent planned - stops, most runs will leave this off or large. + solve done — and composes with `--iter0-from-pickle`). Checked at `enditer`. + (This is the former `--checkpoint-every k`, renamed for symmetry with + `--checkpoint-every-seconds`.) Given infrequent planned stops, most runs leave + the periodic triggers off and rely on the terminal checkpoint alone. + +**Other options** + - **`--checkpoint-dir `** — where per-rank files and the manifest are written - (§10). + (§10); its presence is what enables checkpointing. - **`--checkpoint-backend {dill-reload, leaf}`** — how scenario-model state is restored (§2.2). Default `dill-reload` (captures the warm start + cuts, dodges an expensive `scenario_creator` re-run). `leaf` is the **low-cost** option — tiny, @@ -494,12 +525,25 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: previous checkpoint, a kill after it keeps the new one. The prior generation can be deleted once the manifest is in place. Keeping a few older generations is optional convenience, not a kill-safety requirement. -8. **A `Checkpointer` extension** (write at `enditer` / on trigger, restore at - `post_iter0_after_sync`). For spokes, the xhatter `main()` loop calls no - per-iteration extension hook — add a single `self.opt.extobject.enditer()` (or a - dedicated checkpoint hook) inside it so **one `Checkpointer` serves hub and - xhatter uniformly** (restore already has a home: `pre_iter0`/`post_iter0` fire - once in `xhat_prep` in `xhatbase.py`). +8. **A `Checkpointer` extension** that writes on its active triggers and restores + at `post_iter0_after_sync`: + - *periodic* (`--checkpoint-every-iterations` / `--checkpoint-every-seconds`) — + at `enditer`. The seconds trigger tests + `allreduce_or(now − last_checkpoint ≥ S)` so all ranks decide together + (mirroring the `time_limit` check in `phbase.py`), avoiding a rank-skew + deadlock at the write barrier. + - *terminal* (`--checkpoint-at-termination`, default on) — in `post_everything` + (`phbase.py`), which fires once after the PH loop however it exited + (convergence, `--max-iterations`, `--time-limit`), capturing the last + completed solve. Caveat: `post_everything` runs *after* `scenario_denouement`; + standard denouements only report, but one that re-solves or mutates a model + would be captured — call this out in user docs. + + For spokes, the xhatter `main()` loop calls no per-iteration extension hook — + add a single `self.opt.extobject.enditer()` (or a dedicated checkpoint hook) + inside it so **one `Checkpointer` serves hub and xhatter uniformly** (restore + already has a home: `pre_iter0`/`post_iter0` fire once in `xhat_prep` in + `xhatbase.py`). --- @@ -537,9 +581,11 @@ value. New tests are wired into `run_coverage.bash` **and** - **Phase 1 — Serial hub checkpoint/resume, dill-reload backend.** `Checkpointer` extension; global iteration counter / resume offset; reload-model resume branch (skip `attach_PH_to_objective`, rebuild `_solver_plugin`, set warm start); - geometry+cfg fingerprint; atomic per-rank writes + manifest; end-of-run/on-signal - trigger (+ optional `--checkpoint-every k`); CLI flags `--checkpoint-dir`, - `--checkpoint-backend`, `--resume-from`/`--resume`. Test: serial **MIP** (e.g. + geometry+cfg fingerprint; atomic per-rank writes + manifest; terminal checkpoint + (`--checkpoint-at-termination`, default on) + optional periodic triggers + (`--checkpoint-every-iterations` / `--checkpoint-every-seconds`); CLI flags + `--checkpoint-dir`, `--checkpoint-at-termination`, `--checkpoint-backend`, + `--resume-from`/`--resume`. Test: serial **MIP** (e.g. `sizes`) stop+resume — run continues correctly, incumbent preserved, warm start taken (mid-run model dill round-trip proven, §6). - **Phase 2 — Multi-rank + bundles.** Barriers, rank-tagged files, single-generation From 24d28c073135d13b028adc49edf8b51211fa995a Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Mon, 20 Jul 2026 18:43:16 -0700 Subject: [PATCH 08/10] doc(checkpointing): mark leaf-rebuild backend / Phase 6 as not currently planned MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The primary use case is fully served by the dill-reload backend (Phases 1-4), so the leaf-rebuild backend and the broader spoke coverage bundled with it are deferred. Reframe §11 Phase 6 as a possible-but-unscheduled future phase (design keeps its hooks and the shared framework/manifest so it can be added later), and reconcile the three surfaces that implied leaf ships alongside dill-reload: §1 ("still supported" -> future option, not currently planned), §2.2 ("remains available" -> designed but not currently planned), and the §8 --checkpoint-backend flag (dill-reload is the only implemented/valid backend until Phase 6 lands). Co-Authored-By: Claude Opus 4.8 --- doc/designs/checkpointing_design.md | 32 ++++++++++++++++++----------- 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md index 63b542d78..74879928b 100644 --- a/doc/designs/checkpointing_design.md +++ b/doc/designs/checkpointing_design.md @@ -18,7 +18,8 @@ are planned, not crash-driven. The scenarios are **large MIPs**. That regime drives the design decisions below (dill the scenario models, restore a MIP warm start), and it is worth stating up front because a *different* use case — frequent checkpoints purely for hard-kill safety — would push toward a -lighter, leaf-data-only checkpoint (still supported; §4, §11 Phase 6). +lighter, leaf-data-only checkpoint — the leaf-rebuild backend, kept in this design +as a future option but **not currently planned** (§4, §11 Phase 6). **Goals** @@ -116,8 +117,8 @@ three days is negligible. And it *avoids re-running an expensive `scenario_creat on every resume, which for large models is itself a real saving. The alternative — rebuild each model via `scenario_creator` and overlay the state -as leaf data (arrays/name→value maps) — remains available as the **low-cost -backend**: +as leaf data (arrays/name→value maps) — is the **low-cost backend**, designed here +but **not currently planned** (§11 Phase 6): its checkpoints are tiny and fast (a handful of `O(first-stage)` arrays, no model structure) and version-robust (plain numbers, not pickled classes). That makes it the right choice for small scenarios, cheap creators, or *frequent* kill-safety @@ -439,10 +440,12 @@ extension is not attached at all — zero overhead, no files. - **`--checkpoint-dir `** — where per-rank files and the manifest are written (§10); its presence is what enables checkpointing. - **`--checkpoint-backend {dill-reload, leaf}`** — how scenario-model state is - restored (§2.2). Default `dill-reload` (captures the warm start + cuts, dodges an - expensive `scenario_creator` re-run). `leaf` is the **low-cost** option — tiny, - fast, version-robust checkpoints — for small/cheap-creator runs or frequent - kill-safety writes. + restored (§2.2). `dill-reload` is the default and the only backend implemented in + the planned phases (captures the warm start + cuts, dodges an expensive + `scenario_creator` re-run). `leaf` — the **low-cost** option (tiny, fast, + version-robust checkpoints, for small/cheap-creator runs or frequent kill-safety + writes) — is designed but **not currently planned** (§11 Phase 6); until that + phase lands, `dill-reload` is the only valid value. - **`--resume-from `** (or `--resume`, auto-selecting the latest *complete* checkpoint from the manifest) — reconstruct the wheel and restore. Resume requires identical geometry (§5.7); a mismatch is refused with a clear error. @@ -604,11 +607,16 @@ value. New tests are wired into `run_coverage.bash` **and** continues, best xhat preserved. - **Phase 5 — Exact spoke continuity (optional).** Hoist `ScenarioCycler`/`xh_iter` onto `self`; checkpoint the cursor (+ RNG getstate if a stream becomes stateful). -- **Phase 6 — Leaf-rebuild backend + broader coverage.** The lighter, - version-robust `--checkpoint-backend leaf` path (rebuild via `scenario_creator`, - overlay W/rho/nonants/fixedness, replay prox `cut_values`, optional all-var warm - start); this is what the PoC prototyped. Plus lagranger, FWPH, subgradient - spokes. +- **Phase 6 — Leaf-rebuild backend + broader coverage (not currently planned).** + A possible future phase, deferred: the primary use case is fully served by the + dill-reload backend (Phases 1–4), so this is recorded for when a lighter, + version-robust checkpoint is actually needed rather than scheduled now. It would + add the `--checkpoint-backend leaf` path (rebuild via `scenario_creator`, overlay + W/rho/nonants/fixedness, replay prox `cut_values`, optional all-var warm start — + what the PoC prototyped) plus lagranger, FWPH, and subgradient spoke coverage. + The design deliberately keeps this backend's hooks and the shared + framework/manifest (§2.2) so it can be added later without disturbing the shipped + dill-reload path. --- From d11832a041f28bb35611d3ab2b237494d9b91d01 Mon Sep 17 00:00:00 2001 From: Dave Woodruff Date: Thu, 23 Jul 2026 10:03:45 -0700 Subject: [PATCH 09/10] Retention policy: exactly one checkpoint generation, older not supported Co-Authored-By: Claude Fable 5 --- doc/designs/checkpointing_design.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md index 74879928b..0200dcb14 100644 --- a/doc/designs/checkpointing_design.md +++ b/doc/designs/checkpointing_design.md @@ -525,9 +525,10 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: checkpoint by atomically rewriting `manifest.json` (itself temp-then-rename) to point at the new complete generation (§10). That flip is the single commit point, so **one committed generation is enough**: a kill before it keeps the - previous checkpoint, a kill after it keeps the new one. The prior generation can - be deleted once the manifest is in place. Keeping a few older generations is - optional convenience, not a kill-safety requirement. + previous checkpoint, a kill after it keeps the new one. The prior generation is + deleted once the manifest is in place. Retaining more than one checkpoint is + **not supported**: exactly one committed generation exists at any time (plus + the in-progress one transiently during a publish). 8. **A `Checkpointer` extension** that writes on its active triggers and restores at `post_iter0_after_sync`: - *periodic* (`--checkpoint-every-iterations` / `--checkpoint-every-seconds`) — @@ -564,7 +565,8 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: # overwritten asynchronously on improvement (§9, item 6) ``` -The hub writes iteration-tagged generations under `hub/`; each spoke keeps a +The hub writes each checkpoint as an iteration-tagged generation under `hub/`, +deleting the prior one after the manifest flip (§9, item 7); each spoke keeps a single latest-wins file under `spokes/` that it overwrites atomically on improvement — the two are deliberately *not* aligned (§9, item 6). `manifest.json` is the single commit point: it names the latest *complete* hub @@ -631,8 +633,8 @@ Resolved (given the §1 use case): - **Warm start.** Worthwhile for MIPs (branch-and-bound benefits), free via the dilled model, fed through the existing `warmstart_subproblems` / `solution_available` path. -- **Checkpoint retention** (§9, item 7): a single manifest-published generation - suffices; older generations are optional history. +- **Checkpoint retention** (§9, item 7): exactly one manifest-published + generation is kept; retaining older generations is not supported. - **Spoke snapshot coordination** (§9, item 6): resolved by *not* coordinating. - **Mid-run MIP model dill round-trip** — was the load-bearing unvalidated assumption; **validated by the MIP dill-reload PoC** (§6), including the @@ -641,9 +643,9 @@ Resolved (given the §1 use case): Still open: -- **Disk footprint.** Dilled large MIP models × scenarios/rank × a few generations - can be large; the single-generation policy (§9, item 7) keeps only one live, but - document the peak (two generations during a publish). +- **Disk footprint.** Dilled large MIP models × scenarios/rank can be large; the + single-generation policy (§9, item 7) keeps only one checkpoint live, but + document the peak (two generations transiently during a publish). - **variable_probability / surrogate vars.** mpi-sppy masks `W` (not prox) for zero-probability nonants and assumes each surrogate var is fixed at 0. Restore must **reproduce that invariant**: dill-reload preserves the mask and fixedness From 3c95ae2208ce1123e25a61ec374a861d5d046c13 Mon Sep 17 00:00:00 2001 From: David L Woodruff Date: Tue, 28 Jul 2026 14:00:15 -0700 Subject: [PATCH 10/10] Design review: ADMM/CVaR coverage, in-core resume branch, A/B test harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - New §8.2: stochastic ADMM contract (naming/file discovery must not use module name creators or extract_num; creator-cost/memory caveats incl. releasing wrapper-held models; variable_probability identity-keying invariant; wrapper-mutated model dill round-trip to validate; AdmmBundler; spoke-set differences). - Restore is now an in-core resume branch in Iter0 replacing the iter-0 solve (precedent: iter0_from_pickle) — no throwaway W=0 solve; dropped the special end-of-iteration-0 checkpoint. - New §11.1: A/B full-run vs stop+resume CI harness with explicit similarity criteria; instances farmer, farmer+cvar, stoch-distr (--stoch-admm), sizes; wired into the phase test lists. - §10: define in file names; fold in the disk-footprint peak. - §12 recast as resolved decisions + deferrals (former open items folded into the sections where they belong). Co-Authored-By: Claude Fable 5 --- doc/designs/checkpointing_design.md | 304 ++++++++++++++++++++++------ 1 file changed, 241 insertions(+), 63 deletions(-) diff --git a/doc/designs/checkpointing_design.md b/doc/designs/checkpointing_design.md index 0200dcb14..53f7bbf43 100644 --- a/doc/designs/checkpointing_design.md +++ b/doc/designs/checkpointing_design.md @@ -114,7 +114,9 @@ serializes by class/closure reference) and **per-checkpoint overhead** (full models are large) — are both moot for this use case: the resume is same-environment and the checkpoints are infrequent, so a heavy write paid a handful of times over three days is negligible. And it *avoids re-running an expensive `scenario_creator`* -on every resume, which for large models is itself a real saving. +on every resume, which for large models is itself a real saving. (Exception: +the ADMM paths, where the wrapper re-runs the creator at startup regardless — +§8.2, item 2.) The alternative — rebuild each model via `scenario_creator` and overlay the state as leaf data (arrays/name→value maps) — is the **low-cost backend**, designed here @@ -206,10 +208,28 @@ already exist: `_populate_W_cache`/`W_from_flat_list` (`phbase.py`), `_save_nonants`/`_restore_nonants` (`spopt.py`, which already captures fixedness in `fixedness_cache`). -**Restore point:** after the scaffolding exists and models are loaded, before the -`iterk` loop. In serial this can be a driver call; in cylinders the hub runs -`ph_main` internally, so restore happens in the **`post_iter0_after_sync`** -extension hook (`post_iter0_after_sync`, end of `Iter0` in `phbase.py`). +**Restore point: a resume branch inside `Iter0`, in place of the iter-0 solve.** +`Iter0` (`phbase.py`) already contains the structural precedent: the +`iter0_from_pickle` option replaces the iter-0 `solve_loop` with +`_iter0_use_pickled_solution()`. Resume is a third branch at the same spot — +after `_create_solvers()`, *instead of* the solve loop — that loads the +checkpoint and proceeds to `iterk_loop`. This matters for the target use case: +restoring in a *post*-iter0 extension hook (what the PoC did, to avoid core +changes) would first solve every fresh model with `W = 0` and then throw those +solutions away — one discarded full MIP solve per scenario per resume, +plausibly hours. The design therefore specifies the core branch, not the hook +(§9, item 2): the PoC's `post_iter0_after_sync` restore was a +zero-core-change validation crutch and is **not** the design. + +On the resume branch, the rest of `Iter0` adjusts as follows: the feasibility +check reads the restored per-scenario feasibility flags; `trivial_bound` / +`best_bound_obj_val` are restored from the checkpoint's leaf data rather than +recomputed via `Ebound`; the spoke sync still runs (publishing the *restored* +W/xbar/nonants to the spokes — they start from checkpointed state +immediately); the `rho_setter` and the deferred +`_attach_PH_to_objective_after_iter0` are skipped (rho and the spliced +objective ride in the reloaded model); the converger is constructed as usual +and extension `restore_state` hooks (§9, item 3) fire before `iterk_loop`. ### 5.2 Recourse variable values — *warm start, in the dilled model* @@ -373,7 +393,11 @@ deterministic solve — the §7 validation crutch): Still to prove in later phases (this PoC was serial and focused on the model round-trip + continuation): the dill-reload backend under **multi-rank** and **cylinders**; carrying the **incumbent** across a dill-reload stop; a measured -warm-start speedup; and the disk/time footprint at true model scale. +warm-start speedup; the disk/time footprint at true model scale; and the +mid-run dill round-trip of a **stoch-ADMM wrapper-mutated model** (§8.2, item +4), which is structurally stranger than anything this PoC dilled. Note also +that both PoCs restored in the `post_iter0_after_sync` hook; the design +replaces that with the in-core resume branch (§5.1), which is itself unproven. --- @@ -428,12 +452,13 @@ extension is not attached at all — zero overhead, no files. checkpoint cannot be taken mid-solve. Distinct from `--time-limit`, which *stops* the run: this keeps it running and snapshots. - **`--checkpoint-every-iterations K` (optional insurance).** Also checkpoint every - `K` PH iterations, **and always at the end of iteration 0** (which establishes - the resume baseline — solvers created, trivial bound computed, initial `W = 0` - solve done — and composes with `--iter0-from-pickle`). Checked at `enditer`. - (This is the former `--checkpoint-every k`, renamed for symmetry with - `--checkpoint-every-seconds`.) Given infrequent planned stops, most runs leave - the periodic triggers off and rely on the terminal checkpoint alone. + `K` PH iterations. Checked at `enditer`. (This is the former + `--checkpoint-every k`, renamed for symmetry with + `--checkpoint-every-seconds`.) There is no special iteration-0 checkpoint: + resume is an in-core branch that restores any checkpointed iteration directly + (§5.1), so iteration 0 is not a privileged baseline. Given infrequent planned + stops, most runs leave the periodic triggers off and rely on the terminal + checkpoint alone. **Other options** @@ -470,6 +495,76 @@ on bundles" (`spopt.py`). That predates proper bundles and refers to the removed loose mechanism; re-verify and refresh it when bundle checkpointing is validated (Phase 2). +### 8.2 ADMM (deterministic and stochastic) + +`--admm` / `--stoch-admm` runs (`generic/admm.py`, `utils/admmWrapper.py`, +`utils/stoch_admmWrapper.py`) are plain PH hubs over *wrapped* scenarios, so +the checkpoint machinery applies in principle — but the wrapper path breaks +several assumptions made elsewhere in this design. Each of the following must +be honored or checkpointing will not work for ADMM: + +1. **Scenario naming and file discovery.** The existing pickle paths that §4 + builds on are *hard-refused* for ADMM (`_check_admm_compatibility`, + `generic/admm.py`) because `scenario_io.py` derives file names from + `module.scenario_names_creator` and `sputils.extract_num` — wrapped names + (`ADMM_STOCH__ADMM____ADMM__`) come from the wrapper, not the + module, and `extract_num` scrapes trailing digits, colliding across ADMM + subproblems that share a stochastic scenario. The checkpoint code must + therefore (a) enumerate `opt.local_scenarios.keys()` — never a module name + creator — for both write and restore, (b) never use `extract_num` in file + names (§10), and (c) not be swept into the ADMM incompatibility checks the + way the pickle flags were: checkpointing is *supposed* to work here, and a + test should pin that it does. +2. **The creator-cost saving does not apply, and a naive resume doubles model + memory.** §2.2 counts "avoids re-running an expensive `scenario_creator`" + as a dill-reload benefit. Not for ADMM: `Stoch_AdmmWrapper.__init__` runs + the user's `scenario_creator` for every local wrapped scenario (plus probe + scenarios) during normal startup — it needs the built models to assemble + consensus lists, `varprob_dict`, node names, and objective scaling — so an + ADMM resume pays the full creator cost regardless. Worse, after the reload + branch swaps the dilled models into `local_scenarios`, the fresh models + remain referenced by `wrapper.local_admm_stoch_subproblem_scenarios` and by + the `cfg._admm_variable_probability` bound method — a *persistent* 2× + per-rank model footprint for large MIPs. The reload branch must release or + replace the wrapper-held fresh models (§9, item 2). +3. **`variable_probability` is object-identity-keyed.** The wrapper's + `varprob_dict` maps scenario *object* → `(id(var), prob)` pairs + (`stoch_admmWrapper.py`; `AdmmBundler._bundle_varprob` likewise). This is + safe today only because `_use_variable_probability_setter` runs exactly + once, in `SPBase.__init__`, against the wrapper's own model objects, and + its results land on the model itself (`s._mpisppy_data.prob_coeff` / + `prob0_mask`) — which the dilled model carries back. The reload branch + depends on that invariant: **variable probabilities are consumed only at + construction; after the swap, the reloaded model's `_mpisppy_data` masks + and fixed-at-0 dummy vars are authoritative, and `var_prob_list` must never + be called with a reloaded model** (it would `KeyError` — or silently + mismatch if the dict were rebuilt with new ids). mpi-sppy masks `W` (not + prox) for zero-probability nonants and assumes each surrogate/dummy var is + fixed at 0; dill-reload preserves the mask and the fixedness together, and + the ADMM resume test must assert both survive. (A leaf-rebuild ADMM resume + would have to re-apply the mask and re-fix the dummies explicitly — one + more reason that backend is deferred, §11 Phase 6.) +4. **The dill round-trip is unvalidated for a wrapper-mutated model.** The MIP + PoC (§6) dilled a plain `sizes` model. A stoch-ADMM scenario is stranger: + inline dummy `pyo.Var()`s added post-construction with bracket-mangled + names, rewritten `ScenarioNode`s carrying *unattached* + `pyo.Expression(expr=0)` cost expressions and `surrogate_vardatas` sets of + vardata references, a rescaled objective, an appended ADMM stage + (a multistage tree even for a 2-stage-origin problem), and + probability-mask arrays on `_mpisppy_data`. dill should handle the cycles, + but this is a load-bearing assumption of the same kind §6 insisted on + PoC-ing — validate a stoch-ADMM mid-run round-trip early + (`mpisppy/tests/examples/stoch_distr` is the vehicle, + `test_stoch_admmWrapper.py` the harness; §11 Phases 2 and 4). +5. **Bundled stoch-ADMM.** `--stoch-admm --scenarios-per-bundle` + (`AdmmBundler`) creates bundles on the fly as EFs; they are first-class + subproblems and should dill like other proper bundles (§8.1). Its + `var_prob_list` has the same identity keying as item 3. +6. **The spoke set differs.** For stoch-ADMM cylinders: FWPH is refused, + `xhatshuffle` requires `--stage2-ef-solver-name`, and `xhatxbar` is the + variable-probability-native inner bounder. The Phase 4 test matrix must + include a stoch-ADMM configuration (§11). + --- ## 9. Core changes required @@ -481,22 +576,30 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: from 1 and its checkpoints collide with the pre-crash ones. Add a resume offset so checkpoint numbering is the global iteration and termination honors the original `max_iterations`. -2. **A reload-model resume branch.** When restoring via dill-reload, startup must - reconstruct comms/windows/solvers but **skip both `attach_Ws_and_prox` and - `attach_PH_to_objective`** — the reloaded model already carries the W/rho/xbars - params, the spliced objective, and the prox cuts, so re-running either would - duplicate components or double the terms. Then strip/rebuild `_solver_plugin` - (`set_instance`) and set `solution_available` for the warm start (§5.2). Two - details the PoC surfaced: **refresh `saved_objectives[sname]`** for each - reloaded model — `Eobjective` reads those objective handles and they otherwise - dangle to the discarded fresh model — and swap the reloaded model into - `local_scenarios` (which `SPOpt.solve_loop` iterates). A plain PH keeps no - `local_subproblems`, but the **generic file-based path** the dill-reload backend - builds on (`scenario_io.py` sets `sp.local_subproblems = sp.local_scenarios`) - maintains that alias, and `CGBase.solve_loop` iterates it — so where the alias - exists the reload branch must refresh it too, or it dangles to the discarded - model. This is a distinct branch from the leaf-rebuild "build fresh, overlay - values" path; the `Checkpointer` picks the branch from `--checkpoint-backend`. +2. **A reload-model resume branch, in `Iter0`, replacing the iter-0 solve.** + The branch lives where `iter0_from_pickle` already branches (§5.1): after + `_create_solvers()`, instead of the iter-0 `solve_loop` — so a resume never + pays a throwaway `W = 0` solve of the fresh models. When restoring via + dill-reload, startup must reconstruct comms/windows/solvers but **skip both + `attach_Ws_and_prox` and `attach_PH_to_objective`** — the reloaded model + already carries the W/rho/xbars params, the spliced objective, and the prox + cuts, so re-running either would duplicate components or double the terms. + Then strip/rebuild `_solver_plugin` (`set_instance`) and set + `solution_available` for the warm start (§5.2). Details the PoC and the ADMM + analysis surfaced: **refresh `saved_objectives[sname]`** for each reloaded + model — `Eobjective` reads those objective handles and they otherwise dangle + to the discarded fresh model; swap the reloaded model into `local_scenarios` + (which `SPOpt.solve_loop` iterates); where the `local_subproblems` alias + exists, refresh it too — a plain PH keeps no `local_subproblems`, but the + **generic file-based path** the dill-reload backend builds on + (`scenario_io.py` sets `sp.local_subproblems = sp.local_scenarios`) + maintains it, and `CGBase.solve_loop` iterates it; and on ADMM runs, + **release or replace the fresh models held by the wrapper** + (`local_admm_stoch_subproblem_scenarios` and the + `cfg._admm_variable_probability` closure), or the run keeps two copies of + every local scenario alive for its whole life (§8.2, item 2). This is a + distinct branch from the leaf-rebuild "build fresh, overlay values" path; + the `Checkpointer` picks the branch from `--checkpoint-backend`. 3. **Extension `checkpoint_state` / `restore_state` contract** on `Extension` (no-ops by default; implemented by rho updaters, `fixer`, `slammer`, convergers). Covers **extension-object** state under both backends; @@ -529,8 +632,9 @@ Touch-points an implementation needs beyond the PoC's extension/subclass hacks: deleted once the manifest is in place. Retaining more than one checkpoint is **not supported**: exactly one committed generation exists at any time (plus the in-progress one transiently during a publish). -8. **A `Checkpointer` extension** that writes on its active triggers and restores - at `post_iter0_after_sync`: +8. **A `Checkpointer` extension** that writes on its active triggers; restore + itself is the in-core resume branch (item 2), with extension + `restore_state` hooks (item 3) fired from it before `iterk_loop`: - *periodic* (`--checkpoint-every-iterations` / `--checkpoint-every-seconds`) — at `enditer`. The seconds trigger tests `allreduce_or(now − last_checkpoint ≥ S)` so all ranks decide together @@ -575,6 +679,17 @@ backend the `.dill` model files are replaced by numeric arrays inside the `hub_rank_*.pkl`. Use plain `pickle` for the numeric/leaf state; `dill` for the scenario models. +`` is the scenario's full name sanitized for the filesystem (or its index in +the rank's local scenario list) — **never** `sputils.extract_num`, which is not +unique for ADMM wrapped names (§8.2, item 1). More generally, both write and +restore enumerate `opt.local_scenarios.keys()`, not a module name creator. + +**Disk footprint.** Dilled large MIP models × scenarios/rank can be large. The +single-generation policy (§9, item 7) keeps exactly one checkpoint live, but +the peak is **two generations transiently during a publish** (the new one is +fully written before the manifest flip deletes the old one) — state the peak in +user docs so disk quotas are sized for it. + --- ## 11. Phased rollout @@ -583,20 +698,72 @@ Each phase is a review-sized PR that is green on its own and adds user-visible value. New tests are wired into `run_coverage.bash` **and** `test_pr_and_main.yml` in the same commit. +### 11.1 The A/B resume harness (every phase's acceptance test) + +The core CI test shape, reused by every phase, is an **A/B comparison**: + +- **Run A (reference):** an uninterrupted run of `N` iterations on a small + instance. +- **Run B (checkpointed):** the same instance stopped at iteration `k < N` + with a checkpoint written, then resumed in a **fresh process** and run to + `N`. +- **Compare A and B** under the §7 determinism contract: + - **Deterministic LP instances** (farmer; farmer + CVaR): `W`, nonants, + `rho`, `xbar` at each common iteration and the final objective must be + **bit-identical** (`max|diff| == 0.0`), and final bounds equal. + - **MIP instances** (`sizes`): with single-thread deterministic solver + settings (`Threads=1`, fixed seed, `MIPGap=0` — the §7 validation crutch) + the same bit-identity check applies; under default settings assert instead + that the run **continues** (global iteration numbering, no re-attach / + duplicate-component errors), the **incumbent never regresses**, bounds + stay valid, and the final objective agrees within a stated tolerance. +- Also assert the negatives: run B performs **no iter-0 subproblem solve** on + resume (§5.1), and a geometry/cfg mismatch is refused with a clear error + (§5.7). + +Instances — all small enough for the pip-installed, size-limited CPLEX/Xpress +CI solvers: + +- **farmer** — deterministic-LP baseline, serial and cylinders. +- **farmer + `--cvar`** (`utils/cvar.py`) — a mutate-after-creation transform: + the deactivated risk-neutral objective, the active `WITH_CVAR` objective, + and the eta var appended to the root nonants must all survive the dill + round-trip, and the resume branch's `saved_objectives` refresh (§9, item 2) + must resolve to `WITH_CVAR`, not the deactivated original. +- **stoch-distr (`--stoch-admm`)** — exercises everything in §8.2: wrapped + names in file discovery, variable-probability masks and fixed-at-0 dummy + vars, the wrapper-mutated model dill round-trip, and release of the + wrapper-held fresh models. +- **`sizes`** — the MIP target: warm start taken on resume, incumbent carried. + +The phase bullets below say where each instance enters (Phase 1: serial +farmer / farmer+CVaR / `sizes`; Phase 2: multi-rank, bundles, stoch-ADMM; +Phase 4: cylinders, including a stoch-ADMM configuration). + - **Phase 1 — Serial hub checkpoint/resume, dill-reload backend.** `Checkpointer` extension; global iteration counter / resume offset; reload-model resume branch - (skip `attach_PH_to_objective`, rebuild `_solver_plugin`, set warm start); - geometry+cfg fingerprint; atomic per-rank writes + manifest; terminal checkpoint - (`--checkpoint-at-termination`, default on) + optional periodic triggers - (`--checkpoint-every-iterations` / `--checkpoint-every-seconds`); CLI flags - `--checkpoint-dir`, `--checkpoint-at-termination`, `--checkpoint-backend`, - `--resume-from`/`--resume`. Test: serial **MIP** (e.g. - `sizes`) stop+resume — run continues correctly, incumbent preserved, warm start - taken (mid-run model dill round-trip proven, §6). -- **Phase 2 — Multi-rank + bundles.** Barriers, rank-tagged files, single-generation - atomic publish. Validate with **proper bundles** (§8.1) and refresh the stale - `_restore_nonants` comment. Test: `mpiexec` MIP stop+resume on every rank, incl. - uneven distribution and `--scenarios-per-bundle`; mismatch refusal. + **in `Iter0`, replacing the iter-0 solve** (§5.1, §9 item 2 — skip + `attach_PH_to_objective`, rebuild `_solver_plugin`, set warm start, no + throwaway solve); geometry+cfg fingerprint; atomic per-rank writes + manifest; + terminal checkpoint (`--checkpoint-at-termination`, default on) + optional + periodic triggers (`--checkpoint-every-iterations` / + `--checkpoint-every-seconds`); CLI flags `--checkpoint-dir`, + `--checkpoint-at-termination`, `--checkpoint-backend`, + `--resume-from`/`--resume`. Tests (the §11.1 A/B harness, serial): **farmer** + and **farmer + `--cvar`** bit-identical A vs B; **`sizes`** (MIP) — + bit-identical under deterministic solver settings, and under default settings + run continues correctly, incumbent preserved, warm start taken (mid-run model + dill round-trip proven, §6); no iter-0 subproblem solve occurs on resume. +- **Phase 2 — Multi-rank + bundles + stoch-ADMM.** Barriers, rank-tagged files, + single-generation atomic publish. Validate with **proper bundles** (§8.1) and + refresh the stale `_restore_nonants` comment. Validate **stoch-ADMM** (§8.2): + the wrapper-mutated model dill round-trip (item 4), file naming with wrapped + scenario names (item 1), wrapper-held fresh models released on resume (item + 2), and the probability mask + dummy-var fixedness surviving restore (item + 3) — `mpisppy/tests/examples/stoch_distr` is the vehicle. Tests (the §11.1 + A/B harness under `mpiexec`): MIP stop+resume compared on every rank, incl. + uneven distribution and `--scenarios-per-bundle`; a stoch-distr + (`--stoch-admm`) A/B stop+resume; mismatch refusal. - **Phase 3 — Extension-object state contract.** `checkpoint_state`/`restore_state` on `Extension`; implement for rho updaters, `fixer`, `slammer`, convergers. (Model-attached `fixer` counter and nonant fixedness ride in the dill.) Test: PH @@ -605,8 +772,12 @@ value. New tests are wired into `run_coverage.bash` **and** - **Phase 4 — Cylinders / spokes.** One-line xhatter write hook; unified `Checkpointer` on spoke opts; each spoke checkpoints its own **best xhat** (by name) asynchronously on improvement — no hub↔spoke coordination (§9, item 6). - Test: farmer/`sizes` cylinders (hub+lagrangian+xhatshuffle) stop+resume — run - continues, best xhat preserved. + Tests (the §11.1 A/B harness on cylinders): farmer/`sizes` + (hub+lagrangian+xhatshuffle) stop+resume — hub primal trajectory compared A + vs B (bit-identical for farmer, per the §6 PoC), best xhat preserved, bounds + valid best-so-far — **plus a stoch-ADMM cylinders configuration** (§8.2, + item 6: no FWPH; `xhatshuffle` with `--stage2-ef-solver-name`, or + `xhatxbar`). - **Phase 5 — Exact spoke continuity (optional).** Hoist `ScenarioCycler`/`xh_iter` onto `self`; checkpoint the cursor (+ RNG getstate if a stream becomes stateful). - **Phase 6 — Leaf-rebuild backend + broader coverage (not currently planned).** @@ -622,36 +793,43 @@ value. New tests are wired into `run_coverage.bash` **and** --- -## 12. Open questions / risks +## 12. Design decisions (resolved) and deferrals Resolved (given the §1 use case): - **Backend choice.** dill the scenario models (§2.2): overhead is negligible at a few checkpoints, version robustness is unneeded (same-environment resume next day), and it captures the warm start + prox cuts + model-attached state for free - while avoiding an expensive `scenario_creator` re-run. + while avoiding an expensive `scenario_creator` re-run (except on ADMM paths — + §8.2, item 2). - **Warm start.** Worthwhile for MIPs (branch-and-bound benefits), free via the dilled model, fed through the existing `warmstart_subproblems` / `solution_available` path. +- **Restore point.** An in-core resume branch in `Iter0` replacing the iter-0 + solve (§5.1; §9, item 2) — no throwaway `W = 0` solve on resume, and + consequently no special iteration-0 checkpoint (§8). The PoCs' extension-hook + restore was a validation crutch, not the design. - **Checkpoint retention** (§9, item 7): exactly one manifest-published - generation is kept; retaining older generations is not supported. + generation is kept; retaining older generations is not supported. The disk + peak is two generations transiently during a publish (§10) — a documented + cost, not an open question. - **Spoke snapshot coordination** (§9, item 6): resolved by *not* coordinating. +- **variable_probability / surrogate vars (incl. ADMM).** Resolved by the §8.2 + contract: probabilities are consumed only at `SPBase` construction; after the + reload swap, the reloaded model's `_mpisppy_data` masks and fixed-at-0 + dummy/surrogate vars are authoritative, and `var_prob_list` is never called + with a reloaded model. Validation is scheduled (Phase 2). - **Mid-run MIP model dill round-trip** — was the load-bearing unvalidated assumption; **validated by the MIP dill-reload PoC** (§6), including the linearized-prox cuts, in-process and cross-process, with serial stop→reload→ - continue bit-identical under a deterministic solver. - -Still open: - -- **Disk footprint.** Dilled large MIP models × scenarios/rank can be large; the - single-generation policy (§9, item 7) keeps only one checkpoint live, but - document the peak (two generations transiently during a publish). -- **variable_probability / surrogate vars.** mpi-sppy masks `W` (not prox) for - zero-probability nonants and assumes each surrogate var is fixed at 0. Restore - must **reproduce that invariant**: dill-reload preserves the mask and fixedness - automatically, but verify; leaf-rebuild must re-apply the mask and re-fix the - surrogate at 0 (§5.1 fixedness), not just reload raw `W`. Among the spokes only - `xhatxbar` supports variable_probability today. -- **Cross-geometry resume** is explicitly deferred; revisit if HPC users need to - resume on a different node count. -``` + continue bit-identical under a deterministic solver. The stoch-ADMM + wrapper-mutated variant of the same assumption is not yet validated — that is + a scheduled validation item (§6; §8.2, item 4; §11 Phase 2), not an open + design question. + +Deferred: + +- **Cross-geometry resume** (different rank count or scenario-to-rank + distribution) — a §1 non-goal; revisit if HPC users need to resume on a + different node count. +- **Leaf-rebuild backend** — designed (§2.2) but not scheduled (§11, Phase 6).