Checkpoint/resume design (doc only, for review)

[DLWoodruff]

Checkpoint / resume — design for review (no library code yet)

This PR adds a design document (doc/designs/checkpointing_design.md) for
checkpointing an mpi-sppy run so it can be stopped and resumed later. It is
doc-only and intended for design review before any implementation.

Use case driving the design

A long (multi-day) run that is intentionally stopped and resumed on a schedule
— e.g. a three-day study that ends each day and picks up the next morning on the
same cluster — with large MIP scenarios. Checkpoints are infrequent (a
handful over the whole run, roughly twice as many writes as resumes). This regime
is what makes the primary design choice below the right one; a different regime
(frequent checkpoints purely for hard-kill safety) is served by the low-cost
backend, also in the doc.

TL;DR

• Do not dill the opt/hub object graph — it holds live MPI communicators, RMA
  windows, and persistent solver handles. Those are always reconstructed via
  normal startup (a fresh MPI job each morning).
• Do dill the mid-run scenario models — this is the recommended backend for
  the use case above. It captures W/rho/nonants/fixedness, the second-stage
  MIP warm start, the linearized-prox cuts, and model-attached extension state
  in one consistent shot, and avoids re-running an expensive scenario_creator.
  At a few checkpoints the write cost is negligible, and same-environment resume
  makes dill's version fragility a non-issue.
• A low-cost leaf-rebuild backend remains available (--checkpoint-backend leaf): rebuild via scenario_creator and overlay tiny numeric arrays — small,
  fast, version-robust — for small/cheap-creator runs or frequent kill-safety
  writes.
• Keep the best xhat, not just the best bound: in a cylinders run the best
  xhat solution values live on the xhat spoke, while the hub holds only the
  incumbent objective. The design checkpoints both.

Validated by proofs of concept

• Framework + leaf-rebuild backend (farmer LP, gurobi_persistent): serial and
  multi-rank (-np 3, uneven -np 2) resume reproduces a full run
  bit-identically for W/nonants/rho; cylinders (PH hub + lagrangian +
  xhatshuffle) hub primal resumes bit-identical inside WheelSpinner; geometry
  mismatch is refused with a clear error.
• dill-reload backend on a MIP (sizes SIZES3, gurobi_persistent,
  single-thread for a deterministic solve): a mid-run scenario model — including
  the linearized-prox case with 845 cuts + 65 ProxApproxManagers — dills and
  reloads in-process and cross-process, re-solving to identical objective and
  decision variables; a serial stop → reload → continue is bit-identical to
  an uninterrupted run for both quadratic and linearized prox.
• Still to prove in later phases: the dill-reload backend under multi-rank and
  cylinders, incumbent carry across a dill-reload stop, a measured warm-start
  speedup, and the disk footprint at true model scale.

Determinism contract

• MIPs (the target): resume continues correctly and warm-started, with the
  best xhat preserved — but is not bit-reproducible (multi-threaded MIP solves
  are nondeterministic and admit multiple optima). The PoC's bit-identity results
  used a single-thread deterministic solve as a validation crutch.
• Leaf-rebuild on a deterministic LP/QP solve: primal trajectory can be
  bit-identical (a bonus, not the target).
• Bounds and incumbent: valid, best-so-far, not bit-reproducible (async).

What the doc specifies

State inventory (reconstructed vs restored; what rides in the dilled model vs
non-model leaf data), the required core changes (global iteration counter, a
reload-model resume branch that skips re-attach and refreshes objective handles, a
one-line xhatter write hook so a single Checkpointer serves hub + spoke, an
extension checkpoint_state/restore_state contract, async per-spoke incumbent
checkpoints with no hub↔spoke coordination, atomic single-generation writes via a
manifest), a proposed file layout, and a 6-phase rollout (each phase a
review-sized PR).

Ready for review. Implementation would land as the phased PRs described in the doc.

---

Update log

Per review (@bknueven): leaned into mpi-sppy's asynchrony — dropped the
hub-triggered snapshot barrier (each spoke checkpoints its own incumbent
asynchronously on improvement) and simplified retention to a single
manifest-published generation.

Design pivot (this revision): re-centered on the concrete use case above
(multi-day planned stop/resume, huge MIP scenarios, same environment). The
recommended backend is now dill the mid-run scenario models rather than
leaf-data-only — at this checkpoint cadence the overhead is negligible and it
captures the MIP warm start + prox cuts for free — with leaf-rebuild kept as the
low-cost alternative. Softened the determinism contract for MIPs, shifted the
primary trigger to end-of-run/on-signal, and validated the load-bearing
assumption (a mid-run MIP model round-trips through dill) with the MIP PoC above.

Design review pass (this revision): added §8.2 stochastic ADMM (wrapped-name
file discovery — no scenario_names_creator/extract_num; the creator-cost saving
does not apply and resume must release the wrapper-held fresh models; the
variable_probability identity-keying invariant; the wrapper-mutated model dill
round-trip flagged for early validation; AdmmBundler; spoke-set differences).
Restore is now an in-core resume branch in Iter0 replacing the iter-0 solve
(precedent: iter0_from_pickle) — no throwaway W = 0 solve on resume, and the
special end-of-iteration-0 checkpoint is dropped. Added §11.1, an A/B
(uninterrupted vs stop+resume) CI harness with explicit similarity criteria over
small instances: farmer, farmer+--cvar, stoch-distr (--stoch-admm), and
sizes. §12 recast as resolved decisions + deferrals.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.49%. Comparing base (64a8fff) to head (3c95ae2).
⚠️ Report is 43 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #777      +/-   ##
==========================================
+ Coverage   76.16%   76.49%   +0.32%     
==========================================
  Files         169      175       +6     
  Lines       22224    23236    +1012     
==========================================
+ Hits        16928    17774     +846     
- Misses       5296     5462     +166     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bknueven]

I'm confused on this point -- doesn't serializing the best solution as already stored by the xhat / incumbent spokes already do this?

[DLWoodruff]

Yes — and that's the direction I took the revision. §5.3 already identified the xhat spoke's best_solution_cache as the thing to serialize; I've now made it explicit (revised §5.3 and §9 item 6) that the spoke checkpoints that cache itself, asynchronously on each improvement (reusing _maybe_write_incumbent_on_improvement), with no separate hub-driven coordination. So there's no extra machinery beyond serializing what the spoke already holds.

[bknueven]

Isn't it sufficient to just keep a single snapshot? After we complete the operations to write the new snapshot, can't we just immediately delete the prior one?

[DLWoodruff]

Agreed — revised §9 item 7 to a single published generation. The new generation is written to fresh files, then manifest.json is atomically renamed to point at it (the single commit point), then the prior generation is deleted. A kill before the flip keeps the previous complete checkpoint; a kill after keeps the new one. So one committed generation is enough for kill-safety; keeping older generations is now just optional history, not a requirement.

[DLWoodruff]

Follow-up: the doc now goes one step further and pins this down — retention of more than one checkpoint is explicitly not supported (d11832a). Exactly one committed generation exists at any time; the prior one is deleted right after the manifest flip, and the only time two exist is transiently during a publish (the documented peak disk footprint). The earlier "older generations are optional history" wording is gone.

[Copilot]

Pull request overview

Adds a design document describing a checkpoint/resume architecture for mpi-sppy runs (including cylinders), focusing on planned stop/resume of long-running, large-scenario MIP studies by reconstructing MPI/solver scaffolding and restoring algorithm/model state (primarily via dilling mid-run scenario models).

Changes:

• Introduces a doc-only design specifying what state is reconstructed vs restored, and proposes two backends (dill-model recommended; leaf alternative).
• Defines checkpoint semantics (triggers, atomic publication via a manifest, geometry/cfg fingerprinting) and details state inventory including incumbents and extension state.
• Proposes a phased implementation plan with testing expectations and rollout milestones.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[DLWoodruff]

Good catch — this was inaccurate, and it flagged a real implementation gotcha. Corrected in 3bcc029: a plain PH has no local_subproblems, but the generic file-based path the dill-reload backend builds on sets sp.local_subproblems = sp.local_scenarios (scenario_io.py) and CGBase.solve_loop iterates it, so the reload branch must refresh that alias where it exists or it dangles to the discarded model. (Re-opened — this had been resolved by mistake.)

[DLWoodruff]

Agreed — replaced the file.py:line references with file + symbol (e.g. warmstart_subproblems in spopt.py) in 3bcc029 so they stay stable as the code moves.

[DLWoodruff]

Fixed in 3bcc029 — the CLI flag is now --checkpoint-backend dill-reload (was dill-model), so the flag and the narrative use one name, with "reload" as the restore step.