Mutable scenario probabilities (issue #797)

[DLWoodruff]

Mutable scenario probabilities (issue #797)

Closes #797.

Today each scenario's probability is folded into the Extensive Form (EF)
objective as a floating-point constant when the model is built, so changing a
probability requires rebuilding the objective and, for a persistent solver,
re-loading the instance. This PR adds an opt-in mutable_probability mode that
stores each probability as a mutable Pyomo Param in the objective, so the
probability vector can be updated in place and a persistent solver re-solved
cheaply — the motivating use case in the issue (a rolling-horizon loop that
re-weights a fixed scenario set between solves), and probability-sensitivity
studies.

What's in this PR

• Persistence detection — sputils.has_persistent_solve_api recognizes
  the APPSI / pyomo.contrib.solver interface (e.g. appsi_highs, the
  solver in the issue) as persistent for the EF workflow, in addition to legacy
  PersistentSolver. Kept separate from is_persistent() on purpose: APPSI's
  LegacySolver lacks the update_var/add_var/add_constraint methods that
  the PH/FWPH sites call behind is_persistent, so broadening that would break
  them.
• EF path — mutable_probability option on ExtensiveForm /
  sputils._create_EF_from_scen_dict builds the objective against a mutable
  Param; ExtensiveForm.set_scenario_probabilities(prob_map) validates and
  applies a new vector (transactionally) and re-pushes the objective;
  solve_extensive_form(reuse_instance=True) skips set_instance so the
  persistent solver keeps its loaded instance across a sweep.
• PH path — SPBase.set_scenario_probabilities(prob_map, check_sum=True, reset_ph_duals=True) updates _mpisppy_probability, forces prob_coeff to
  be recomputed, re-applies any variable-probability overrides, and (by
  default) zeroes the PH multipliers W. Two-stage for now.
• Docs + example — doc/src/mutable_probability.rst and a runnable
  examples/farmer/farmer_prob_sensitivity.py that sweeps the weight on one
  scenario and reuses the persistent instance across the sweep.

Normalization: require sum-to-1 (option B)

On the mutable path the supplied probabilities must sum to 1 (within 1e-9);
there is no re-normalization, so the objective stays a plain probability-weighted
sum with no division node. The existing float path — including the normalization
divisor that scenario bundles rely on — is untouched, because the mutable
flag is opt-in and bundles never set it (requesting mutable_probability for a
bundle raises). Partial mappings are allowed: omitted scenarios keep their
current probability as long as the full vector still sums to 1. A rejected call
leaves the model unchanged.

PH warm-start caveat (why reset_ph_duals defaults to True)

At a converged PH solution every scenario sits at the same nonanticipative
point, so the probability-weighted xbar is independent of the weights.
Re-solving from there with new probabilities but stale W leaves xbar
unmoved and PH reports immediate (false) convergence at the old solution.
Zeroing W breaks that consensus so PH re-converges for the new probabilities.
Covered by a regression test that shows reset_ph_duals=False stays stuck.

Cross-solver behavior (verified)

• APPSI (appsi_highs) auto-tracks the objective change on the next
  solve(); the explicit set_objective re-push is redundant but harmless.
• Legacy persistent (gurobi_persistent) does not auto-track, so the
  re-push is required — set_scenario_probabilities does it.

Verified end-to-end on the farmer example against a rebuild oracle to machine
precision across a probability sweep, with set_instance called once, for both
appsi_highs and gurobi_persistent.

Tests

Test_mutable_probability and Test_mutable_probability_ph in
mpisppy/tests/test_ef_ph.py (EF-vs-rebuild-oracle across a sweep, single
set_instance, guards/transactionality, PH prob_coeff refresh,
fresh-PH-vs-EF-oracle, in-place reuse, and the stuck-consensus case). Full file
passes; ruff clean; docs build.

Design and follow-up

doc/designs/mutable_scenario_probabilities_design.md records the full
rationale. A follow-up (design §9 phase 3) will extend
SPBase.set_scenario_probabilities to multistage node probabilities
(ScenarioNode.cond_prob) and the variable-probability interaction; a
multistage node list currently raises NotImplementedError.

[codecov]

Codecov Report

❌ Patch coverage is 92.85714% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.21%. Comparing base (1e1b734) to head (a4a23b8).
⚠️ Report is 42 commits behind head on main.

Files with missing lines	Patch %	Lines
mpisppy/utils/sputils.py	82.35%	3 Missing ⚠️
mpisppy/spbase.py	92.85%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #799      +/-   ##
==========================================
+ Coverage   76.18%   76.21%   +0.02%     
==========================================
  Files         169      169              
  Lines       22224    22287      +63     
==========================================
+ Hits        16932    16986      +54     
- Misses       5292     5301       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bknueven]

Since you included PH -- I'm guessing just disabling prox is not a good thing, as you could still converge immediately for some problems. But could we do something less extreme than resetting W, and just do a partial reset by taking a convex combination of the converged re-normalized or re-projected W and 0?

Or, we can disable prox and keep the re-normalized or re-projected W's for the first iteration.