Should PET use MMORPG?

[patnr]

Look into compatibility and relevance.

MMORPG

• Pro: MMORPG already specialises for NORCE cluster.
• Con: MMORPG targets dispatching inhomogeneous experiment arrays, not homogeneous ensemble runs.

[patnr]

Findings

Recommendation: no — PET shouldn't depend on MMORPG. The real blockers:

1. dispatch() is one-shot; our loop is iterative. Each call stamps a fresh
   timestamp/tag dir, copytrees all of proj_dir into it, and rsyncs to a new
   remote dir — no reuse across calls. MMORPG is "upload once, run a huge grid";
   we're "run many rounds against a fixed deck".
2. fun must be a module-level named function. MMORPG never pickles it
3. MMORPG can't submit SLURM from on the cluster.

Also: PET is requires-python >=3.8 vs MMORPG's >=3.12, and we'd use only its
dispatch/ subpackage (results/ is parameter-sweep tables — irrelevant).

Worth stealing

• Split batch count from CPU count. ensemble.py passes batch_size=nparallel into
  run_on_HPC — a CPU count used as a SLURM array size. Different knobs, different
  constraints.
• scancel on interrupt. run_on_HPC leaves jobs queued if you Ctrl-C out of
  wait_for_jobs.
• Keep the exception, not bare False. A crashed member is silently replaced by a
  random successful one; the reason is lost.