Skip to content

feat(compute-hosts): per-host path_namespace for runs on hosts with a shadowed NAS path - #725

Merged
dmlguq456 merged 5 commits into
mainfrom
framed-sr-corrnet-peer-ref-e87708b381-9dda6d6356
Oct 21, 2026
Merged

dmlguq456 merged 5 commits into
mainfrom
framed-sr-corrnet-peer-ref-e87708b381-9dda6d6356

Conversation

@dmlguq456

@dmlguq456 dmlguq456 commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

On mpwav, /home/nas is now the company NAS stacked over the lab NAS; the lab NAS is at /home/nas_IIP. The company NAS also has /home/nas/user/Uihyeop. compute-hosts runs there therefore saw the wrong tree: three TASLP trainings died on 2026-10-21 with FileNotFoundError and left no exit_code, so no completion notice came. Any write would have landed on the company NAS.

This adds an optional per-host inventory key:

  mpwav:
    path_namespace:
      /home/nas: /home/nas_IIP   # target: source, block mapping only

For such a host, only the run-directory creation and the runner/payload (log, exit_code, the training) run inside a per-process user+mount namespace that rbinds the source onto the target. The uid/gid map is the caller's own (no unshare -r, no uid 0). The shared tmux server, probe, claim, stop and has-session stay in the host namespace. Hosts without the key are unchanged.

Behavior

  • Fail closed. Before anything is created, the launch refuses with path_namespace refused: <reason> and a synchronous exit 1 when: the source is missing or not a mount point, the logical run_root's (st_dev, st_ino) inside differs from the physical source path, a bound target's identity differs, or unshare, the map writes, the private mount or the rbind fail (EPERM included). If the runner's own re-check fails, it writes stop_reason and then exit_code=125 through a descriptor of the verified source-side run dir, so the launcher's watcher still gets a terminal. It never falls back to the logical target, creates parents or overwrites a natural exit.
  • Two checks, two processes. A forked creation child verifies and mkdirs the run dir before tmux exists. The detached runner re-enters and re-verifies, because the pane inherits the server's namespace. The runner is written to <lease state dir>/compute-runners/<run_id>.py on the host's local disk (refused if that lies under a mapped target) and unlinks itself on start. This keeps the tmux command short; an inline runner exceeded tmux's message limit.
  • Lease parity with launch_compute: acquire, then the wrapper-bind wait, then release on every failure.
  • Self launcher. On a host whose own entry declares path_namespace, local run_root commands (run incl. --dry-run, claim, runs, tail, stop, watch-run) refuse. list/probe skip local claims. route_parent_close.linked_resources does not scan run_root.
  • Inventory validation: canonical absolute paths, source ≠ target, no overlapping targets or source/target overlap, run_root below exactly one target. Malformed config fails the inventory, like a missing ssh_host. A valid but unavailable source never blocks list/probe.
  • Byte identity for unconfigured hosts. gpu_leases.py keeps its old bytes as a prefix before # BEGIN PATH_NAMESPACE EXTENSION. Unconfigured launches ship only that prefix.

Docs: core/OPERATIONS.md §5.13 and the installer template comment.

Test

  • tools/run-tests.py --select utilities/owner_resource_continuation.test.py --select utilities/route_parent_close.test.py --select utilities/compute_hosts.test.py --select utilities/gpu_leases.test.py: PASS=4 (fingerprint 7b9040f1656f0fe8); clean env 79/79, 40/40, 45/45, 6/6.
  • CI fix 56bdad1ac: the compute-hosts stand-in in owner_resource_continuation.test.py gains _namespace_self, which route_parent_close.linked_resources now calls. Shard 2 of the first run hard-failed on this.
  • One-time byte comparison against base 7a855f24a (cycle evidence, not a repo golden): run with nested tmux and nohup runners, probe, claim, stop kill and has-session for an unconfigured host — all exact.
  • tools/generate.py --check, tools/check-adaptation-boundary.sh: pass.
  • Live on mpwav (CPU only): mpwav-20261021-175708-ns-smoke2 exit 0. The log shows /home/nas source 163.239.13.40:/volume1/nas1_data, uid 1004, run_root visible and /home/DB resolving. A temp inventory with a missing source refused in 1.95 s with no directory on either NAS. Nothing new appeared on the company NAS (one pre-existing foreign leftover is untouched).

Notes

  • The installed release ignores path_namespace, so mpwav is protected only after this is released and installed.
  • Training-scale CUDA/NCCL inside the namespace is not exercised here (nvidia-smi/cuInit were checked during framing).
  • Follow-up suggestion from review: one integration test that runs the generated runner with the real helper and payload sources, patching only syscalls.

🤖 Generated with Claude Code

@dmlguq456
dmlguq456 merged commit 4130499 into main Oct 21, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant