[MakazhanAlpamys]

Summary

On the disk tier the training step waits on the read, and one NVMe drive is the ceiling. This PR
adds SOUP_LAYER_STREAM_STRIPE_DIRS: extra folders on OTHER NVMe drives. Decoder layer i is
stored on drive i mod N (drive 0 is the primary cache root), and the async reader keeps one layer
in flight per drive.

Measured on a cold 70B-shaped NF4 store, seq 512, batch 1, RTX 5070 Laptop (8 GB) with two
Samsung PM9B1 drives: the training step went from 17.4 s to 9.97 s (1.75x), and the read rate
from about 4.0 to 7.0 GB/s. The decision rule was committed before any run; the record is
benchmarks/gate-two-drive-striping.md. Three earlier attempts gave no verdict (the box lost AC
power twice, and one sequence saw both drives slow down mid-run); they are reported in the record
as measured. That figure is from a rested box. Under sustained back-to-back load both drives
throttled and the speed-up measured 1.28-1.41x (benchmarks/probe-rtx5070-two-drive-sustained.md,
no formal verdict yet), so the docs say to plan a long run on roughly 1.3-1.4x.

With the variable unset nothing changes: one root, the configured read-ahead, and the reader's
one-drive path.

What changes

• utils/stripe_roots.py (new): parses and validates the entries. Each must be absolute, an
  existing directory, not a link, free of control characters, on a different volume from the
  primary cache root and from every other entry, and classify as NVMe
  (training.stream_disk_kind overrides a wrong probe); at most 6. Anything else refuses the run
  and names the entry. Also the placement (layer_roots_for) and the effective read-ahead.
• utils/owner_only_dir.py (new): the per-model folder Soup creates inside each stripe root is
  owner-only (POSIX 0o700, checked on reuse; Windows: a protected DACL for the current user and
  SYSTEM, re-applied on reuse). A failure refuses the run and names the folder.
• utils/layer_shard.py: the index records the stripe roots and each layer's root; a
  stripe.json marker per stripe folder; the stripe folder is <slug>-<hash of the primary cache path>, so two primary caches of the same model never share one; a link or junction at that
  folder is refused before any read, write or delete; changing or unsetting the list re-shards and
  names the folder left behind. Nothing is ever deleted inside a stripe root.
• utils/layer_stream_runtime.py, utils/async_disk_source.py: per-layer roots reach the
  sources; the async reader becomes a non-blocking per-drive dispatcher (one pending read and one
  range pool per drive; direct I/O is decided per drive).
• trainer/stream_setup.py: resolves the roots, splits the disk pre-flight across them, and
  defaults training.stream_read_ahead to drives + 1 (a configured value equal to the default
  counts as the default). Advice to lower the depth names a value the bump does not undo.
• Benchmarks: stream_probe.py --stripe-dir, the per-arm gate driver, a cache-compare harness, the
  sustained-load and per-drive throttle drivers, and four records indexed in
  benchmarks/README.md: the gate (gate-two-drive-striping.md), the two-drive read-ceiling probe
  (probe-rtx5070-two-drive-read.md, no formal verdict), the sustained-load probe
  (probe-rtx5070-two-drive-sustained.md, no verdict) and the per-drive throttle check
  (probe-rtx5070-drive-throttle.md, no verdict).
• Docs: docs/performance-and-quantization.md.

Verification

• Gate: SHIP by the committed rule (striped at or under 12.0 s in every round). Striped
  10.050 / 9.971 / 9.929 s, one drive 17.552 / 17.431 / 17.438 s; direct I/O and pinned staging
  on all six arms. The striped cache is byte-identical to the one-root cache, tensor by tensor.
• A real soup train (SmolLM2-135M-Instruct, 8 steps) with the variable set: first striped
  run, cache reuse, unset -> re-shard plus the orphan notice, and refusals for a missing folder and
  for a folder on the primary's volume. The adapter was byte-identical across the three training
  runs.
• Tests: nine new files tests/test_disk_tier_stripe_*.py (roots, cache, sharder, runtime,
  reader incl. a steady-state two-drive overlap test, setup, read-ahead advice, folders, terminal
  output). Full suite on Windows / Python 3.12 / RTX 5070: the only failures are three that fail
  identically on a clean checkout of main (test_issue1174_head_prefetch_after_lookup.py::... test_untied[kto], and the PRM and MoLE resume tests in test_issue1220_resume_prm_moe.py).
• Reviews: a whole-branch review and a security review of the stripe roots (a write location
  outside the home/cwd/tmp containment); their findings are fixed on this branch.

Not covered / follow-ups

• The gate times three-step bursts. Sustained load was measured twice without reaching a formal
  verdict (a hibernation, then another process reading more than the validity limit): in the two
  complete rounds both drives throttled and the speed-up was 1.28x and 1.41x. A valid sustained
  verdict is still open.
• The gate measured the branch before the last main merge and the review fixes; feat(stream): make NF4 fused forward checkpoint-visible (#842) #1143 changed
  the runtime in that merge. The soup train smoke and the suite ran on the final tree.
• The six POSIX owner-only tests skip on Windows; they ran and passed in CI's Linux and macOS
  cells.
• Soup restricts only its own folder; the docs ask the operator to make the stripe root itself
  writable only by their account.
• layer streaming: a pinned staging arena can start off a 4096-byte boundary, which silently turns direct I/O off #1531 (pinned staging arenas off a 4096-byte boundary turn direct I/O off) is separate and
  pre-existing.

[MakazhanAlpamys]

Follow-up measurement: sustained load. No verdict yet, but it changes how the 1.75x should be read.

The gate times three-step bursts from a rested box. I pre-registered a sustained probe: each arm reads
continuously for about 6 minutes, rounds run back to back, and the rule was committed before any run.
Two sequences have run. Neither reached a verdict. The first ended when the laptop slept. In the
second, Windows Search (SearchIndexer.exe) read 1.3-2.4 GB during four arms, which the rule
counts as void. The numbers below are descriptive only:

arm	early median (s/step)	late median (s/step)
sequence 1, round 0, one drive	17.52	17.40
sequence 1, round 0, two drives	8.38	8.43
sequence 2, round 0, one drive	17.39	17.30
sequence 2, round 0, two drives	8.35	13.53
sequence 2, round 1, two drives	13.58	13.59
sequence 2, round 1, one drive	17.30	19.19
sequence 2, round 2, one drive	17.71	19.78

• From rest, two drives are about 2.1x faster than one: 8.4 s against 17.4 s.
• Under sustained back-to-back load, the striped pair settles at about 2.5 GB/s per drive after a
  few minutes and does not recover within an arm. Late-window speed-ups were 1.28x and 1.41x.
• A single drive also throttles, but later and less. After about 7 minutes of continuous reading,
  C: alone drops to about 2.6 GB/s for 11-12 s roughly every 60 s.
• Drive temperatures are not readable without elevation, so "thermal" is the likely reading, not
  a measured one.

So on this laptop, a long training run should expect roughly 1.3-1.4x from striping, not 1.75x,
until a valid sustained sequence says otherwise. The striped cache was compared tensor by tensor
after each sequence, and both copies were equal.

A valid run needs Windows Search stopped for the window. The records will follow in a separate
benchmarks PR. I can amend this PR's docs paragraph and changelog fragment to quote the sustained
range instead of the burst figure, if the maintainer prefers that before merging.

[MakazhanAlpamys]

Design notes from the development of this PR

The working notes for this branch are being removed. These are the decisions recorded there
that the code and the benchmark records do not explain by themselves.

The cache on a second drive

1. Stripe roots are outside the home/cwd/tmp containment on purpose. Containment would
   refuse every second drive, and setting SOUP_LAYER_STREAM_STRIPE_DIRS is itself the opt-in.
   What containment gave the primary cache is enforced directly instead:

   • the per-model folder inside a stripe root is owner-only (POSIX: mode 0700 at creation,
     owner and mode checked on reuse; Windows: a protected ACL for the current account and
     SYSTEM, applied at creation and again on reuse);
   • that folder is never followed when it is a link or junction, checked before any read,
     write or delete;
   • its name carries a hash of the primary cache's path, so two primary caches of one model
     never share a folder.

   A folder that cannot be made private refuses the run and names the folder. Two consequences:
   a stripe drive whose file system has no ACLs (FAT, exFAT) is refused on Windows (this follows
   from the code; no such volume was tried), and Soup restricts only its own folder, which is
   why the docs ask the operator to keep the stripe root itself writable only by their account.

2. The old copy of a moved layer is deleted only after the new index is written. The index
   is the cache's commit point: every file it names must exist at every instant, so an
   interrupted re-shard leaves the old index consistent. A copy that then cannot be deleted is
   a warning, not a failure, because the shard is already committed.

3. Cache-miss reasons print a stripe path as plain text. Only a file's base name in the
   primary root keeps repr. repr doubles every Windows backslash, and the full path has to
   stay readable in the message.

The reader

4. AsyncDiskSource._read_layer dispatches and returns. It is still the one call per layer
   read, but it hands the ranges to that drive's workers instead of blocking until they land.
   Code that brackets the call times the dispatch; the warm-stage harness waits on the outcomes
   for that reason and refuses a striped source.
5. With one drive the range workers keep their old thread names (soup-layer-range-<n>),
   because existing tests match them. With several drives each pool is named per drive, so a
   stack dump shows which drive a worker serves.
6. The steady-state test gives the consumer 50 ms of work per layer on purpose. With a
   consumer that does no work the drives run in phase, and no threshold separates this reader
   from one that reads in batches. A separate posed-state test covers the exclude argument of
   _claim_slot, which no timed walk reaches.
7. The pinned two-drive path has no CI test. Two reads in flight while a staging slot's copy
   to the GPU is still draining needs a GPU, and CI has none. The gate ran that path on the
   70B-shaped store.

The measurements

8. The gate passes --read-ahead 3 for the striped arm explicitly (2 for the single-drive
   arm). stream_probe builds the model directly and does not go through the setup code that
   raises the default depth to drives + 1. The arms therefore differ in depth; section 7 of the
   gate record says why that is expected to matter little and that no single-drive arm at
   depth 3 was run.
9. A verdict is read from one complete six-arm sequence. Arms of different sequences are
   never pooled, since that would pick rounds after seeing them. That is why the record reports
   three sequences without a verdict beside the one that shipped. --run-prefix exists so that
   a new sequence does not overwrite files an earlier part of the record cites.
10. The gate measured the branch before the last merge of main (which brought feat(stream): make NF4 fused forward checkpoint-visible (#842) #1143, a
    change to the streaming runtime) and before the review fixes. The soup train check and the
    full test suite ran after both.
11. The eight commits of the two-drive read probe are on this branch because
    utils/stripe_roots.py cites that record: evidence that code cites travels with the code.
12. The docs say striping "changes the speed, not the result", not that training is
    bit-identical. The gate showed that GPU losses are not bit-identical from one process to
    the next even within one arm, while the cache bytes are identical (compared tensor by
    tensor). So the claim is about the bytes.