Repository navigation
Layer streaming: a bad SOUP_LAYER_STREAM_STRIPE_DIRS entry is refused only after the dataset loads; --dry-run and soup doctor never check it #1613
Description
Follow-up to the layer-cache striping merged in #1536. Line numbers are from main at the merge
commit, 45b8470c.
What happens today
The entries of SOUP_LAYER_STREAM_STRIPE_DIRS are parsed and validated in one place: inside
_setup_streaming_transformers, after the tokenizer and the model config have been loaded
(src/soup_cli/trainer/stream_setup.py:636-645, then resolve_stripe_roots at :673-678).
That method runs from trainer_wrapper.setup(dataset), which is late in soup train
(src/soup_cli/commands/train.py):
| step | line |
|---|---|
| confirmation prompt | :1525 |
--dry-run: validate the data, print "Config valid. Ready to train!", exit |
:1537-1553 |
| load the dataset | :1559 |
create the run row, print Run ID |
:1578-1590 |
trainer_wrapper.setup(dataset) |
:1680 |
Observed in a real run on Windows, on the #1536 branch before it merged (SmolLM2-135M-Instruct,
an 8-row dataset, the variable set to a folder that does not exist): exit code 1 after about
21 s, after the dataset had loaded, a Run ID had been printed and the tokenizer had been
loaded. The run tracker kept a failed row for it. The output was:
Error: StripeRootError: SOUP_LAYER_STREAM_STRIPE_DIRS entry
'D:\\does-not-exist-xyz': must be an existing directory. Soup creates only its
per-model folder inside a stripe root, never the root itself, so a drive that
is not mounted cannot be mistaken for an empty one. Reconnect the drive, or
remove this entry (unset SOUP_LAYER_STREAM_STRIPE_DIRS to re-shard to one
root).
Run with --verbose for the full traceback.
Three more things follow from the same placement:
--dry-runcannot report the problem. Its branch exits before a trainer exists, so it never
reaches the check. (Read from the code; a dry run with a bad entry was not executed.)soup doctordoes not read the variable.soup doctor --diskclassifies the current
directory only (src/soup_cli/commands/doctor.py:948-960).- The message starts with the exception class name, because
format_friendly_errorhas no
entry for it and falls back to<type>: <message>(src/soup_cli/utils/errors.py:364).
What should happen
- An entry that breaks a rule which needs no disk probe is refused before the dataset is loaded
and before a run row is created. soup train --dry-runreports the same refusal.soup doctorshows the variable and what Soup makes of each entry.- The refusal reads as a refusal with a way out, not as
StripeRootError: ....
Suggested direction
- When
training.stream_layersis on and the variable is set, run the cheap rules right after
the config is loaded:parse_stripe_dirs, the count cap,validate_stripe_rootfor each
entry, and the same-volume rule (volume_of, oneos.stat).utils/stripe_roots.pyhas no
top-level torch import, so the CLI stays light if the import is inside the function. - The rules need the primary cache ROOT, not the per-model folder.
resolve_shard_dircreates
the per-model folder as a side effect (src/soup_cli/utils/layer_shard.py:696-709); an early
check should take the root without creating anything. - Leave the NVMe rule where it is: the disk-kind probe costs about 9 s on Windows. Keep the full
validation in the setup as the authority; the early call only moves the common mistakes
forward. - Call the same helper from the
--dry-runbranch. - In
soup doctor, add a row per entry.--diskalready pays for the probe, so with that flag
the NVMe rule can run too. - Give
StripeRootErrora friendly form inutils/errors.py, or catch it in the command.
Acceptance test
- A
CliRunnertest ofsoup trainwithstream_layers: trueand the variable naming a folder
that does not exist: exit code 1, the output contains "must be an existing directory", it does
not contain the "Loaded:" line or aRun ID, andExperimentTracker.start_runwas not called. - The same config with
--dry-run: the same refusal, and no "Config valid" line. - A same-volume entry is refused at the same early point.
soup doctorwith the variable set to a missing folder prints a row that names the entry and
the rule; with the variable unset the output is unchanged.- With the variable unset,
soup trainbehaves exactly as before (no new output, no new call). - Output assertions go through the ANSI-strip and whitespace-collapse helper the test suite
already uses, and the tests pass withFORCE_COLOR=1. - Mutation check: remove the early call, and the first two tests fail.
Activity
Nari-Ab commentedon Oct 5, 2026
Hi @MakazhanAlpamys, I would like to work on this issue!
Here is my plan based on the acceptance criteria and suggested direction:
Lightweight early validation helper (
validate_early_stripe_roots):src/soup_cli/utils/stripe_roots.py, add a cheap validation helper forSOUP_LAYER_STREAM_STRIPE_DIRS:parse_stripe_dirs, count cap (MAX_STRIPE_DIRS),validate_stripe_root(syntax, existence, absolute path, symlinks, overlap with primary cache root and peer entries), and the distinct-volume rule (volume_of).resolve_cache_rootwithout creating per-model directories or shard folders early).stream_setup.py(andsoup doctor --disk).Wire early check into
soup train&--dry-run:src/soup_cli/commands/train.py, run this check right after config validation whentraining.stream_layersis enabled, ahead of confirmation prompts,--dry-run, dataset loading, and run row creation (start_run).Wire into
soup doctor:src/soup_cli/commands/doctor.py, whenSOUP_LAYER_STREAM_STRIPE_DIRSis set, add diagnostic rows for the variable and what Soup makes of each entry (syntax, existence, volume).--diskis specified, also execute the disk-kind probe on each stripe entry.Friendly error formatting:
StripeRootErrortoERROR_MAPinsrc/soup_cli/utils/errors.pyso refusals render cleanly with actionable guidance rather than falling back toError: StripeRootError: ....Acceptance tests & mutation verification:
CliRunnertests forsoup trainandsoup train --dry-runwith non-existent and same-volume stripe folders, confirming exit code 1, absence ofLoaded:andRun ID, and thatstart_runwas never called.soup doctorwith invalid stripe entries vs unset variable.FORCE_COLOR=1.changelog.d/.Could you please assign this issue to me?
MakazhanAlpamys commentedon Oct 5, 2026
Confirmed, @Nari-Ab: this one is yours, and I've marked it claimed. Thank you for the detailed plan, and for opening #1628 so quickly; I'll answer there.
One heads-up for the thread: #1611 (@vaputa, PR #1626) also edits
utils/stripe_roots.py, so whichever of the two lands second may need a rebase.fix(layer-stream): address review feedback on doctor, helper, tests a…
test(doctor): pin non-NVMe stripe refusal and advisory issue recording (
test(train): pin that dry-run with a real dataset exits before data v…
fix(layer-stream): validate SOUP_LAYER_STREAM_STRIPE_DIRS early in tr…
fix(layer-stream): address review feedback on doctor, helper, tests a…
test(doctor): pin non-NVMe stripe refusal and advisory issue recording (
test(train): pin that dry-run with a real dataset exits before data v…
fix(layer-stream): validate SOUP_LAYER_STREAM_STRIPE_DIRS early in tr…
fix(layer-stream): address review feedback on doctor, helper, tests a…
test(doctor): pin non-NVMe stripe refusal and advisory issue recording (
test(train): pin that dry-run with a real dataset exits before data v…
fix(layer-stream): validate SOUP_LAYER_STREAM_STRIPE_DIRS early in tr…
fix(layer-stream): address review feedback on doctor, helper, tests a…
test(doctor): pin non-NVMe stripe refusal and advisory issue recording (
test(train): pin that dry-run with a real dataset exits before data v…
test(doctor): pin issue recording for refused stripe entry without --…
fix(layer-stream): validate SOUP_LAYER_STREAM_STRIPE_DIRS early in tr…
fix(layer-stream): address review feedback on doctor, helper, tests a…
test(doctor): pin non-NVMe stripe refusal and advisory issue recording (
test(train): pin that dry-run with a real dataset exits before data v…
test(doctor): pin issue recording for refused stripe entry without --…
fix(layer-stream): validate SOUP_LAYER_STREAM_STRIPE_DIRS early in tr…
MakazhanAlpamys commentedon Oct 7, 2026
Closed by #1628 (merged as
4b34d2c9), by @Nari-Ab. Thank you.A bad
SOUP_LAYER_STREAM_STRIPE_DIRSentry is now refused whensoup trainorsoup train --dry-runstarts, before the confirmation prompt and before the dataset loads, with a message that names the entry and the rule.soup doctorlists every stripe entry with its verdict and applies the NVMe rule under--disk. Try:soup doctor --disk