Skip to content

Layer streaming: a bad SOUP_LAYER_STREAM_STRIPE_DIRS entry is refused only after the dataset loads; --dry-run and soup doctor never check it #1613

Closed

Description

@MakazhanAlpamys

Follow-up to the layer-cache striping merged in #1536. Line numbers are from main at the merge
commit, 45b8470c.

What happens today

The entries of SOUP_LAYER_STREAM_STRIPE_DIRS are parsed and validated in one place: inside
_setup_streaming_transformers, after the tokenizer and the model config have been loaded
(src/soup_cli/trainer/stream_setup.py:636-645, then resolve_stripe_roots at :673-678).
That method runs from trainer_wrapper.setup(dataset), which is late in soup train
(src/soup_cli/commands/train.py):

step line
confirmation prompt :1525
--dry-run: validate the data, print "Config valid. Ready to train!", exit :1537-1553
load the dataset :1559
create the run row, print Run ID :1578-1590
trainer_wrapper.setup(dataset) :1680

Observed in a real run on Windows, on the #1536 branch before it merged (SmolLM2-135M-Instruct,
an 8-row dataset, the variable set to a folder that does not exist): exit code 1 after about
21 s, after the dataset had loaded, a Run ID had been printed and the tokenizer had been
loaded. The run tracker kept a failed row for it. The output was:

Error: StripeRootError: SOUP_LAYER_STREAM_STRIPE_DIRS entry
'D:\\does-not-exist-xyz': must be an existing directory. Soup creates only its
per-model folder inside a stripe root, never the root itself, so a drive that
is not mounted cannot be mistaken for an empty one. Reconnect the drive, or
remove this entry (unset SOUP_LAYER_STREAM_STRIPE_DIRS to re-shard to one
root).
Run with --verbose for the full traceback.

Three more things follow from the same placement:

  • --dry-run cannot report the problem. Its branch exits before a trainer exists, so it never
    reaches the check. (Read from the code; a dry run with a bad entry was not executed.)
  • soup doctor does not read the variable. soup doctor --disk classifies the current
    directory only (src/soup_cli/commands/doctor.py:948-960).
  • The message starts with the exception class name, because format_friendly_error has no
    entry for it and falls back to <type>: <message> (src/soup_cli/utils/errors.py:364).

What should happen

  • An entry that breaks a rule which needs no disk probe is refused before the dataset is loaded
    and before a run row is created.
  • soup train --dry-run reports the same refusal.
  • soup doctor shows the variable and what Soup makes of each entry.
  • The refusal reads as a refusal with a way out, not as StripeRootError: ....

Suggested direction

  • When training.stream_layers is on and the variable is set, run the cheap rules right after
    the config is loaded: parse_stripe_dirs, the count cap, validate_stripe_root for each
    entry, and the same-volume rule (volume_of, one os.stat). utils/stripe_roots.py has no
    top-level torch import, so the CLI stays light if the import is inside the function.
  • The rules need the primary cache ROOT, not the per-model folder. resolve_shard_dir creates
    the per-model folder as a side effect (src/soup_cli/utils/layer_shard.py:696-709); an early
    check should take the root without creating anything.
  • Leave the NVMe rule where it is: the disk-kind probe costs about 9 s on Windows. Keep the full
    validation in the setup as the authority; the early call only moves the common mistakes
    forward.
  • Call the same helper from the --dry-run branch.
  • In soup doctor, add a row per entry. --disk already pays for the probe, so with that flag
    the NVMe rule can run too.
  • Give StripeRootError a friendly form in utils/errors.py, or catch it in the command.

Acceptance test

  • A CliRunner test of soup train with stream_layers: true and the variable naming a folder
    that does not exist: exit code 1, the output contains "must be an existing directory", it does
    not contain the "Loaded:" line or a Run ID, and ExperimentTracker.start_run was not called.
  • The same config with --dry-run: the same refusal, and no "Config valid" line.
  • A same-volume entry is refused at the same early point.
  • soup doctor with the variable set to a missing folder prints a row that names the entry and
    the rule; with the variable unset the output is unchanged.
  • With the variable unset, soup train behaves exactly as before (no new output, no new call).
  • Output assertions go through the ANSI-strip and whitespace-collapse helper the test suite
    already uses, and the tests pass with FORCE_COLOR=1.
  • Mutation check: remove the early call, and the first two tests fail.

Activity

  1. Nari-Ab commented on Oct 5, 2026

    @Nari-Ab
    Contributor

    Hi @MakazhanAlpamys, I would like to work on this issue!

    Here is my plan based on the acceptance criteria and suggested direction:

    1. Lightweight early validation helper (validate_early_stripe_roots):

      • In src/soup_cli/utils/stripe_roots.py, add a cheap validation helper for SOUP_LAYER_STREAM_STRIPE_DIRS: parse_stripe_dirs, count cap (MAX_STRIPE_DIRS), validate_stripe_root (syntax, existence, absolute path, symlinks, overlap with primary cache root and peer entries), and the distinct-volume rule (volume_of).
      • Obtain the primary cache root without side effects (using resolve_cache_root without creating per-model directories or shard folders early).
      • Leave the ~9s NVMe probe to the full runtime setup in stream_setup.py (and soup doctor --disk).
    2. Wire early check into soup train & --dry-run:

      • In src/soup_cli/commands/train.py, run this check right after config validation when training.stream_layers is enabled, ahead of confirmation prompts, --dry-run, dataset loading, and run row creation (start_run).
    3. Wire into soup doctor:

      • In src/soup_cli/commands/doctor.py, when SOUP_LAYER_STREAM_STRIPE_DIRS is set, add diagnostic rows for the variable and what Soup makes of each entry (syntax, existence, volume).
      • When --disk is specified, also execute the disk-kind probe on each stripe entry.
      • When the variable is unset, keep output completely unchanged.
    4. Friendly error formatting:

      • Add StripeRootError to ERROR_MAP in src/soup_cli/utils/errors.py so refusals render cleanly with actionable guidance rather than falling back to Error: StripeRootError: ....
    5. Acceptance tests & mutation verification:

      • Add CliRunner tests for soup train and soup train --dry-run with non-existent and same-volume stripe folders, confirming exit code 1, absence of Loaded: and Run ID, and that start_run was never called.
      • Test soup doctor with invalid stripe entries vs unset variable.
      • Verify output assertions with ANSI-stripping / whitespace collapsing and compatibility under FORCE_COLOR=1.
      • Add the changelog entry in changelog.d/.

    Could you please assign this issue to me?

  2. MakazhanAlpamys commented on Oct 5, 2026

    @MakazhanAlpamys
    OwnerAuthor

    Confirmed, @Nari-Ab: this one is yours, and I've marked it claimed. Thank you for the detailed plan, and for opening #1628 so quickly; I'll answer there.

    One heads-up for the thread: #1611 (@vaputa, PR #1626) also edits utils/stripe_roots.py, so whichever of the two lands second may need a rebase.

  3. added
    claimedSomeone is actively working on this — check the thread before starting
    on Oct 5, 2026
  4. MakazhanAlpamys commented on Oct 7, 2026

    @MakazhanAlpamys
    OwnerAuthor

    Closed by #1628 (merged as 4b34d2c9), by @Nari-Ab. Thank you.
    A bad SOUP_LAYER_STREAM_STRIPE_DIRS entry is now refused when soup train or soup train --dry-run starts, before the confirmation prompt and before the dataset loads, with a message that names the entry and the rule. soup doctor lists every stripe entry with its verdict and applies the NVMe rule under --disk. Try: soup doctor --disk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    claimedSomeone is actively working on this — check the thread before startingdifficulty:4Difficulty 4/10: a few files, existing pattern.enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions