Skip to content

fix(artifacts): pin Hugging Face directory downloads to listed commit - #18

Merged
ykhrustalev merged 3 commits into
Liquid4All:mainfrom
200lz:fix/hf-directory-snapshot-pin
Sep 24, 2026
Merged

ykhrustalev merged 3 commits into
Liquid4All:mainfrom
200lz:fix/hf-directory-snapshot-pin

Conversation

@200lz

@200lz 200lz commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Hugging Face directory models are listed at a requested revision, but the
subsequent file downloads currently resolve that revision again for every file.

For a mutable revision such as main, a branch, or a tag, the ref can move
between the model-info listing and those downloads. The resulting directory is
therefore not guaranteed to come from the same repository snapshot as the
listing that planned it.

Solution

Use the sha returned by the same Hugging Face model-info response that
provides the directory listing, and pin every planned directory download to
that snapshot.

The requested revision remains unchanged for model identity and storage. This
affects the shared MLX, Torch, and OpenVINO directory-fetch path only.

No additional HTTP request is introduced.

If model-info does not provide a sha, directory planning fails through the
existing response-decoding path instead of falling back to the mutable
requested ref. The returned sha is otherwise treated as an opaque snapshot
identifier.

The fetch path also logs the requested revision and the resolved snapshot so
the exact resolution remains observable after planning.

Testing

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets
  • cargo nextest run --workspace
  • cargo test --doc --workspace
  • ./ci/lint.sh
  • git diff --check

Added regression coverage for both default (main) and named revisions,
verifying that directory download URLs use the sha returned with the listing
rather than the requested mutable ref.

@m3m0ryh0l3 m3m0ryh0l3 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I have a few suggestions for improvements, but in general, it looks fairly tidy.

Comment on lines +318 to +322
/// One model-info response kept intact through directory download planning.
struct HfRepoListing {
resolved_revision: String,
files: Vec<String>,
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: HfRepoListing is a second shape of the model-info response that HfModelInfo already deserializes. Consider a required sha: String on HfModelInfo and &HfModelInfo as the plan_dir_downloads parameter: this removes HfRepoListing, list_repo_files, the hf_listing test helper, and UnresolvedHfCommit, and a missing sha becomes the existing Http decode error. The tradeoff is hf_dir_size_bytes, which shares HfModelInfo through repo_info and would return None for a response without sha, so the quota check would silently turn off.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — simplified this around the existing HfModelInfo. sha is now required and plan_dir_downloads consumes the same model-info response directly, so HfRepoListing, list_repo_files, and the custom unresolved-commit error are gone.

.collect())
) -> Result<HfRepoListing, ModelFetchError> {
let HfModelInfo { sha, siblings } = repo_info(http, hf_endpoint, repo, false)?;
let resolved_revision = sha

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: resolved_revision builds the download URLs and is then dropped, so the log does not record which commit a movable ref such as main resolved to for this fetch. Add one log::info! here that names the repo, the requested revision, and the resolved commit, in the style of the log::warn! in hf_dir_size_bytes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an info log on the actual directory-planning path that records the repo, effective requested revision, and returned snapshot sha.

) -> Result<HfRepoListing, ModelFetchError> {
let HfModelInfo { sha, siblings } = repo_info(http, hf_endpoint, repo, false)?;
let resolved_revision = sha
.filter(|sha| sha.len() == 40 && sha.chars().all(|c| c.is_ascii_hexdigit()))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: The len() == 40 and hex-digit filter turns any sha that is not SHA-1 shaped into a hard planning failure, and the pin only needs a present sha that names an immutable object. Drop the .filter and keep sha.ok_or_else(...). Then drop the Some(\"main\") case from list_repo_files_rejects_missing_or_invalid_snapshot_sha and the 40-character wording from the UnresolvedHfCommit message.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — removed the SHA-1 shape assumption entirely. The model-info sha is now treated as an opaque snapshot identifier and used directly for the resolve URLs.

}

#[test]
fn list_repo_files_rejects_missing_or_invalid_snapshot_sha() -> anyhow::Result<()> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: This test hand-rolls its two cases in a for loop, while dir_plan_pins_downloads_to_the_listed_commit in the same change uses #[rstest] for the same shape. Convert it to #[rstest] with #[case::missing_sha(None)] and #[case::non_sha_revision(Some(\"main\"))] on a #[case] sha: Option<&str> parameter, so each case gets its own name in the test output.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The simplification above removes the custom missing/invalid-SHA validation path and its hand-written loop entirely, so there is no longer a case set to convert here. The existing snapshot-pinning rstest remains.

@ykhrustalev ykhrustalev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The four review suggestions are addressed in d78d832. Verified locally on the PR merged with main: cargo fmt --check, cargo clippy -p pipette-artifacts --all-targets, and cargo nextest run -p pipette-artifacts (211 passed).

@ykhrustalev
ykhrustalev merged commit 461a2c0 into Liquid4All:main Sep 24, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants