Repository navigation
fix(artifacts): pin Hugging Face directory downloads to listed commit - #18
Conversation
m3m0ryh0l3
left a comment
There was a problem hiding this comment.
Thanks for the PR! I have a few suggestions for improvements, but in general, it looks fairly tidy.
| /// One model-info response kept intact through directory download planning. | ||
| struct HfRepoListing { | ||
| resolved_revision: String, | ||
| files: Vec<String>, | ||
| } |
There was a problem hiding this comment.
suggestion: HfRepoListing is a second shape of the model-info response that HfModelInfo already deserializes. Consider a required sha: String on HfModelInfo and &HfModelInfo as the plan_dir_downloads parameter: this removes HfRepoListing, list_repo_files, the hf_listing test helper, and UnresolvedHfCommit, and a missing sha becomes the existing Http decode error. The tradeoff is hf_dir_size_bytes, which shares HfModelInfo through repo_info and would return None for a response without sha, so the quota check would silently turn off.
There was a problem hiding this comment.
Thanks — simplified this around the existing HfModelInfo. sha is now required and plan_dir_downloads consumes the same model-info response directly, so HfRepoListing, list_repo_files, and the custom unresolved-commit error are gone.
| .collect()) | ||
| ) -> Result<HfRepoListing, ModelFetchError> { | ||
| let HfModelInfo { sha, siblings } = repo_info(http, hf_endpoint, repo, false)?; | ||
| let resolved_revision = sha |
There was a problem hiding this comment.
suggestion: resolved_revision builds the download URLs and is then dropped, so the log does not record which commit a movable ref such as main resolved to for this fetch. Add one log::info! here that names the repo, the requested revision, and the resolved commit, in the style of the log::warn! in hf_dir_size_bytes.
There was a problem hiding this comment.
Added an info log on the actual directory-planning path that records the repo, effective requested revision, and returned snapshot sha.
| ) -> Result<HfRepoListing, ModelFetchError> { | ||
| let HfModelInfo { sha, siblings } = repo_info(http, hf_endpoint, repo, false)?; | ||
| let resolved_revision = sha | ||
| .filter(|sha| sha.len() == 40 && sha.chars().all(|c| c.is_ascii_hexdigit())) |
There was a problem hiding this comment.
suggestion: The len() == 40 and hex-digit filter turns any sha that is not SHA-1 shaped into a hard planning failure, and the pin only needs a present sha that names an immutable object. Drop the .filter and keep sha.ok_or_else(...). Then drop the Some(\"main\") case from list_repo_files_rejects_missing_or_invalid_snapshot_sha and the 40-character wording from the UnresolvedHfCommit message.
There was a problem hiding this comment.
Agreed — removed the SHA-1 shape assumption entirely. The model-info sha is now treated as an opaque snapshot identifier and used directly for the resolve URLs.
| } | ||
|
|
||
| #[test] | ||
| fn list_repo_files_rejects_missing_or_invalid_snapshot_sha() -> anyhow::Result<()> { |
There was a problem hiding this comment.
suggestion: This test hand-rolls its two cases in a for loop, while dir_plan_pins_downloads_to_the_listed_commit in the same change uses #[rstest] for the same shape. Convert it to #[rstest] with #[case::missing_sha(None)] and #[case::non_sha_revision(Some(\"main\"))] on a #[case] sha: Option<&str> parameter, so each case gets its own name in the test output.
There was a problem hiding this comment.
The simplification above removes the custom missing/invalid-SHA validation path and its hand-written loop entirely, so there is no longer a case set to convert here. The existing snapshot-pinning rstest remains.
ykhrustalev
left a comment
There was a problem hiding this comment.
The four review suggestions are addressed in d78d832. Verified locally on the PR merged with main: cargo fmt --check, cargo clippy -p pipette-artifacts --all-targets, and cargo nextest run -p pipette-artifacts (211 passed).
Problem
Hugging Face directory models are listed at a requested revision, but the
subsequent file downloads currently resolve that revision again for every file.
For a mutable revision such as
main, a branch, or a tag, the ref can movebetween the model-info listing and those downloads. The resulting directory is
therefore not guaranteed to come from the same repository snapshot as the
listing that planned it.
Solution
Use the
shareturned by the same Hugging Face model-info response thatprovides the directory listing, and pin every planned directory download to
that snapshot.
The requested revision remains unchanged for model identity and storage. This
affects the shared MLX, Torch, and OpenVINO directory-fetch path only.
No additional HTTP request is introduced.
If model-info does not provide a
sha, directory planning fails through theexisting response-decoding path instead of falling back to the mutable
requested ref. The returned
shais otherwise treated as an opaque snapshotidentifier.
The fetch path also logs the requested revision and the resolved snapshot so
the exact resolution remains observable after planning.
Testing
cargo fmt --all --checkcargo clippy --workspace --all-targetscargo nextest run --workspacecargo test --doc --workspace./ci/lint.shgit diff --checkAdded regression coverage for both default (
main) and named revisions,verifying that directory download URLs use the
shareturned with the listingrather than the requested mutable ref.