[Wenjix]

Closes #265

Summary

The dataset builders download their non-Hub files into --raw, which defaults to /tmp/devtools-raw and /tmp/breadth-raw (finding 1 of a security scan of 2074cc4). fetch wrote each download to the predictable <target>.part with curl -o, in a directory it created with exist_ok=True and never checked. On a machine shared with other local users, someone could create the directory first and plant <target>.part as a symlink. curl follows it and overwrites the symlink's target with the download, with the builder's permissions.

1, downloads could follow a planted symlink in a directory someone else controls. fetch now does two things:

• It requires the download directory to be a real directory that you own and that has no group or other write bit. A missing directory is created with mode 0700. Anything else, including a symlink in its place, is refused with the fix: chmod go-w <dir>, or --raw pointing at a directory of your own.
• It downloads into a file made by mkstemp, which has a random name and is created exclusively, so nothing planted beside the target is ever opened. The file is moved into place with os.replace and removed if the download fails.

main checks --raw before its first use. That also protects subdirectories such as flakeflagger/: in a directory only you can write, no one else can create or rename entries.

The /tmp/*-raw defaults stay, so caches from earlier runs (created by you with mode 0755) still work and nothing is downloaded again. Downloaded bytes are checked against the same pins as before, so --reproduce-v1 and breadth-v1's byte-for-byte rebuild are unaffected.

Behavior relative to main

Flow	main	this PR
--raw missing	created with the umask's mode	created 0700
--raw owned by another user, writable by group or others, or a symlink	used	refused, with the fix
Download target	<target>.part: predictable, follows symlinks	mkstemp file in the checked directory, then os.replace
Failed download	.part left behind	temporary file removed
Hash checks	sha256 / md5 pins	unchanged

Red / green

• tests/test_devtools_v1.py::test_fetch_downloads_through_a_private_temp_file: a data.bin.part symlink planted next to the target, pointing at a victim file. On main curl writes through it onto the victim. Here the victim and the symlink are untouched, the target has the right bytes and hash, and no temporary file remains.
• test_fetch_refuses_a_writable_directory: a 0777 download directory is refused before anything is fetched.
• test_private_dir_accepts_owned_dirs_and_creates_missing: a missing directory is created 0700, an existing 0755 directory you own passes, and a symlink to it is refused.
• test_fetch_a_failed_download_leaves_nothing: a failing curl leaves neither the temporary file nor the target.

Verification

• uv run --extra serve python -m pytest tests/test_devtools_v1.py tests/test_breadth_v1.py -q: 82 passed, 1 skipped (macOS).
• scripts/build_devtools_v1.py --help and scripts/build_breadth_v1.py --help run.
• The CI test set with all five hardening branches applied together: 603 passed, 26 skipped.

Merge Danger

• A --raw directory that is group-writable (for example one created under umask 002) is now refused; chmod go-w it. A symlinked --raw is refused too; pass the real path.
• The ownership check runs on POSIX only. On Windows the directory is still created as before.

Not in this PR: the FlakeFlagger GitHub tarball is still fetched without a sha256 pin (its hash is recorded in the manifest after the build). With the directory private, only you can place a file there. Pinning it is a separate decision, because GitHub's codeload archives are not promised to stay byte-identical.

Checklist

• The why is documented in the linked issue and this description.
• tests/test_devtools_v1.py and tests/test_breadth_v1.py pass locally.
• The --raw requirement is noted in AGENTS.md.
• No frozen suite changes; the same download pins.

Generated with Devin

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.