Repository navigation
Dataset builders download only into a directory you own - #273
Draft
jaredpalmer wants to merge 2 commits into
Draft
jaredpalmer wants to merge 2 commits into
jaredpalmer wants to merge 2 commits into
Conversation
fetch wrote each download to a predictable <target>.part in a /tmp directory it never checked, so another local user could plant a symlink there and have curl overwrite its target. Downloads now go only into a directory owned by the caller with no group or other write bit (created 0700), through an mkstemp file moved into place with os.replace and removed on failure. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
private_dir's docstring and its error are the contract. The suite paragraph is about labels and ids, and breadth-v1 does not belong inside it. Co-authored-by: Jared Palmer <hello@jaredpalmer.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/build_devtools_v1.pydownloads withcurl -o <target>.partinto--raw, which defaults to/tmp/devtools-raw.scripts/build_breadth_v1.pyuses the samefetchand defaults to/tmp/breadth-raw. The directory is created withexist_okand its owner and mode are never checked. On a shared POSIX machine another local user can create that directory first and plant<target>.partas a symlink.curl -ofollows it and overwrites the symlink's target as the user running the builder. Most of those files are then checked against a pinned sha256, so a swapped file fails the build. The FlakeFlagger GitHub tarball is not pinned at fetch time. Its hash is recorded in the manifest after the build, so a planted copy would be used as it is.fetchnow writes only into a directory the caller owns, with no group or other write bit. A missing directory is created mode0700. An existing directory the user owns at mode0755still works. That is what the oldmkdirproduced under a normal umask. A symlink is refused rather than followed. The bytes go to anmkstempfile in that directory, andos.replacemoves it onto the target. A failed curl removes the temp file and leaves no target. On Windows the ownership check is skipped and the directory is still created.This keeps Wenjix's commit from #268. I dropped the
AGENTS.mdsentence. It was inserted into the devtools-v1 parenthetical, which is about CodeReviewer ids and state keys, and it mentioned breadth-v1 from inside that parenthesis.private_dir's docstring and the error text are the contract.Not in this PR: pinning the FlakeFlagger tarball. GitHub codeload archives are not promised to stay byte-identical. Also not here: the kev-finetune scripts. Those stay standard library. This builder already imports
kev.Test plan
tests/test_devtools_v1.pyandtests/test_conventions.py: 79 passed. Covers a planted<name>.partsymlink left untouched, a mode0777directory refused before any download, a missing directory created0700, and a failed curl leaving the directory empty.