Repository navigation
Prefix synthetic raw tables with SYNTHETIC - #37
Merged
Merged
Conversation
… for dbGaP exports
✅ Deploy Preview for study-palette canceled.
|
There was a problem hiding this comment.
🟢 Approval recommended
The change is low-risk and localized to filename emission, with existing CI already regenerating and validating the synthetic corpus.
Pull request overview
This PR updates the synthetic raw-table generator to prefix every emitted dbGaP-style .txt.gz with SYNTHETIC. so a stray file can’t be mistaken for a controlled-access export, aligning the generator with the synthetic-data publication policy described in CLAUDE.md.
Changes:
- Prefix raw table filenames with
SYNTHETIC.insynthetic/generate.py. - Update
synthetic/README.mdto describe theSYNTHETIC.filename marker in the raw-table pipeline overview.
File summaries
| File | Description |
|---|---|
| synthetic/README.md | Documents the new SYNTHETIC. prefix in the raw-table naming scheme and pipeline diagram. |
| synthetic/generate.py | Changes raw table output filenames to include a SYNTHETIC. prefix while preserving the same table/accession content. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟢 Approval recommended
The changes are narrowly scoped to synthetic raw-table naming and accompanying documentation, with no additional code paths impacted in-repo.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CLAUDE.mdrequires generated data to carrySYNTHETICin the filename as well as a header, so that a stray.txt.gzis identifiable without opening it. The raw tables had the header but not the filename — they followed the dbGaP convention exactly, which is precisely what makes one lifted out of its directory look like a controlled-access export. The policy landed in #28, after the generator was written.Raw tables are now
SYNTHETIC.phs000101.v1.pht000111.v1.p1.c1.ex0_1s.HMB.txt.gz. A prefix rather than a suffix: it is what a listing sorts on and shows first, and it survives the truncation that hides the middle of a long name.No effect on the pipeline or its output. dm-bip finds the table accession with an unanchored search for
pht[0-9]+and already expects prefixed names (CARDIA_pht001562.txt.gzis the example in its own source). Ran both studies through and diffed: every harmonized file is byte-identical, since ids derive from accessions and SUBJECT_ID, not filenames. Specs andsample/unchanged, 70 validation checks pass.Found while staging the rc2 corpus release, which is blocked on this.