Skip to content

Prefix synthetic raw tables with SYNTHETIC #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
amc-corey-cox merged 2 commits into from
Sep 10, 2026
Merged

Prefix synthetic raw tables with SYNTHETIC #37

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5ba858f
Prefix synthetic raw tables with SYNTHETIC so they cannot be mistaken…
amc-corey-cox Sep 10, 2026
4d22e20
Address PR review feedback
amc-corey-cox Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions synthetic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,15 @@ second implementation of the transformation, free to drift from the real one in
ways nobody would notice until a portal built on it met real data.

```
generate.py -> data/raw/*.txt.gz -> dm-bip map-data -> harmonized BDCHM
specs.py -> specs/*/*.yaml -> ^
generate.py -> data/raw/SYNTHETIC.*.txt.gz -> dm-bip map-data -> harmonized BDCHM
specs.py -> specs/*/*.yaml -> ^
Comment thread
amc-corey-cox marked this conversation as resolved.
Outdated
```

The raw tables follow the dbGaP naming convention exactly apart from a
`SYNTHETIC.` prefix. Without it a file lifted out of its directory is
indistinguishable from a controlled-access export, and a file is identified by
its name rather than by the citation line in its header.

## Running it

```bash
Expand Down
13 changes: 12 additions & 1 deletion synthetic/generate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,17 @@

CITATION = "Synthetic corpus for portal development. Not derived from participant data."

# Every raw table is named so it cannot be mistaken for a controlled-access
# dbGaP export. The header already says so, but a file that escapes its
# directory is identified by its name, not by its contents, and these otherwise
# follow the dbGaP naming convention exactly.
#
# A prefix rather than a suffix: it is what a directory listing sorts on and
# shows first, and it survives the truncation that hides the middle of a long
# name. dm-bip finds the table accession with an unanchored search for
# `pht[0-9]+`, so the prefix does not disturb the pipeline.
SYNTHETIC_MARKER = "SYNTHETIC"

# Column layouts. The phv accessions are fictional but well-formed, and are
# what the transformation specs reference.
LAYOUTS = {
Expand Down Expand Up @@ -93,7 +104,7 @@ def write_table(out_dir, study, table, rows):
"""Write one table in dbGaP raw format, returning its path."""
columns, phv_count = LAYOUTS[table]
pht = study.tables[table]
path = out_dir / f"{study.phs}.v1.{pht}.v1.p1.c1.ex0_1s.HMB.txt.gz"
path = out_dir / f"{SYNTHETIC_MARKER}.{study.phs}.v1.{pht}.v1.p1.c1.ex0_1s.HMB.txt.gz"

with gzip.open(path, "wt", newline="") as fh:
fh.write(f"# Study accession: {study.phs}.v1.p1\n")
Expand Down
Loading