Add synthetic corpus generator for two example cohorts

.github/workflows/test-synthetic.yaml

@@ -0,0 +1,61 @@
+name: Test Synthetic Corpus
+
+on:
+  pull_request:
+    paths:
+      - "synthetic/**"
+      - ".github/workflows/test-synthetic.yaml"
+  push:
+    branches: [main]
+    paths:
+      - "synthetic/**"
+      - ".github/workflows/test-synthetic.yaml"
+
+jobs:
+  test-synthetic:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      # The corpus is a build product, not a committed artifact: it is
+      # regenerated here and checked, rather than diffed against something
+      # stored in the repo.
+      - name: Generate raw tables
+        working-directory: synthetic
+        run: python generate.py
+
+      - name: Generate transformation specs
+        working-directory: synthetic
+        run: python specs.py
+
+      - name: Validate distributions and invariants
+        working-directory: synthetic
+        run: python validate.py
+
+      - name: Check specs are well-formed YAML
+        working-directory: synthetic
+        run: |
+          pip install --quiet pyyaml
+          python - <<'PY'
+          import glob
+          import sys
+
+          import yaml
+
+          bad = []
+          paths = sorted(glob.glob("specs/*/*.yaml"))
+          for path in paths:
+              try:
+                  yaml.safe_load(open(path))
+              except yaml.YAMLError as exc:
+                  bad.append(f"{path}: {exc}")
+          print(f"{len(paths) - len(bad)}/{len(paths)} specs parse")
+          if bad:
+              print("\n".join(bad))
+              sys.exit(1)
+          PY

pyproject.toml

@@ -33,6 +33,7 @@ target-version = "py312"
 line-length = 120
 include = [
     "api/**/*.py",
+    "synthetic/**/*.py",
 ]
 
 [tool.ruff.lint]

synthetic/.gitignore

@@ -0,0 +1,7 @@
+# Generated corpus — reproducible from SEED, distributed as release assets
+data/
+output/
+# sample/ is committed on purpose — see README
+# Vendored upstream schema, fetched not committed
+bdchm.yaml
+__pycache__/

synthetic/README.md

@@ -0,0 +1,134 @@
+# Synthetic corpus
+
+Two fictional cohorts, structurally faithful to harmonized BDC data, for teams
+building against the portal without touching participant data.
+
+Nothing here derives from real participants. Every coded value — CURIEs, enum
+members, units — is taken from RTI's `priority_variables_transform` specs or
+from BDCHM itself, so the corpus resolves against the same vocabulary as real
+harmonized data.
+
+## Why it goes through dm-bip
+
+The generator emits **dbGaP-style raw tables**, not harmonized output, and those
+are transformed by dm-bip against BDCHM. Emitting BDCHM directly would be a
+second implementation of the transformation, free to drift from the real one in
+ways nobody would notice until a portal built on it met real data.
+
+```
+generate.py  ->  data/raw/*.txt.gz  ->  dm-bip map-data  ->  harmonized BDCHM
+specs.py     ->  specs/*/*.yaml     ->  ^
+```
+
+## Running it
+
+```bash
+python generate.py          # raw tables, per study
+python specs.py             # BDCHM-targeted transformation specs
+python validate.py          # distributions and invariants
+```
+
+After the pipeline has run:
+
+```bash
+python schema.py --study study_one   # JSONL -> Parquet, typed from BDCHM
+```
+
+The pipeline emits YAML and JSONL side by side — YAML is what makes the
+published corpus readable, JSONL is the machine-facing form with the same
+nesting. `schema.py` reads BDCHM for leaf types and the transformation specs for
+which slots nest, because the model alone does not decide that: BDCHM gives
+`associated_participant` a range of `Participant`, but the spec materialises it
+as a uuid5 string while `value_quantity` is nested inline.
+
+It also reports where the model and the data disagree on cardinality. That is
+not hypothetical — BDCHM declares `identity` multivalued and every
+transformation spec, RTI's included, emits a scalar.
+
+Then, from a dm-bip checkout:
+
+```bash
+SYNTH=/path/to/synthetic
+make pipeline CONFIG=$SYNTH/pipeline/example_study_one.mk \
+              SYNTH_DIR=$SYNTH SYNTH_OUTPUT_DIR=$SYNTH/output/study_one
+```
+
+BDCHM is fetched rather than vendored, pinned to a release so an upstream change
+is a deliberate bump here rather than a silent change in what the pipeline
+produces:
+
+```bash
+./fetch-bdchm.sh          # currently v1.3.0
+```
+
+## What is and isn't committed
+
+The generator is committed. The corpus is not: it is 14MB of YAML, deterministic
+given `SEED`, and would produce a 14MB diff every time the seed or the model
+changed. Built corpora are distributed as release assets — 3MB compressed, with
+a stable URL and a version, which is what a consuming team needs anyway.
+
+`sample/` **is** committed — 40KB of hand-selected records covering every
+structural feature the corpus claims. It exists so a reviewer, or a team
+deciding whether this is the reference data they want, can see the output shape
+without running the pipeline. Regenerate it with `python sample.py` after a run.
+
+## The cohorts
+
+|  | Example Study One | Example Study Two |
+|---|---|---|
+| Participants | 500 | 500 |
+| Visits | 3 | 5 |
+| Sex skew | slightly male | slightly female |
+| Race | 50/30/10/10 white, black, Asian, American Indian | 60/20/10/10 white, black, Middle Eastern, Native Hawaiian |
+| BMI | continuous | categorical |
+
+5% of Study Two's participants are the same individuals as in Study One. They
+share a `dbGaP_Subject_ID`, so harmonization resolves them to one `Person` with
+two `Participant` records — the same way real cross-study participation appears.
+
+One visit per participant is `TELEHEALTH`; the rest are `STUDY_SITE_VISIT`.
+Deceased participants stop attending, so nobody is measured after they die.
+
+## Known shapes that surprise people
+
+**`cause_of_death` is present for living participants**, multivalued, with a null
+cause:
+
+```yaml
+cause_of_death:
+- cause: null
+  order: null
+  id: 73e3cc59-...
+vital_status: OMOP:4230556
+```
+
+Test `vital_status`, not the presence of `cause_of_death`.
+
+This is not an artifact of the synthetic corpus — it is how the real
+transformation behaves. RTI's MESA spec derives `cause_of_death` the same way,
+with `cause` set to `None` for the living and no mechanism to suppress the
+object, because `ClassDerivation` in linkml-map has no conditional emission
+(see linkml/linkml-map#187). Real harmonized BDC data therefore carries the same
+shape, and the corpus reproduces it deliberately. A portal built against a
+tidied-up version would break on real data.
+
+## Structural features exercised
+
+Beyond the clinical values, the corpus is meant to exercise the shapes a portal
+has to render:
+
+- `MeasurementObservationSet` with nested systolic and diastolic observations
+- `Quantity.operator` for results censored below an assay's detection limit
+- `Assay` with `lower_limit_of_detection` / `upper_limit_of_detection` (HDL)
+- `associated_assay` referencing a CBC instance (WBC)
+- `qualifier` marking a value as an average (BUN)
+- `Condition.relationship_to_participant` for family history
+- `associated_evidence` on study-record-sourced conditions
+- `exposure_status` distinguishing absent from present drug exposures
+- Continuous and categorical presentations of the same concept across cohorts
+
+Coverage against BDCHM's full slot inventory — the percentage of slots appearing
+at least once — is intended as the acceptance number for broadening the corpus
+beyond this first pass. **That measurement is not built yet.** This pass covers
+the brief only.

synthetic/fetch-bdchm.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Fetch the pinned BDCHM schema used as the transformation target.
+#
+# Pinned by commit rather than by tag: git tags are mutable and can be
+# retargeted, so a tag alone would let the schema change underneath us. The
+# checksum then verifies the bytes, covering the case where the fetch itself
+# returns something unexpected.
+#
+# To bump: change all three values together and re-run the pipeline.
+set -euo pipefail
+
+BDCHM_VERSION="v1.3.0"
+BDCHM_COMMIT="84222624fb550e47ce7ec3f6c4a3754d80a1cd16"
+BDCHM_SHA256="01af15d50ba1ce3929344a30698cf83776fc10f19902b392cc1fcf7469c10029"
+
+REPO="RTIInternational/NHLBI-BDC-DMC-HM"
+URL="https://raw.githubusercontent.com/${REPO}/${BDCHM_COMMIT}/src/bdchm/schema/bdchm.yaml"
+
+DEST="$(cd "$(dirname "$0")" && pwd)/bdchm.yaml"
+TMP="$(mktemp)"
+trap 'rm -f "$TMP"' EXIT
+
+curl -fsSL "$URL" -o "$TMP"
+
+ACTUAL="$(sha256sum "$TMP" | cut -d' ' -f1)"
+if [ "$ACTUAL" != "$BDCHM_SHA256" ]; then
+    echo "Checksum mismatch for BDCHM ${BDCHM_VERSION} (${BDCHM_COMMIT})" >&2
+    echo "  expected ${BDCHM_SHA256}" >&2
+    echo "  actual   ${ACTUAL}" >&2
+    exit 1
+fi
+
+mv "$TMP" "$DEST"
+trap - EXIT
+echo "Fetched BDCHM ${BDCHM_VERSION} (${BDCHM_COMMIT:0:12}) -> ${DEST}"