Expand synthetic corpus with diabetes, temporal metadata, and broader HTN/T2D codes

[amc-corey-cox]

Closes #32, closes #33.

Temporal metadata. Conditions now carry age_at_condition_start,
age_at_condition_end where they resolve, and the visit that recorded them.
Measurements carry age_at_observation — including the blood pressure
observations nested inside a set, which previously had to be placed in time by
joining back through the set. A quarter of diagnoses are made mid-study, and a
condition only makes the measurements it explains abnormal from that point on,
so glucose before a T2D diagnosis runs at the corpus-wide 10% pathological and
after it at the diabetic 52%. validate.py checks that separation, not just
that the slots are non-null.

Diabetes. T2D Conditions plus fasting glucose and HbA1c, using the OMOP
types, units and method_type strings from MESA-ingest.

Concept coverage. Hypertension goes from one hardcoded HP:0000822 to 13
subtypes; T2D has 11. SNOMED CT and ICD-10-CM stay in the raw dbGaP-style
tables — BDCHM has no slot for source terminology, and that is where the T2D
complication detail lives.

Three judgment calls to check: pregnancy-related hypertension is excluded (both
cohorts enrol at 45-78), T2D with ophthalmic complications is excluded (its only
code is the wildcard E11.3*, so it would be indistinguishable from any other
T2D record), and reference cells naming a hierarchy rather than a term become
None rather than a guess.

Also wires up associated_evidence, which the README claimed but no spec
emitted, distinguishing an ECG-backed infarction from a self-reported one.

Both studies were run end to end through dm-bip against the pinned BDCHM v1.3.0
plus the Parquet build; 70 validation checks pass and sample/ is regenerated.
Note that adding columns shifts the positionally-derived phv accessions, so
every downstream id moves — fine for a build product, but not backwards
compatible for anyone holding an older build.

[netlify]

✅ Deploy Preview for study-palette canceled.

Name	Link
🔨 Latest commit	00d974a
🔍 Latest deploy log	https://app.netlify.com/projects/study-palette/deploys/6a9acf1db7821b000897158e

[Copilot]

🟡 Changes recommended

The updated sampling selectors can label ABSENT screening records as “Type 2 diabetes,” making the committed synthetic/sample/Condition.yaml misleading.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Expands the synthetic corpus to better support temporal cohort queries and broader hypertension/T2D concept coverage by adding diagnosis timing metadata, new diabetes measurements, and validation checks to enforce before/after behavior.

Changes:

• Add hypertension and Type 2 diabetes subtype vocab (MONDO/HPO + raw SNOMED/ICD-10-CM) and wire these through generation and validation.
• Add temporal anchors (age_at_*, associated_visit) for Conditions and MeasurementObservations (including nested BP observations).
• Add diabetes Conditions plus fasting glucose and HbA1c measurements, and validate that glucose distributions differ before vs after incident diagnosis.

File summaries

File	Description
synthetic/vocab.py	Adds diabetes measurement constants, evidence strings, and HTN/T2D subtype code lists.
synthetic/validate.py	Adds temporal, before/after, and code-coverage validation checks.
synthetic/specs.py	Extends spec generation to emit temporal anchors, diabetes measures, and condition evidence.
synthetic/generate.py	Updates raw table layouts/rows for AGE_DAYS, diabetes labs, and expanded condition columns.
synthetic/population.py	Adds diabetes + temporal condition modeling and diabetes measurement generation.
synthetic/sample.py	Updates sample selectors to include new temporal/diabetes/evidence examples.
synthetic/README.md	Documents temporal metadata, concept/source-code strategy, and associated_evidence behavior.
synthetic/specs/example_study_one/weight.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_one/wbc.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_one/height.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_one/hdl.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_one/bun.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_one/bmi.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_one/blood_pressure.yaml	Adds associated_visit + age_at_observation to nested BP observations and shifts phv mappings.
synthetic/specs/example_study_one/glucose.yaml	New spec for fasting glucose measurement.
synthetic/specs/example_study_one/hba1c.yaml	New spec for HbA1c measurement.
synthetic/specs/example_study_one/cond_hypertension.yaml	Adds condition temporal anchors, visit linkage, subtype concept population, and evidence.
synthetic/specs/example_study_one/cond_diabetes.yaml	New spec for diabetes Condition with temporal anchors and evidence.
synthetic/specs/example_study_one/cond_heart_failure.yaml	Adds condition temporal anchors, visit linkage, and evidence.
synthetic/specs/example_study_one/cond_heart_attack.yaml	Adds condition temporal anchors, visit linkage, and evidence branching via case().
synthetic/specs/example_study_one/cond_family_stroke.yaml	Adds associated_visit and evidence (no age start/end).
synthetic/specs/example_study_two/weight.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_two/wbc.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_two/height.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_two/hdl.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_two/bun.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_two/bmi.yaml	Adds age_at_observation and shifts phv mappings for new AGE_DAYS column.
synthetic/specs/example_study_two/blood_pressure.yaml	Adds associated_visit + age_at_observation to nested BP observations and shifts phv mappings.
synthetic/specs/example_study_two/glucose.yaml	New spec for fasting glucose measurement.
synthetic/specs/example_study_two/hba1c.yaml	New spec for HbA1c measurement.
synthetic/specs/example_study_two/cond_hypertension.yaml	Adds condition temporal anchors, visit linkage, subtype concept population, and evidence.
synthetic/specs/example_study_two/cond_diabetes.yaml	New spec for diabetes Condition with temporal anchors and evidence.
synthetic/specs/example_study_two/cond_heart_failure.yaml	Adds condition temporal anchors, visit linkage, and evidence.
synthetic/specs/example_study_two/cond_heart_attack.yaml	Adds condition temporal anchors, visit linkage, and evidence branching via case().
synthetic/specs/example_study_two/cond_family_stroke.yaml	Adds associated_visit and evidence (no age start/end).
synthetic/sample/Visit.yaml	Regenerated sample reflecting updated IDs/ages.
synthetic/sample/Person.yaml	Regenerated sample reflecting updated IDs and cause-of-death examples.
synthetic/sample/Participant.yaml	Regenerated sample reflecting updated participant/person linkage.
synthetic/sample/MeasurementObservationSet.yaml	Regenerated sample now includes age_at_observation/associated_visit on nested BP observations.
synthetic/sample/MeasurementObservation.yaml	Regenerated sample including glucose/HbA1c and age_at_observation on measurements.
synthetic/sample/DrugExposure.yaml	Regenerated sample reflecting updated participant IDs and drug concept selection.
synthetic/sample/Demography.yaml	Regenerated sample reflecting updated participant IDs and demographic draws.
synthetic/sample/Condition.yaml	Regenerated sample including temporal anchors and associated_evidence examples.

Review details

• Files reviewed: 43/43 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟢 Approval recommended

The changes are cohesive and internally consistent across generation, specs, samples, and validation, with only a minor docstring wording nit flagged in review comments.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

synthetic/vocab.py:7

• The module docstring says "Every CURIE here..." but this file now also defines non-CURIE literals (e.g., SNOMED numeric codes, ICD-10 codes, and evidence/method strings). This is misleading documentation; consider broadening the wording to cover identifiers/literals rather than only CURIEs.

• Files reviewed: 43/43 changed files
• Comments generated: 0 new
• Review effort level: Lite