diff --git a/synthetic/README.md b/synthetic/README.md index ea4fec3..86114be 100644 --- a/synthetic/README.md +++ b/synthetic/README.md @@ -42,8 +42,16 @@ which slots nest, because the model alone does not decide that: BDCHM gives as a uuid5 string while `value_quantity` is nested inline. It also reports where the model and the data disagree on cardinality. That is -not hypothetical — BDCHM declares `identity` multivalued and every -transformation spec, RTI's included, emits a scalar. +not hypothetical — BDCHM declares `identity` and `Condition.associated_evidence` +multivalued, and every transformation spec, RTI's included, emits a scalar for +both. + +The `associated_evidence` case has a sharp edge worth knowing about: in +linkml-map a `value:` derivation emits a scalar, while `populated_from` with +`value_mappings` honours the multivalued declaration and emits a list. Using +both for one slot leaves it carrying two shapes, and `schema.py` types a column +from the first record it sees — so it would get the rest wrong. The specs here +branch with `case()` instead, which stays scalar and matches what RTI emits. Then, from a dm-bip checkout: @@ -83,6 +91,10 @@ without running the pipeline. Regenerate it with `python sample.py` after a run. | Race | 50/30/10/10 white, black, Asian, American Indian | 60/20/10/10 white, black, Middle Eastern, Native Hawaiian | | BMI | continuous | categorical | +Both carry the same five conditions — heart failure, family history of stroke, +hypertension, Type 2 diabetes, and myocardial infarction — and the same +measurements, including fasting glucose and HbA1c. + 5% of Study Two's participants are the same individuals as in Study One. They share a `dbGaP_Subject_ID`, so harmonization resolves them to one `Person` with two `Participant` records — the same way real cross-study participation appears. @@ -90,6 +102,68 @@ two `Participant` records — the same way real cross-study participation appear One visit per participant is `TELEHEALTH`; the rest are `STUDY_SITE_VISIT`. Deceased participants stop attending, so nobody is measured after they die. +## Time + +Conditions carry `age_at_condition_start`, `age_at_condition_end` where they +resolved, and the `associated_visit` that recorded them. Measurements carry +`age_at_observation` and their own `associated_visit` — including the systolic +and diastolic observations nested inside a blood pressure set, which are +queryable on their own and should not need a join back through the set to be +placed in time. + +The reason to populate those slots is that a quarter of diagnoses are **incident**: +made between two visits rather than before enrolment. A condition only makes +the measurements it explains abnormal from its diagnosis onwards, and stops +once it resolves, so the same participant's results fall into two distinct +distributions: + +``` +glucose at or above 126 mg/dL, participants diagnosed with T2D mid-study + before the diagnosis 10.3% (the corpus-wide rate) + after the diagnosis 52.2% (the diabetic rate) +``` + +A corpus where every diagnosis predates every visit answers "measurements after +diagnosis" with the whole cohort, and the temporal slots are decoration. +`validate.py` checks the separation rather than merely checking the slots are +non-null. + +Family history is the exception: `age_at_condition_start` is defined as the +*participant's* age, which means nothing for a condition a parent had, so +family-history records carry the recording visit and no ages. + +## Concepts and source terminology + +Hypertension and Type 2 diabetes are coded from the BDC cohort-readiness +code-set reference — 13 hypertension subtypes and 11 for T2D, against the +single `HP:0000822` and no diabetes at all that the corpus started with. A +participant screened and found negative is coded to the hierarchy root, since a +questionnaire asks about hypertension in general; a subtype is recorded once +there is a diagnosis. + +SNOMED CT and ICD-10-CM appear **only in the raw dbGaP-style tables**. BDCHM has +no slot for source terminology anywhere — `condition_concept` is single-valued +over `ConditionConceptEnum`, which is MONDO ∪ HPO — so mapping source codes into +harmonized output would mean inventing a place to put them. Leaving them in the +raw layer is also what real data looks like, and it means the transformation +exercises the code-mapping path rather than assuming it away. + +That matters most for T2D, where the reference gives `MONDO:0005148` for nearly +every row and puts the complication-level detail in ICD-10-CM. The granularity +this corpus offers for diabetes complications lives in the source codes. + +Two reference rows are deliberately not used, and one class of cell is not +guessed at: + +- **Pregnancy-related hypertension** — both cohorts enrol at 45-78. +- **T2D with ophthalmic complications** — its ICD-10-CM cell is the bare + wildcard `E11.3*` and its MONDO is the shared root, so it would be + indistinguishable from every other T2D record. +- Cells naming a hierarchy rather than a term (`"44054006 hierarchy + renal + complication descendants"`) or a wildcard rather than a code become `None` + rather than a guess. A subtype with no concrete source code emits none, which + is a shape real data has too. + ## Known shapes that surprise people **`cause_of_death` is present for living participants**, multivalued, with a null @@ -124,7 +198,9 @@ has to render: - `associated_assay` referencing a CBC instance (WBC) - `qualifier` marking a value as an average (BUN) - `Condition.relationship_to_participant` for family history -- `associated_evidence` on study-record-sourced conditions +- `associated_evidence` distinguishing an ECG-backed infarction from self-report +- `age_at_condition_start` / `_end` and `associated_visit` on conditions +- `age_at_observation` on every measurement, nested ones included - `exposure_status` distinguishing absent from present drug exposures - Continuous and categorical presentations of the same concept across cohorts diff --git a/synthetic/generate.py b/synthetic/generate.py index 0131ac2..ee0c136 100644 --- a/synthetic/generate.py +++ b/synthetic/generate.py @@ -26,29 +26,58 @@ 7, ), "visit": (["dbGaP_Subject_ID", "SUBJECT_ID", "VISIT_NUM", "VISIT_TYPE", "AGE_DAYS"], 4), + # AGE_DAYS repeats the visit's age on every measurement table. It is + # derivable by joining to the visit table, but real dbGaP measurement tables + # carry the age alongside the result, and it is what age_at_observation is + # populated from without requiring the transformation to join. "clinical": ( - ["dbGaP_Subject_ID", "SUBJECT_ID", "VISIT_NUM", "HEIGHT_CM", "WEIGHT_KG", "BMI", "SBP", "DBP"], - 7, + ["dbGaP_Subject_ID", "SUBJECT_ID", "VISIT_NUM", "AGE_DAYS", + "HEIGHT_CM", "WEIGHT_KG", "BMI", "SBP", "DBP"], + 8, ), "labs": ( - ["dbGaP_Subject_ID", "SUBJECT_ID", "VISIT_NUM", "HDL", "HDL_OPERATOR", "BUN", "WBC"], - 6, + ["dbGaP_Subject_ID", "SUBJECT_ID", "VISIT_NUM", "AGE_DAYS", + "HDL", "HDL_OPERATOR", "BUN", "WBC", "GLUCOSE", "HBA1C"], + 9, ), + # One row per participant, so the visit that recorded each condition is a + # per-condition column rather than the table-level VISIT_NUM the measurement + # tables carry. SNOMED and ICD-10-CM appear only here: BDCHM has no slot for + # source terminology, so they stay in the raw layer, which is where they sit + # in real dbGaP tables too. "conditions": ( [ "dbGaP_Subject_ID", "SUBJECT_ID", "HEART_FAILURE", "HF_CONCEPT", + "HF_AGE_START", + "HF_VISIT", "FAM_STROKE", "FS_CONCEPT", "FS_RELATIVE", + "FS_VISIT", "HYPERTENSION", + "HTN_CONCEPT", + "HTN_SNOMED", + "HTN_ICD10", + "HTN_AGE_START", + "HTN_AGE_END", + "HTN_VISIT", + "DIABETES", + "DM_CONCEPT", + "DM_SNOMED", + "DM_ICD10", + "DM_AGE_START", + "DM_AGE_END", + "DM_VISIT", "HEART_ATTACK", "HA_CONCEPT", "HA_SOURCE", + "HA_AGE_START", + "HA_VISIT", ], - 10, + 28, ), "meds": (["dbGaP_Subject_ID", "SUBJECT_ID", "CCB_STATUS", "CCB_CONCEPT"], 3), } @@ -98,25 +127,33 @@ def build_rows(participants, study): for visit in p.visits: rows["visit"].append([gid, sid, visit.number, visit.category, visit.age_days]) rows["clinical"].append([ - gid, sid, visit.number, + gid, sid, visit.number, visit.age_days, visit.height_cm, visit.weight_kg, visit.bmi_category if study.bmi_categorical else visit.bmi, visit.systolic, visit.diastolic, ]) rows["labs"].append([ - gid, sid, visit.number, + gid, sid, visit.number, visit.age_days, visit.hdl, visit.hdl_operator, visit.bun, visit.wbc, + visit.glucose, visit.hba1c, ]) c = p.conditions + hf, fs, htn, dm, ha = ( + c["heart_failure"], c["family_stroke"], c["hypertension"], + c["diabetes"], c["heart_attack"], + ) rows["conditions"].append([ gid, sid, - c["heart_failure"]["status"], c["heart_failure"]["concept"], - c["family_stroke"]["status"], c["family_stroke"]["concept"], - c["family_stroke"]["relationship"], - c["hypertension"]["status"], - c["heart_attack"]["status"], c["heart_attack"]["concept"], - "STUDY_RECORD" if c["heart_attack"]["from_study_record"] else "SELF_REPORT", + hf["status"], hf["concept"], hf["age_start"], hf["visit"], + fs["status"], fs["concept"], fs["relationship"], fs["visit"], + htn["status"], htn["concept"], htn["snomed"], htn["icd10"], + htn["age_start"], htn["age_end"], htn["visit"], + dm["status"], dm["concept"], dm["snomed"], dm["icd10"], + dm["age_start"], dm["age_end"], dm["visit"], + ha["status"], ha["concept"], + "STUDY_RECORD" if ha["from_study_record"] else "SELF_REPORT", + ha["age_start"], ha["visit"], ]) rows["meds"].append([ diff --git a/synthetic/population.py b/synthetic/population.py index 5a4826d..06853a6 100644 --- a/synthetic/population.py +++ b/synthetic/population.py @@ -29,10 +29,22 @@ "heart_failure": ("systolic", "diastolic", "bmi", "hdl", "bun"), "hypertension": ("systolic", "diastolic"), "heart_attack": ("systolic", "hdl"), + "diabetes": ("glucose", "hba1c", "bmi"), } # Exactly this many HDL results fall below the assay's lower limit. HDL_BELOW_LLOD = 5 +# A quarter of the conditions a participant ever has are diagnosed during +# follow-up rather than before enrolment. Without incident cases there is +# nothing for a measurement to fall before or after, and the temporal slots +# carry no query value — a corpus where every diagnosis predates every visit +# answers 'measurements after diagnosis' with the whole cohort. +INCIDENT_FRACTION = 0.25 +# A prevalent condition starts this many years before the first visit. +PREVALENT_YEARS = (5, 20) +# Nobody is diagnosed before this age, however far back the draw reaches. +MIN_DIAGNOSIS_AGE_DAYS = 18 * 365 + HDL_LLOD = 5.0 HDL_ULOD = 150.0 @@ -135,6 +147,8 @@ class Visit: hdl_operator: str | None = None bun: float | None = None wbc: float | None = None + glucose: float | None = None + hba1c: float | None = None def _weighted(rng, mapping): @@ -152,15 +166,28 @@ def _measure(rng, normal, pathological, rate=PATHOLOGY_RATE): return round(rng.uniform(lo, hi), 1) -def _elevated(conditions): - """Which measures this participant's conditions make abnormal more often.""" +def _elevated(conditions, age_days): + """ + Which measures this participant's conditions make abnormal at this age. + + A condition only elevates measurements taken while it is active: at or + after diagnosis, and not after it resolved. That is what makes 'glucose + before diagnosis' a different distribution from 'glucose after it', rather + than the same elevated draw applied to every visit a participant attended. + """ if not conditions: return frozenset() measures = set() for name, linked in COMORBIDITY.items(): entry = conditions.get(name) - if entry and entry["status"] in (v.PRESENT, v.HISTORICAL): - measures.update(linked) + if not entry or entry["status"] not in (v.PRESENT, v.HISTORICAL): + continue + start, end = entry["age_start"], entry["age_end"] + if start is None or age_days < start: + continue + if end is not None and age_days > end: + continue + measures.update(linked) return frozenset(measures) @@ -210,7 +237,7 @@ def _make_visits(rng, study, person): def _fill_measurements(rng, study, visit, conditions=None): - elevated = _elevated(conditions) + elevated = _elevated(conditions, visit.age_days) def rate(measure): return COMORBID_PATHOLOGY_RATE if measure in elevated else PATHOLOGY_RATE @@ -236,17 +263,87 @@ def rate(measure): visit.hdl = _maybe_null(rng, _measure(rng, (40, 72), (22, 39), rate("hdl"))) visit.bun = _maybe_null(rng, _measure(rng, (7, 20), (21, 46), rate("bun"))) visit.wbc = _maybe_null(rng, _measure(rng, (4.5, 11.0), (1.8, 19.5), rate("wbc"))) + # Fasting glucose and HbA1c step straight from normal to diabetic; the + # corpus does not model the prediabetic band between them, in keeping with + # every other measure here being drawn from one range or the other. + visit.glucose = _maybe_null(rng, _measure(rng, (70, 99), (126, 260), rate("glucose"))) + visit.hba1c = _maybe_null(rng, _measure(rng, (4.8, 5.6), (6.5, 11.5), rate("hba1c"))) + + +def _diagnosis(rng, visits, resolves=False): + """ + Date a diagnosis: when it started, when it ended, and which visit recorded it. + + Prevalent conditions start before enrolment and are recorded at the first + visit — the common case, and the only one the corpus had before. Incident + ones start between two visits and are recorded at the next, which is what + puts some of a participant's measurements before the diagnosis and some + after. + + Ages are in days, matching BDCHM's age_at_condition_start and _end. + """ + if not visits: + return None, None, None + + if len(visits) > 1 and rng.random() < INCIDENT_FRACTION: + i = rng.randrange(1, len(visits)) + start = rng.randint(visits[i - 1].age_days + 1, visits[i].age_days) + recorded = visits[i] + else: + lo, hi = PREVALENT_YEARS + start = visits[0].age_days - rng.randint(lo * 365, hi * 365) + start = max(start, MIN_DIAGNOSIS_AGE_DAYS) + recorded = visits[0] + # A resolved condition ends somewhere between diagnosis and the last visit + # the participant attended, so nothing resolves after they stop being seen. + end = None + if resolves and start < visits[-1].age_days: + end = rng.randint(start + 1, visits[-1].age_days) + return start, end, recorded.number -def _make_conditions(rng): - """Condition statuses follow the proportions in the brief.""" + +def _absent(concept, subtype=None): + """ + Build a condition that was asked about and answered no. + + No diagnosis means no dates and no source code: a screening question + records the concept it screened for, not a coded diagnosis. + """ + entry = { + "status": v.ABSENT, + "concept": concept, + "age_start": None, + "age_end": None, + "visit": None, + } + if subtype is not None: + entry.update(snomed=None, icd10=None, subtype=subtype.label) + return entry + + +def _make_conditions(rng, visits): + """ + Condition statuses follow the proportions in the brief. + + Each condition the participant has carries the temporal metadata BDCHM + provides for it — age at start, age at end where it resolved, and the visit + that recorded it. Absent conditions carry none of the three. + """ conditions = {} r = rng.random() - conditions["heart_failure"] = { - "status": v.PRESENT if r < 0.15 else v.ABSENT, - "concept": rng.choice(v.HEART_FAILURE), - } + if r < 0.15: + start, end, visit = _diagnosis(rng, visits) + conditions["heart_failure"] = { + "status": v.PRESENT, + "concept": rng.choice(v.HEART_FAILURE), + "age_start": start, + "age_end": end, + "visit": visit, + } + else: + conditions["heart_failure"] = _absent(rng.choice(v.HEART_FAILURE)) r = rng.random() if r < 0.35: @@ -255,30 +352,102 @@ def _make_conditions(rng): status = v.ABSENT else: status = v.UNKNOWN + # Family history is about a relative, so age_at_condition_start — defined as + # the *participant's* age — has no meaning here. The visit that recorded the + # answer does, and is populated. conditions["family_stroke"] = { "status": status, "concept": rng.choice(v.STROKE), "relationship": rng.choice([v.NATURAL_FATHER, v.NATURAL_MOTHER]), + "age_start": None, + "age_end": None, + "visit": visits[0].number if visits else None, } + conditions["hypertension"] = _make_hypertension(rng, visits) + conditions["diabetes"] = _make_diabetes(rng, visits) + + if rng.random() < 0.10: + start, _, visit = _diagnosis(rng, visits) + conditions["heart_attack"] = { + "status": v.PRESENT, + "concept": rng.choice(v.HEART_ATTACK), + # Half of those with an MI have it from the study record, with an + # ECG in evidence; the rest self-report. + "from_study_record": rng.random() < 0.5, + "age_start": start, + # An infarction is an event, not a state that resolves, so it has a + # start and no end. + "age_end": None, + "visit": visit, + } + else: + entry = _absent(rng.choice(v.HEART_ATTACK)) + entry["from_study_record"] = False + conditions["heart_attack"] = entry + + return conditions + + +def _make_hypertension(rng, visits): + """ + Hypertension, coded to a subtype from the code-set reference when diagnosed. + + HISTORICAL means resolved, so those carry an age_at_condition_end. + """ r = rng.random() if r < 0.10: status = v.PRESENT elif r < 0.20: status = v.HISTORICAL else: - status = v.ABSENT - conditions["hypertension"] = {"status": status, "concept": v.HYPERTENSION} - - present = rng.random() < 0.10 - conditions["heart_attack"] = { - "status": v.PRESENT if present else v.ABSENT, - "concept": rng.choice(v.HEART_ATTACK), - # Half of those with an MI have it from the study record, with an ECG - # in evidence; the rest self-report. - "from_study_record": present and rng.random() < 0.5, + return _absent(v.HYPERTENSION_ROOT.concept, v.HYPERTENSION_ROOT) + + subtype = _weighted(rng, {s: s.weight for s in v.HYPERTENSION_SUBTYPES}) + start, end, visit = _diagnosis(rng, visits, resolves=status == v.HISTORICAL) + return { + "status": status, + "concept": subtype.concept, + "subtype": subtype.label, + "snomed": subtype.snomed, + "icd10": rng.choice(subtype.icd10) if subtype.icd10 else None, + "age_start": start, + "age_end": end, + "visit": visit, + } + + +def _make_diabetes(rng, visits): + """ + Type 2 diabetes, coded to a subtype from the code-set reference. + + The reference gives MONDO:0005148 for nearly every T2D row and puts the + complication detail in ICD-10-CM, so the concept is often the same across + subtypes and the source code is what distinguishes them. 'In remission' is + the one subtype that resolves, and is recorded HISTORICAL with an end age. + """ + r = rng.random() + if r < 0.02: + entry = _absent(v.DIABETES_ROOT.concept, v.DIABETES_ROOT) + entry["status"] = v.UNKNOWN + return entry + if r >= 0.22: + return _absent(v.DIABETES_ROOT.concept, v.DIABETES_ROOT) + + subtype = _weighted(rng, {s: s.weight for s in v.DIABETES_SUBTYPES}) + remission = subtype is v.DIABETES_REMISSION + status = v.HISTORICAL if remission else v.PRESENT + start, end, visit = _diagnosis(rng, visits, resolves=remission) + return { + "status": status, + "concept": subtype.concept, + "subtype": subtype.label, + "snomed": subtype.snomed, + "icd10": rng.choice(subtype.icd10) if subtype.icd10 else None, + "age_start": start, + "age_end": end, + "visit": visit, } - return conditions def build(): @@ -315,11 +484,13 @@ def build(): ) next_subject[study.name] += 1 - # Conditions first: measurements are drawn conditional on them, so - # that an abnormal result and the diagnosis explaining it co-occur. - participant.conditions = _make_conditions(rng) - + # Visits, then conditions, then measurements. Conditions are dated + # against the visit schedule, and measurements are drawn conditional + # on which conditions were active at the visit — so an abnormal + # result and the diagnosis explaining it co-occur, and only from the + # diagnosis onwards. participant.visits = _make_visits(rng, study, person) + participant.conditions = _make_conditions(rng, participant.visits) for visit in participant.visits: _fill_measurements(rng, study, visit, participant.conditions) diff --git a/synthetic/sample.py b/synthetic/sample.py index 2e5f70d..16ef304 100644 --- a/synthetic/sample.py +++ b/synthetic/sample.py @@ -56,11 +56,41 @@ ("BMI as a category", lambda d: d.get("observation_type") == "OMOP:3038553" and d.get("value_quantity", {}).get("value_concept") is not None), + ("fasting glucose, anchored to an age and a visit", + lambda d: d.get("observation_type") == "OMOP:4156660" + and d.get("age_at_observation") is not None), + ("HbA1c, reported as a percentage", + lambda d: d.get("observation_type") == "OMOP:4184637" + and d.get("value_quantity", {}).get("unit") == "%"), ], "Condition": [ ("family history, with the relative recorded", lambda d: d.get("relationship_to_participant", "").startswith("OMOP:")), ("about oneself", lambda d: d.get("relationship_to_participant") == "ONESELF"), + # The temporal slots, which are what make a condition queryable against + # the measurements taken around it. + ("ongoing, dated to the age at diagnosis", + lambda d: d.get("age_at_condition_start") and not d.get("age_at_condition_end")), + ("resolved, with both a start and an end age", + lambda d: d.get("age_at_condition_start") and d.get("age_at_condition_end")), + # Scoped to diabetes rather than any absent condition, so the record + # shown is the screening answer the neighbouring T2D entry is the + # diagnosed counterpart to. + ("undiagnosed: screened for, so no dates and no source code", + lambda d: d.get("condition_concept") == "MONDO:0005148" + and d.get("condition_status") == "ABSENT"), + # Concept breadth: T2D and a hypertension subtype more specific than the + # hierarchy root everyone else is coded to. Both must be diagnosed — + # MONDO:0005148 is also what a negative diabetes screen is coded to, so + # matching on the concept alone would label a participant who does not + # have diabetes 'Type 2 diabetes'. + ("Type 2 diabetes", + lambda d: d.get("condition_concept") == "MONDO:0005148" + and d.get("condition_status") in ("PRESENT", "HISTORICAL")), + ("hypertensive heart disease, not just 'hypertension'", + lambda d: d.get("condition_concept") == "MONDO:0001302"), + ("an infarction with an ECG in evidence rather than self-report", + lambda d: d.get("associated_evidence") == "electrocardiogram"), ], "DrugExposure": [ ("taking a calcium channel blocker", lambda d: d.get("drug_concept")), @@ -76,13 +106,23 @@ def load(path): def pick(records, selectors): - """Take the first record matching each selector, keeping the labels.""" - chosen = [] + """ + Take the first unused record matching each selector, keeping the labels. + + Selectors overlap — a record can be about oneself, undiagnosed, and coded + to diabetes all at once — so without skipping what earlier selectors took, + one record turns up several times under several labels and the sample stops + showing as many distinct shapes as it claims to. + """ + chosen, seen = [], set() for label, predicate in selectors: for record in records: + if record.get("id") in seen: + continue try: if predicate(record): chosen.append((label, record)) + seen.add(record.get("id")) break except (AttributeError, TypeError): continue diff --git a/synthetic/sample/Condition.yaml b/synthetic/sample/Condition.yaml index 0faa180..4f253bc 100644 --- a/synthetic/sample/Condition.yaml +++ b/synthetic/sample/Condition.yaml @@ -1,31 +1,185 @@ # study_one: family history, with the relative recorded +associated_evidence: self-report questionnaire associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a -condition_concept: MONDO:0005264 +associated_visit: 75122f58-efd2-559d-9eac-2619199795aa +condition_concept: MONDO:0013792 condition_provenance: PATIENT_SELF-REPORTED_CONDITION condition_status: ABSENT id: 1bacbc56-4aaf-5017-b70c-42fa03f9d48b relationship_to_participant: OMOP:4321888 --- # study_one: about oneself +age_at_condition_end: null +age_at_condition_start: null +associated_evidence: self-report questionnaire associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a -condition_concept: MONDO:0005068 +associated_visit: null +condition_concept: MONDO:0005148 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: ABSENT +id: 93dc8788-7f85-5c52-b78f-0244bd7f1f3b +relationship_to_participant: ONESELF +--- +# study_one: ongoing, dated to the age at diagnosis +age_at_condition_end: null +age_at_condition_start: 22043 +associated_evidence: self-report questionnaire +associated_participant: 1e3b97bf-f17d-53ab-b05b-da66139cf65a +associated_visit: 91a85e60-c7b1-55f6-9c7a-1caef7e28fc1 +condition_concept: MONDO:0005148 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: PRESENT +id: fe9fd9e2-68c7-541e-a276-f5410ab6c5ae +relationship_to_participant: ONESELF +--- +# study_one: resolved, with both a start and an end age +age_at_condition_end: 23679 +age_at_condition_start: 20465 +associated_evidence: self-report questionnaire +associated_participant: 005f1434-9203-5c31-8d4f-4426ea0d1edc +associated_visit: 912a37e2-3cd7-5e43-90e4-ea31243a63cc +condition_concept: MONDO:0005148 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: HISTORICAL +id: 2e1fc958-7841-5bb9-ab91-051f7d35e061 +relationship_to_participant: ONESELF +--- +# study_one: undiagnosed: screened for, so no dates and no source code +age_at_condition_end: null +age_at_condition_start: null +associated_evidence: self-report questionnaire +associated_participant: e805802b-da55-5d52-9db3-810ad3ed2058 +associated_visit: null +condition_concept: MONDO:0005148 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: ABSENT +id: 7421193c-9e57-5018-ad8e-956ccb1df1a5 +relationship_to_participant: ONESELF +--- +# study_one: Type 2 diabetes +age_at_condition_end: null +age_at_condition_start: 17887 +associated_evidence: self-report questionnaire +associated_participant: 94470960-2d15-5ae5-8eb7-664b56478e73 +associated_visit: d3e958b4-898c-5942-be79-765f4890d4f6 +condition_concept: MONDO:0005148 condition_provenance: PATIENT_SELF-REPORTED_CONDITION condition_status: PRESENT -id: 4058ed2d-92d4-5b8a-af0e-aee32c2c9960 +id: 696a19ca-77f3-5443-af49-36d06f7a33ca +relationship_to_participant: ONESELF +--- +# study_one: hypertensive heart disease, not just 'hypertension' +age_at_condition_end: null +age_at_condition_start: 22145 +associated_evidence: self-report questionnaire +associated_participant: 4bf678da-a746-5d29-a100-2814a8973b28 +associated_visit: 8c37d465-a2ae-5a97-b0fa-6ffc0e0625ec +condition_concept: MONDO:0001302 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: PRESENT +id: d4033a16-0e38-522b-99a3-a7f7d21bbc53 +relationship_to_participant: ONESELF +--- +# study_one: an infarction with an ECG in evidence rather than self-report +age_at_condition_start: 15441 +associated_evidence: electrocardiogram +associated_participant: c033e47e-9147-55f9-a2f2-d9aee07f19d9 +associated_visit: 35c0c7fc-fb59-567f-903f-cac28727e381 +condition_concept: MONDO:0006803 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: PRESENT +id: 4b55221e-27aa-5882-9b26-485714d8a2c9 relationship_to_participant: ONESELF --- # study_two: family history, with the relative recorded +associated_evidence: self-report questionnaire associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb -condition_concept: MONDO:0013792 +associated_visit: 22bf721d-ce80-5f08-b1c5-d19857b89b15 +condition_concept: MONDO:0006809 condition_provenance: PATIENT_SELF-REPORTED_CONDITION -condition_status: UNKNOWN +condition_status: ABSENT id: 87bc5b14-516d-598a-a33d-0bd3d7b92615 relationship_to_participant: OMOP:4321888 --- # study_two: about oneself +age_at_condition_end: null +age_at_condition_start: null +associated_evidence: self-report questionnaire associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb -condition_concept: MONDO:0005068 +associated_visit: null +condition_concept: MONDO:0005148 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: ABSENT +id: ac60fd9e-c1ff-55b9-829b-261592dbaaf5 +relationship_to_participant: ONESELF +--- +# study_two: ongoing, dated to the age at diagnosis +age_at_condition_end: null +age_at_condition_start: 16623 +associated_evidence: self-report questionnaire +associated_participant: 95a9eb19-e6ee-59bd-bb86-1a9f46b34601 +associated_visit: 1f8fd2ab-290a-551e-9f62-14dc76f5d60a +condition_concept: MONDO:0005148 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: PRESENT +id: cd68a5aa-db88-5dfa-9e87-86cc123056be +relationship_to_participant: ONESELF +--- +# study_two: resolved, with both a start and an end age +age_at_condition_end: 26131 +age_at_condition_start: 25125 +associated_evidence: self-report questionnaire +associated_participant: 16f1434c-7148-5af2-bf4b-3c9b0f5345bd +associated_visit: 1e7adf97-1060-5efd-be0b-af4b2b0357d8 +condition_concept: MONDO:0005148 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: HISTORICAL +id: dfd1a68f-e83b-5bbe-b087-0fa6986fa01f +relationship_to_participant: ONESELF +--- +# study_two: undiagnosed: screened for, so no dates and no source code +age_at_condition_end: null +age_at_condition_start: null +associated_evidence: self-report questionnaire +associated_participant: fd79135a-f146-5bd8-8774-cd330887be23 +associated_visit: null +condition_concept: MONDO:0005148 condition_provenance: PATIENT_SELF-REPORTED_CONDITION condition_status: ABSENT -id: 67332483-bbf9-57fd-98de-9f54e47bcd9e +id: 359d1efa-1663-55eb-ae26-49e37732b615 +relationship_to_participant: ONESELF +--- +# study_two: Type 2 diabetes +age_at_condition_end: null +age_at_condition_start: 25238 +associated_evidence: self-report questionnaire +associated_participant: e562d1f8-aea8-5991-b75f-5f135d78015f +associated_visit: f4071f3b-98cf-5adf-9e62-ee3ed69111c6 +condition_concept: MONDO:0005148 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: PRESENT +id: 1abb7978-2941-5bfd-ae1a-f3ad8f1716ef +relationship_to_participant: ONESELF +--- +# study_two: hypertensive heart disease, not just 'hypertension' +age_at_condition_end: 27128 +age_at_condition_start: 20216 +associated_evidence: self-report questionnaire +associated_participant: 6d489c3a-0277-5196-8034-a232e2b76e3a +associated_visit: 6c3f6f5d-788d-5eda-a287-e6f7598e9d95 +condition_concept: MONDO:0001302 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: HISTORICAL +id: 93201bc3-a85a-5db5-ac2e-7f0a02d48eb3 +relationship_to_participant: ONESELF +--- +# study_two: an infarction with an ECG in evidence rather than self-report +age_at_condition_start: 19714 +associated_evidence: electrocardiogram +associated_participant: cd0524eb-56c3-53f2-bd42-3a8aa2cceb36 +associated_visit: c8edc65c-3eb1-5a90-a683-b28aee2660e6 +condition_concept: MONDO:0005068 +condition_provenance: PATIENT_SELF-REPORTED_CONDITION +condition_status: PRESENT +id: daf67a3e-8982-50fe-ba22-aba5987760d3 relationship_to_participant: ONESELF diff --git a/synthetic/sample/Demography.yaml b/synthetic/sample/Demography.yaml index b9c45d5..ebf8a71 100644 --- a/synthetic/sample/Demography.yaml +++ b/synthetic/sample/Demography.yaml @@ -1,10 +1,10 @@ # study_one: Hispanic ethnicity -associated_participant: dd37092b-1515-5c8b-aebf-18507b1f98f6 +associated_participant: 0875292e-8eee-572d-9876-7a3d49bb7e6a ethnicity: OMOP:38003563 -id: 27107a6a-5d26-54ac-90cd-c243d4102d3a +id: ca8665d8-d9d6-567e-a492-5430c33bc9f3 race: -- OMOP:8527 -sex: OMOP:8507 +- OMOP:8516 +sex: OMOP:8532 --- # study_one: non-Hispanic associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a @@ -15,11 +15,11 @@ race: sex: OMOP:8507 --- # study_two: Hispanic ethnicity -associated_participant: 390f76f9-c667-5487-a1b9-ecd8b5fe9927 +associated_participant: 8c4e8af1-e644-58f4-8dc4-0c3bba8a3df0 ethnicity: OMOP:38003563 -id: 021e4f47-6504-5725-b1e0-d79450ce064a +id: 90b255f1-7ac0-5283-b31e-9d0574f5c263 race: -- OMOP:8527 +- OMOP:8516 sex: OMOP:8532 --- # study_two: non-Hispanic @@ -27,5 +27,5 @@ associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb ethnicity: OMOP:38003564 id: 18e9e092-8ad5-5414-b9e8-677ff7d886d1 race: -- OMOP:8515 -sex: OMOP:8532 +- OMOP:8527 +sex: OMOP:8507 diff --git a/synthetic/sample/DrugExposure.yaml b/synthetic/sample/DrugExposure.yaml index 9ec6e0b..95c42b2 100644 --- a/synthetic/sample/DrugExposure.yaml +++ b/synthetic/sample/DrugExposure.yaml @@ -13,11 +13,11 @@ exposure_status: ABSENT id: f9592e79-ad0d-5e45-b130-a5823ecc447a --- # study_two: taking a calcium channel blocker -associated_participant: 0a0a6844-cd9d-5401-95b0-f7dfa5d910db -drug_concept: NDFRT:N0000175421 +associated_participant: 49cc38d7-99f6-5d03-ba66-901c6bf3fa4a +drug_concept: RxCUI:7417 exposure_provenance: PATIENT_SELF_REPORTED_MEDICATION exposure_status: PRESENT -id: 5cd32ad9-48ef-570f-a1bb-c7ef61bbc57c +id: 06fb143b-6517-5844-90e4-9e68f8f85281 --- # study_two: not taking one associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb diff --git a/synthetic/sample/MeasurementObservation.yaml b/synthetic/sample/MeasurementObservation.yaml index 8d8a406..4c941eb 100644 --- a/synthetic/sample/MeasurementObservation.yaml +++ b/synthetic/sample/MeasurementObservation.yaml @@ -1,4 +1,5 @@ # study_one: HDL below the assay's lower limit, carrying the operator +age_at_observation: 20075 associated_assay: id: ddcb07cc-0d54-5828-93a8-495a3f8a3f07 lower_limit_of_detection: @@ -10,17 +11,18 @@ associated_assay: id: b0655aa0-8a6e-5719-809d-e5678af1e9e6 unit: mg/dL value_decimal: 150.0 -associated_participant: 9e9b7cbd-07b5-5ea9-9912-6cc95a1e8ca5 -associated_visit: 7b35a86e-1813-56e3-97d2-0fbb66803db0 -id: b01f532f-d4cd-5dc7-b494-79b5876a6e8c +associated_participant: 161a3327-cd8f-5d53-b3fd-1e0d94063646 +associated_visit: 8cfdc8e5-8d5d-561e-8f30-800564338c57 +id: 5d27f1d2-8527-5998-9b9d-ec0f2520cd74 observation_type: OMOP:3007070 value_quantity: - id: be2ab2d1-121b-547f-aae9-c647ddf3d69f + id: be0ac0b6-6f27-5a00-842d-133d8cb0d990 operator: < unit: mg/dL value_decimal: 5.0 --- # study_one: HDL with assay detection limits +age_at_observation: 27375 associated_assay: id: ddcb07cc-0d54-5828-93a8-495a3f8a3f07 lower_limit_of_detection: @@ -40,9 +42,10 @@ value_quantity: id: 16ec110d-a004-5269-a4f0-33574ed63357 operator: null unit: mg/dL - value_decimal: 58.4 + value_decimal: 45.1 --- # study_one: WBC referencing the CBC assay +age_at_observation: 27375 associated_assay: id: 25bf7403-bd81-5764-9e90-8af5246333ff method: MMO:0000533 @@ -53,9 +56,10 @@ observation_type: OMOP:3000905 value_quantity: id: 071da59d-a293-55d0-a8ab-141a15d99ab3 unit: 10*3/uL - value_decimal: 10.4 + value_decimal: 6.0 --- # study_one: BUN qualified as an average +age_at_observation: 27375 associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a associated_visit: 75122f58-efd2-559d-9eac-2619199795aa id: 74e323bb-6fa2-5a03-b8d4-01d1b37713bb @@ -64,9 +68,10 @@ qualifier: AVERAGE value_quantity: id: 54f1a4f3-26f9-532c-af11-0f9a032fcc7e unit: mg/dL - value_decimal: 18.4 + value_decimal: 8.1 --- # study_one: BMI as a continuous value +age_at_observation: 27375 associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a associated_visit: 75122f58-efd2-559d-9eac-2619199795aa id: dcfddca5-aea6-55ea-a3f5-05c8ea5e87b6 @@ -74,9 +79,34 @@ observation_type: OMOP:3038553 value_quantity: id: 69a41584-68c4-5f4c-ab51-419b2f2416f7 unit: kg/m2 - value_decimal: 21.9 + value_decimal: 22.2 +--- +# study_one: fasting glucose, anchored to an age and a visit +age_at_observation: 27375 +associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a +associated_visit: 75122f58-efd2-559d-9eac-2619199795aa +id: f0660717-b1ad-5ee2-971f-dafe2acb6f1f +method_type: CALIBRATED +observation_type: OMOP:4156660 +value_quantity: + id: 8a984fc0-87d8-5de3-9273-85a9e3a40853 + unit: mg/dL + value_decimal: 147.3 +--- +# study_one: HbA1c, reported as a percentage +age_at_observation: 27375 +associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a +associated_visit: 75122f58-efd2-559d-9eac-2619199795aa +id: 9d71fb65-1fa1-5b33-ba61-e3e178121ed1 +method_type: blood assay +observation_type: OMOP:4184637 +value_quantity: + id: e446c271-5140-5a09-836f-c4517ab63d46 + unit: '%' + value_decimal: null --- # study_two: HDL below the assay's lower limit, carrying the operator +age_at_observation: 17155 associated_assay: id: 1533b5fa-b7d7-5d34-a075-88950c88b9b3 lower_limit_of_detection: @@ -88,17 +118,18 @@ associated_assay: id: cd40e959-ee21-55fb-bcd2-a6a84fdae679 unit: mg/dL value_decimal: 150.0 -associated_participant: f82b92bc-cc81-5d6d-9fcc-e2b1df68b733 -associated_visit: ca601664-8b89-5304-8f58-200d2ccaa4a2 -id: e5e4b790-b5ed-57ee-8679-9c75b4af3960 +associated_participant: 69772f79-909d-5669-bcde-83fa2132b25d +associated_visit: c8f93751-d44c-5b98-9276-94a6eb2bf682 +id: 7dc60dd9-677a-5926-a045-9a458dbc4e87 observation_type: OMOP:3007070 value_quantity: - id: 31829a46-4891-541c-b0ae-437badfdb600 + id: 14c0713c-1c86-56e2-b011-093e6b9c80d2 operator: < unit: mg/dL value_decimal: 5.0 --- # study_two: HDL with assay detection limits +age_at_observation: 28470 associated_assay: id: 1533b5fa-b7d7-5d34-a075-88950c88b9b3 lower_limit_of_detection: @@ -118,9 +149,10 @@ value_quantity: id: 9c071689-64ca-58df-b23b-1c664ee4336f operator: null unit: mg/dL - value_decimal: 63.7 + value_decimal: 51.2 --- # study_two: WBC referencing the CBC assay +age_at_observation: 28470 associated_assay: id: c1a9ff74-d562-5bd5-b97f-15a2ed7a8e57 method: MMO:0000533 @@ -131,9 +163,10 @@ observation_type: OMOP:3000905 value_quantity: id: a91f6e17-efb3-5168-b09b-0a66df0abe43 unit: 10*3/uL - value_decimal: 9.9 + value_decimal: 8.2 --- # study_two: BUN qualified as an average +age_at_observation: 28470 associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb associated_visit: 22bf721d-ce80-5f08-b1c5-d19857b89b15 id: d5c1a7cf-457f-5274-b8f5-41644eb4b1c5 @@ -142,9 +175,10 @@ qualifier: AVERAGE value_quantity: id: 6e3ab74d-99e2-5419-984a-29361d647941 unit: mg/dL - value_decimal: 17.9 + value_decimal: 13.8 --- # study_two: BMI as a category +age_at_observation: 28470 associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb associated_visit: 22bf721d-ce80-5f08-b1c5-d19857b89b15 id: 6b2572f5-a244-504c-9f4f-7dae5a26ceb0 @@ -152,3 +186,27 @@ observation_type: OMOP:3038553 value_quantity: id: f427eb8f-c038-5c83-9ea3-ffa4586a6728 value_concept: normal weight +--- +# study_two: fasting glucose, anchored to an age and a visit +age_at_observation: 28470 +associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb +associated_visit: 22bf721d-ce80-5f08-b1c5-d19857b89b15 +id: 5a62b826-238f-59ea-964e-798aa9a21311 +method_type: CALIBRATED +observation_type: OMOP:4156660 +value_quantity: + id: 1caaceec-be4a-51fa-85c0-9bf51827e9b3 + unit: mg/dL + value_decimal: 73.5 +--- +# study_two: HbA1c, reported as a percentage +age_at_observation: 28470 +associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb +associated_visit: 22bf721d-ce80-5f08-b1c5-d19857b89b15 +id: 53448361-7e8f-509e-9da6-2686214541d5 +method_type: blood assay +observation_type: OMOP:4184637 +value_quantity: + id: ce0c92b9-08a7-5b36-bd9b-f2a617efd1dd + unit: '%' + value_decimal: 5.4 diff --git a/synthetic/sample/MeasurementObservationSet.yaml b/synthetic/sample/MeasurementObservationSet.yaml index 15b032b..1184281 100644 --- a/synthetic/sample/MeasurementObservationSet.yaml +++ b/synthetic/sample/MeasurementObservationSet.yaml @@ -3,7 +3,9 @@ associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a associated_visit: 75122f58-efd2-559d-9eac-2619199795aa id: f443b266-2d72-5429-8df4-43dfaf1bcc12 observations: -- associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a +- age_at_observation: 27375 + associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a + associated_visit: 75122f58-efd2-559d-9eac-2619199795aa body_position: SITTING_POSITION body_site: Right arm id: daae1547-43eb-5db9-88a0-1fc442022951 @@ -11,8 +13,10 @@ observations: value_quantity: id: 7fe894d1-919d-573a-a17d-d36adc78f486 unit: mm[Hg] - value_decimal: 164.3 -- associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a + value_decimal: 128.7 +- age_at_observation: 27375 + associated_participant: ded1d538-866f-5c18-8aa2-0b24f1f08a5a + associated_visit: 75122f58-efd2-559d-9eac-2619199795aa body_position: SITTING_POSITION body_site: Right arm id: ecee49c5-c3c5-540b-b723-f43b78f05656 @@ -20,14 +24,16 @@ observations: value_quantity: id: f2c228df-56c8-56a0-946d-d8c06f69d7e0 unit: mm[Hg] - value_decimal: 82.6 + value_decimal: 82.5 --- # study_two: blood pressure, systolic and diastolic in one set associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb associated_visit: 22bf721d-ce80-5f08-b1c5-d19857b89b15 id: e9806428-dd0d-5ea0-b9d4-a7b21d8156c5 observations: -- associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb +- age_at_observation: 28470 + associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb + associated_visit: 22bf721d-ce80-5f08-b1c5-d19857b89b15 body_position: SITTING_POSITION body_site: Right arm id: 2d648e6e-ea07-5879-a4f8-569930bc20d0 @@ -35,8 +41,10 @@ observations: value_quantity: id: d7463d3b-6bbb-53a5-9ee8-d61c723822bd unit: mm[Hg] - value_decimal: 128.2 -- associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb + value_decimal: 124.8 +- age_at_observation: 28470 + associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb + associated_visit: 22bf721d-ce80-5f08-b1c5-d19857b89b15 body_position: SITTING_POSITION body_site: Right arm id: f3d69c77-bf43-5e06-9185-ce6cac66e8de @@ -44,4 +52,4 @@ observations: value_quantity: id: cbe14372-7714-55f0-a567-db4b1735f728 unit: mm[Hg] - value_decimal: 94.6 + value_decimal: 72.0 diff --git a/synthetic/sample/Participant.yaml b/synthetic/sample/Participant.yaml index 97a6123..9882bd2 100644 --- a/synthetic/sample/Participant.yaml +++ b/synthetic/sample/Participant.yaml @@ -5,7 +5,7 @@ identity: '100001' member_of_research_study: Example Study One --- # study_two: enrolment, linked to a Person -associated_person: b9733cac-ed81-52b8-bb35-ac8f1fccb21f +associated_person: 383327ec-dd83-5a40-a8a5-07aa34bd31d6 id: e9d16f51-a014-5b89-bb71-d3550b9a7cbb identity: '200001' member_of_research_study: Example Study Two diff --git a/synthetic/sample/Person.yaml b/synthetic/sample/Person.yaml index f547a0a..0c47024 100644 --- a/synthetic/sample/Person.yaml +++ b/synthetic/sample/Person.yaml @@ -20,20 +20,20 @@ vital_status: OMOP:4230556 --- # study_two: deceased, with cause of death cause_of_death: -- cause: ICD10CM:R99 - id: c810c2ab-9b85-58f1-a7ba-009eccd3b843 +- cause: ICD10CM:I20-I25 + id: dec11e22-1e3d-5caa-8f2e-431632c49f26 order: 1 -id: 79153ada-4607-5077-a30d-93cc240aa7df -identity: '900441' +id: 5dcbaad9-f657-5cfe-b8f3-c180aaf0fc56 +identity: '900449' species: NCBITaxon:9606 vital_status: OMOP:434489 --- # study_two: living, note the empty cause_of_death entry cause_of_death: - cause: null - id: c92da1ee-347b-585a-a12c-bda32ec3c201 + id: 77455809-ed44-58e7-8916-5fb170defbf4 order: null -id: b9733cac-ed81-52b8-bb35-ac8f1fccb21f -identity: '900093' +id: 383327ec-dd83-5a40-a8a5-07aa34bd31d6 +identity: '900483' species: NCBITaxon:9606 vital_status: OMOP:4230556 diff --git a/synthetic/sample/Visit.yaml b/synthetic/sample/Visit.yaml index b8797ee..83f195d 100644 --- a/synthetic/sample/Visit.yaml +++ b/synthetic/sample/Visit.yaml @@ -13,15 +13,15 @@ visit_category: STUDY_SITE_VISIT visit_provenance: CASE_REPORT_FORM --- # study_two: telehealth -age_at_visit_start: 22630 +age_at_visit_start: 28470 associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb -id: 17f3fa62-5837-571d-9e3a-5b089b4c1d9d +id: 22bf721d-ce80-5f08-b1c5-d19857b89b15 visit_category: TELEHEALTH visit_provenance: CASE_REPORT_FORM --- # study_two: study site visit -age_at_visit_start: 21170 +age_at_visit_start: 29200 associated_participant: e9d16f51-a014-5b89-bb71-d3550b9a7cbb -id: 22bf721d-ce80-5f08-b1c5-d19857b89b15 +id: 3e539a4f-bd85-5be1-a262-e65e539ba5e9 visit_category: STUDY_SITE_VISIT visit_provenance: CASE_REPORT_FORM diff --git a/synthetic/specs.py b/synthetic/specs.py index ec83de1..488772b 100644 --- a/synthetic/specs.py +++ b/synthetic/specs.py @@ -54,13 +54,34 @@ def ident(study, table): return subject, f"'uuid5(\"{PARTICIPANT_NS}\", str({{{subject}}}) + \":{study.name}\")'" -def visit_ref(study, table): - """Build the uuid5 reference to a row's visit.""" +def visit_ref(study, table, column="VISIT_NUM"): + """ + Build the uuid5 reference to a row's visit. + + The measurement tables carry one VISIT_NUM for the row. The conditions + table is one row per participant, so each condition names the visit that + recorded it in its own column. + """ subject = phv(study, table, "SUBJECT_ID") - num = phv(study, table, "VISIT_NUM") + num = phv(study, table, column) return f"'uuid5(\"{VISIT_NS}\", str({{{subject}}}) + \":{study.name} VISIT \" + str({{{num}}}))'" +def observed_at(study, table): + """ + Build the two temporal anchors every MeasurementObservation should carry. + + RTI derives age_at_observation as '{age_in_years} * 365' because their raw + columns hold years; ours hold days already, so this is a plain + populated_from and nulls propagate rather than needing a case() guard. + """ + return ( + f" associated_visit:\n expr: {visit_ref(study, table)}\n" + f" age_at_observation:\n" + f" populated_from: {phv(study, table, 'AGE_DAYS')}\n" + ) + + def person_and_participant(study): """Build the Person and Participant derivations.""" t = study.tables["subject"] @@ -157,6 +178,9 @@ def blood_pressure(study): dbp = phv(study, "clinical", "DBP") def observation(concept, source, label): + # The set carries associated_visit too, but an observation is also + # queryable on its own, and a consumer filtering MeasurementObservation + # by time should not have to join back through the set to do it. return f""" - MeasurementObservation: populated_from: {t} slot_derivations: @@ -164,6 +188,10 @@ def observation(concept, source, label): expr: {uid(OBSERVATION_NS, row_key(study, "clinical", label))} associated_participant: expr: {participant} + associated_visit: + expr: {visit} + age_at_observation: + populated_from: {phv(study, "clinical", "AGE_DAYS")} observation_type: value: {concept} body_position: @@ -208,7 +236,6 @@ def simple_measure(study, table, column, concept, unit, extra="", categorical=Fa """ t = study.tables[table] _, participant = ident(study, table) - visit = visit_ref(study, table) key = row_key(study, table, column) if categorical: value_slot = "value_concept" @@ -224,9 +251,7 @@ def simple_measure(study, table, column, concept, unit, extra="", categorical=Fa expr: {uid(OBSERVATION_NS, key)} associated_participant: expr: {participant} - associated_visit: - expr: {visit} - observation_type: +{observed_at(study, table)} observation_type: value: {concept} {extra} value_quantity: class_derivations: @@ -245,7 +270,6 @@ def hdl(study): t = study.tables["labs"] operator = phv(study, "labs", "HDL_OPERATOR") _, participant = ident(study, "labs") - visit = visit_ref(study, "labs") return f"""- class_derivations: MeasurementObservation: populated_from: {t} @@ -254,9 +278,7 @@ def hdl(study): expr: {uid(OBSERVATION_NS, row_key(study, "labs", "HDL"))} associated_participant: expr: {participant} - associated_visit: - expr: {visit} - observation_type: +{observed_at(study, "labs")} observation_type: value: OMOP:3007070 associated_assay: class_derivations: @@ -309,7 +331,6 @@ def wbc(study): """WBC references a CBC assay instance through associated_assay.""" t = study.tables["labs"] _, participant = ident(study, "labs") - visit = visit_ref(study, "labs") return f"""- class_derivations: MeasurementObservation: populated_from: {t} @@ -318,9 +339,7 @@ def wbc(study): expr: {uid(OBSERVATION_NS, row_key(study, "labs", "WBC"))} associated_participant: expr: {participant} - associated_visit: - expr: {visit} - observation_type: +{observed_at(study, "labs")} observation_type: value: OMOP:3000905 associated_assay: class_derivations: @@ -345,8 +364,28 @@ def wbc(study): """ -def condition(study, label, status_col, concept_col, provenance, relationship=None): - """One Condition. relationship defaults to ONESELF for self-reported history.""" +def condition( + study, + label, + status_col, + concept_col, + provenance, + relationship=None, + visit_col=None, + age_start_col=None, + age_end_col=None, + evidence=None, +): + """ + One Condition. relationship defaults to ONESELF for self-reported history. + + The temporal slots are optional because not every condition has a meaningful + one: family history is about a relative, so the participant's age at its + start is undefined, and an infarction is an event rather than a state that + resolves, so it has no end. Where a column is given, it is populated + straight through — the raw columns hold days and are already null when the + condition is absent, so no case() guard is needed to suppress them. + """ t = study.tables["conditions"] _, participant = ident(study, "conditions") relationship_line = ( @@ -354,11 +393,21 @@ def condition(study, label, status_col, concept_col, provenance, relationship=No if relationship else f" value: {v.ONESELF}" ) - concept = ( - f" populated_from: {phv(study, 'conditions', concept_col)}" - if concept_col - else " value: HP:0000822" - ) + + temporal = "" + if visit_col: + temporal += f" associated_visit:\n expr: {visit_ref(study, 'conditions', visit_col)}\n" + if age_start_col: + temporal += ( + f" age_at_condition_start:\n" + f" populated_from: {phv(study, 'conditions', age_start_col)}\n" + ) + if age_end_col: + temporal += ( + f" age_at_condition_end:\n" + f" populated_from: {phv(study, 'conditions', age_end_col)}\n" + ) + return f"""- class_derivations: Condition: populated_from: {t} @@ -367,15 +416,49 @@ def condition(study, label, status_col, concept_col, provenance, relationship=No expr: {uid(v.CONDITION_NS, row_key(study, "conditions", label))} associated_participant: expr: {participant} - condition_concept: -{concept} +{temporal} condition_concept: + populated_from: {phv(study, 'conditions', concept_col)} condition_status: populated_from: {phv(study, "conditions", status_col)} condition_provenance: value: {provenance} relationship_to_participant: {relationship_line} -""" +{evidence or ""}""" + + +def self_report_evidence(): + """ + Evidence for a condition taken from a questionnaire. + + BDCHM ranges associated_evidence on Entity and marks it multivalued, but + MESA-ingest puts a bare descriptive string there. The corpus follows the + real spec rather than the model, so schema.py reports the cardinality + disagreement instead of the corpus hiding it. + """ + return f' associated_evidence:\n value: "{v.SELF_REPORT_EVIDENCE}"\n' + + +def mi_evidence(study): + """ + Evidence for an infarction, which differs by where the record came from. + + Half of the MIs are in the study record with an ECG behind them; the rest + are self-reported. + + This branches with case() rather than the value_mappings the real specs use + for coded columns, because the two disagree on cardinality: value_mappings + honours BDCHM's multivalued declaration and emits a list, while a plain + `value:` emits a scalar. Mixing them would leave one slot carrying both + shapes, which schema.py types from the first record it sees and would + therefore get wrong for the rest. RTI emits the scalar, so the corpus does. + """ + source = phv(study, "conditions", "HA_SOURCE") + return ( + f" associated_evidence:\n" + f" expr: 'case(({{{source}}} == \"STUDY_RECORD\", \"{v.ECG_EVIDENCE}\")," + f" ({{{source}}} == \"SELF_REPORT\", \"{v.SELF_REPORT_EVIDENCE}\"))'\n" + ) def drug_exposure(study): @@ -420,16 +503,42 @@ def build(study): extra=' qualifier:\n value: "AVERAGE"\n', ), "wbc": wbc(study), + # Fasting glucose and HbA1c follow MESA-ingest's specs: the same OMOP + # observation types, units, and method_type strings. + "glucose": simple_measure( + study, "labs", "GLUCOSE", v.FASTING_GLUCOSE, "mg/dL", + extra=f' method_type:\n value: {v.GLUCOSE_METHOD}\n', + ), + "hba1c": simple_measure( + study, "labs", "HBA1C", v.HBA1C, "%", + extra=f' method_type:\n value: "{v.HBA1C_METHOD}"\n', + ), "cond_heart_failure": condition( - study, "heart_failure", "HEART_FAILURE", "HF_CONCEPT", v.SELF_REPORTED_CONDITION), + study, "heart_failure", "HEART_FAILURE", "HF_CONCEPT", v.SELF_REPORTED_CONDITION, + visit_col="HF_VISIT", age_start_col="HF_AGE_START", + evidence=self_report_evidence(), + ), "cond_family_stroke": condition( study, "family_stroke", "FAM_STROKE", "FS_CONCEPT", v.SELF_REPORTED_CONDITION, relationship=phv(study, "conditions", "FS_RELATIVE"), + visit_col="FS_VISIT", + evidence=self_report_evidence(), ), "cond_hypertension": condition( - study, "hypertension", "HYPERTENSION", None, v.SELF_REPORTED_CONDITION), + study, "hypertension", "HYPERTENSION", "HTN_CONCEPT", v.SELF_REPORTED_CONDITION, + visit_col="HTN_VISIT", age_start_col="HTN_AGE_START", age_end_col="HTN_AGE_END", + evidence=self_report_evidence(), + ), + "cond_diabetes": condition( + study, "diabetes", "DIABETES", "DM_CONCEPT", v.SELF_REPORTED_CONDITION, + visit_col="DM_VISIT", age_start_col="DM_AGE_START", age_end_col="DM_AGE_END", + evidence=self_report_evidence(), + ), "cond_heart_attack": condition( - study, "heart_attack", "HEART_ATTACK", "HA_CONCEPT", v.SELF_REPORTED_CONDITION), + study, "heart_attack", "HEART_ATTACK", "HA_CONCEPT", v.SELF_REPORTED_CONDITION, + visit_col="HA_VISIT", age_start_col="HA_AGE_START", + evidence=mi_evidence(study), + ), "drug_exposure": drug_exposure(study), } return specs diff --git a/synthetic/specs/example_study_one/blood_pressure.yaml b/synthetic/specs/example_study_one/blood_pressure.yaml index 9bb3b4e..6903309 100644 --- a/synthetic/specs/example_study_one/blood_pressure.yaml +++ b/synthetic/specs/example_study_one/blood_pressure.yaml @@ -17,6 +17,10 @@ expr: 'uuid5("https://w3id.org/bdchm/MeasurementObservation", str({phv10111300}) + ":Example Study One:" + str({phv10111301}) + ":systolic")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111300}) + ":Example Study One")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111300}) + ":Example Study One VISIT " + str({phv10111301}))' + age_at_observation: + populated_from: phv10111302 observation_type: value: OMOP:4152194 body_position: @@ -31,7 +35,7 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111300}) + ":Example Study One:" + str({phv10111301}) + ":systolic")' value_decimal: - populated_from: phv10111305 + populated_from: phv10111306 unit: value: "mm[Hg]" - MeasurementObservation: @@ -41,6 +45,10 @@ expr: 'uuid5("https://w3id.org/bdchm/MeasurementObservation", str({phv10111300}) + ":Example Study One:" + str({phv10111301}) + ":diastolic")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111300}) + ":Example Study One")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111300}) + ":Example Study One VISIT " + str({phv10111301}))' + age_at_observation: + populated_from: phv10111302 observation_type: value: OMOP:4154790 body_position: @@ -55,6 +63,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111300}) + ":Example Study One:" + str({phv10111301}) + ":diastolic")' value_decimal: - populated_from: phv10111306 + populated_from: phv10111307 unit: value: "mm[Hg]" diff --git a/synthetic/specs/example_study_one/bmi.yaml b/synthetic/specs/example_study_one/bmi.yaml index 3002d9a..332826c 100644 --- a/synthetic/specs/example_study_one/bmi.yaml +++ b/synthetic/specs/example_study_one/bmi.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111300}) + ":Example Study One")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111300}) + ":Example Study One VISIT " + str({phv10111301}))' + age_at_observation: + populated_from: phv10111302 observation_type: value: OMOP:3038553 value_quantity: @@ -18,6 +20,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111300}) + ":Example Study One:" + str({phv10111301}) + ":BMI")' value_decimal: - populated_from: phv10111304 + populated_from: phv10111305 unit: value: "kg/m2" diff --git a/synthetic/specs/example_study_one/bun.yaml b/synthetic/specs/example_study_one/bun.yaml index 74fb89a..723fe78 100644 --- a/synthetic/specs/example_study_one/bun.yaml +++ b/synthetic/specs/example_study_one/bun.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111400}) + ":Example Study One")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111400}) + ":Example Study One VISIT " + str({phv10111401}))' + age_at_observation: + populated_from: phv10111402 observation_type: value: OMOP:3013682 qualifier: @@ -20,6 +22,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111400}) + ":Example Study One:" + str({phv10111401}) + ":BUN")' value_decimal: - populated_from: phv10111404 + populated_from: phv10111405 unit: value: "mg/dL" diff --git a/synthetic/specs/example_study_one/cond_diabetes.yaml b/synthetic/specs/example_study_one/cond_diabetes.yaml new file mode 100644 index 0000000..1eccb17 --- /dev/null +++ b/synthetic/specs/example_study_one/cond_diabetes.yaml @@ -0,0 +1,24 @@ +- class_derivations: + Condition: + populated_from: pht000115 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10111500}) + ":Example Study One:diabetes")' + associated_participant: + expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111500}) + ":Example Study One")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111500}) + ":Example Study One VISIT " + str({phv10111522}))' + age_at_condition_start: + populated_from: phv10111520 + age_at_condition_end: + populated_from: phv10111521 + condition_concept: + populated_from: phv10111517 + condition_status: + populated_from: phv10111516 + condition_provenance: + value: PATIENT_SELF-REPORTED_CONDITION + relationship_to_participant: + value: ONESELF + associated_evidence: + value: "self-report questionnaire" diff --git a/synthetic/specs/example_study_one/cond_family_stroke.yaml b/synthetic/specs/example_study_one/cond_family_stroke.yaml index e9f0aea..a7515e2 100644 --- a/synthetic/specs/example_study_one/cond_family_stroke.yaml +++ b/synthetic/specs/example_study_one/cond_family_stroke.yaml @@ -6,11 +6,15 @@ expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10111500}) + ":Example Study One:family_stroke")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111500}) + ":Example Study One")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111500}) + ":Example Study One VISIT " + str({phv10111508}))' condition_concept: - populated_from: phv10111504 + populated_from: phv10111506 condition_status: - populated_from: phv10111503 + populated_from: phv10111505 condition_provenance: value: PATIENT_SELF-REPORTED_CONDITION relationship_to_participant: - populated_from: phv10111505 + populated_from: phv10111507 + associated_evidence: + value: "self-report questionnaire" diff --git a/synthetic/specs/example_study_one/cond_heart_attack.yaml b/synthetic/specs/example_study_one/cond_heart_attack.yaml index b9e4774..26562da 100644 --- a/synthetic/specs/example_study_one/cond_heart_attack.yaml +++ b/synthetic/specs/example_study_one/cond_heart_attack.yaml @@ -6,11 +6,17 @@ expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10111500}) + ":Example Study One:heart_attack")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111500}) + ":Example Study One")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111500}) + ":Example Study One VISIT " + str({phv10111527}))' + age_at_condition_start: + populated_from: phv10111526 condition_concept: - populated_from: phv10111508 + populated_from: phv10111524 condition_status: - populated_from: phv10111507 + populated_from: phv10111523 condition_provenance: value: PATIENT_SELF-REPORTED_CONDITION relationship_to_participant: value: ONESELF + associated_evidence: + expr: 'case(({phv10111525} == "STUDY_RECORD", "electrocardiogram"), ({phv10111525} == "SELF_REPORT", "self-report questionnaire"))' diff --git a/synthetic/specs/example_study_one/cond_heart_failure.yaml b/synthetic/specs/example_study_one/cond_heart_failure.yaml index 5bfeeb3..2693d26 100644 --- a/synthetic/specs/example_study_one/cond_heart_failure.yaml +++ b/synthetic/specs/example_study_one/cond_heart_failure.yaml @@ -6,6 +6,10 @@ expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10111500}) + ":Example Study One:heart_failure")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111500}) + ":Example Study One")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111500}) + ":Example Study One VISIT " + str({phv10111504}))' + age_at_condition_start: + populated_from: phv10111503 condition_concept: populated_from: phv10111502 condition_status: @@ -14,3 +18,5 @@ value: PATIENT_SELF-REPORTED_CONDITION relationship_to_participant: value: ONESELF + associated_evidence: + value: "self-report questionnaire" diff --git a/synthetic/specs/example_study_one/cond_hypertension.yaml b/synthetic/specs/example_study_one/cond_hypertension.yaml index 060428d..d163168 100644 --- a/synthetic/specs/example_study_one/cond_hypertension.yaml +++ b/synthetic/specs/example_study_one/cond_hypertension.yaml @@ -6,11 +6,19 @@ expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10111500}) + ":Example Study One:hypertension")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111500}) + ":Example Study One")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111500}) + ":Example Study One VISIT " + str({phv10111515}))' + age_at_condition_start: + populated_from: phv10111513 + age_at_condition_end: + populated_from: phv10111514 condition_concept: - value: HP:0000822 + populated_from: phv10111510 condition_status: - populated_from: phv10111506 + populated_from: phv10111509 condition_provenance: value: PATIENT_SELF-REPORTED_CONDITION relationship_to_participant: value: ONESELF + associated_evidence: + value: "self-report questionnaire" diff --git a/synthetic/specs/example_study_one/glucose.yaml b/synthetic/specs/example_study_one/glucose.yaml new file mode 100644 index 0000000..7b2da1e --- /dev/null +++ b/synthetic/specs/example_study_one/glucose.yaml @@ -0,0 +1,27 @@ +- class_derivations: + MeasurementObservation: + populated_from: pht000114 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/MeasurementObservation", str({phv10111400}) + ":Example Study One:" + str({phv10111401}) + ":GLUCOSE")' + associated_participant: + expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111400}) + ":Example Study One")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111400}) + ":Example Study One VISIT " + str({phv10111401}))' + age_at_observation: + populated_from: phv10111402 + observation_type: + value: OMOP:4156660 + method_type: + value: CALIBRATED + value_quantity: + class_derivations: + - Quantity: + populated_from: pht000114 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111400}) + ":Example Study One:" + str({phv10111401}) + ":GLUCOSE")' + value_decimal: + populated_from: phv10111407 + unit: + value: "mg/dL" diff --git a/synthetic/specs/example_study_one/hba1c.yaml b/synthetic/specs/example_study_one/hba1c.yaml new file mode 100644 index 0000000..6870c3b --- /dev/null +++ b/synthetic/specs/example_study_one/hba1c.yaml @@ -0,0 +1,27 @@ +- class_derivations: + MeasurementObservation: + populated_from: pht000114 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/MeasurementObservation", str({phv10111400}) + ":Example Study One:" + str({phv10111401}) + ":HBA1C")' + associated_participant: + expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111400}) + ":Example Study One")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111400}) + ":Example Study One VISIT " + str({phv10111401}))' + age_at_observation: + populated_from: phv10111402 + observation_type: + value: OMOP:4184637 + method_type: + value: "blood assay" + value_quantity: + class_derivations: + - Quantity: + populated_from: pht000114 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111400}) + ":Example Study One:" + str({phv10111401}) + ":HBA1C")' + value_decimal: + populated_from: phv10111408 + unit: + value: "%" diff --git a/synthetic/specs/example_study_one/hdl.yaml b/synthetic/specs/example_study_one/hdl.yaml index 7374b82..76a9e6b 100644 --- a/synthetic/specs/example_study_one/hdl.yaml +++ b/synthetic/specs/example_study_one/hdl.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111400}) + ":Example Study One")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111400}) + ":Example Study One VISIT " + str({phv10111401}))' + age_at_observation: + populated_from: phv10111402 observation_type: value: OMOP:3007070 associated_assay: @@ -49,8 +51,8 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111400}) + ":Example Study One:" + str({phv10111401}) + ":HDL")' value_decimal: - populated_from: phv10111402 - operator: populated_from: phv10111403 + operator: + populated_from: phv10111404 unit: value: "mg/dL" diff --git a/synthetic/specs/example_study_one/height.yaml b/synthetic/specs/example_study_one/height.yaml index f44ebb2..df299ab 100644 --- a/synthetic/specs/example_study_one/height.yaml +++ b/synthetic/specs/example_study_one/height.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111300}) + ":Example Study One")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111300}) + ":Example Study One VISIT " + str({phv10111301}))' + age_at_observation: + populated_from: phv10111302 observation_type: value: OMOP:3036277 value_quantity: @@ -18,6 +20,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111300}) + ":Example Study One:" + str({phv10111301}) + ":HEIGHT_CM")' value_decimal: - populated_from: phv10111302 + populated_from: phv10111303 unit: value: "cm" diff --git a/synthetic/specs/example_study_one/wbc.yaml b/synthetic/specs/example_study_one/wbc.yaml index ef3c46e..0188987 100644 --- a/synthetic/specs/example_study_one/wbc.yaml +++ b/synthetic/specs/example_study_one/wbc.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111400}) + ":Example Study One")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111400}) + ":Example Study One VISIT " + str({phv10111401}))' + age_at_observation: + populated_from: phv10111402 observation_type: value: OMOP:3000905 associated_assay: @@ -27,6 +29,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111400}) + ":Example Study One:" + str({phv10111401}) + ":WBC")' value_decimal: - populated_from: phv10111405 + populated_from: phv10111406 unit: value: "10*3/uL" diff --git a/synthetic/specs/example_study_one/weight.yaml b/synthetic/specs/example_study_one/weight.yaml index a5e78c4..9172e16 100644 --- a/synthetic/specs/example_study_one/weight.yaml +++ b/synthetic/specs/example_study_one/weight.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10111300}) + ":Example Study One")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10111300}) + ":Example Study One VISIT " + str({phv10111301}))' + age_at_observation: + populated_from: phv10111302 observation_type: value: OMOP:3025315 value_quantity: @@ -18,6 +20,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10111300}) + ":Example Study One:" + str({phv10111301}) + ":WEIGHT_KG")' value_decimal: - populated_from: phv10111303 + populated_from: phv10111304 unit: value: "kg" diff --git a/synthetic/specs/example_study_two/blood_pressure.yaml b/synthetic/specs/example_study_two/blood_pressure.yaml index 043fbff..127e130 100644 --- a/synthetic/specs/example_study_two/blood_pressure.yaml +++ b/synthetic/specs/example_study_two/blood_pressure.yaml @@ -17,6 +17,10 @@ expr: 'uuid5("https://w3id.org/bdchm/MeasurementObservation", str({phv10212300}) + ":Example Study Two:" + str({phv10212301}) + ":systolic")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212300}) + ":Example Study Two")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212300}) + ":Example Study Two VISIT " + str({phv10212301}))' + age_at_observation: + populated_from: phv10212302 observation_type: value: OMOP:4152194 body_position: @@ -31,7 +35,7 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212300}) + ":Example Study Two:" + str({phv10212301}) + ":systolic")' value_decimal: - populated_from: phv10212305 + populated_from: phv10212306 unit: value: "mm[Hg]" - MeasurementObservation: @@ -41,6 +45,10 @@ expr: 'uuid5("https://w3id.org/bdchm/MeasurementObservation", str({phv10212300}) + ":Example Study Two:" + str({phv10212301}) + ":diastolic")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212300}) + ":Example Study Two")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212300}) + ":Example Study Two VISIT " + str({phv10212301}))' + age_at_observation: + populated_from: phv10212302 observation_type: value: OMOP:4154790 body_position: @@ -55,6 +63,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212300}) + ":Example Study Two:" + str({phv10212301}) + ":diastolic")' value_decimal: - populated_from: phv10212306 + populated_from: phv10212307 unit: value: "mm[Hg]" diff --git a/synthetic/specs/example_study_two/bmi.yaml b/synthetic/specs/example_study_two/bmi.yaml index 97e28fb..341d3c6 100644 --- a/synthetic/specs/example_study_two/bmi.yaml +++ b/synthetic/specs/example_study_two/bmi.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212300}) + ":Example Study Two")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212300}) + ":Example Study Two VISIT " + str({phv10212301}))' + age_at_observation: + populated_from: phv10212302 observation_type: value: OMOP:3038553 value_quantity: @@ -18,4 +20,4 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212300}) + ":Example Study Two:" + str({phv10212301}) + ":BMI")' value_concept: - populated_from: phv10212304 + populated_from: phv10212305 diff --git a/synthetic/specs/example_study_two/bun.yaml b/synthetic/specs/example_study_two/bun.yaml index 4b26615..d728ecb 100644 --- a/synthetic/specs/example_study_two/bun.yaml +++ b/synthetic/specs/example_study_two/bun.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212400}) + ":Example Study Two")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212400}) + ":Example Study Two VISIT " + str({phv10212401}))' + age_at_observation: + populated_from: phv10212402 observation_type: value: OMOP:3013682 qualifier: @@ -20,6 +22,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212400}) + ":Example Study Two:" + str({phv10212401}) + ":BUN")' value_decimal: - populated_from: phv10212404 + populated_from: phv10212405 unit: value: "mg/dL" diff --git a/synthetic/specs/example_study_two/cond_diabetes.yaml b/synthetic/specs/example_study_two/cond_diabetes.yaml new file mode 100644 index 0000000..bd36226 --- /dev/null +++ b/synthetic/specs/example_study_two/cond_diabetes.yaml @@ -0,0 +1,24 @@ +- class_derivations: + Condition: + populated_from: pht000125 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10212500}) + ":Example Study Two:diabetes")' + associated_participant: + expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212500}) + ":Example Study Two")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212500}) + ":Example Study Two VISIT " + str({phv10212522}))' + age_at_condition_start: + populated_from: phv10212520 + age_at_condition_end: + populated_from: phv10212521 + condition_concept: + populated_from: phv10212517 + condition_status: + populated_from: phv10212516 + condition_provenance: + value: PATIENT_SELF-REPORTED_CONDITION + relationship_to_participant: + value: ONESELF + associated_evidence: + value: "self-report questionnaire" diff --git a/synthetic/specs/example_study_two/cond_family_stroke.yaml b/synthetic/specs/example_study_two/cond_family_stroke.yaml index c51161a..b92db66 100644 --- a/synthetic/specs/example_study_two/cond_family_stroke.yaml +++ b/synthetic/specs/example_study_two/cond_family_stroke.yaml @@ -6,11 +6,15 @@ expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10212500}) + ":Example Study Two:family_stroke")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212500}) + ":Example Study Two")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212500}) + ":Example Study Two VISIT " + str({phv10212508}))' condition_concept: - populated_from: phv10212504 + populated_from: phv10212506 condition_status: - populated_from: phv10212503 + populated_from: phv10212505 condition_provenance: value: PATIENT_SELF-REPORTED_CONDITION relationship_to_participant: - populated_from: phv10212505 + populated_from: phv10212507 + associated_evidence: + value: "self-report questionnaire" diff --git a/synthetic/specs/example_study_two/cond_heart_attack.yaml b/synthetic/specs/example_study_two/cond_heart_attack.yaml index e1d3965..2ae326f 100644 --- a/synthetic/specs/example_study_two/cond_heart_attack.yaml +++ b/synthetic/specs/example_study_two/cond_heart_attack.yaml @@ -6,11 +6,17 @@ expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10212500}) + ":Example Study Two:heart_attack")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212500}) + ":Example Study Two")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212500}) + ":Example Study Two VISIT " + str({phv10212527}))' + age_at_condition_start: + populated_from: phv10212526 condition_concept: - populated_from: phv10212508 + populated_from: phv10212524 condition_status: - populated_from: phv10212507 + populated_from: phv10212523 condition_provenance: value: PATIENT_SELF-REPORTED_CONDITION relationship_to_participant: value: ONESELF + associated_evidence: + expr: 'case(({phv10212525} == "STUDY_RECORD", "electrocardiogram"), ({phv10212525} == "SELF_REPORT", "self-report questionnaire"))' diff --git a/synthetic/specs/example_study_two/cond_heart_failure.yaml b/synthetic/specs/example_study_two/cond_heart_failure.yaml index 16afe07..16b29aa 100644 --- a/synthetic/specs/example_study_two/cond_heart_failure.yaml +++ b/synthetic/specs/example_study_two/cond_heart_failure.yaml @@ -6,6 +6,10 @@ expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10212500}) + ":Example Study Two:heart_failure")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212500}) + ":Example Study Two")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212500}) + ":Example Study Two VISIT " + str({phv10212504}))' + age_at_condition_start: + populated_from: phv10212503 condition_concept: populated_from: phv10212502 condition_status: @@ -14,3 +18,5 @@ value: PATIENT_SELF-REPORTED_CONDITION relationship_to_participant: value: ONESELF + associated_evidence: + value: "self-report questionnaire" diff --git a/synthetic/specs/example_study_two/cond_hypertension.yaml b/synthetic/specs/example_study_two/cond_hypertension.yaml index b980ae9..68ebde2 100644 --- a/synthetic/specs/example_study_two/cond_hypertension.yaml +++ b/synthetic/specs/example_study_two/cond_hypertension.yaml @@ -6,11 +6,19 @@ expr: 'uuid5("https://w3id.org/bdchm/Condition", str({phv10212500}) + ":Example Study Two:hypertension")' associated_participant: expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212500}) + ":Example Study Two")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212500}) + ":Example Study Two VISIT " + str({phv10212515}))' + age_at_condition_start: + populated_from: phv10212513 + age_at_condition_end: + populated_from: phv10212514 condition_concept: - value: HP:0000822 + populated_from: phv10212510 condition_status: - populated_from: phv10212506 + populated_from: phv10212509 condition_provenance: value: PATIENT_SELF-REPORTED_CONDITION relationship_to_participant: value: ONESELF + associated_evidence: + value: "self-report questionnaire" diff --git a/synthetic/specs/example_study_two/glucose.yaml b/synthetic/specs/example_study_two/glucose.yaml new file mode 100644 index 0000000..9fdde6b --- /dev/null +++ b/synthetic/specs/example_study_two/glucose.yaml @@ -0,0 +1,27 @@ +- class_derivations: + MeasurementObservation: + populated_from: pht000124 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/MeasurementObservation", str({phv10212400}) + ":Example Study Two:" + str({phv10212401}) + ":GLUCOSE")' + associated_participant: + expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212400}) + ":Example Study Two")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212400}) + ":Example Study Two VISIT " + str({phv10212401}))' + age_at_observation: + populated_from: phv10212402 + observation_type: + value: OMOP:4156660 + method_type: + value: CALIBRATED + value_quantity: + class_derivations: + - Quantity: + populated_from: pht000124 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212400}) + ":Example Study Two:" + str({phv10212401}) + ":GLUCOSE")' + value_decimal: + populated_from: phv10212407 + unit: + value: "mg/dL" diff --git a/synthetic/specs/example_study_two/hba1c.yaml b/synthetic/specs/example_study_two/hba1c.yaml new file mode 100644 index 0000000..9afc2ba --- /dev/null +++ b/synthetic/specs/example_study_two/hba1c.yaml @@ -0,0 +1,27 @@ +- class_derivations: + MeasurementObservation: + populated_from: pht000124 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/MeasurementObservation", str({phv10212400}) + ":Example Study Two:" + str({phv10212401}) + ":HBA1C")' + associated_participant: + expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212400}) + ":Example Study Two")' + associated_visit: + expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212400}) + ":Example Study Two VISIT " + str({phv10212401}))' + age_at_observation: + populated_from: phv10212402 + observation_type: + value: OMOP:4184637 + method_type: + value: "blood assay" + value_quantity: + class_derivations: + - Quantity: + populated_from: pht000124 + slot_derivations: + id: + expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212400}) + ":Example Study Two:" + str({phv10212401}) + ":HBA1C")' + value_decimal: + populated_from: phv10212408 + unit: + value: "%" diff --git a/synthetic/specs/example_study_two/hdl.yaml b/synthetic/specs/example_study_two/hdl.yaml index cccf6c0..00d0a59 100644 --- a/synthetic/specs/example_study_two/hdl.yaml +++ b/synthetic/specs/example_study_two/hdl.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212400}) + ":Example Study Two")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212400}) + ":Example Study Two VISIT " + str({phv10212401}))' + age_at_observation: + populated_from: phv10212402 observation_type: value: OMOP:3007070 associated_assay: @@ -49,8 +51,8 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212400}) + ":Example Study Two:" + str({phv10212401}) + ":HDL")' value_decimal: - populated_from: phv10212402 - operator: populated_from: phv10212403 + operator: + populated_from: phv10212404 unit: value: "mg/dL" diff --git a/synthetic/specs/example_study_two/height.yaml b/synthetic/specs/example_study_two/height.yaml index fdca786..4657e1f 100644 --- a/synthetic/specs/example_study_two/height.yaml +++ b/synthetic/specs/example_study_two/height.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212300}) + ":Example Study Two")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212300}) + ":Example Study Two VISIT " + str({phv10212301}))' + age_at_observation: + populated_from: phv10212302 observation_type: value: OMOP:3036277 value_quantity: @@ -18,6 +20,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212300}) + ":Example Study Two:" + str({phv10212301}) + ":HEIGHT_CM")' value_decimal: - populated_from: phv10212302 + populated_from: phv10212303 unit: value: "cm" diff --git a/synthetic/specs/example_study_two/wbc.yaml b/synthetic/specs/example_study_two/wbc.yaml index 6f34c7a..1295b42 100644 --- a/synthetic/specs/example_study_two/wbc.yaml +++ b/synthetic/specs/example_study_two/wbc.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212400}) + ":Example Study Two")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212400}) + ":Example Study Two VISIT " + str({phv10212401}))' + age_at_observation: + populated_from: phv10212402 observation_type: value: OMOP:3000905 associated_assay: @@ -27,6 +29,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212400}) + ":Example Study Two:" + str({phv10212401}) + ":WBC")' value_decimal: - populated_from: phv10212405 + populated_from: phv10212406 unit: value: "10*3/uL" diff --git a/synthetic/specs/example_study_two/weight.yaml b/synthetic/specs/example_study_two/weight.yaml index 930ac0d..c228bb6 100644 --- a/synthetic/specs/example_study_two/weight.yaml +++ b/synthetic/specs/example_study_two/weight.yaml @@ -8,6 +8,8 @@ expr: 'uuid5("https://w3id.org/bdchm/Participant", str({phv10212300}) + ":Example Study Two")' associated_visit: expr: 'uuid5("https://w3id.org/bdchm/Visit", str({phv10212300}) + ":Example Study Two VISIT " + str({phv10212301}))' + age_at_observation: + populated_from: phv10212302 observation_type: value: OMOP:3025315 value_quantity: @@ -18,6 +20,6 @@ id: expr: 'uuid5("https://w3id.org/bdchm/Quantity", str({phv10212300}) + ":Example Study Two:" + str({phv10212301}) + ":WEIGHT_KG")' value_decimal: - populated_from: phv10212303 + populated_from: phv10212304 unit: value: "kg" diff --git a/synthetic/validate.py b/synthetic/validate.py index 42c9a37..116a356 100644 --- a/synthetic/validate.py +++ b/synthetic/validate.py @@ -36,12 +36,125 @@ def exact(label, actual, expected): failures.append(label) +def at_least(label, actual, floor): + """Record a coverage check: enough distinct values to be worth querying.""" + ok = actual >= floor + notes.append(f"{'ok ' if ok else 'FAIL'} {label}: {actual} (at least {floor})") + if not ok: + failures.append(label) + + def fraction(items, predicate): """Return the proportion of items satisfying the predicate.""" items = list(items) return sum(1 for x in items if predicate(x)) / len(items) if items else 0.0 +def diagnosed(entry): + """Whether a condition entry represents a diagnosis rather than a screen.""" + return entry["status"] in (v.PRESENT, v.HISTORICAL) + + +def check_temporal(participants): + """ + Every temporal anchor must be usable, not merely present. + + A start age nobody could have reached, an end before its start, or a + recording visit the participant never attended would each pass a + completeness check while making the slot useless for a cohort query. + """ + bad_order = bad_floor = bad_visit = bad_absent = orphan_end = 0 + + for p in participants: + ages = {x.number: x.age_days for x in p.visits} + for entry in p.conditions.values(): + start, end, visit = entry["age_start"], entry["age_end"], entry["visit"] + + if not diagnosed(entry) and (start is not None or end is not None): + bad_absent += 1 + if start is not None and start < pop.MIN_DIAGNOSIS_AGE_DAYS: + bad_floor += 1 + if end is not None: + if start is None: + orphan_end += 1 + elif end <= start: + bad_order += 1 + # A condition recorded at a visit must name one the participant + # attended, and cannot be diagnosed after that visit took place. + if visit is not None: + if visit not in ages: + bad_visit += 1 + elif start is not None and start > ages[visit]: + bad_visit += 1 + + exact("conditions dated while absent", bad_absent, 0) + exact("diagnoses before the minimum age", bad_floor, 0) + exact("conditions ending before they started", bad_order, 0) + exact("conditions ending without a start", orphan_end, 0) + exact("conditions recorded at an unattended or earlier visit", bad_visit, 0) + + +def check_before_and_after(participants): + """ + Check the point of the temporal metadata: measurements sort by diagnosis. + + A condition only elevates the measures it explains once it has been + diagnosed, so for participants diagnosed mid-study the glucose values + before the diagnosis should look like the corpus as a whole and the ones + after should look like the diabetic subgroup. If these two converge, the + temporal slots are decoration. + """ + before, after = [], [] + for p in participants: + entry = p.conditions["diabetes"] + start = entry["age_start"] + if entry["status"] != v.PRESENT or start is None or not p.visits: + continue + if start <= p.visits[0].age_days: + continue # prevalent: nothing falls before the diagnosis + for visit in p.visits: + if visit.glucose is None: + continue + (before if visit.age_days < start else after).append(visit.glucose) + + # The pathological glucose range starts at 126, so 'pathological draw' and + # 'at or above 126' are the same set. + exact("incident diabetes with glucose on both sides", len(before) > 0 and len(after) > 0, True) + if before and after: + # Both samples are small — a few dozen values each — so these carry a + # wider tolerance than the corpus-wide distribution checks. + near("glucose pathological before diagnosis", + fraction(before, lambda x: x >= 126), pop.PATHOLOGY_RATE, tolerance=0.10) + near("glucose pathological after diagnosis", + fraction(after, lambda x: x >= 126), pop.COMORBID_PATHOLOGY_RATE, tolerance=0.18) + + +def check_code_coverage(participants): + """ + Enough distinct concepts and source codes to test a broad cohort definition. + + Floors rather than exact counts: the rarest subtypes carry weights low + enough that whether they appear at all is a coin flip, and a corpus that + failed because one 0.5% subtype went unsampled would be failing on noise. + """ + htn = [p.conditions["hypertension"] for p in participants] + dm = [p.conditions["diabetes"] for p in participants] + + at_least("distinct hypertension concepts", + len({e["concept"] for e in htn if diagnosed(e)}), 10) + at_least("distinct hypertension ICD-10-CM codes", + len({e["icd10"] for e in htn if e["icd10"]}), 8) + at_least("distinct diabetes subtypes", + len({e["subtype"] for e in dm if diagnosed(e)}), 8) + at_least("distinct diabetes ICD-10-CM codes", + len({e["icd10"] for e in dm if e["icd10"]}), 18) + + # Source codes belong to diagnoses. A screening answer of 'no' records the + # concept that was screened for and nothing else. + stray = sum(1 for e in htn + dm if not diagnosed(e) and (e["icd10"] or e["snomed"])) + exact("source codes on undiagnosed conditions", stray, 0) + + def main(): """Run every check and report.""" people, participants = pop.build() @@ -97,6 +210,23 @@ def main(): near(f"{label}: heart attack PRESENT", fraction(group, lambda p: p.conditions["heart_attack"]["status"] == v.PRESENT), 0.10) + # Diabetes is drawn as 20% diagnosed and 2% unknown; the diagnosed split + # into PRESENT and the HISTORICAL 'in remission' subtype, so check the + # two together rather than pinning a share to one subtype's weight. + near(f"{label}: diabetes diagnosed", + fraction(group, lambda p: diagnosed(p.conditions["diabetes"])), 0.20) + near(f"{label}: diabetes UNKNOWN", + fraction(group, lambda p: p.conditions["diabetes"]["status"] == v.UNKNOWN), 0.02) + + # Incident diagnoses are what make a before-and-after query meaningful. + dx = [e for p in group for e in p.conditions.values() if e["age_start"] is not None] + incident = [ + e for p in group for e in p.conditions.values() + if e["age_start"] is not None and p.visits and e["age_start"] > p.visits[0].age_days + ] + near(f"{label}: diagnoses made during follow-up", + len(incident) / len(dx) if dx else 0.0, pop.INCIDENT_FRACTION, tolerance=0.07) + mi = [p for p in group if p.conditions["heart_attack"]["status"] == v.PRESENT] near(f"{label}: MI from study record", fraction(mi, lambda p: p.conditions["heart_attack"]["from_study_record"]), 0.50, tolerance=0.15) @@ -122,7 +252,8 @@ def main(): getattr(x, field) for p in participants for x in p.visits - for field in ("height_cm", "weight_kg", "systolic", "diastolic", "hdl", "bun", "wbc") + for field in ("height_cm", "weight_kg", "systolic", "diastolic", "hdl", + "bun", "wbc", "glucose", "hba1c") ] near("null rate", sum(1 for c in cells if c is None) / len(cells), pop.NULL_RATE, tolerance=0.002) @@ -139,6 +270,10 @@ def main(): for concept in v.CALCIUM_CHANNEL_BLOCKERS: near(f"CCB share {concept}", ccb[concept] / total, 0.25, tolerance=0.09) + check_temporal(participants) + check_before_and_after(participants) + check_code_coverage(participants) + print("\n".join(notes)) print() if failures: diff --git a/synthetic/vocab.py b/synthetic/vocab.py index e67b744..619ed8a 100644 --- a/synthetic/vocab.py +++ b/synthetic/vocab.py @@ -2,11 +2,18 @@ """ Coded values used by the synthetic corpus. -Every CURIE here is attested in RTI's priority_variables_transform specs or in -BDCHM itself, so the corpus resolves against the same vocabulary as real -harmonized data. Nothing in this file is invented. +Every CURIE here is attested in one of three places, so the corpus resolves +against the same vocabulary as real harmonized data. Nothing in this file is +invented: + + 1. RTI's priority_variables_transform specs + 2. BDCHM itself + 3. The BDC cohort-readiness code-set reference, for the hypertension and + Type 2 diabetes subtypes (see CODE_SET_REFERENCE below) """ +from dataclasses import dataclass + # Demography — BDCHM RaceEnum / EthnicityEnum / SexEnum meanings MALE = "OMOP:8507" FEMALE = "OMOP:8532" @@ -43,7 +50,6 @@ "MONDO:0011057", "MONDO:0013792", ] -HYPERTENSION = "HP:0000822" HEART_ATTACK = ["MONDO:0005068", "MONDO:0006803"] # Observation types — from blood_pressure.yaml and the lab specs @@ -82,12 +88,28 @@ # even though it is required; a reference corpus should populate it. VISIT_PROVENANCE = "CASE_REPORT_FORM" +# Glycemic measures. Unlike HDL, BUN and WBC — whose OMOP codes come from RTI's +# specs and are not in the model — these two are permissible values of BDCHM's +# MeasurementObservationTypeEnum (FAST_GLUC_BLD, HEMO_A1C) *and* are what +# MESA-ingest/fast_gluc_bld.yaml and hemo_a1c.yaml emit. Units and method_type +# are taken from those same specs. +FASTING_GLUCOSE = "OMOP:4156660" +HBA1C = "OMOP:4184637" +GLUCOSE_METHOD = "CALIBRATED" +HBA1C_METHOD = "blood assay" + # Assay methods — BDCHM AssayMethodEnum resolves through MMO. No MMO term is # attested in the real specs, so these were looked up in MMO directly rather # than invented. ASSAY_METHOD_HDL = "MMO:0000133" # serum high-density lipoprotein-cholesterol measurement test ASSAY_METHOD_WBC = "MMO:0000533" # white blood cell counting method +# Evidence supporting a condition assertion. BDCHM ranges associated_evidence on +# Entity, but MESA-ingest puts a bare descriptive string there, so this is what +# real harmonized data actually carries. +SELF_REPORT_EVIDENCE = "self-report questionnaire" +ECG_EVIDENCE = "electrocardiogram" + # Relationship of a condition to the participant. ONESELF has no OMOP meaning # in BDCHM — it is a bare permissible value. ONESELF = "ONESELF" @@ -96,3 +118,126 @@ DRUG_EXPOSURE_NS = "https://w3id.org/bdchm/DrugExposure" OBSERVATION_SET_NS = "https://w3id.org/bdchm/MeasurementObservationSet" CAUSE_OF_DEATH_NS = "https://w3id.org/bdchm/CauseOfDeath" + + +# --- Hypertension and Type 2 diabetes subtypes ------------------------------ +# +# Source: the BDC cohort-readiness code-set reference, which lists MONDO, HPO, +# SNOMED CT and ICD-10-CM for each subtype. +CODE_SET_REFERENCE = ( + "https://docs.google.com/spreadsheets/d/1AciqkeEaQfA9IptDpUI9yQYYzN7PWnt7" +) + + +@dataclass(frozen=True) +class Subtype: + """ + One row of the code-set reference. + + `concept` is what BDCHM's condition_concept takes — MONDO where the + reference gives one, HPO otherwise, since ConditionConceptEnum is the union + of the two. `snomed` and `icd10` have no slot anywhere in BDCHM, so they + live only in the raw dbGaP-style tables, which is where source terminology + sits in real data too. + + Several reference cells name a hierarchy rather than a term ("MONDO:0005148 + hierarchy", "44054006 hierarchy + renal complication descendants") or a + wildcard rather than a code ("E11.3*"). Those are recorded as None or as an + empty tuple rather than guessed at: a subtype with no concrete source code + simply emits none, which is a shape real data has as well. + """ + + label: str + concept: str + hpo: str | None + snomed: str | None + icd10: tuple[str, ...] + weight: float + + +# Weights are illustrative. The ordering is real — essential hypertension +# dominates any hypertensive population — but the numbers are not drawn from any +# prevalence study and must not be read as a finding. +# +# Pregnancy-related hypertension (MONDO:0005081) is in the reference but is left +# out here: both cohorts enrol at 45-78, so no participant is plausibly +# pregnancy-hypertensive during follow-up. +HYPERTENSION_SUBTYPES = ( + Subtype("essential (primary) hypertension", + "MONDO:0001134", "HP:0000822", "59621000", ("I10",), 0.56), + Subtype("systemic hypertensive disorder, unspecified", + "MONDO:0005044", "HP:0000822", "38341003", ("I10",), 0.15), + Subtype("hypertensive heart disease", + "MONDO:0001302", "HP:0000822", "64715009", ("I11.0", "I11.9"), 0.07), + Subtype("secondary hypertension", + "MONDO:0001200", "HP:0000822", "31992008", + ("I15.0", "I15.1", "I15.2", "I15.8", "I15.9"), 0.05), + Subtype("resistant hypertension", + "MONDO:0100078", None, "461301000124109", ("I1A.0",), 0.04), + Subtype("renovascular hypertension", + "MONDO:0006947", "HP:0100817", "123799005", ("I15.0",), 0.03), + Subtype("renal hypertension", + "MONDO:0001105", "HP:0000822", "28119000", ("I15.0", "I15.1"), 0.03), + Subtype("pulmonary arterial hypertension", + "MONDO:0015924", "HP:0002092", "11399002", (), 0.02), + Subtype("hypertensive urgency", + "MONDO:1030007", "HP:0100735", "443482000", ("I16.0",), 0.015), + Subtype("malignant hypertension", + "MONDO:0006846", "HP:0100735", "70272006", (), 0.01), + # The reference gives no single MONDO term for hypertensive crisis, only a + # pointer into the hierarchy — so this one is HPO-coded, which BDCHM allows. + Subtype("hypertensive crisis", + "HP:0100735", "HP:0100735", "706882009", ("I16.0", "I16.1", "I16.9"), 0.01), + Subtype("portal hypertension", + "MONDO:0005080", "HP:0001409", "34742003", ("K76.6",), 0.01), + Subtype("hypertensive encephalopathy", + "MONDO:0006796", None, "50490005", (), 0.005), +) + +# The screening concept for participants who answered no: a questionnaire asks +# about hypertension in general, and a subtype is only recorded once diagnosed. +HYPERTENSION_ROOT = HYPERTENSION_SUBTYPES[1] + +# Type 2 diabetes. The reference gives MONDO:0005148 for nearly every row and +# puts the complication-level detail in ICD-10-CM, so that is where the +# granularity in this corpus lives too. +# +# T2D with ophthalmic complications is in the reference but omitted here: its +# ICD-10-CM cell is the bare wildcard "E11.3*" and its MONDO is the shared +# root, so it would be indistinguishable from every other T2D record. +DIABETES_SUBTYPES = ( + Subtype("Type 2 diabetes without complications", + "MONDO:0005148", "HP:0005978", "44054006", ("E11.9",), 0.37), + Subtype("Type 2 diabetes with kidney complications", + "MONDO:0005148", "HP:0005978", None, + ("E11.21", "E11.22", "E11.29"), 0.11), + Subtype("Type 2 diabetes with neurologic complications", + "MONDO:0005148", "HP:0005978", "421326000", + ("E11.40", "E11.41", "E11.42", "E11.43", "E11.44", "E11.49"), 0.11), + Subtype("Type 2 diabetes with other specified complications", + "MONDO:0005148", "HP:0005978", None, + ("E11.610", "E11.618", "E11.620", "E11.621", "E11.622", "E11.628", + "E11.630", "E11.638", "E11.641", "E11.649", "E11.65", "E11.69"), 0.09), + # In remission is the one subtype that resolves, and is what gives the + # corpus its Conditions with an age_at_condition_end. + Subtype("Type 2 diabetes in remission", + "MONDO:0005148", "HP:0005978", None, ("E11.A",), 0.08), + Subtype("Type 2 diabetes with unspecified complications", + "MONDO:0005148", "HP:0005978", None, ("E11.8",), 0.07), + Subtype("Type 2 diabetes with circulatory complications", + "MONDO:0005148", "HP:0005978", None, + ("E11.51", "E11.52", "E11.59"), 0.07), + # MONDO:0005015 is what MESA-ingest/diabetes.yaml actually emits, so a + # portal built on this corpus meets the same broad code real data carries. + Subtype("diabetes mellitus, type unspecified", + "MONDO:0005015", "HP:0000819", "73211009", (), 0.05), + Subtype("Type 2 diabetes with hyperosmolarity", + "MONDO:0005148", "HP:0005978", None, ("E11.00", "E11.01"), 0.02), + Subtype("Type 2 diabetes with ketoacidosis", + "MONDO:0005148", "HP:0005978", None, ("E11.10", "E11.11"), 0.02), + Subtype("lipoatrophic diabetes", + "MONDO:0005827", "HP:0005978", "127012008", (), 0.01), +) + +DIABETES_ROOT = DIABETES_SUBTYPES[0] +DIABETES_REMISSION = DIABETES_SUBTYPES[4]