data-pipelines: DAG shown at both levels — collapsed modules and expanded tables

src/explanation/data-pipelines.md

@@ -24,18 +24,29 @@ These components work together: code defines the schema and computations, the da
 
 ## Pipeline as a DAG
 
-A DataJoint pipeline forms a **Directed Acyclic Graph (DAG)** at two levels:
+A DataJoint pipeline forms a **Directed Acyclic Graph (DAG)** at two levels, and the same
+pipeline can be viewed at either one.
 
-![Pipeline DAG Structure](../images/pipeline-modules.svg)
+**Module level — the collapsed view.** Each node is a Python module, which corresponds to a
+database schema. Each edge represents a bundle of dependencies: the foreign key references
+between tables of the two schemas, together with the Python import dependency between their
+modules. A heavier edge carries a larger bundle — below, the `lab → session` and
+`session → imaging` edges each bundle two separate foreign keys, and `reference` supplies
+both `session` and `imaging`. Fourteen tables (including parts) collapse to four nodes.
 
-**Nodes** represent Python modules, which correspond to database schemas — the dashed clusters in the diagram above.
+![Pipeline DAG at the module level](../images/pipeline-modules-collapsed.svg)
 
-**Edges** represent:
+**Table level — the expanded view.** Each node is a table; the dashed clusters group the
+tables of each module. Each edge is an individual foreign key constraint — the two foreign
+keys that cross the `session → imaging` boundary (`Scan → ScanQuality` and
+`ScanInfo → MotionCorrection`) appear separately here, having collapsed into the single
+bundled edge above; likewise `Subject → Session` and `User → Session` across
+`lab → session`.
 
-- Python import dependencies between modules
-- Bundles of foreign key references between schemas
+![Pipeline DAG at the table level](../images/pipeline-modules.svg)
 
-This dual structure ensures that both code dependencies and data dependencies flow in the same direction.
+This dual structure ensures that both code dependencies and data dependencies flow in the
+same direction: collapsing every module's tables into a single node must itself yield a DAG.
 
 ### DAG Constraints
 
@@ -63,23 +74,23 @@ This model treats the database schema as an **executable workflow specification*
 
 ## Schema Organization
 
-Each schema corresponds to a dedicated Python module. The module import structure mirrors the foreign key dependencies between schemas:
-
-Within a schema, tables of different tiers form their own DAG — here, a `scan` schema with lookup, manual, and imported tables, including a master table with its part tables:
-
-![Schema Structure](../images/scan-schema.svg)
+Each schema corresponds to a dedicated Python module, and the module import structure mirrors the foreign key dependencies between schemas. The pipeline above is organized as:
 
 ```
 my_pipeline/
 ├── src/
 │   └── my_pipeline/
 │       ├── __init__.py
-│       ├── subject.py      # subject schema (no dependencies)
-│       ├── session.py      # session schema (depends on subject)
-│       ├── acquisition.py  # acquisition schema (depends on session)
-│       └── analysis.py     # analysis schema (depends on acquisition)
+│       ├── lab.py        # lab schema (no dependencies)
+│       ├── reference.py  # reference schema (no dependencies)
+│       ├── session.py    # session schema (imports lab, reference)
+│       └── imaging.py    # imaging schema (imports session, reference)
 ```
 
+Within a schema, tables form their own DAG. Drilling into the `imaging` module: computed tables, including two masters with their part tables — `Segmentation` with `Segmentation.Roi`, and `Fluorescence` with `Fluorescence.Trace`, whose rows reference individual ROIs:
+
+![The imaging module expanded](../images/imaging-schema.svg)
+
 For practical guidance on organizing multi-schema pipelines, configuring repositories, and managing team access, see [Manage a Pipeline Project](../how-to/manage-pipeline-project.md/).
 
 ## Object-Augmented Schemas
@@ -119,26 +130,30 @@ When a database row is deleted, its associated stored objects are garbage-collec
 Different attributes can route to different stores:
 
 ```python
-class Recording(dj.Imported):
+class ScanInfo(dj.Imported):
     definition = """
-    -> Session
+    -> Scan
     ---
-    raw_data : <blob@fast>       # Hot storage for active analysis
+    nframes : int32
+    fps : float32
+    raw_movie : <blob@fast>      # Hot storage for active analysis
     archive : <blob@cold>        # Cold storage for long-term retention
     """
 ```
 
+Here the pipeline's `ScanInfo` table keeps its scalar metadata (`nframes`, `fps`) in the database while routing its payloads to two different stores.
+
 This architecture lets teams work with terabyte-scale datasets while retaining the query power, integrity guarantees, and reproducibility of the relational model.
 
 ## Pipeline Workflow
 
 A typical data pipeline workflow:
 
-1. **Acquisition** — Data is collected from instruments, experiments, or external sources. Raw files land in object storage; metadata populates Manual tables.
+1. **Acquisition** — Data is collected from instruments, experiments, or external sources. Raw files land in object storage; metadata populates Manual tables (`Session`, `Scan`).
 
-2. **Import** — Automated processes parse raw data, extract signals, and populate Imported tables with structured results.
+2. **Import** — Automated processes parse raw data, extract signals, and populate Imported tables with structured results (`ScanInfo`).
 
-3. **Computation** — The `populate()` mechanism identifies new data and triggers downstream processing. Compute resources execute transformations and populate Computed tables.
+3. **Computation** — The `populate()` mechanism identifies new data and triggers downstream processing. Compute resources execute transformations and populate Computed tables (`MotionCorrection`, `Segmentation`, `Fluorescence`).
 
 4. **Query & Analysis** — Users query results across the pipeline, combining data from multiple stages to generate insights, reports, or visualizations.
 

src/explanation/relational-workflow-model.md

@@ -13,10 +13,10 @@ the lineage of relational modeling follows.
 ## A worked example
 
 Diagrams in this documentation use the same notation as `dj.Diagram` in
-`datajoint-python`: **Manual** tables are green rectangles, **Lookup**
-tables are gray rectangles, **Imported** tables are blue ovals, and
-**Computed** tables are red ovals. A **Part** table is a plain rectangle
-grouped with its master inside a light box. Tier is conveyed by shape and
+`datajoint-python`: **Manual** tables are green rounded boxes, **Lookup**
+tables are gray rounded boxes, **Imported** tables are blue ellipses, and
+**Computed** tables are orange ellipses. A **Part** table is a small plain box
+grouped with its master inside a light box (the *entity cluster*). Tier is conveyed by shape and
 color, and **edge thickness** shows how a child relates to its parent — a
 thick line means the child **extends** the parent (one per parent); a thin
 line means the child is **contained within** the parent (many per parent).
@@ -41,7 +41,7 @@ dimension. The legend below the figure keys the full notation.
 
 ![Worked-example imaging pipeline diagram spanning two schemas: experiment (Mouse → Session → Scan) and analysis (AverageFrame → Segmentation → Fluorescence, with Lookup SegmentationParam feeding Segmentation, and the Part tables Roi on Segmentation and Trace on Fluorescence).](../images/rwm-pipeline.svg)
 
-![Legend: table tiers — Manual (green rectangle), Lookup (gray rectangle), Imported (blue oval), Computed (red oval), Part (smaller plain rectangle); an underlined name is a new entity type (a new schema dimension, many rows per parent) while a plain name is composed from existing entities (one row per parent); edge thickness — thick means the child extends the parent, thin means the child is contained within the parent; a dashed rounded box is a schema module (labeled in the corner); a gray box encloses a master with its parts; edges have no arrowheads, so direction follows the layout.](../images/rwm-legend.svg)
+![Legend: table tiers — Manual (green rounded box), Lookup (gray rounded box), Imported (blue ellipse), Computed (orange ellipse), Part (smaller plain box); an underlined name is a new entity type (a new schema dimension, many rows per parent) while a plain name is composed from existing entities (one row per parent); edge thickness — thick means the child extends the parent, thin means the child is contained within the parent; a dashed rounded box is a schema module (labeled in the corner); a gray box encloses a master with its parts; edges have no arrowheads, so direction follows the layout.](../images/rwm-legend.svg)
 
 The notation is specified in full in the [Diagram specification](../reference/specs/diagram.md). The concepts it depicts are explained in depth elsewhere: [entity integrity](entity-integrity.md) (keys, entity types, and schema dimensions), [master–part tables](../reference/specs/master-part.md) (the entity group and its all-or-nothing populate), the [computation model](computation-model.md) (how `make()` produces Imported and Computed tables), and [semantic matching](semantic-matching.md) (why a name means the same thing everywhere it appears).
 

src/how-to/master-part.ipynb

@@ -102,30 +102,30 @@
        "  [fill="#fffde7"] { fill: #3A3620; }\n",
        "  [stroke="#c9bc5b"] { stroke: #C9BC5B; }\n",
        "  [fill="#6b6420"] { fill: #EBE3A0; }\n",
-       "  [fill="#e7f3ec"] { fill: #16281F; }\n",
-       "  [stroke="#2f7d5b"] { stroke: #4FA97F; }\n",
-       "  [fill="#1b5138"] { fill: #BCE6CF; }\n",
-       "  [fill="#f2f4f7"] { fill: #242832; }\n",
-       "  [stroke="#a9b1bd"] { stroke: #8A93A1; }\n",
-       "  [fill="#495261"] { fill: #C9CFD9; }\n",
-       "  [fill="#fbeaec"] { fill: #331A1F; }\n",
-       "  [stroke="#b23a48"] { stroke: #D0687A; }\n",
-       "  [fill="#7c2430"] { fill: #F3C2CB; }\n",
-       "  [fill="#e2ecfa"] { fill: #152538; }\n",
-       "  [stroke="#2a5fa5"] { stroke: #5E92D6; }\n",
-       "  [fill="#123a6d"] { fill: #C3DAF6; }\n",
+       "  [fill="#E8F0E9"] { fill: #16281F; }\n",
+       "  [stroke="#3E7A52"] { stroke: #6BBF94; }\n",
+       "  [fill="#28513A"] { fill: #BCE6CF; }\n",
+       "  [fill="#F0F0F1"] { fill: #242832; }\n",
+       "  [stroke="#808285"] { stroke: #9DA0A4; }\n",
+       "  [fill="#5A5C5F"] { fill: #C9CFD9; }\n",
+       "  [fill="#FFEDE5"] { fill: #331B12; }\n",
+       "  [stroke="#FF5113"] { stroke: #FF7A47; }\n",
+       "  [fill="#B23200"] { fill: #FFC7AE; }\n",
+       "  [fill="#E0F4FC"] { fill: #0F2433; }\n",
+       "  [stroke="#00A0DF"] { stroke: #33B8E8; }\n",
+       "  [fill="#00537A"] { fill: #BEE7F9; }\n",
        "  [fill="#ffffff"] { fill: #1E232C; }\n",
-       "  [stroke="#9aa6b8"] { stroke: #7B879B; }\n",
-       "  [fill="#46536b"] { fill: #C4CCDB; }\n",
+       "  [stroke="#B9BBBE"] { stroke: #8F959D; }\n",
+       "  [fill="#55585C"] { fill: #C8CED8; }\n",
        "  [fill="#edeef0"] { fill: #242730; }\n",
        "  [stroke="#808890"] { stroke: #8890A0; }\n",
        "  [fill="#404040"] { fill: #C7CDD6; }\n",
-       "  [stroke="#3a424f"] { stroke: #AEB6C2; stroke-opacity: 0.753; }\n",
+       "  [stroke="#171C39"] { stroke: #AEB6C2; stroke-opacity: 0.753; }\n",
        "  [stroke="#c77d3a"] { stroke: #D68C4A; stroke-opacity: 0.753; }\n",
        "  [fill="#f3f5f8"] { fill: #2A313D; }\n",
        "  [stroke="#f3f5f8"] { stroke: #2A313D; }\n",
        "  [stroke="gray"] { stroke: #606875; }\n",
-       "  [fill="gray"] { fill: #8A93A1; }\n",
+       "  [fill="gray"] { fill: #9DA0A4; }\n",
        "}\n",
        "</style>\n",
        "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 152)\">\n",
@@ -147,8 +147,8 @@
        "------------------------------\r",
        "session_date         \r",
        "\">\n",
-       "<path fill=\"#e7f3ec\" stroke=\"#2f7d5b\" d=\"M150.91,-58.28C150.91,-58.28 118.36,-58.28 118.36,-58.28 112.6,-58.28 106.84,-52.52 106.84,-46.76 106.84,-46.76 106.84,-35.24 106.84,-35.24 106.84,-29.48 112.6,-23.72 118.36,-23.72 118.36,-23.72 150.91,-23.72 150.91,-23.72 156.67,-23.72 162.43,-29.48 162.43,-35.24 162.43,-35.24 162.43,-46.76 162.43,-46.76 162.43,-52.52 156.67,-58.28 150.91,-58.28\"/>\n",
-       "<text text-anchor=\"start\" x=\"114.76\" y=\"-38.1\" font-family=\"Helvetica,sans-Serif\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"#1b5138\">Session</text>\n",
+       "<path fill=\"#E8F0E9\" stroke=\"#3E7A52\" d=\"M150.91,-58.28C150.91,-58.28 118.36,-58.28 118.36,-58.28 112.6,-58.28 106.84,-52.52 106.84,-46.76 106.84,-46.76 106.84,-35.24 106.84,-35.24 106.84,-29.48 112.6,-23.72 118.36,-23.72 118.36,-23.72 150.91,-23.72 150.91,-23.72 156.67,-23.72 162.43,-29.48 162.43,-35.24 162.43,-35.24 162.43,-46.76 162.43,-46.76 162.43,-52.52 156.67,-58.28 150.91,-58.28\"/>\n",
+       "<text text-anchor=\"start\" x=\"114.76\" y=\"-38.1\" font-family=\"Helvetica,sans-Serif\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"#28513A\">Session</text>\n",
        "</a>\n",
        "</g>\n",
        "</g>\n",
@@ -162,15 +162,15 @@
        "response             \r",
        "reaction_time        \r",
        "\">\n",
-       "<path fill=\"#ffffff\" stroke=\"#9aa6b8\" d=\"M145.55,-97.69C145.55,-97.69 123.72,-97.69 123.72,-97.69 120.15,-97.69 116.59,-94.13 116.59,-90.56 116.59,-90.56 116.59,-83.44 116.59,-83.44 116.59,-79.87 120.15,-76.31 123.72,-76.31 123.72,-76.31 145.55,-76.31 145.55,-76.31 149.12,-76.31 152.68,-79.87 152.68,-83.44 152.68,-83.44 152.68,-90.56 152.68,-90.56 152.68,-94.13 149.12,-97.69 145.55,-97.69\"/>\n",
-       "<text text-anchor=\"start\" x=\"124.51\" y=\"-84.88\" font-family=\"Helvetica,sans-Serif\" text-decoration=\"underline\" font-size=\"10.00\" fill=\"#46536b\">Trial</text>\n",
+       "<path fill=\"#ffffff\" stroke=\"#B9BBBE\" d=\"M145.55,-97.69C145.55,-97.69 123.72,-97.69 123.72,-97.69 120.15,-97.69 116.59,-94.13 116.59,-90.56 116.59,-90.56 116.59,-83.44 116.59,-83.44 116.59,-79.87 120.15,-76.31 123.72,-76.31 123.72,-76.31 145.55,-76.31 145.55,-76.31 149.12,-76.31 152.68,-79.87 152.68,-83.44 152.68,-83.44 152.68,-90.56 152.68,-90.56 152.68,-94.13 149.12,-97.69 145.55,-97.69\"/>\n",
+       "<text text-anchor=\"start\" x=\"124.51\" y=\"-84.88\" font-family=\"Helvetica,sans-Serif\" text-decoration=\"underline\" font-size=\"10.00\" fill=\"#55585C\">Trial</text>\n",
        "</a>\n",
        "</g>\n",
        "</g>\n",
        "<!-- Session&#45;&gt;Session.Trial -->\n",
        "<g id=\"edge2\" class=\"edge\">\n",
        "<title>Session-&gt;Session.Trial</title>\n",
-       "<path fill=\"none\" stroke=\"#3a424f\" stroke-width=\"0.75\" stroke-opacity=\"0.619608\" d=\"M134.63,-58.61C134.63,-64.38 134.63,-70.14 134.63,-75.91\"/>\n",
+       "<path fill=\"none\" stroke=\"#171C39\" stroke-width=\"0.75\" stroke-opacity=\"0.619608\" d=\"M134.63,-58.61C134.63,-64.38 134.63,-70.14 134.63,-75.91\"/>\n",
        "</g>\n",
        "<!-- Subject -->\n",
        "<g id=\"node2\" class=\"node\">\n",
@@ -179,15 +179,15 @@
        "------------------------------\r",
        "species              \r",
        "\">\n",
-       "<path fill=\"#e7f3ec\" stroke=\"#2f7d5b\" d=\"M59.32,-58.28C59.32,-58.28 27.52,-58.28 27.52,-58.28 21.76,-58.28 16,-52.52 16,-46.76 16,-46.76 16,-35.24 16,-35.24 16,-29.48 21.76,-23.72 27.52,-23.72 27.52,-23.72 59.32,-23.72 59.32,-23.72 65.08,-23.72 70.84,-29.48 70.84,-35.24 70.84,-35.24 70.84,-46.76 70.84,-46.76 70.84,-52.52 65.08,-58.28 59.32,-58.28\"/>\n",
-       "<text text-anchor=\"start\" x=\"23.92\" y=\"-38.1\" font-family=\"Helvetica,sans-Serif\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"#1b5138\">Subject</text>\n",
+       "<path fill=\"#E8F0E9\" stroke=\"#3E7A52\" d=\"M59.32,-58.28C59.32,-58.28 27.52,-58.28 27.52,-58.28 21.76,-58.28 16,-52.52 16,-46.76 16,-46.76 16,-35.24 16,-35.24 16,-29.48 21.76,-23.72 27.52,-23.72 27.52,-23.72 59.32,-23.72 59.32,-23.72 65.08,-23.72 70.84,-29.48 70.84,-35.24 70.84,-35.24 70.84,-46.76 70.84,-46.76 70.84,-52.52 65.08,-58.28 59.32,-58.28\"/>\n",
+       "<text text-anchor=\"start\" x=\"23.92\" y=\"-38.1\" font-family=\"Helvetica,sans-Serif\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"#28513A\">Subject</text>\n",
        "</a>\n",
        "</g>\n",
        "</g>\n",
        "<!-- Subject&#45;&gt;Session -->\n",
        "<g id=\"edge1\" class=\"edge\">\n",
        "<title>Subject-&gt;Session</title>\n",
-       "<path fill=\"none\" stroke=\"#3a424f\" stroke-width=\"0.75\" stroke-opacity=\"0.619608\" d=\"M71.19,-41C82.38,-41 95.36,-41 106.58,-41\"/>\n",
+       "<path fill=\"none\" stroke=\"#171C39\" stroke-width=\"0.75\" stroke-opacity=\"0.619608\" d=\"M71.19,-41C82.38,-41 95.36,-41 106.58,-41\"/>\n",
        "</g>\n",
        "</g>\n",
        "</svg>"