mggg · peterrrock2 · Jun 2, 2026 · Jun 2, 2026 · Jun 2, 2026 · Jun 3, 2026
diff --git a/.gitignore b/.gitignore
@@ -11,7 +11,7 @@ __pycache__/
 
 # Distribution / packaging
 .Python
-build/
+/build/
 develop-eggs/
 dist/
 downloads/
@@ -130,6 +130,7 @@ venv/
 ENV/
 env.bak/
 venv.bak/
+.vscode/
 
 # Spyder project settings
 .spyderproject
@@ -162,3 +163,14 @@ cython_debug/
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
 
+# macOS
+**/.DS_Store
+
+# Project data and generated files
+/cbsas/
+/census_geographies/
+/study_areas/
+/census_raw/
+/data/
+/archive/
+/outputs/
diff --git a/AGENTS.md b/AGENTS.md
@@ -0,0 +1,69 @@
+## Project Purpose
+
+This project downloads population and geography data from the Census Bureau API and IPUMS/NHGIS, constructs adjacency graphs where census units (e.g. tracts) within study areas (e.g. CBSAs) are connected if they share a border, and applies a battery of residential segregation metrics. The goal is to assess segregation and disagreement among metrics across geographies and decades.
+
+## Pipeline overview
+
+The full pipeline is driven by `scripts/reproduce.sh`, which sources `scripts/pipeline_config.sh` for all configuration. Steps run in order:
+
+1. **`scripts/setup.sh`** — scaffolds the directory tree (`census_raw/`, `census_geographies/`, `study_area_sources/`, `study_areas/`, `outputs/`, etc.)
+2. **`pipeline/download_population_tables.py`** — downloads decennial census race/ethnicity counts (TOTPOP, WHITE, BLACK, POC, etc.) via Census API; uses IPUMS/NHGIS extracts for 1980 and 1990
+3. **`pipeline/download_geographies.py`** — downloads TIGER/Line shapefiles (2000–2020 via Census API; 1980/1990 via IPUMS NHGIS)
+4. **`pipeline/build_census_geographies.py`** — joins population tables to shapefiles, producing one attributed shapefile per year/level in `census_geographies/`
+5. **`scripts/build_study_areas.sh`** → **`pipeline/build_study_areas.py`** — builds study area boundary polygons (e.g. CBSA outlines from county-component `.xls` files) into `study_areas/definitions/`
+6. **`scripts/overlaps.sh`** → **`pipeline/overlaps.py`** — clips census geography shapefiles to each study area boundary (parallelized over years); outputs clipped shapefiles to `study_areas/<year>/` and coverage stats to `outputs/<run>/coverage_stats.csv`
+7. **`pipeline/gen_duals.py`** — builds the dual adjacency graph from each clipped shapefile; contracts zero-population nodes and ensures full connectivity; outputs `*_orig.json` and `*_connected.json` alongside each shapefile
+8. **`pipeline/calculate_metrics.py`** — computes ~80 segregation metrics per study area / year from each connected graph JSON; outputs one CSV row per area; errors logged to `outputs/<run>/metric_failures.csv`
+9. **`pipeline/generate_figures.py`** — reads aggregated metric CSVs and produces publication figures
+
+## Configuration
+
+All pipeline behavior is controlled by environment variables (with defaults in `scripts/pipeline_config.sh`):
+
+| Variable | Default | Options |
+|---|---|---|
+| `STUDY_AREA_TYPE` | `cbsa` | `cbsa`, `county` |
+| `CENSUS_GEOGRAPHY_TYPE` | `tracts` | `tracts`, `block_groups`, `blocks`, `counties` |
+| `CENSUS_GEOGRAPHY_YEARS` | `2020 2010 2000 1990 1980` | space-separated year list |
+| `STUDY_AREA_VINTAGE` | `2020` | year |
+| `RUN_NAME` | `<geo_type>_in_<study_area_type>` | string |
+| `RUN_OUTPUT_DIR` | `outputs/<RUN_NAME>` | path |
+
+For `STUDY_AREA_TYPE=cbsa`, a delineation file matching `list1_*_<vintage>.xls` must exist in `study_area_sources/`. A Census API key and IPUMS API key are required for downloads.
+
+## Important directories
+
+| Directory | Contents | Notes |
+|---|---|---|
+| `study_area_sources/` | CBSA delineation `.xls` files | Required input; not generated |
+| `census_raw/geographies/` | Raw downloaded TIGER/NHGIS shapefiles | Do not modify |
+| `census_raw/population/` | Raw downloaded population CSV tables | Do not modify |
+| `census_geographies/` | Merged population+shapefile per year/level (`<year>_<level>.shp`) | Generated; heavy |
+| `study_areas/definitions/` | Study area boundary shapefiles | Generated |
+| `study_areas/<year>/` | Clipped census shapefiles + dual graph JSONs per study area | Generated; heavy |
+| `outputs/<run_name>/` | Metric CSVs (`white_black.csv`, `white_poc.csv`), `coverage_stats.csv`, `metric_failures.csv`, figures, `run.log` | Generated outputs |
+
+`_orig.json` = raw dual graph; `_connected.json` = fully connected, zero-pop nodes contracted (used for metrics).
+
+## Important commands
+
+- **Setup:** `bash scripts/setup.sh`
+- **Full reproduction:** `bash scripts/reproduce.sh` (takes many hours on full dataset)
+- **Generate dual graphs:** `python pipeline/gen_duals.py <shapefile.shp> <out_orig.json> <out_connected.json>`
+- **Calculate metrics:** `python pipeline/calculate_metrics.py <connected.json> <x_col> <y_col> <tot_col>`
+- **Generate figures:** `python pipeline/generate_figures.py --filename <metrics.csv> --prefix <prefix>`
+- **Check overlaps:** `bash scripts/overlaps.sh`
+- **Parse output:** `python pipeline/parse_output.py` (consumes CSV format)
+
+## Testing / Verification
+
+- Run `pytest` to execute `tests/test_gen_duals.py` and `tests/test_pipeline_config.py`
+- Run `python pipeline/gen_duals.py ...` on a small fixture for a quick sanity check
+- Run `bash scripts/reproduce.sh` only when full reproduction is needed
+
+## Outputs
+
+- Metric CSVs: `outputs/<run_name>/white_black.csv`, `outputs/<run_name>/white_poc.csv`
+- Coverage stats: `outputs/<run_name>/coverage_stats.csv`
+- Figures: under `outputs/<run_name>/`
+- Output formats must not change.
diff --git a/Makefile b/Makefile
@@ -1,31 +1,19 @@
-.PHONY: all
+.PHONY: install setup test run clean
 
-CHICAGO_REF := 9618502_16980_march_2020
-OUTPUT_DIR := chicago-maup
+# Override with: make run EXPERIMENT=initial_paper_reproduction
+EXPERIMENT ?= baseline
 
-../data.mggg.org/census-2020/il/il_%.shp: pipeline/overlaps.py pipeline/gen_duals.py cbsas/defs/$(CHICAGO_REF).shp
-	python3 pipeline/overlaps.py $@ "cbsas/defs/$(CHICAGO_REF).shp" $(OUTPUT_DIR) --prefix "$*_"
-	python3 pipeline/gen_duals.py $(OUTPUT_DIR)/$*_$(CHICAGO_REF)_cbsa_tracts.shp $(OUTPUT_DIR)/$*_$(CHICAGO_REF)_orig.json $(OUTPUT_DIR)/$*_$(CHICAGO_REF)_connected.json --attr GEOID20 --pop-col TOTPOP20
+install:
+	poetry install
 
-chicago_duals: $(wildcard ../data.mggg.org/census-2020/il/il_*.shp)
+setup:
+	bash scripts/setup.sh
 
-chicago_black_header: pipeline/calculate_metrics.py ../data.mggg.org/census-2020/il/il_sldl.shp
-	python3 pipeline/calculate_metrics.py $(OUTPUT_DIR)/il_tract_cbsa_sldl_connected.json BLACK WHITE TOTPOP --headers-only > outputs/white_black_chicago.csv
-chicago_poc_header: ../data.mggg.org/census-2020/il/il_sldl.shp
-	python3 pipeline/calculate_metrics.py $(OUTPUT_DIR)/il_tract_cbsa_sldl_connected.json POC WHITE TOTPOP --headers-only > outputs/white_poc_chicago.csv
+test:
+	poetry run pytest pipeline/tests/ -v
 
-chicago-maup/%_$(CHCIAGO_REF)_connected.json: pipeline/calculate_metrics.py chicago_black_header chicago_poc_header ../data.mggg.org/census-2020/il/il_%.shp
-	python3 pipeline/calculate_metrics.py $@ BLACK WHITE TOTPOP >> outputs/white_black_chicago.csv
-	python3 pipeline/calculate_metrics.py $@ POC WHITE TOTPOP >> outputs/white_poc_chicago.csv
+run: setup
+	bash scripts/run_experiment.sh experiments/$(EXPERIMENT)/config.json
 
-outputs/white_black_chicago.csv: chicago_black_header chicago_duals
-	$(MAKE) $(wildcard chicago-maup/*connected.json)
-outputs/white_poc_chicago.csv: chicago_poc_header chicago_duals
-	$(MAKE) $(wildcard chicago-maup/*connected.json)
-
-chicago: outputs/white_black_chicago.csv outputs/white_poc_chicago.csv
-
-all: chicago
-
-clean: 
-	rm chicago-maup/*
+clean:
+	rm -rf outputs/
diff --git a/README.md b/README.md
@@ -1,32 +1,106 @@
-## CAPY
+## Project Purpose
+
+This project downloads population and geography data from the Census Bureau API and IPUMS/NHGIS, constructs adjacency graphs where census units (e.g. tracts) within study areas (e.g. CBSAs) are connected if they share a border, and applies a battery of residential segregation metrics. The goal is to assess segregation and disagreement among metrics across geographies and decades.
+
+## Folder structure
 
-## Testing
 ```
-pytest
+capy-bara/
+├── data/
+│   ├── raw/                        # downloaded source files (gitignored)
+│   │   ├── geographies/            # TIGER/Line and NHGIS shapefiles
+│   │   ├── population/             # Census API / NHGIS population tables
+│   │   └── ipums_extracts/         # IPUMS extracts (1980, 1990)
+│   ├── interim/                    # processed intermediates (gitignored)
+│   │   ├── census_geographies/     # population-attributed shapefiles per year/level
+│   │   ├── cbsas/                  # CBSA definitions by decade
+│   │   ├── study_areas/            # clipped shapefiles + dual graph JSONs per study area
+│   │   └── study_area_sources/     # CBSA delineation .xls files
+│   └── outputs/                    # pipeline run outputs
+│       ├── tracts_in_cbsa/
+│       ├── block_groups_in_cbsa/
+│       ├── blocks_in_cbsa/
+│       └── cross_level_comparisons/  # figures comparing results across runs
+│
+├── pipeline/                       # core pipeline modules
+│   ├── download/                   # download_geographies.py, download_population_tables.py
+│   ├── build/                      # build_census_geographies.py, build_study_areas.py,
+│   │                               #   overlaps.py, gen_duals.py, filter_cbsas.py
+│   ├── metrics/                    # calculate_metrics.py, parse_output.py
+│   ├── viz/                        # generate_figures.py
+│   ├── utils/                      # definitions.py
+│   └── tests/
+│
+├── experiments/                    # hypothesis-testing experiments
+│   ├── notebooks/                  # scratch notebooks before a hypothesis becomes a script
+│   ├── comparisons/                # cross-experiment analyses and figures
+│   └── exp_<name>/                 # one folder per experiment: run.py + figures/
+│
+├── working_paper_reproduction/     # materials for reproducing paper results
+│   ├── notebooks/                  # reproduction notebooks
+│   ├── misc_analysis/
+│   └── figures/
+│
+├── scripts/                        # shell scripts for running the pipeline
+└── archive/                        # inactive code and old outputs
 ```
 
-## Reproducing
-First, install the [dependencies](#dependencies).
+## Pipeline overview
 
-Next, download the [nhgis](https://nhgis.org) race and shapefile data for Census tracts (from 1970-2020 inclusive) into the `nhgis` folder.
+The full pipeline is driven by `scripts/reproduce.sh`, which sources `scripts/pipeline_config.sh` for all configuration. Steps run in order:
 
-Finally, run the reproduce script to rerun the analysis pipeline:
-```
-bash reproduce.sh
-```
+1. **`scripts/setup.sh`** — scaffolds the directory tree
+2. **`pipeline/download/download_population_tables.py`** — downloads decennial census race/ethnicity counts (TOTPOP, WHITE, BLACK, POC, etc.) via Census API; uses IPUMS/NHGIS extracts for 1980 and 1990
+3. **`pipeline/download/download_geographies.py`** — downloads TIGER/Line shapefiles (2000–2020 via Census API; 1980/1990 via IPUMS NHGIS)
+4. **`pipeline/build/build_census_geographies.py`** — joins population tables to shapefiles, producing one attributed shapefile per year/level in `data/interim/census_geographies/`
+5. **`scripts/build_study_areas.sh`** → **`pipeline/build/build_study_areas.py`** — builds study area boundary polygons (e.g. CBSA outlines from county-component `.xls` files) into `data/interim/study_areas/definitions/`
+6. **`scripts/overlaps.sh`** → **`pipeline/build/overlaps.py`** — clips census geography shapefiles to each study area boundary (parallelized over years); outputs clipped shapefiles to `data/interim/study_areas/<year>/` and coverage stats to `data/outputs/<run>/coverage_stats.csv`
+7. **`pipeline/build/gen_duals.py`** — builds the dual adjacency graph from each clipped shapefile; contracts zero-population nodes and ensures full connectivity; outputs `*_orig.json` and `*_connected.json` alongside each shapefile
+8. **`pipeline/metrics/calculate_metrics.py`** — computes ~80 segregation metrics per study area / year from each connected graph JSON; outputs one CSV row per area; errors logged to `data/outputs/<run>/metric_failures.csv`
+9. **`pipeline/viz/generate_figures.py`** — reads aggregated metric CSVs and produces publication figures
+
+`_orig.json` = raw dual graph; `_connected.json` = fully connected, zero-pop nodes contracted (used for metrics).
+
+## Configuration
+
+All pipeline behavior is controlled by environment variables (with defaults in `scripts/pipeline_config.sh`):
 
-Your outputs will be in the `outputs/` folder.
+| Variable | Default | Options |
+|---|---|---|
+| `STUDY_AREA_TYPE` | `cbsa` | `cbsa`, `county` |
+| `CENSUS_GEOGRAPHY_TYPE` | `tracts` | `tracts`, `block_groups`, `blocks`, `counties` |
+| `CENSUS_GEOGRAPHY_YEARS` | `2020 2010 2000 1990 1980` | space-separated year list |
+| `STUDY_AREA_VINTAGE` | `2020` | year |
+| `RUN_NAME` | `<geo_type>_in_<study_area_type>` | string |
+| `RUN_OUTPUT_DIR` | `data/outputs/<RUN_NAME>` | path |
 
-### Dependencies
-Python deps are managed via Poetry. Install Python `3.9.10`, then install the Python deps by running:
+For `STUDY_AREA_TYPE=cbsa`, a delineation file matching `list1_*_<vintage>.xls` must exist in `data/interim/study_area_sources/`. A Census API key and IPUMS API key are required for downloads.
+
+## Important commands
+
+- **Setup:** `bash scripts/setup.sh`
+- **Full reproduction:** `bash scripts/reproduce.sh` (takes many hours on full dataset)
+- **Generate dual graphs:** `python pipeline/build/gen_duals.py <shapefile.shp> <out_orig.json> <out_connected.json>`
+- **Calculate metrics:** `python pipeline/metrics/calculate_metrics.py <connected.json> <x_col> <y_col> <tot_col>`
+- **Generate figures:** `python pipeline/viz/generate_figures.py --filename <metrics.csv> --prefix <prefix>`
+- **Check overlaps:** `bash scripts/overlaps.sh`
+- **Parse output:** `python pipeline/metrics/parse_output.py`
+
+## Dependencies
+
+Python deps are managed via Poetry:
 ```
 pip install poetry
 poetry install
 ```
 
-To launch a shell with the poetry enviroment activated, run:
+To launch a shell with the poetry environment activated:
 ```
 poetry shell
 ```
 
-[`fd`](https://github.com/sharkdp/fd) is also required to run the reproduce script.
+Set API keys in your shell before downloading data:
+```
+export CENSUS_API_KEY="..."
+export IPUMS_API_KEY="..."
+```
diff --git a/chicago.sh b/chicago.sh
diff --git a/configs/README.md b/configs/README.md
diff --git a/configs/list1_april_2018.xls b/configs/list1_april_2018.xls
diff --git a/configs/list1_august_2017.xls b/configs/list1_august_2017.xls
diff --git a/configs/list1_february_2013.xls b/configs/list1_february_2013.xls
diff --git a/configs/list1_july_2015.xls b/configs/list1_july_2015.xls
diff --git a/configs/list1_march_2020.xls b/configs/list1_march_2020.xls
diff --git a/configs/list1_september_2018.xls b/configs/list1_september_2018.xls
diff --git a/experiments/README.md b/experiments/README.md
@@ -0,0 +1,5 @@
+This folder stores hypothesis testing scripts (finalized python scripts, notebooks, and figures).
+
+Folders are of the form `h*_t*_<short name>`, corresponding to the hypothesis and test number in the `hypotheses_notebook`.
+
+`notebooks` are for figuring out what should be in python scripts.
diff --git a/experiments/baseline/config.json b/experiments/baseline/config.json
@@ -0,0 +1,3 @@
+{
+  "name": "baseline"
+}
diff --git a/experiments/baseline/run.py b/experiments/baseline/run.py
@@ -0,0 +1,6 @@
+from pathlib import Path
+
+from experiments.experiment_orchestration import run_experiment
+
+if __name__ == "__main__":
+    run_experiment(Path(__file__).parent / "config.json")
diff --git a/experiments/defaults.json b/experiments/defaults.json
@@ -0,0 +1,7 @@
+{
+  "study_area_type": "cbsa",
+  "census_geography_type": "tracts",
+  "census_geography_years": [2020, 2010, 2000, 1990, 1980],
+  "study_area_vintage": 2020,
+  "study_area_definition_geography_type": "counties"
+}