Poetry packaging + release 0.2.0 (numpy 2 / econml 0.16 / dowhy 0.14); RunExperiments harness

.github/workflows/format.yml

@@ -8,6 +8,11 @@ on:
 
   workflow_dispatch:
 
+# Cancel superseded runs (e.g. a force-push) for the same workflow + ref.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   check:
     runs-on: ubuntu-latest

.github/workflows/lint.yml

@@ -8,6 +8,11 @@ on:
 
   workflow_dispatch:
 
+# Cancel superseded runs (e.g. a force-push) for the same workflow + ref.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   check:
     runs-on: ubuntu-latest

.github/workflows/test.yml

@@ -8,6 +8,11 @@ on:
 
   workflow_dispatch:
 
+# Cancel superseded runs (e.g. a force-push) for the same workflow + ref.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   check:
     runs-on: ${{ matrix.os }}

.gitignore

@@ -1,7 +1,19 @@
 htmlcov/
 venv/
+.venv/
 env
 notebooks/scratch/
+
+# Simulation run results / generated artifacts (large, never commit)
+notebooks/RunExperiments/EXPERIMENT_RESULTS_*/
+notebooks/RunExperiments/RunDatasets/
+notebooks/RunExperiments/runners/plots/
+notebooks/RunExperiments/runners/tex/
+notebooks/RunExperiments/runners/causaltune
+
+# Accidentally-dropped credentials — never commit
+*ray-gcp-key
+*ray-gcp-key.pub
 .ropeproject/
 .DS_Store
 .vscode
@@ -108,4 +120,8 @@ data/*
 docs/_build/
 docs/_any
 docs/_cache/
-docs/notebooks
+docs/notebooks
+# Build artifacts
+build/
+dist/
+*.egg-info/

CLAUDE.md

@@ -0,0 +1,26 @@
+# CLAUDE.md — causaltune
+
+## Committing: DCO sign-off is REQUIRED
+
+This repo enforces the **Developer Certificate of Origin (DCO)** in CI. Every
+commit MUST carry a `Signed-off-by:` trailer matching the commit author, or the
+DCO check fails and the PR is blocked.
+
+ALWAYS commit with `-s` (sign-off):
+
+```bash
+git commit -s -m "your message"
+```
+
+This appends `Signed-off-by: Egor Kraev <egor.kraev@gmail.com>` (from
+`git config user.name` / `user.email`). When amending or rebasing, keep the
+trailer — use `git rebase --signoff` or `git commit -s --amend` as needed.
+
+Never create a commit here without the sign-off trailer.
+
+## Do NOT commit simulation run results
+
+`notebooks/RunExperiments/EXPERIMENT_RESULTS_*/`, `RunDatasets/`, and the
+`runners/plots/` + `runners/tex/` output dirs are large generated artifacts
+(tens of GB) and are gitignored. Only source code (`.py`, tests, docs,
+`runners/*.py` scripts) gets committed.

notebooks/Estimate assignment outcomes.ipynb

@@ -0,0 +1,76 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "139511bd",
+   "metadata": {},
+   "source": [
+    "Use the results of a past A/B test to estimate outcomes of any other assignment\n",
+    "\n",
+    "(checkbox A)\n",
+    "* I have the results of a past A/B/N test with strictly random assignment (chosen by default)\n",
+    "* (advanced) I have the results of a past test using random assignment with variable probabilities\n",
+    "\n",
+    "(checkbox B)\n",
+    "* I have the variants I'd like to assign to each user in a new hypothetical assignment policy (chosen by default)\n",
+    "* (advanced) I have the assignment probabilities of each variant to each user, my new policy is probabilistic\n",
+    "\n",
+    "Then display the following text:\n",
+    "\n",
+    "Please upload a csv file with the following columns, spelled just like that:\n",
+    "- ActualAssignment: number of the variant actually assigned to that user in the test, sequential integers starting with zero, 0 being control (so for an A/B test 0 or 1)\n",
+    "- (if the first option is chosen in B) HypotheticalAssignment: number of the variant assigned to each user in the hypothetical assignment whose outcome you want to estimate\n",
+    "- (if the second option is chosen in B) HypotheticalAssignment0, HypotheticalAssignment1, ...: probabilities of the respective variant \n",
+    "- (show this only if the second option is chosen in A) PTT: The ex ante probability of choosing the variant that was chosen in the test (also known as propensity to treat)\n",
+    "\n",
+    "[Upload file]\n",
+    "\n",
+    "The estimated outcome for the new assignment is ...\n",
+    "The confidence interval for it is ..."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4181fc89",
+   "metadata": {},
+   "source": [
+    "Then display the following text:\n",
+    "\n",
+    "Please upload a csv file with the following columns, spelled just like that:\n",
+    "- ActualAssignment: number of the variant actually assigned to that user in the test, sequential integers starting with zero, 0 being control (so for an A/B test 0 or 1)\n",
+    "- (if the first option is chosen in B) HypotheticalAssignment: number of the variant assigned to each user in the hypothetical assignment whose outcome you want to estimate\n",
+    "- (if the second option is chosen in B) HypotheticalAssignment0, HypotheticalAssignment1, ...: probabilities of the respective variant \n",
+    " "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d1432d63",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python [conda env:.conda-causaltune]",
+   "language": "python",
+   "name": "conda-env-.conda-causaltune-py"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/RunExperiments/cluster_config.yaml

@@ -6,7 +6,7 @@ cluster_name: default
 
 # The maximum number of workers nodes to launch in addition to the head
 # node.
-max_workers: 8
+max_workers: 7
 
 # The autoscaler will scale up the cluster faster with higher upscaling speed.
 # E.g., if the task requires adding more nodes then autoscaler will gradually
@@ -93,7 +93,7 @@ available_node_types:
         min_workers: 1
         # The maximum number of worker nodes of this type to launch.
         # This takes precedence over min_workers.
-        max_workers: 8
+        max_workers: 7
         # The resources provided by this node type.
         resources: {"CPU": 2}
         # Provider-specific config for the head node, e.g. instance type. By default

notebooks/RunExperiments/runners/estimator_level_trials.py

@@ -0,0 +1,28 @@
+import os
+
+from experiment_runner import run_batch, get_estimator_list
+
+use_ray = True
+
+for kind in ["KC"]:
+    metrics = [
+        "erupt",
+        "qini",
+        "auc",
+        "psw_energy_distance",
+        "codec",  # NEW
+        "bite",  # NEW
+    ]
+    estimators = get_estimator_list(kind)
+    for estimator in estimators:
+        short_name = estimator.split(".")[-1]
+        print(f"Running {kind} with {estimator}")
+        out_dir = run_batch(
+            identifier=f"Estimator_{short_name}",
+            kind=kind,
+            metrics=metrics,
+            estimators=[estimator],
+            dataset_path=os.path.realpath("../RunDatasets"),
+            num_trials=25,
+            use_ray=use_ray,
+        )