Replication package - The Score Is Not the Model

Code, per-run results, and every number the manuscript prints, as JSON.

Shuaidong Gao Chongqing Institute of Foreign Studies, Chongqing, China gaoshuaidong@stu.cqifs.edu.cn

The Score Is Not the Model: Four Protocol Failure Modes in Open Graph Benchmark Reporting, submitted to Knowledge and Information Systems.

https://github.com/sgao-academics/ogb-protocol-audit

Four protocol probes for Open Graph Benchmark reporting, a protocol-clean re-measurement of five OGB benchmarks, and every number the manuscript prints, as JSON.

Start here

python audit/run_all.py --fast      # reuse the shipped JSON: rebuild tables and figures, then re-check
python audit/run_all.py             # the full audit (~25 min, no GPU; downloads the OGB datasets)
python audit/run_all.py --verify    # manuscript numbers against the JSON; exit 0 means they agree

INDEX.json records the entry point, the verifier, and the SHA-256 of every audit script. audit/README.md states what each probe tests and where its evidence lives.

Layout

audit/          probe scripts, their JSON output, and the consistency checker
                verify_paper.py re-checks every printed number; run_all.py drives
                the pipeline
results/raw/    per-run metrics as the training scripts wrote them:
                ogbn-*/<tag>/metrics.json and linkpred/<dataset>/<protocol>/result.json
paper/          the bibliography (refs.bib), the journal style files
                (sn-jnl.cls, sn-basic.bst), the generated appendix tables, and
                the figures the package regenerates into paper/figures/

Paths and environment

Every script resolves its inputs relative to its own location, so the package runs from wherever it is unpacked. Four variables override the defaults:

Variable	Meaning	Default
OGB_AUDIT_RAW	the per-run metrics	./results/raw
FIGDIR	where figures are written	./paper/figures
APPENDIX_TEX	the generated appendix tables	./paper/appendix_tables.tex
CONTACT_MAIL	your address, if you want the CrossRef/OpenAlex polite pool	unset

Two scripts (audit/verify_bib.py, audit/verify_bib_authors.py) contact CrossRef, OpenAlex and the arXiv API. They are reference checks, not part of the audit, and they are the only part of the package that needs the network.

Absolute paths do appear inside the shipped records under audit/ and results/raw/. Those are the paths the scripts printed on the machine that produced them, left verbatim because those records are the primary evidence the audit reads and the paper's whole claim is that they were not edited afterwards. The in-package equivalent of each of them is results/raw/.

What this repository does not contain

The manuscript source is supplied to the journal as Supplementary Material and is not mirrored here.

The models whose reporting the audit diagnoses. Those are described in the manuscript. The probes read their recorded outputs; they do not retrain them.

An earlier, superseded study by the same author, whose artefacts are shipped with the formal submission for transparency, is not included here; the present paper cites none of it.

Determinism

The structural heuristics are exact integer or rational quantities, so probes P1, P2 and the symmetric arms of P3 carry no seed variance at all. The learned probes fix the numpy, torch and scikit-learn seeds and report per-run standard deviations.

Citation

@misc{gao2026score,
  author = {Gao, Shuaidong},
  title  = {The Score Is Not the Model: Four Protocol Failure Modes in
            Open Graph Benchmark Reporting},
  year   = {2026},
  note   = {Manuscript submitted to Knowledge and Information Systems},
  url    = {https://github.com/sgao-academics/ogb-protocol-audit}
}

Licence

MIT - see LICENSE.