engineering-patterns.json.bak.1784130641

[
 "np.array(list_of_tuples) silently builds a 2-D array, not a 1-D object array of tuples; iterating it yields row-arrays and dict/DataFrame lookups then fail with KeyError. Fix the wrapping (keep a plain list, or fill an explicit object array) rather than the labels.",
 "pandas MultiIndex columns mean column labels are TUPLES. Code that assumes scalar labels (string keys, str() calls, np.array wrapping) breaks on them. Handle tuple labels as first-class, preserve them.",
 "Never fix a lookup error (KeyError/IndexError) by changing the IDENTITY of the data — stringifying tuples, joining keys, renaming labels. That cures the symptom by amputation and violates the library contract. Fix the code path that mishandles the identity instead.",
 "Before submitting, re-read your diff and ask: does this patch change the TYPE or identity of anything public (attribute, return value, label)? If yes, the fix is probably wrong even though your reproduction passes.",
 "Prefer the smallest patch that makes the failing case pass: a fix that deletes or bypasses the problematic wrapper is usually more correct than one that transforms the data to survive the wrapper.",
 "Agent scaffold: when a verification GATE auto-runs a check and then REJECTS the agent's declare/submit, feed back the check's SPECIFIC diagnostic, never a generic \"gate not passed\". A generic rejection gives the agent nothing to act on, so it re-declares or thrashes the same tool until its turn/token budget dies (observed: sphinx phase-1 bootstrap burned 20-36 run_smoke_test turns into a 50-turn budget death on envs that were actually fine). Capture the checker's result and surface the actionable reason (missing test dependency, uncollectable suite, no green test). Keep the payload byte-identical when there is no extra detail so unrelated gate paths (e.g. a different phase) are provably unaffected.",
 "Test-runner flags must be version-gated to the tool era of the code under test. A benchmark harness runs each instance against ITS OWN pinned test-framework version, so a CLI flag that is valid for your newest instances can be a fatal usage error for older ones. Concretely: pytest's --no-header/--no-summary exist only in pytest>=6.0; on pytest 4.x/5.x they are 'unrecognized arguments' -> exit code 4 with ZERO tests collected/run. If that flag lives in THE single test path used for SCORING, every old-framework instance is silently auto-scored UNRESOLVED regardless of whether the patch is correct (a whole false-negative family). Detection signature: misses whose captured tail is a bare framework header line (e.g. 'rootdir: ...') with no pass/fail counts, clustered in the repo that pins the oldest framework. Fix: probe the framework major version once per environment (CACHE KEYED ON THE ENV DIR, never on a relative interpreter path that is identical across instances in a long-running multi-instance process) and include era-specific flags only when supported. Verify by running the SHIPPED scoring function (not an ad-hoc command) on one affected instance and confirming the result flips, plus a newest-version instance to prove no regression, interleaved in one process to expose cache-key collisions.",
 "A test suite that fails to COLLECT (ImportError / VersionRequirementError / \"needs at least <framework> vX\" / \"found no collectors\") instead of failing an assertion is an ENVIRONMENT problem, not your code — never edit source to silence it. Testing an OLD framework version requires its plugin/theme/support ecosystem (e.g. sphinxcontrib-*, alabaster, pyparsing) pinned to that version’s era; the newest release of a support package often hard-requires a newer framework and aborts collection. Let the harness own env setup and get a green baseline (run_smoke_test with no args) before trusting any repro.",
 "Warnings-as-errors suite still dies at COLLECTION after you pinned the obvious too-new dep? Look for a SECOND too-new pure-python dep that raises a DeprecationWarning during import — especially version machinery. A newer package can HIJACK a library's entry point (e.g. a 'vcs-versioning' fork taking over setuptools_scm's version-scheme entry points) and turn a config value the project still uses (an old scheme name like 'release-branch-semver') into a fatal deprecation at import. Downgrading the primary package is NOT enough while the hijacking sibling stays installed — you must UNINSTALL it, not just pin. Reproduce the reference env by clearing EVERY too-new build/version dep, not only the first. Whether it bites can be version-dependent (only builds that invoke the machinery at import), so an env fix that is a no-op for some instances is still correct for all.",
 "Auditing a coding-benchmark scorer for FALSE POSITIVES needs a two-state CAUSATION test, not a bare re-run of the withheld checks. SWE-bench resolution = all FAIL_TO_PASS flip to pass AND all PASS_TO_PASS stay green; a home scorer that runs only FAIL_TO_PASS can mark a solve resolved even if the patch broke a P2P. But you CANNOT detect that by just re-running P2P at home: many P2P tests fail in the home env even at base commit (home/gold ENV DISCREPANCY), so a bare P2P failure misclassifies a correct solve. Only P2P that PASSES at base+test_patch and FAILS at base+test_patch+model_patch is a genuine model-induced regression. Corollary: naively enforcing PASS_TO_PASS in the home scorer would trade a rare false positive for many false negatives -> enforce P2P ONLY in the authoritative Docker eval (end-of-run audit), symmetric to the false-negative reclaim workflow. Empirically (140-run, 56 resolved w/ surviving work-dirs): 47 fully green, 9 P2P-fail, ALL 9 = env discrepancy, 0 genuine regressions.",
 "Benchmark tests that hardcode an EXTERNAL service URL (e.g. requests' suite defaults HTTPBIN to http://httpbin.org/) become a false-negative family in a network-isolated scorer: every such FAIL_TO_PASS raises ConnectionError regardless of patch correctness, so a correct patch is scored UNRESOLVED. Detection signature: misses in the repo whose F2P are network endpoints (httpbin('get')/'post'/'basic-auth'/...), with either a LONG runtime (DNS/connect timeouts, tens of seconds) or a fast connection-refused; confirm the host is unreachable and no local stub is installed. General fix: provide a LOCAL stand-in (pip install pytest-httpbin, start its bundled httpbin Flask app on 127.0.0.1, export HTTPBIN_URL before pytest -- the module reads HTTPBIN at import time so a fixture is too late) -- do NOT edit source. Verification method that also discriminates real misses: re-run F2P against the local service; genuine misses still fail with real assertions/TypeErrors (requests-2317's patched compat.py raises TypeError even with httpbin up) while env-FNs flip to all-pass (requests-2148 10/10). Caveat: a test that scheme-swaps http->https on the SAME host needs a dual-scheme local server, else it SSL-fails locally though it passes against a real dual-scheme host.",
 "When a benchmark test reads configuration (e.g. a service URL like HTTPBIN_URL) from the ENVIRONMENT at MODULE-IMPORT time, the earliest robust injection point is a repo-root conftest.py MODULE BODY: pytest imports conftest before collecting/importing test modules, so setting os.environ there (and starting any needed local server) beats the test's import-time read, whereas a session-scoped fixture runs too late. Mark any background server thread daemon (+atexit stop) so a never-stopped serve_forever cannot hang pytest at exit. Keep the conftest UNTRACKED so it stays out of the candidate's git-diff and carries no answer/instance data, and write it AFTER applying the scoring test patch so a suite-provided conftest is never clobbered. General, leakage-safe way to de-network an external-service test family (shipped for psf/requests as ensure_local_httpbin, commit 9bab933).",
 "A test RUNNER's default PARALLELISM can corrupt a benchmark scorer, independent of the code under test. django's tests/runtests.py defaults to a multiprocessing worker pool; on ANY failing test the worker cannot pickle the traceback, so django prints 'tracebacks cannot be pickled, making it impossible for the parallel test runner to handle this exception cleanly', the pool is torn down uncleanly (Pool state=RUN at __del__), and the terminal OK/FAILED result summary is DROPPED while the exit code can be perturbed by the teardown exception. Two harms: (1) score_tail telemetry becomes a bare 'ResourceWarning: unclosed running multiprocessing pool' instead of 'FAILED (failures=N)'/'OK', defeating false-negative triage; (2) a genuinely PASSING patch can score as a miss if the 'OK' line is suppressed or the exit code is perturbed (false negative). Fix: force SERIAL execution at scoring time (django: '--parallel 1'), which matches the authoritative SWE-bench django harness and makes results deterministic. Detection signature: a django (unittest) miss whose score_tail is the pool ResourceWarning with no result summary; passing runs are unaffected because no traceback pickling occurs, so the bug hides on green instances and only bites on runs containing a failure/error. GENERAL RULE: pin a test runner to serial/deterministic mode for scoring, and version-gate any runner flag on the runner actually supporting it (static source read / --help probe, cached) so an older codebase never receives an unrecognized flag -- the same guard the harness uses for pytest '--no-header' (_pytest_major).",
 "A REGRESSION/ACCEPTANCE signal in a bug-fix harness is only as good as WHICH tests it samples. Sampling 'neighbor' tests by the first-N in pytest collection order (top-of-tree/alphabetical) means that in large repos (django/sympy) the sampled files are unrelated to the model's edit, so a locally-introduced regression is almost never exercised and the signal is near-worthless. Bias the sample toward the FAULT AREA: take the in-repo traceback frames of the reproduction as hint paths and rank candidate test files by proximity to them (same directory >> shared leading package segments >> module-name match, e.g. test_expr.py near expr.py). Keep it a STABLE reorder so that with no hints every score is 0 and the result is byte-identical to the old first-N selection -> the change is provably behavior-preserving where no fault signal exists. Leakage-safety falls out for free: FAIL_TO_PASS tests fail pre-patch so they are auto-excluded from a 'passing-baseline' sample, and the neighbor node ids surfaced to the model are PASS_TO_PASS-like (green pre-patch), never the graded hidden tests. Make the ranker a PURE function (paths in, ordered ids out) so it is unit-testable offline and cannot touch the scoring path.",
 "A 'warnings-as-errors' test suite makes EVERY too-new tooling dependency a latent false-negative source, and the TEST RUNNER ITSELF is one of those deps. matplotlib sets filterwarnings=error, so a fresh env where the resolver pulls the LATEST pytest (9.x / >=8.4) dies at COLLECTION: pytest raises PytestRemovedIn10Warning for any same-file test that passes a generator to @parametrize (e.g. test_rcparams.py::test_validator_valid) -> fatal -> the whole module fails to collect -> 'found no collectors' -> a genuinely correct target patch is scored a miss. This is the SAME NO_COLLECTORS symptom as too-new pyparsing and the setuptools-scm/vcs-versioning hijack — a THIRD sub-cause in the same family. Fix by pinning the runner to the era of the code under test (pytest<8, era-appropriate 7.x, which still supports the harness's --no-header) alongside the other pure-python pins. Verify BOTH directions before shipping: reproduce the collection error under the too-new runner, confirm the pin restores collection (target test runs to a real verdict), AND reconstruct a currently-RESOLVED instance (base+model+test) to confirm the downgrade keeps it green (no regression). Note the pin can surface a real miss (the masked verdict may still be a FAIL) — restoring collection is about scorer CORRECTNESS, not about manufacturing reclaims.",
 "Before hypothesizing an ENVIRONMENT false-negative for a repo's miss cluster, VERIFY by re-running FAIL_TO_PASS in the retained work-dir venv (model+test patch applied): a repo can be dominated by verification-frontier (DECLARED-WRONG) misses with ZERO env-FN family. It is ENV when the failure is a collection error / skip / network timeout / fatal warning from a too-new dep; it is GENUINE when env_ok holds, the interpreter is healthy, and 'N failed' comes with a deterministic assertion/logic error in <1s -- frequently with the model's OWN self-verification passing. sympy is the second case: 6/6 verifiable self-verified misses (24102/22005/21847/20639/21171/21379) reproduce as genuine parser/solver/printing edge-case failures the model's happy-path self-verify never covered (plus 23191 where the model edited only a test file). The lesson is symmetric: a home audit both RECLAIMS false-negatives AND RULES OUT a whole class of env investigation, redirecting effort to the real bottleneck -- a non-self acceptance signal. Corollary: model self-verification is systematically over-optimistic on parse/print/format tasks; weight it accordingly.",
 "A self-verifying agent's RED->GREEN reproduction gate is only as strong as the reproduction's ASSERTION. A script that goes RED via an uncaught exception and GREEN once that exception stops firing proves only 'no crash', NOT 'correct output'. Reproductions whose sole check is a constant (assert True/1) or that carry no value assertion are a systematic self-deception vector for OUTPUT-correctness bugs (parse/print/format/compute): the agent declares success while hidden acceptance tests that check the actual value still fail. Measured on 147 SWE-bench traces: 68% of self-verified MISSES had such weak reproductions vs 55% of self-verified RESOLVES -- real and directional but NOT a clean classifier (many correct crash-fixes legitimately have no value assertion). Therefore act on it as ADVISORY STEERING, never a hard gate (a hard block would manufacture false-negatives on genuine crash-fixes). Implementation: a pure, leakage-safe strength classifier over the model's OWN script (value_check / vacuous_constant / weak); at GREEN, if not value_check, surface a note urging an expected-VALUE assertion. Keep it OUT of the score path (the scorer runs the hidden tests directly) so it can only steer, never change a verdict. (swe_fix_tools _reproduction_strength, commit 644f3b2)",
 "A subprocess that can DEADLOCK (not merely run slow) needs a PROGRESS-WATCHDOG, not just a hard timeout: watch a strictly-growing progress signal and abort in ~grace seconds when it FREEZES, keeping the hard timeout only as a backstop. A lone hard timeout forces a bad choice between killing legitimately-slow work early and silently burning the whole budget on a hang. Localize WHERE a hang lives before trying to fix it: all worker threads parked in futex_do_wait at ~0% CPU with ZERO downstream side effects (no build-cache growth, no new images/containers, empty run logs) == an ORCHESTRATION/lock deadlock, DISTINCT from a network or client-handshake stall. Prove the distinction cheaply and POSITIVELY instead of assuming: SWE-bench run_evaluation hung on pop-os 2026-07-15 even though the docker-py 7.1.0 handshake returned in 0.02s and CLI + low-level + high-level SDK builds of a trivial image all finished in <0.1s -- so the hang was inside the eval build orchestration, not the registry (pulled fine) or the SDK. Corollary: a recurring \"blocked until the cache/network warms\" assumption should be TESTED, not deferred cycle after cycle -- the real blocker may be internal and warming will never clear it. General hygiene: fail fast + fail visibly (distinct exit code for stall vs error), verify the watchdog decision logic with a dependency-free selftest (stalled child killed within grace; progressing child left alone), and restore any scratch state (e.g. throwaway test images) you created while probing."
]