Skip to content

docs(ci): align sync resolver/fixer/analyzer prompts with wiring-aware reasoning - #126

Merged
mateo-di merged 1 commit into
fix/upstream-sync-agent-reactionfrom
fix/upstream-sync-prompt-alignment
Aug 4, 2026
Merged

docs(ci): align sync resolver/fixer/analyzer prompts with wiring-aware reasoning#126
mateo-di merged 1 commit into
fix/upstream-sync-agent-reactionfrom
fix/upstream-sync-prompt-alignment

Conversation

@mateo-di

Copy link
Copy Markdown
Collaborator

Summary

Aligns the upstream-sync automation prompts with the reasoning that actually found and fixed the v1.92.0 regressions. That sync shipped three CARTO features which passed every automated check yet were functionally broken, and two of the prompts' own heuristics caused the loss. This is the prompt/doc half of the long-term hardening (Workplan B); the mechanical test-reaction wiring is #125 (Workplan A), which this is stacked on.

The three failure classes, none catchable by "does the pattern still grep":

  1. Dropped call site across an auto-merged filestreaming_iterator.py (conflicted) lost a param while handler.py (auto-merged) kept passing it → TypeError on every streaming request.
  2. Orphaned helper_patch_get_session_from_redis survived byte-for-byte but its caller was replaced by upstream's → sessions written to Redis, read from the batch-delayed DB → multi-turn context loss.
  3. Cross-version data-format drift — upstream began b64-encoding response ids; CARTO's unchanged store keyed Redis by the encoded id while lookup used the decoded id → 100% session-lookup miss.

Changes

Resolver prompt (carto-upstream-sync-resolver.yml):

  • Verbatim-first rule (restore CARTO blocks byte-identical from carto/main, diff-verified; adapt only where an upstream API change forces it, minimal + # CARTO PATCH). Replaces "preserve the BEHAVIOR, not necessarily the exact file versions".
  • Removes the harmful heuristics: "bigger file = probably correct" deleted; "upstream TAG code WORKS" reframed as "works for upstream's call graph, not necessarily CARTO's".
  • Adds three post-resolution checks: orphan sweep, cross-file wiring (auto-merged siblings), cross-version data-flow.
  • References the regression canaries as the definition of "wired correctly".

Fixer prompt (carto-upstream-sync-ci-fixer.yml):

  • Same verbatim-first rule + orphan sweep + call-graph reframing.
  • Manifest-aware loop guard: the "3+ fixes → sync entire file from upstream" escape hatch now excludes manifest files (blind sync is how wirings get erased).

Analyzer prompt (carto-upstream-sync-customizations-analyzer.yml):

  • Judge PRESERVED_CARTO by WIRING, not string presence: an orphan / missing call site / broken data-flow is INCORRECTLY_DROPPED even when the pattern greps OK. It had reported the dead session read as PASS ✅.

Docs (CARTO_UPSTREAM_SYNC.md): new troubleshooting section with the three wiring-loss classes and detection commands.

Ordering / dependencies

Test plan

  • All three workflows parse as YAML; 73 run: blocks pass bash -n
  • No prose references stale/removed heuristics
  • Acceptance (manual, post-merge): replay the pre-resolution state of a sync against the updated resolver prompt and confirm it produces the wiring fixes (or fails loudly) without human intervention

…e reasoning

The v1.92.0 sync shipped three CARTO features that passed every automated
check yet were functionally broken (dropped call site across an auto-merged
file, an orphaned helper with no caller, and a store/lookup key that
mismatched after upstream changed id encoding). None were catchable by the
prompts' existing "does the pattern still grep" verification, and two of the
prompts' heuristics actively caused the loss. This aligns the prompts with
the reasoning that actually found and fixed those bugs.

Resolver prompt:
- Verbatim-first rule: restore CARTO blocks byte-identical from carto/main
  (diff-verified); adapt only where an upstream API change makes verbatim
  impossible, minimally and marked # CARTO PATCH. Replaces "preserve the
  BEHAVIOR, not necessarily the exact file versions", which licensed the
  paraphrase that dropped a session read-path.
- Remove harmful heuristics: delete "bigger file = probably correct" and
  reframe "upstream TAG code WORKS" as "works for upstream's call graph, not
  necessarily CARTO's - re-verify callers, attributes, data formats".
- Add three post-resolution checks the grep cannot do: orphan sweep
  (helper with no caller), cross-file wiring (auto-merged siblings), and
  cross-version data-flow (format drift across the version boundary).
- Reference the regression canaries as the definition of "wired correctly".

Fixer prompt:
- Same verbatim-first rule, orphan sweep, and call-graph reframing.
- Manifest-aware loop guard: the "3+ fixes -> sync entire file from upstream"
  escape hatch now excludes manifest files; for those, take upstream as base
  and re-apply the CARTO block verbatim (blind sync is how wirings get erased).
- Log extraction / verification already retargeted to CARTO Feature Tests in
  the agent-reaction PR this is stacked on.

Analyzer prompt:
- Judge PRESERVED_CARTO by WIRING, not string presence: an orphaned helper,
  a missing call site, or broken data-flow is INCORRECTLY_DROPPED even when
  the def/pattern greps OK. It had reported the dead session read as PASS.

CARTO_UPSTREAM_SYNC.md:
- New troubleshooting section documenting the three v1.92.0 wiring-loss
  classes with detection commands.

Manifest wiring-pattern enrichment (adding call-site patterns to
carto-features.yml) is deliberately deferred to a follow-up to avoid
clobbering the manifest changes on the open v1.92.0 sync PR.
@mateo-di mateo-di self-assigned this Jul 17, 2026
@mateo-di
mateo-di marked this pull request as ready for review August 4, 2026 12:22
@mateo-di
mateo-di merged commit a87eb5f into fix/upstream-sync-agent-reaction Aug 4, 2026
mateo-di added a commit that referenced this pull request Aug 4, 2026
…sts (#125)

* fix(ci): restore agent reaction to test failures via CARTO Feature Tests

The upstream-sync fixer and ready-checker were wired to react to "LiteLLM
Mock Tests" and "LiteLLM Linting" workflow_run completions. Both are dead
on carto/main: Mock Tests is upstream-deprecated (workflow_dispatch only)
and Linting only triggers for upstream's own branches. So no test ever
ran on a carto/main sync PR, the fixer never fired, and the only failure
signal was cloud-native integration tests three repos downstream (see the
v1.92.0 sync, where three broken CARTO wirings shipped that way).

Wire both workflows to "CARTO Feature Tests" (carto-feature-tests.yml,
CARTO's own unit-test gate) instead:

- ready-checker: replace the two dead names with Docker CI + Feature
  Tests. workflow_run only re-invokes ready-checker when a listed
  workflow completes, so listing Feature Tests is what makes sync-ready
  wait for the tests rather than just the build.
- ci-fixer: same trigger list; swap the log-extraction query from the
  dead Mock Tests workflow to Feature Tests; align the prompt's local
  verification with the gate's real command (uv + make install-test-deps
  + manifest-derived scope) instead of the stale pip/pytest invocation.

Supersedes #123 (which only removed the dead names). Depends on the
CARTO Feature Tests workflow from #124.

* docs(ci): align sync resolver/fixer/analyzer prompts with wiring-aware reasoning (#126)

The v1.92.0 sync shipped three CARTO features that passed every automated
check yet were functionally broken (dropped call site across an auto-merged
file, an orphaned helper with no caller, and a store/lookup key that
mismatched after upstream changed id encoding). None were catchable by the
prompts' existing "does the pattern still grep" verification, and two of the
prompts' heuristics actively caused the loss. This aligns the prompts with
the reasoning that actually found and fixed those bugs.

Resolver prompt:
- Verbatim-first rule: restore CARTO blocks byte-identical from carto/main
  (diff-verified); adapt only where an upstream API change makes verbatim
  impossible, minimally and marked # CARTO PATCH. Replaces "preserve the
  BEHAVIOR, not necessarily the exact file versions", which licensed the
  paraphrase that dropped a session read-path.
- Remove harmful heuristics: delete "bigger file = probably correct" and
  reframe "upstream TAG code WORKS" as "works for upstream's call graph, not
  necessarily CARTO's - re-verify callers, attributes, data formats".
- Add three post-resolution checks the grep cannot do: orphan sweep
  (helper with no caller), cross-file wiring (auto-merged siblings), and
  cross-version data-flow (format drift across the version boundary).
- Reference the regression canaries as the definition of "wired correctly".

Fixer prompt:
- Same verbatim-first rule, orphan sweep, and call-graph reframing.
- Manifest-aware loop guard: the "3+ fixes -> sync entire file from upstream"
  escape hatch now excludes manifest files; for those, take upstream as base
  and re-apply the CARTO block verbatim (blind sync is how wirings get erased).
- Log extraction / verification already retargeted to CARTO Feature Tests in
  the agent-reaction PR this is stacked on.

Analyzer prompt:
- Judge PRESERVED_CARTO by WIRING, not string presence: an orphaned helper,
  a missing call site, or broken data-flow is INCORRECTLY_DROPPED even when
  the def/pattern greps OK. It had reported the dead session read as PASS.

CARTO_UPSTREAM_SYNC.md:
- New troubleshooting section documenting the three v1.92.0 wiring-loss
  classes with detection commands.

Manifest wiring-pattern enrichment (adding call-site patterns to
carto-features.yml) is deliberately deferred to a follow-up to avoid
clobbering the manifest changes on the open v1.92.0 sync PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant