feat: Add Claude AI error analysis workflow (Epic #616 Phase 2) - #815
Conversation
Implements Epic #616 Phase 2 / sub-issue #806: a `repository_dispatch` (`event_type: analyze-error`) workflow that asks Claude (Anthropic) for a structured analysis of a Sentry-detected API error and PUTs the result back to the API callback (`PUT /api/webhooks/github/ai-result/:id`) landed in #805. - `.github/workflows/analyze-error.yml` — repository_dispatch + workflow_dispatch (with `dry_run` / `skip_callback` defaults of true so it stays green pre-secrets-rollout). Mints a short-lived installation token via `actions/create-github-app-token@v2`. Concurrency-grouped by `sentry_issue_id` so duplicate webhooks do not double-bill Anthropic. - `.github/actions/claude-analyze/action.yml` — composite action orchestrating bun install → analyze script → curl PUT (2 retries on 5xx, immediate fail on 4xx). - `.github/actions/claude-analyze/analyze.mjs` — derives keywords from title/route, runs `git grep -l` to gather up to 6 candidate files (head 80 lines each), renders `prompt.md`, calls Anthropic SDK with retry, validates output via Zod, writes JSON to `$GITHUB_OUTPUT` path. - `.github/actions/claude-analyze/schema.mjs` — Zod schema mirroring the server's `updateAiAnalysis` boundary (severity enum + suspected-file shape) so a malformed Claude response fails CI rather than writing garbage back. - `.github/actions/claude-analyze/prompt.md` — bilingual prompt template with the Epic #616 severity rubric (high/medium/low/unknown) and a fixed JSON output contract. - `__tests__/schema.test.mjs` + `fixtures/*.json` — 12 fixture cases using Node 24's built-in test runner (no new vitest workspace). - README.md documents the `client_payload` contract, required secrets, local + workflow_dispatch dry-run recipes, and retry semantics. Closes #806
|
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a new Analyze API error GitHub workflow and a local ChangesAI Error Analysis Workflow
Sequence DiagramsequenceDiagram
participant GH as GitHub Actions
participant WF as analyze-error<br/>Workflow
participant CA as claude-analyze<br/>Action
participant Repo as Repository
participant Claude as Anthropic<br/>Claude API
participant Zedi as Zedi Callback<br/>API
GH->>WF: repository_dispatch or workflow_dispatch (error metadata)
WF->>CA: invoke composite action (env/inputs)
CA->>Repo: git grep for candidate files
CA->>Repo: read file excerpts (first N lines)
CA->>Claude: render prompt + call Messages API (with retry)
Claude-->>CA: text response (may include prose + JSON)
CA->>CA: extract JSON, parse & validate (Zod)
alt Validation Success
CA->>Zedi: PUT /ai-result/{api_error_id} (if skip_callback=false, with retry rules)
Zedi-->>CA: 2xx/4xx/5xx
CA->>WF: set outputs (severity, output_path)
else Validation Failure
CA->>WF: exit non-zero (no callback PUT)
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Code Review
This pull request introduces the claude-analyze composite action, which leverages Anthropic's Claude to analyze Sentry-reported API errors and return structured JSON results to a callback endpoint. The implementation includes a Zod-based validation schema, a script for gathering repository context via git grep, and a comprehensive test suite with fixtures. Review feedback identifies a critical version error in the setup-node action, suggests adopting import.meta.dirname for modern Node.js environments, recommends Unicode support for keyword extraction to handle Japanese error titles, and proposes optimizing git grep performance by batching patterns.
| using: composite | ||
| steps: | ||
| - name: Setup Node | ||
| uses: actions/setup-node@v6 |
| import Anthropic from "@anthropic-ai/sdk"; | ||
| import { parseAndValidate } from "./schema.mjs"; | ||
|
|
||
| const HERE = path.dirname(new URL(import.meta.url).pathname); |
| "into", | ||
| ]); | ||
| const tokens = `${title} ${route}` | ||
| .split(/[^A-Za-z0-9_/.\-]+/) |
| for (const kw of keywords) { | ||
| const res = spawnSync( | ||
| "git", | ||
| [ | ||
| "grep", | ||
| "-l", | ||
| "--", | ||
| kw, | ||
| // 巨大な lockfile / 生成物 / バイナリは検索対象外。 | ||
| // Skip lockfiles, build outputs, and binaries. | ||
| ":!*.lock", | ||
| ":!*lock.json", | ||
| ":!dist/**", | ||
| ":!**/dist/**", | ||
| ":!node_modules/**", | ||
| ":!**/node_modules/**", | ||
| ":!**/*.png", | ||
| ":!**/*.jpg", | ||
| ":!**/*.svg", | ||
| ":!**/*.pdf", | ||
| ], | ||
| { cwd: workspace, encoding: "utf8", timeout: 15_000 }, | ||
| ); | ||
| if (res.status === 0 && typeof res.stdout === "string") { | ||
| for (const line of res.stdout.split("\n")) { | ||
| const trimmed = line.trim(); | ||
| if (trimmed) hits.add(trimmed); | ||
| } | ||
| } | ||
| if (hits.size >= MAX_EXCERPT_FILES * 4) break; | ||
| } |
There was a problem hiding this comment.
キーワードごとに git grep を個別に実行するのは非効率です。git grep は複数のパターン (-e) を一度に受け取ることができるため、一つのコマンドで実行することでプロセス生成のオーバーヘッドを削減できます。
const patterns = keywords.flatMap((kw) => ["-e", kw]);
const res = spawnSync(
"git",
[
"grep",
"-l",
...patterns,
"--",
":!*.lock",
":!*lock.json",
":!dist/**",
":!**/dist/**",
":!node_modules/**",
":!**/node_modules/**",
":!**/*.png",
":!**/*.jpg",
":!**/*.svg",
":!**/*.pdf",
],
{ cwd: workspace, encoding: "utf8", timeout: 15_000 },
);
if (res.status === 0 && typeof res.stdout === "string") {
for (const line of res.stdout.split("\n")) {
const trimmed = line.trim();
if (trimmed) hits.add(trimmed);
}
}There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/actions/claude-analyze/action.yml:
- Around line 120-124: The curl call that assigns http_status (the command using
http_status=$(curl ... -X PUT "${CALLBACK_URL}" -H "Authorization: Bearer
${INSTALLATION_TOKEN}" -H "Content-Type: application/json" --data-binary
"@${OUTPUT_PATH}" || echo "000")) needs explicit timeouts to avoid hung TCP
sessions; update that curl invocation to include suitable timeout flags (for
example --connect-timeout <seconds> and --max-time <seconds>) so the PUT returns
quickly on network hangs and the retry loop can proceed.
In @.github/actions/claude-analyze/analyze.mjs:
- Around line 113-132: The readEnv function currently calls
must("ANTHROPIC_API_KEY") unconditionally which fails in dry-run environments;
change the logic so dryRun is computed first (via CLAUDE_ANALYZE_DRY_RUN) and
only require the API key when dryRun is false—e.g., compute dryRun early, then
set anthropicApiKey conditionally (use must("ANTHROPIC_API_KEY") when dryRun is
false, otherwise allow empty or undefined). Reference symbols: readEnv, must,
dryRun, anthropicApiKey.
- Around line 355-357: The console.log call that prints env.title and env.route
may expose sensitive Sentry text; update the logging in analyze.mjs (the
console.log that references env.apiErrorId, env.sentryIssueId, env.title,
env.route) to avoid printing raw values — instead log metadata-only (e.g.,
redacted values, lengths, truncated safe prefix, or a stable hash) and strip
query params from route so CI logs never contain full text fields; ensure the
message still includes apiErrorId and sentryIssueId but replaces env.title and
env.route with the chosen redaction/metadata representation.
In @.github/actions/claude-analyze/prompt.md:
- Around line 36-38: Add a language hint to the fenced code block that contains
the template variable "{{repo_excerpts}}": change the opening triple-backtick to
include the language token (e.g., "text") so the block reads ```text
{{repo_excerpts}} ``` to satisfy MD040; update the fenced block in the prompt.md
content where "{{repo_excerpts}}" is rendered.
In @.github/actions/claude-analyze/schema.mjs:
- Line 59: The ai_suspected_files array schema currently allows unbounded
entries; update the definition that uses suspectedFileSchema so the array
enforces a maximum of 5 items (e.g., add a .max(5) constraint to the
z.array(...) for ai_suspected_files in schema.mjs), keeping the existing
.nullable()/.optional() behavior intact; locate the ai_suspected_files
declaration and add the max constraint to ensure payloads larger than 5 are
rejected.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 2ef91728-12c2-4fbf-8a01-d8f82526e88f
📒 Files selected for processing (12)
.github/actions/claude-analyze/README.md.github/actions/claude-analyze/__tests__/fixtures/invalid-bad-severity.json.github/actions/claude-analyze/__tests__/fixtures/invalid-missing-summary.json.github/actions/claude-analyze/__tests__/fixtures/invalid-suspected-file.json.github/actions/claude-analyze/__tests__/fixtures/valid-high.json.github/actions/claude-analyze/__tests__/fixtures/valid-low-nulls.json.github/actions/claude-analyze/__tests__/schema.test.mjs.github/actions/claude-analyze/action.yml.github/actions/claude-analyze/analyze.mjs.github/actions/claude-analyze/prompt.md.github/actions/claude-analyze/schema.mjs.github/workflows/analyze-error.yml
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9a07b0f848
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // route is nullable on the server side, so we permit empty string here. | ||
| route: process.env.CLAUDE_ANALYZE_ROUTE?.trim() ?? "", | ||
| repository: must("CLAUDE_ANALYZE_REPOSITORY"), | ||
| anthropicApiKey: must("ANTHROPIC_API_KEY"), |
There was a problem hiding this comment.
Skip Anthropic key validation during dry-run
The dry-run path is intended to work without external secrets (the workflow defaults dry_run=true and skip_callback=true), but readEnv() always requires ANTHROPIC_API_KEY before checking dryRun. In repositories/forks where that secret is intentionally absent, the analyze step fails before it can emit the stub JSON, so manual dry-run validation cannot run at all. Only enforce ANTHROPIC_API_KEY when dry_run is false.
Useful? React with 👍 / 👎.
CodeRabbit / Codex / Gemini review fixes — all minimal, no architectural
changes:
- analyze.mjs: replace `path.dirname(new URL(...).pathname)` with
`import.meta.dirname` (Node 20.11+, Windows-safe).
- analyze.mjs: switch keyword extraction regex to Unicode property
classes (`\p{L}\p{N}`, `u` flag) so non-ASCII Sentry titles
(Japanese error messages, …) keep their tokens instead of being
stripped to empty.
- analyze.mjs: collapse the per-keyword `git grep` loop into a single
`git grep -e KW1 -e KW2 ...` call. OR semantics are preserved (set
union); avoids per-keyword process startup overhead.
- analyze.mjs: only require `ANTHROPIC_API_KEY` when `dryRun` is false,
so the default `workflow_dispatch` (`dry_run=true` + `skip_callback=true`)
works in fork PRs and pre-secrets environments.
- analyze.mjs: redact `title` / `route` in the startup log; emit
`title_len` / `route_present` and a `keywords_count` only. Sentry's
data scrubbing remains the primary defense; CI logs are a separate
retention plane (defense-in-depth per Epic #616).
- analyze.mjs: cap the dry-run stub's `ai_suspected_files` at 5 entries
to keep it consistent with the new schema cap (below).
- action.yml: add `--connect-timeout 10` and `--max-time 30` to the
callback `curl` so a hung TCP session can't burn the 10-minute job
timeout on a single PUT before the retry loop progresses.
- prompt.md: add `text` language hint to the `{{repo_excerpts}}` fence
(markdownlint MD040).
- schema.mjs: add `.max(5)` to `ai_suspected_files`. The 5-entry cap
was already documented in `prompt.md` and the README; enforce it at
the schema layer so a Claude response that ignores the instruction
fails CI rather than being PUT to the API.
- __tests__: add `invalid-too-many-files.json` fixture + a 13th test
case asserting `parseAndValidate` rejects 6+ entries.
Skipped: `actions/setup-node@v6` Gemini suggestion — `v6` is the
established repo convention used 16+ times in existing workflows
(ci.yml, deploy-dev.yml, deploy-prod.yml, …) and the PR's Lint / Build
jobs already pass on the same major.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
.github/actions/claude-analyze/analyze.mjs (1)
448-449: 💤 Low valueConsider using
fileURLToPathfor cross-platform path comparison.
new URL(import.meta.url).pathnameproduces/C:/path/...on Windows, which won't matchpath.resolve(process.argv[1])(C:\path\...). Since CI runs on Linux this is fine, but usingfileURLToPathwould improve local Windows dev experience:+import { fileURLToPath } from "node:url"; + const invokedDirectly = - process.argv[1] && path.resolve(process.argv[1]) === new URL(import.meta.url).pathname; + process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/actions/claude-analyze/analyze.mjs around lines 448 - 449, The current invokedDirectly check uses new URL(import.meta.url).pathname which yields a POSIX-style path on Windows and can mismatch process.argv[1]; instead import fileURLToPath from 'url' and use fileURLToPath(import.meta.url) for the comparison so path.resolve(process.argv[1]) will correctly match across platforms; update the invokedDirectly declaration to call fileURLToPath(import.meta.url) (and add the import) while preserving the existing use of process.argv[1] and path.resolve.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/actions/claude-analyze/analyze.mjs:
- Around line 448-449: The current invokedDirectly check uses new
URL(import.meta.url).pathname which yields a POSIX-style path on Windows and can
mismatch process.argv[1]; instead import fileURLToPath from 'url' and use
fileURLToPath(import.meta.url) for the comparison so
path.resolve(process.argv[1]) will correctly match across platforms; update the
invokedDirectly declaration to call fileURLToPath(import.meta.url) (and add the
import) while preserving the existing use of process.argv[1] and path.resolve.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 1d007782-d1bf-4595-8092-9f6d638ffeb2
📒 Files selected for processing (6)
.github/actions/claude-analyze/__tests__/fixtures/invalid-too-many-files.json.github/actions/claude-analyze/__tests__/schema.test.mjs.github/actions/claude-analyze/action.yml.github/actions/claude-analyze/analyze.mjs.github/actions/claude-analyze/prompt.md.github/actions/claude-analyze/schema.mjs
✅ Files skipped from review due to trivial changes (1)
- .github/actions/claude-analyze/tests/fixtures/invalid-too-many-files.json
🚧 Files skipped from review as they are similar to previous changes (3)
- .github/actions/claude-analyze/schema.mjs
- .github/actions/claude-analyze/action.yml
- .github/actions/claude-analyze/tests/schema.test.mjs
CodeRabbit review nitpick on PR #815: the `invokedDirectly` check still used `new URL(import.meta.url).pathname`, which yields a POSIX-style `/C:/path/...` on Windows that never matches `path.resolve(process.argv[1])` (`C:\path\...`). CI runs on Linux so this didn't affect the workflow, but the local-dev path on Windows would silently skip `main()` and exit 0 with no output. `fileURLToPath` normalizes correctly across platforms and is consistent with the earlier `import.meta.dirname` migration for `HERE`.
There was a problem hiding this comment.
🧹 Nitpick comments (2)
.github/actions/claude-analyze/analyze.mjs (2)
223-229: ⚡ Quick winUse fixed-string matching for keyword grep.
On Line 223,
git grepis currently regex-based. Since keywords come from title/route, regex metacharacters (for example.) can broaden matches and reduce excerpt relevance. Prefer literal matching with-F.Proposed change
const res = spawnSync( "git", [ "grep", + "-F", "-l", ...patternFlags, "--",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/actions/claude-analyze/analyze.mjs around lines 223 - 229, The git grep invocation built with spawnSync uses regex matching via patternFlags; change it to fixed-string matching by adding the -F flag to the git grep arguments (e.g., include "-F" in the args array passed to spawnSync alongside "grep" and before ...patternFlags). Update the invocation that constructs the args (the spawnSync call and any code that builds patternFlags) so git grep runs with -F (literal matching) rather than regex.
337-369: ⚡ Quick winDisable SDK retries to keep retry logic localized.
The Anthropic SDK (
@anthropic-ai/sdkv0.92.0) defaults tomaxRetries: 2forclient.messages.create()calls. Combined with theMAX_ATTEMPTS: 2retry loop already in place, this creates nested retry logic that can generate 4+ upstream calls unexpectedly, exceeding the intended retry budget. Explicitly setmaxRetries: 0on both client instantiations (lines ~337 and 435-436) so this module controls all retry behavior.Proposed change
- const client = new Anthropic({ apiKey: env.anthropicApiKey }); + const client = new Anthropic({ apiKey: env.anthropicApiKey, maxRetries: 0 });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/actions/claude-analyze/analyze.mjs around lines 337 - 369, The Anthropic SDK's built-in retries are causing nested retries; disable them by setting maxRetries: 0 on the Anthropic client(s) so only this module's MAX_ATTEMPTS loop controls retries—update the client instantiation(s) that produce the `client` used in `client.messages.create()` (and any other instantiation that returns a `client` in this file) to pass maxRetries: 0 in the options; keep the existing retry loop (the for loop around `client.messages.create`) unchanged so retry behavior is centralized here.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/actions/claude-analyze/analyze.mjs:
- Around line 223-229: The git grep invocation built with spawnSync uses regex
matching via patternFlags; change it to fixed-string matching by adding the -F
flag to the git grep arguments (e.g., include "-F" in the args array passed to
spawnSync alongside "grep" and before ...patternFlags). Update the invocation
that constructs the args (the spawnSync call and any code that builds
patternFlags) so git grep runs with -F (literal matching) rather than regex.
- Around line 337-369: The Anthropic SDK's built-in retries are causing nested
retries; disable them by setting maxRetries: 0 on the Anthropic client(s) so
only this module's MAX_ATTEMPTS loop controls retries—update the client
instantiation(s) that produce the `client` used in `client.messages.create()`
(and any other instantiation that returns a `client` in this file) to pass
maxRetries: 0 in the options; keep the existing retry loop (the for loop around
`client.messages.create`) unchanged so retry behavior is centralized here.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 2e02a37e-3a4b-4361-96d3-56846863880b
📒 Files selected for processing (1)
.github/actions/claude-analyze/analyze.mjs
Two more CodeRabbit nitpicks on PR #815: - analyze.mjs (grepCandidateFiles): add `git grep -F` so keywords match literally. Sentry titles often contain regex metacharacters (`.`, `(`, …) and the previous regex mode was broadening matches and diluting the candidate-file ranking with unrelated hits. - analyze.mjs (Claude client): pass `maxRetries: 0` when constructing the Anthropic client. The SDK defaults to `maxRetries: 2`, which combined with the outer `callClaudeWithRetry` loop (`MAX_ATTEMPTS=2`) produced a worst-case 2×2=4 upstream calls — outside issue #806's "1〜2 回まで" budget. The outer loop now owns retries exclusively.
概要
Sentry が検知した API エラーを Claude (Anthropic) で自動解析し、構造化 JSON 結果を Zedi API のコールバックに書き戻す GitHub Actions ワークフローを実装しました。Epic #616 Phase 2 / Issue #806 の実装です。
変更点
.github/actions/claude-analyze/: 新規 composite actionanalyze.mjs: Claude 呼び出しのオーケストレーション、リポジトリコンテキスト収集、出力検証schema.mjs: Zod による出力 JSON スキーマ定義(サーバ側updateAiAnalysisと同期)prompt.md: Claude へのプロンプトテンプレート(重大度判定基準を含む)action.yml: composite action 定義、入力/出力、API コールバック処理.github/workflows/analyze-error.yml: 新規ワークフローrepository_dispatch(event_type: analyze-error) で自動起動workflow_dispatchで手動ドライラン対応.github/actions/claude-analyze/__tests__/: テストスイートschema.test.mjs: Node 24 組み込みテストランナーで動作する fixture テスト__tests__/fixtures/*.json: 有効・無効ペイロードのサンプル主な機能
dry_run=trueで Anthropic 呼び出しをスキップ、skip_callback=trueで API PUT をスキップ変更の種類
テスト方法
https://claude.ai/code/session_01B6eQAKTBtn2wxjdcRpfZhk
Summary by CodeRabbit
New Features
Tests
Documentation