Check-run evaluations fan out across stacked PRs and exhaust the GitHub App rate limit

[acornet-plume]

Hi there, Palantir-alumni speaking :)

Over the past weeks, policy-bot often exhausted the GitHub App installation primary rate limit (5,000 req/h on GitHub Team) on our stack. Opening this ticket to investigate.

Symptoms

• Details pages returned 500 Internal Server Error (failed to get user permission level → GitHub 403 rate limit).
• PR evaluations failed with Error loading policy from <org>/<repo>@<branch> and Failed to post repo status.
• The webhook handler returned 503 under load; GitHub-Hookshot then retried, which made the spike worse.

Our setup

• Self-hosted policy-bot (Cloud Run), GitHub App plume-policy-bot.
• Org on GitHub Team (not Enterprise Cloud) → 5k req/h installation floor.
• Monorepo with Graphite stacked PRs/branches (a mid-stack commit SHA is contained in every branch above it).
• CI via depot.dev: typically ~32 check runs per PR .
• .policy.yml uses has_status on a single check, eslint-disable approval (success vs neutral). That is the only status we actually need for policy.
• App is subscribed to check_run (and other standard events from the README).

A typical event that blows the quota: gt restack on a 20–25 PR stack, which rewrites heads from that point up and starts a full CI matrix on each PR.

What we saw in logs

For 2026-09-02 10:00–11:00 UTC (one restack + CI):

Signal	Value
POST /api/github/hook	~5,786
Of those, 503	~1,936
Peak minute (10:00)	~2,064 webhooks
Rate-limit errors that hour	~588
Dominant handler in error stacks	CheckRun.Handle

On a single commit SHA we counted 50–70 Setting "policy-bot" status … lines in ~8 minutes. Hottest SHAs were mid-stack commits from the restacked Graphite stack. GitHub’s “PRs for this commit” list for those SHAs included every PR above that commit.

Webhook volume from check_run completions far exceeded pull_request / review traffic.

Hypothesis (from reading policy-bot source, not a confirmed root-cause analysis)

An agent in our loop read current develop (check_run.go, eval_context.go, base.go). The following reading would explain the numbers — we’d like you to confirm or correct it:

1. CheckRun.Handle evaluates every PR in check_run.pull_requests.
2. Those PRs are “PRs that contain the SHA,” not “PRs whose head is the SHA.” On a Graphite stack, one successful check on a mid-stack commit therefore schedules a full eval for every PR above it.
3. Each of those evals uses TriggerStatus. Because our policy includes has_status, ParseConfig does not skip — so every successful Depot check (lint, tests, …) re-runs the full policy, not only eslint-disable approval, the only check actually referenced by has_status
4. Base.Evaluate → NewEvalContext (load PR, fetch .policy.yml) runs before the trigger skip, so even a future “policy doesn’t need TriggerStatus” change would still spend API calls per PR per check.
5. Rough bound for one full restack, if all 32 checks succeed:
   32 × n(n+1)/2 evals (n = stack depth). For n=25 that is ~10k evals, above a 5k installation cap.

We are not claiming this is a bug versus intended behavior, it matches the comment in check_run.go about PRs that contain the SHA. It does make stacked-PR + dense Checks-API CI a hard quota problem on teams.

Soft suggestions

If you agree with the reading, two changes would help a lot (either would have kept us under 5k):

1. Only evaluate PRs whose head SHA equals the check-run head SHA
   Skip ancestor / stacked PRs that merely contain the commit. pull_request.synchronize on the rewritten heads would still eval those PRs.

2. Allowlist check names and/or apps for TriggerStatus
   e.g. only eslint-disable approval, or only checks that appear in the loaded policy’s has_status / has_workflow predicates. Everything else could no-op before NewEvalContext.

Short-term we will likely unsubscribe check_run and workarroud our need for has_status so we are not status-triggered on every Depot job.

Happy to provide more context if needed, thanks! :)

[bluekeyes]

Sorry about the rate limit issues; the increased use of stacked PRs has been revealing all kinds of interesting edge cases recently. Policies that use status conditions are a known performance liability because status events tend to happen more frequently than other events and can have write amplification properties (statuses from once source cause other integrations to post more statuses.)

The two suggestions sound plausible to me:

1. The intent is that we only process PRs where the status applies to the head commit, and that was my understanding of the webhook payload. I'll want to confirm that the payload actually includes all PRs where the commit is in the unmerged history, but if that's true, I think this is a clear bug. Fixing it has some cost, in that we'll probably need to fetch each PR to check the head commit, but we were going to do that anyway if we proceeded with evaluation.
2. Checking the incoming status or workflow against the policy has some precedent: the issue handler checks whether the incoming comment can influence approval. However, it does add some complexity to the handler logic and doesn't fit cleanly in to the existing trigger model. If we went with this approach, I think the design would need some consideration: do we special-case another handler or do we generalize the trigger mechanism?

Disabling the event should work for now and has_status will still work in policies, as long as there's some other event that usually triggers an evaluation after that status arrives.

[acornet-plume]

Thanks for the quick response @bluekeyes !

Good to hear if the first optimization is a path you'd be happy with, please keep up posted.

Disabling the event should work for now and has_status will still work in policies, as long as there's some other event that usually triggers an evaluation after that status arrives.

I just gave it a try and I think it would be too painful. In our case the successful check relaxes the policy, so on the happy path (most PRs), this would require a manual trigger, e.g. manually open the policy bot detail page after the CI has completed. This would be quite painful for large stacks, especially during restacks where every PR in the stack gets updated.

We still have a decent workaround : we used to have our policy depend on the CI check "is event X true?" and require an approval if not; we can just extract that behavior into the CI check : "is event X true?" becomes "is event X true or has the PR been approved?" and require that check to pass for merging.