Skip to content

Read the agent status through its Arc instead of cloning the record - #3829

Draft
kmatasfp wants to merge 1 commit into
mainfrom
gol-539-invocation-results
Draft

Read the agent status through its Arc instead of cloning the record#3829
kmatasfp wants to merge 1 commit into
mainfrom
gol-539-invocation-results

Conversation

@kmatasfp

@kmatasfp kmatasfp commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Reading one field out of AgentStatusRecord currently deep-copies the whole
record, and the record owns invocation_results: a map that gains an entry for
every invocation a worker has accepted and is never pruned. The clone therefore
copies the worker's entire invocation history, and the hottest of the sites that
does it is lookup_invocation_result, which runs once per invocation. Cost per
invocation grows with the number of invocations already served, so the total is
quadratic in a long-lived worker's invocation count.

last_known_status is an ArcSwap, and load_full() already returns a
consistent snapshot with the lifetime every one of these callers needs. Each of
them only reads. So the fix is to keep the Arc instead of cloning out of it —
seven sites in worker/mod.rs plus the NonDetachedStatus job in
state_actor.rs, which now matches the AttachedStatus job sitting next to it
that already returned an Arc.

Three call sites moved a small field out of the returned value and now clone that
field instead: pending_card_events twice and current_idempotency_key once.
None of them is the growing map.

Nothing about what is read changes, so agent behaviour is unaffected. The
Arc<AgentStatusRecord> return type is also a small guard against the same
mistake coming back, since a future .clone() on it is visible at the call site.

How it showed up

A chaos run driving 800 ops/s across ~200 agents for 12 minutes showed executor
CPU climbing steadily from start to finish and RSS growing over the run. A run of
the same shape at 100 ops/s — same agents, same duration, an eighth of the
invocations per agent — stayed flat, which is the signature of a per-invocation
cost that scales with invocation count rather than with agent count or wall time.

What this does not fix

The CPU half only. Two things still grow without bound for the life of a worker,
and neither is addressed here:

  • AgentStatusRecord::invocation_results keeps one (IdempotencyKey, OplogIndex) entry per invocation forever. Pruning it would change what a
    repeated idempotency key resolves to, so it needs its own design.
  • The in-memory Worker::invocation_results cache stores each successful
    invocation's full AgentInvocationOutput and nothing evicts it. This is the
    larger of the two and is the likelier source of the observed RSS growth.
    Eviction looks safe in principle — a missing entry falls back to
    InvocationResult::Lazy and re-reads the result from the oplog — but that is a
    behaviour change and a separate piece of work.

So a build carrying this commit should show flat executor CPU across a long run
and still show memory growing.

Testing

cargo check --workspace --all-targets clean; cargo test -p golem-worker-executor --lib 1568 passed, 0 failed.

@kmatasfp
kmatasfp requested a review from a team September 4, 2026 09:27
@netlify

netlify Bot commented Sep 4, 2026

Copy link
Copy Markdown

Deploy Preview for golemcloud canceled.

Name Link
🔨 Latest commit a6da4a3
🔍 Latest deploy log https://app.netlify.com/projects/golemcloud/deploys/6a9a98ab1adfe90008985c56

@kmatasfp
kmatasfp marked this pull request as draft September 4, 2026 09:53
@kmatasfp

kmatasfp commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Testing similar change out against 1.5.x first

@kmatasfp
kmatasfp force-pushed the gol-539-invocation-results branch from 7270a21 to a6da4a3 Compare September 4, 2026 10:08
@blacksmith-sh

blacksmith-sh Bot commented Sep 4, 2026

Copy link
Copy Markdown

Found 5 test failures on Blacksmith runners:

Failures

Test View Logs
golem.integration.GolemExamplesIntegrationSpec/ - auto-snapshot-counter View Logs
golem.integration.GolemExamplesIntegrationSpec/ - fork View Logs
golem.integration.GolemExamplesIntegrationSpec/ - snapshot-counter View Logs
golem.integration.GolemExamplesIntegrationSpec/
- snapshot-oplog-auto: Snapshotted[S] produces JSON snapshot entries in oplog
View Logs
golem.integration.GolemExamplesIntegrationSpec/
- snapshot-oplog-custom: custom saveSnapshot/
loadSnapshot produces snapshot entries in oplog
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant