Stop cloning the agent status record on the invocation path - #3830
Merged
Conversation
kmatasfp
force-pushed
the
gol-539-15x
branch
from
September 4, 2026 10:04
94576cf to
15a087a
Compare
This comment has been minimized.
This comment has been minimized.
Contributor
|
This is probably going to be conflicting with #3820 when ported to main |
vigoo
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AgentStatusRecordownsinvocation_results, a map that gains an entry forevery invocation an agent has ever served and is never pruned. Anything that
clones the record therefore costs more on every invocation, and the executor was
cloning it roughly a dozen times per invocation:
commit_and_update_state_innerruns on every oplog commit (two or three perinvocation, more with durable host calls) and copied the record three times,
then
update_last_known_statuscopied it a fourth time and the!=checkwalked the whole map.
drain_pending_from_statuscopied it two or three times per invocation toread
pending_updates.front(),pending_invocations.first()and onetimestamp.
get_latest_metadata, to check the agent had not failed.ForwardingOplog, which wraps every agent's oplogwhether or not it has plugins, copied it every third commit
(
plugin_max_commit_count) intry_flush, and again on its timer.checkpoint_status_mid_invocationcopied it after every durable function,before the throttle that discards nearly all of those calls.
lookup_invocation_resultcopied it once per invocation (the first commit onthis branch).
The cost of serving an invocation grew with the number already served: linear
per invocation, quadratic over the agent's life, while throughput stayed flat.
That is the creep.
Measured
A throwaway in-process test (one durable
Counteragent,incrementinvoked20,000 times sequentially, debug build, Redis-backed) with
perf statcountingretired user-space instructions of the executor process per window of 1,000:
Retired instructions rather than CPU seconds because the machine is shared;
CPU per invocation tells the same story (6.4ms rising to 23.7ms unfixed, flat at
3.4 to 4.2ms fixed). Every intermediate step was measured the same way: the
first three sites above took the slope from 15.8M to 1.65M per 1,000 of
history, the forwarding oplog halved that, and the invoke path removed the rest.
The fresh-agent control is what showed the remainder was per-agent history and
not process state.
perf recordon the unfixed build late in the run put about 70% of user CPU inmalloc,freeand hashbrown's clone, iterate and drop of the(IdempotencyKey, OplogIndex)map.What changed
it and puts the result back, copying nothing.
update_status_with_new_entriesgained a
Resulttwin,fold_status_with_new_entries, that hands the recordback on the detach path so the lock never holds a placeholder. The
!=checkis replaced by "were there any new entries": the fold advances
oplog_idxonany entry and is the identity on none, so the two are equivalent.
get_non_detached_last_known_statusis nowwith_non_detached_last_known_status(|status| ...): callers read the field ortwo they need under the guard. Every caller was converted; none needed more
than a scalar, a
PendingInvocationRefor the (small)current_retry_state.StatusCheckpointer::maybe_checkpointreads the live status through its lockand copies it only once it has decided to write, and the
get_oplog_indexmarker guard moved in with it. Regression test added for the marker.
ForwardingOplogStatereconciles plugin state fromactive_pluginsandoplog_processor_checkpointsalone, and copies the record only when there isa plugin to send to.
ensure_not_failedtakes the agent status anddeleted_regions, which isall
last_errorreads, soWorkerCtx::get_last_error_and_retry_counttakes&DeletedRegionsinstead of the record. The invoke path reads both under theresident worker's status lock; a non-resident worker still goes through
get_latest_metadataas before.on_status_changedtakes the previous record's tracking bitrather than the record, and
WorkerService::set_assignment_trackingtakesthat bit rather than a record. The ephemeral guard moved to the two callers.
Recovery-index writes fire on exactly the same transitions as before; the
flusher test now also asserts the direction of each write.
No ordering changes: the write lock is held only for the synchronous fold,
the recovery-index write and dirty-marking happen after the new record is
installed, as before.
What this does not fix
AgentStatusRecord::invocation_resultsand the in-memoryWorker::invocation_resultscache still retain one entry per invocation forthe life of the agent, so RSS still grows with invocations served. Bounding
them is a retention-policy change (Bound invocation result state in worker status #3820 on
main).dirty agent and walks the map to compute its delta. That is proportional to
resident history rather than to invocation rate, and small at the history
sizes the chaos runs reach; it stops being small for agents with hundreds of
thousands of invocations. Same root cause as the retention point above.
Testing
cargo test -p golem-worker-executor --lib535 passed;cargo clippy --workspace --all-targets --no-deps -- -D warningsandcargo check --workspace --all-targetsclean.