Repository navigation
Measure heap memory alongside the benchmarks, with heaptrack - #28
Merged
Merged
Conversation
The Benchmarks workflow gets a memory job beside the timing job. It covers the same targets, measuring the base and then the change on one runner. scripts/bench-memory.py runs a release build of de under heaptrack once per workload, each in a process of its own, because peak heap is a high-water mark for the whole process: - trainmarks: create from the N-Triples and from the Turtle, each query, and q1_count straight from the Turtle; - WatDiv: each template's five instances. For each workload it records what heaptrack_print reports: peak heap, calls to allocation functions, temporary allocations, memory leaked and peak RSS. It compares the change with its base in the job summary, flags changes over MEMORY_THRESHOLD (5%), and fails when a peak heap grows by more than MEMORY_FAIL_ABOVE (15%). Changes under 256 KB of heap or 1,000 allocations aren't counted. The change's recordings are kept as the memory-<target> artifacts, to open with heaptrack --analyze. Heap figures don't depend on the runner's CPU. Two runs of one build agree to within 0.1% on allocations and to within 2% on peak heap. make bench-memory runs the same measurement locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first CI run matched the local allocation counts exactly, but its peak heap was up to 11% lower than a developer machine's: 9.4% lower for create_hdt_from_nt, and about 36 KB lower for every query. So the docs no longer say that runs from different runners compare well. Allocation counts do; peak heap compares only between runs on the same machine, which is how the memory job measures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GregHanson
marked this pull request as ready for review
November 13, 2026 19:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds heap-memory measurement to the Benchmarks workflow, using heaptrack, next to the criterion timings.
What it does
memoryjob inbench.yml, with the checkstrainmarks memoryandwatdiv memory. It uses the same targets and fixtures as the timing job, measures the base and then the change on one runner, and compares the two in the job summary. It runs in parallel with the timing job.scripts/bench-memory.py.runexecutes a release build ofdeunderheaptrack --record-only, once per workload, each in a process of its own: peak heap is a high-water mark for the whole process, which criterion's repeated in-process iterations would blur.createfrom the N-Triples and from the Turtle, each query, andq1_countstraight from the Turtle. For WatDiv: each template's five instances.heaptrack_printreports: peak heap, calls to allocation functions, temporary allocations, memory leaked, and peak RSS (which includes heaptrack's own overhead).comparewrites the table to the job summary, and adds a warning or error annotation for each change.MEMORY_THRESHOLD(5%) flags changes in peak heap or in allocation calls.MEMORY_FAIL_ABOVE(15%) fails the job when a workload's peak heap grows by more than that.memory-trainmarksandmemory-watdiv, kept 90 days frommainand 14 days from pull requests. Open one withheaptrack --analyze, or diff two withheaptrack_print -f NEW -d OLD.make bench-memoryruns the same measurement locally, intotarget/bench-memory/.Numbers, measured locally on
main(94bc7f3)These use CI's settings:
RAYON_NUM_THREADS=4, trainmarkslargeand WatDiv10.create_hdt_from_ntcreate_hdt_from_ttlq1_count_from_ttlq1–q5C3C3makes 64M allocations for its five instances.First CI run (
333757e)All ten checks passed. The memory jobs took about 3.5 minutes each, faster than locally. On the runner, one side of trainmarks takes about 55 s under heaptrack, against 138 s locally.
create_hdt_from_nt), and about 36 KB lower for every query and WatDiv template. The second commit corrects the docs to match. Allocation counts compare across machines; peak heap compares only between runs on the same machine, which is how the job measures.Testing
memory.json:create_hdt_from_ntis marked "more heap" and fails.C3is a warning only.S7is only 664 allocations, so it isn't marked.heaptrack --record-only: without it, heaptrack opensheaptrack_guiafter each recording whenever a display is available, which happened locally under WSLg.make -n bench-memoryexpands as expected, and the workflow YAML parses.The memory job runs only when the
bench.ymlpath filter matches, as the timing job does. This pull request changesscripts/andbench.yml, so its first run will show both jobs.🤖 Generated with Claude Code