Skip to content

Measure heap memory alongside the benchmarks, with heaptrack - #28

Merged
GregHanson merged 2 commits into
mainfrom
memory-bench
Nov 13, 2026
Merged

GregHanson merged 2 commits into
mainfrom
memory-bench

Conversation

@GregHanson

@GregHanson GregHanson commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Adds heap-memory measurement to the Benchmarks workflow, using heaptrack, next to the criterion timings.

What it does

  • A memory job in bench.yml, with the checks trainmarks memory and watdiv memory. It uses the same targets and fixtures as the timing job, measures the base and then the change on one runner, and compares the two in the job summary. It runs in parallel with the timing job.
  • scripts/bench-memory.py.
    • run executes a release build of de under heaptrack --record-only, once per workload, each in a process of its own: peak heap is a high-water mark for the whole process, which criterion's repeated in-process iterations would blur.
    • Its workloads mirror the criterion benchmarks. For trainmarks: create from the N-Triples and from the Turtle, each query, and q1_count straight from the Turtle. For WatDiv: each template's five instances.
    • It records what heaptrack_print reports: peak heap, calls to allocation functions, temporary allocations, memory leaked, and peak RSS (which includes heaptrack's own overhead).
    • compare writes the table to the job summary, and adds a warning or error annotation for each change.
  • Thresholds.
    • MEMORY_THRESHOLD (5%) flags changes in peak heap or in allocation calls.
    • MEMORY_FAIL_ABOVE (15%) fails the job when a workload's peak heap grows by more than that.
    • Changes under 256 KB of heap or 1,000 allocations don't count, because a WatDiv query peaks at about 0.5 MB, so 5% of that is just a small buffer.
    • The base is measured with this commit's script, so a base that predates it is still compared.
  • Artifacts. The change's recordings are uploaded as memory-trainmarks and memory-watdiv, kept 90 days from main and 14 days from pull requests. Open one with heaptrack --analyze, or diff two with heaptrack_print -f NEW -d OLD.
  • make bench-memory runs the same measurement locally, into target/bench-memory/.
  • README. A "Memory" section under "Benchmarks".

Numbers, measured locally on main (94bc7f3)

These use CI's settings: RAYON_NUM_THREADS=4, trainmarks large and WatDiv 10.

Workload Peak heap Allocations Time natively, then under heaptrack
create_hdt_from_nt 60.3 MB 12.8M 4.2 s, then 35 s
create_hdt_from_ttl 61.4 MB 22.5M 11.4 s, then 34 s
q1_count_from_ttl 60.7 MB 25.7M
trainmarks queries q1–q5 0.85–18 MB 1.4M–5.6M
WatDiv C3 0.51 MB 64.3M 49 s under heaptrack
the other 19 WatDiv templates about 0.51 MB 2.2k–507k
  • Run time. One side takes about 2.3 minutes for trainmarks and about 1.1 minutes for WatDiv, under heaptrack. The recordings come to 16 MB and 8 MB.
  • WatDiv's peak heap is flat. The HDT is memory-mapped rather than allocated, and results stream out, so there the signal is the allocation count. C3 makes 64M allocations for its five instances.
  • Noise. Two runs of the same build agree to within 0.1% on allocations. Peak heap is identical for every query, and moves by −2.0% to +1.2% only for the three workloads that build an HDT on rayon's four threads. So the 5% and 15% thresholds sit well above it.

First CI run (333757e)

All ten checks passed. The memory jobs took about 3.5 minutes each, faster than locally. On the runner, one side of trainmarks takes about 55 s under heaptrack, against 138 s locally.

  • The comparison worked as intended. For trainmarks, every workload was within ±0.8% of the base, so nothing was marked. This pull request doesn't change de's code, so that's the expected result.
  • Allocation counts matched the local run exactly: 12.82M, 22.49M, 3.149M and 64.29M.
  • Peak heap didn't match. It was up to 11% lower on the runner (9.4% for create_hdt_from_nt), and about 36 KB lower for every query and WatDiv template. The second commit corrects the docs to match. Allocation counts compare across machines; peak heap compares only between runs on the same machine, which is how the job measures.

Testing

  • Two identical runs, compared at the CI thresholds: no rows marked, and exit 0.
  • Simulated regressions in a copy of one run's memory.json:
    • +20% peak heap on create_hdt_from_nt is marked "more heap" and fails.
    • +10% allocations on C3 is a warning only.
    • +30% allocations on S7 is only 664 allocations, so it isn't marked.
  • heaptrack --record-only: without it, heaptrack opens heaptrack_gui after each recording whenever a display is available, which happened locally under WSLg.
  • make -n bench-memory expands as expected, and the workflow YAML parses.

The memory job runs only when the bench.yml path filter matches, as the timing job does. This pull request changes scripts/ and bench.yml, so its first run will show both jobs.

🤖 Generated with Claude Code

GregHanson and others added 2 commits November 10, 2026 12:17
The Benchmarks workflow gets a memory job beside the timing job. It covers
the same targets, measuring the base and then the change on one runner.
scripts/bench-memory.py runs a release build of de under heaptrack once per
workload, each in a process of its own, because peak heap is a high-water
mark for the whole process:

- trainmarks: create from the N-Triples and from the Turtle, each query,
  and q1_count straight from the Turtle;
- WatDiv: each template's five instances.

For each workload it records what heaptrack_print reports: peak heap,
calls to allocation functions, temporary allocations, memory leaked and
peak RSS. It compares the change with its base in the job summary, flags
changes over MEMORY_THRESHOLD (5%), and fails when a peak heap grows by
more than MEMORY_FAIL_ABOVE (15%). Changes under 256 KB of heap or 1,000
allocations aren't counted. The change's recordings are kept as the
memory-<target> artifacts, to open with heaptrack --analyze.

Heap figures don't depend on the runner's CPU. Two runs of one build agree
to within 0.1% on allocations and to within 2% on peak heap.

make bench-memory runs the same measurement locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first CI run matched the local allocation counts exactly, but its peak
heap was up to 11% lower than a developer machine's: 9.4% lower for
create_hdt_from_nt, and about 36 KB lower for every query. So the docs no
longer say that runs from different runners compare well. Allocation counts
do; peak heap compares only between runs on the same machine, which is how
the memory job measures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GregHanson
GregHanson marked this pull request as ready for review November 13, 2026 19:31
@GregHanson
GregHanson merged commit 7f1770b into main Nov 13, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant