Skip to content

fix(circuits): bench each protocol circuit under the scheme its VK uses - #25507

Merged
iAmMichaelConnor merged 1 commit into
nextfrom
cb/bench-scheme-classification
Sep 18, 2026
Merged

iAmMichaelConnor merged 1 commit into
nextfrom
cb/bench-scheme-classification

Conversation

@AztecBot

@AztecBot AztecBot commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Problem

The rollup circuits have not been benched since the per-kind VK change in May. bench_cmds in noir-projects/fnd/noir-protocol-circuits/bootstrap.sh picks the bb gates scheme for each artifact by grepping its name against ivc_patterns. That array does not exist: the chonk patterns are read from chonk_circuits.json into ivc_kernel_patterns and ivc_app_patterns, and write_vk matches them through the split regexes, while bench_cmds kept the old name. An unset array expands to one empty line, and an empty pattern in a grep -f file matches every input, so the first branch always wins and every artifact is benched with --scheme chonk.

bb rejects that for the rollup circuits, and run_bench.sh pipes the failure into jq without pipefail, so the command exits 0 and writes an empty bench file that bench_merge then drops. In the last merge-queue bench run (engine log) every run_bench.sh ./target/rollup_*.json --scheme chonk line is PASSED (0s); the rollup_tx_merge sub-log is just Rollup Honk proof type not supported on MegaBuilder. The chonk circuits happen to land on the right scheme, so the kernel series were unaffected.

Surfaced while reviewing #25506, which relies on the bench to show reset-variant cost changes.

Change

  • VK generation and the bench classify a circuit through one function. circuit_kind maps an artifact name to hiding, kernel, app, rollup_honk, rollup_root or ultra_honk, in the order write_vk already applied. write_vk and bench_cmds each turn the kind into their own bb flags: write_vk keeps --circuit_kind for the chonk kinds and --oracle_hash keccak for the root, neither of which bb gates takes. The combined ivc_regex export had no remaining caller and is removed.
  • run_bench.sh runs under set -euo pipefail, so a bb error fails the bench command and, through the bench engine's fail-fast, the bench run. It also resolves bb through find-bb, the AVM-enabled binary the build keys the circuits with, instead of the hard-coded non-AVM build/bin/bb: rollup_tx_base_public verifies an AVM proof and the non-AVM binary refuses it.

Effect on the bench

The rollup circuits' _gates and _opcodes series reappear. The kernel, app and hiding series are unchanged.

Tests

  • scripts/circuit_kind.test.sh pins the flavor each circuit family lands on, including the mock circuits' names, which share this script. It is wired into test_cmds, keyed on the bootstrap script, the test and the two pattern files it reads, so CI runs it under noir-protocol-circuits-tests. It exists because the build only runs the classifier for a VK that misses the cache: on a warm cache (example) every VK is downloaded and write_vk never executes, so a misclassification would otherwise surface on the next cold-cache build.
  • Run locally against the published 6.0.0-nightly.20260902 artifacts with the release bb of the same version: every non-simulated artifact measured under the flags the fixed bench_cmds assigns it. 49 of 50 succeed; the one failure is rollup_tx_base_public under the non-AVM binary ("AVM recursion is not supported in this build"), which is what the find-bb change addresses. The bench-foundation target builds bb-cpp-native, which produces bb-avm alongside bb.
  • run_bench.sh exercised directly: a correct invocation writes the two-entry bench file; a wrong-scheme invocation now exits 1.
  • ./bootstrap.sh bench_cmds against placeholder artifacts covering every class plus a _simulated one: simulated skipped, chonk kinds get --scheme chonk, rollup honk gets --scheme ultra_honk --ipa_accumulation, root and the rest get --scheme ultra_honk.
  • bash -n on both scripts.

Created by claudebox · group: slackbot · requested by Mike (@iAmMichaelConnor) · Slack thread

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Sep 17, 2026
@AztecBot
AztecBot force-pushed the cb/bench-scheme-classification branch from 67bc356 to 8783d32 Compare September 18, 2026 13:51
bench_cmds selected the bb scheme for each artifact by grepping its name against ivc_patterns,
an array that no longer exists: the chonk patterns are read into ivc_kernel_patterns and
ivc_app_patterns. An unset array expands to one empty line, and an empty pattern in a grep -f
file matches every input, so every artifact was benched with --scheme chonk. bb rejects that
for the rollup circuits ("Rollup Honk proof type not supported on MegaBuilder"), and run_bench.sh
piped the failure into jq without pipefail, so each rollup bench passed in zero seconds and
wrote an empty bench file. The rollup gate series have been absent from the bench since.

VK generation and the bench now classify a circuit through one function, circuit_kind, and each
turns the kind into its own bb flags. run_bench.sh fails on a bb error instead of hiding it, and
uses the bb the build keys the circuits with, so the public tx-base circuit, which verifies an
AVM proof, measures under bb-avm rather than failing under the non-AVM binary.

The classifier only runs in the build for a VK that misses the cache, so a test pins the flavor
each circuit family lands on, keyed on the script and the pattern files it reads.
@iAmMichaelConnor
iAmMichaelConnor enabled auto-merge (squash) September 18, 2026 14:17
@iAmMichaelConnor
iAmMichaelConnor merged commit 017942f into next Sep 18, 2026
17 of 19 checks passed
@iAmMichaelConnor
iAmMichaelConnor deleted the cb/bench-scheme-classification branch September 18, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants