Skip to content

Run CI tests with cargo-nextest - #203

Merged
illuzen merged 2 commits into
mainfrom
ci-nextest
Nov 9, 2026
Merged

illuzen merged 2 commits into
mainfrom
ci-nextest

Conversation

@illuzen

@illuzen illuzen commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Switches the CI test runs to cargo-nextest. Motivation: on #202 a single memory-heavy test swapped on the macOS runner and, because cargo test runs all tests in one process, dragged every other test with it until the job's 30-minute limit killed the whole step with no per-test attribution.

With nextest:

  • every test runs in its own process, so one stuck or swapping test can't slow the others, and memory is released as each test exits;
  • .config/nextest.toml gives a per-test slow-timeout (flagged at 60 s, killed at 15 min in the ci profile), so a hang fails as a named test rather than a job timeout;
  • a production-circuit test group (max-threads = 1) pins the two #[ignore]d ~12 GB production-circuit builds so they never run concurrently, replacing the hand-written --test-threads=1 in slow-tests;
  • the --skip a --skip b … lists become one -E filterset.

What changed

  • .config/nextest.toml — default and ci profiles, the production-circuit group and its override.
  • .github/workflows/ci.yml — taiki-e/install-action@v2 installs cargo-nextest@${NEXTEST_VERSION} (pinned 0.9.151 in env) in test-workspace and slow-tests; the cargo test invocations become cargo nextest run --profile ci … with equivalent target/filter selection.
  • README.md — one paragraph noting CI uses nextest.

Kept on cargo test:

  • all --doc runs (nextest does not support doctests; the aggregator run in slow-tests gained an explicit --doc step so no coverage is lost);
  • cargo test -p tests -- aggregator_tests … in slow-tests: that crate generates its circuit artifacts once per process with std::sync::Once, which under process-per-test would be regenerated for every test.

Testing (local, M4)

Every converted invocation run under nextest with the same pass counts as under cargo test:

invocation result
workspace --lib --bins with the skip filterset 275 passed, 43 skipped
-p tests --test encoding_safety --test spec_differential 19 passed
-p qp-wormhole-circuit --test heap_zeroization --test formal_trace 3 passed
-p qp-wormhole-gpu --features constraint-export 27 passed, 23 skipped
-p qp-wormhole-aggregator --features gpu --lib -E 'test(gpu)' 2 passed
--features gpu --example public_batch_compare 1 passed
--run-ignored only production guards (serial via group) 2 passed

Back-to-back on the aggregator lib (the dominant binary): cargo test 95.0 s vs nextest 97.0 s. cargo nextest show-config test-groups --run-ignored all confirms both production tests land in production-circuit. taplo format --check passes on the new TOML.

Each test gets its own process and a per-test timeout, so a stuck or
swapping test fails on its own instead of taking the job to its limit.
The production-circuit test group keeps the two ~12 GB builds serial.
The tests crate's Once-based artifact fixtures stay on cargo test, as
do doctests, which nextest does not support.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread .github/workflows/ci.yml Fixed
Comment thread .github/workflows/ci.yml Fixed

@n13 n13 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Request changes
Reviewer: GPT-6.1 Sol

Reviewed head e7e95abc165fcaf8768a0d56b67c784f322b1357 against base 62459294753db7a669562aab488f09163321a2dd.

[P2] Preserve the feature-enabled GPU doctests — .github/workflows/ci.yml:78.

The previous cargo test -p qp-wormhole-gpu --features constraint-export also ran the two compile_fail doctests guarding ProofEncoder::upload and ProofEncoder::copy against overwriting FixedFieldSlice (runtime.rs:963, runtime.rs:1010). Nextest does not run doctests, and the retained workspace --doc command does not enable wgpu/constraint-export: the GPU crate has no default features and gates its runtime module on wgpu. Neither the new aggregator doctest command nor the analysis jobs restores this coverage. Consequently CI would stop detecting an API regression that permits writes to fixed circuit buffers.

Reproduced locally: cargo test --locked -p qp-wormhole-gpu --doc -- --list finds 0 tests, while cargo test --locked -p qp-wormhole-gpu --features constraint-export --doc runs and passes both compile-fail tests. Add cargo test --release --locked -p qp-wormhole-gpu --features constraint-export --doc alongside the converted GPU invocation to preserve the existing checks.

Validation:

  • git diff --check, repository Taplo checks, CI YAML parsing and locked Cargo metadata passed.
  • With pinned nextest 0.9.151, the workspace filterset and previous libtest skip selection match exactly: 275 selected tests. show-config test-groups --profile ci --run-ignored all places both production guards in production-circuit with max-threads = 1.
  • Focused debug runs under nextest's ci profile: GPU 27 passed, 23 skipped; encoding safety/spec differential 19 passed. The feature-enabled GPU doctest run above passed 2 tests.
  • Full release and production-circuit runs were not repeated locally. GitHub's workspace jobs on Ubuntu/macOS and slow-test job were still in progress; the completed checks were successful.

@n13 n13 removed the bot-review label Nov 9, 2026
The compile_fail doctests on ProofEncoder::upload/copy only exist with
the gpu crate's features enabled, which the workspace --doc run does not
do. CodeQL flags the unpinned taiki-e/install-action@v2 tag.

Co-authored-by: Cursor <cursoragent@cursor.com>
@illuzen

illuzen commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in c396c38:

  • P2: added cargo test --release --locked -p qp-wormhole-gpu --features constraint-export --doc next to the converted GPU invocation, so the two compile_fail doctests on ProofEncoder::upload/copy run again (verified locally: 2 passed). I also checked the aggregator with and without --features gpu: same 7 doctests either way, so the workspace --doc run already covers those.
  • CodeQL: taiki-e/install-action is now pinned to cfdb7ed4ccc7c7dce37759eae89dc5674b469b39 (v2.90.3, the commit the v2 tag currently resolves to).

First CI run on e7e95ab was otherwise fully green: macOS test-workspace 17m32s, ubuntu 17m05s, slow-tests 23m27s.

@illuzen
illuzen requested a review from n13 November 9, 2026 10:12

@n13 n13 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Approve
Reviewer: GPT-6.1 Sol

Reviewed head c396c386680b5a96d1ce84411531e58c9a6e3fc7 against base 62459294753db7a669562aab488f09163321a2dd, including the full diff and previous review discussion. No blocking findings remain.

The previous GPU doctest coverage finding is resolved: the explicit feature-enabled cargo test --doc command runs both ProofEncoder::upload/copy compile-fail tests. Both install-action uses now pin the verified v2.90.3 commit. The other converted invocations preserve their target and feature selection, while the artifact-sharing tests remain on libtest.

Validation in an isolated detached worktree:

  • git diff --check, repository Taplo formatting, CI YAML parsing, and locked Cargo metadata passed.
  • With pinned nextest 0.9.151, compared the workspace filterset with the previous libtest --skip selection by enumerating the actual test binaries: exactly the same 275 tests are selected.
  • show-config test-groups --profile ci --run-ignored all confirms both production guards inherit production-circuit with max-threads = 1.
  • Focused debug runs using nextest's ci profile: GPU 27 passed, 23 skipped; encoding safety/spec differential 19 passed.
  • cargo test --locked -p qp-wormhole-gpu --features constraint-export --doc: 2 passed.

Full release and production-circuit runs were not repeated locally. At posting, the current head's workspace jobs on Ubuntu/macOS, slow tests, and security audit were still running; format, Clippy/doc, formal spec/bridge, and CodeQL checks had passed. The earlier head's complete CI run also passed, but that is separate from the current head's pending results.

@n13 n13 removed the bot-review label Nov 9, 2026
@illuzen
illuzen merged commit 080bc89 into main Nov 9, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants