Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
[workspace]
members = ["crates/*", "fuzz"]
exclude = ["benchmarks"]
resolver = "3"

[workspace.package]
Expand Down
2 changes: 2 additions & 0 deletions benchmarks/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
target/
Cargo.lock
38 changes: 38 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Real-World Devirt Benchmarks

Benchmarks measuring `devirt`'s impact on dispatch patterns from real Rust projects.

## tantivy search engine — scorer dispatch (`tantivy-devirt/`)

Reproduces tantivy's per-document scoring loop. tantivy iterates over
matching documents calling `scorer.score()` (BM25: ~10 arithmetic ops)
and `scorer.advance()` through `&mut dyn Scorer` — mandatory dynamic
dispatch since the scorer type comes from the user's search query parsed
at runtime.

Reference: [tantivy `src/query/weight.rs`](https://github.com/quickwit-oss/tantivy/blob/main/src/query/weight.rs)

Note: tantivy already manually devirtualizes its hottest paths
(`TermWeight::for_each` calls the concrete `TermScorer` directly).
The `dyn Scorer` fallback runs for uncommon query types (phrase, regex,
fuzzy). This benchmark measures the speedup devirt would provide on that
fallback path, and demonstrates what tantivy achieves manually that devirt
could automate.

### Results (TermScorer with BM25 scoring)

| n documents | devirt | plain vtable | Speedup |
|-------------|--------|-------------|---------|
| 1,000 | 3.1 µs | 9.9 µs | **3.2×** |
| 10,000 | 27.8 µs | 98.9 µs | **3.6×** |
| 100,000 | 276 µs | 961 µs | **3.5×** |
| 1,000,000 | 3.54 ms | 10.3 ms | **2.9×** |

Shuffled 80/20 hot/cold (100 scorers × 1000 docs): 333 µs vs 821 µs — **2.5×**

### Running

```bash
cd benchmarks/tantivy-devirt
cargo bench
```
Comment on lines +1 to +38

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Document the tracing benchmarks and ConstScorer results mentioned in the PR objectives.

The PR objectives explicitly describe two benchmark case studies:

  1. tantivy (documented here)
  2. tracing subscriber showing ~0% improvement (not documented)

Additionally, the PR mentions that "ConstScorer (very cheap score()) shows ~0% difference," which is valuable context missing from the README.

The negative results are just as important as the positive ones—they help users understand when devirt is and isn't beneficial. Consider adding:

  • A section documenting the tracing benchmarks and their ~0% improvement
  • Discussion of the ConstScorer results showing when devirt doesn't help
  • A conclusion section synthesizing both findings: "devirt helps when dyn calls dominate; no benefit when vtable dispatch is dwarfed by surrounding work"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@benchmarks/README.md` around lines 1 - 38, Update the benchmarks/README.md to
add a new section summarizing the tracing subscriber benchmark (mentioning the
~0% improvement) and a subsection reporting the ConstScorer results (explicitly
stating "ConstScorer (very cheap score()) shows ~0% difference"), include brief
numeric or qualitative results and a short explanation why devirt had no effect,
and add a one‑paragraph conclusion synthesizing both findings (e.g., "devirt
helps when dyn calls dominate; no benefit when vtable dispatch is dwarfed by
surrounding work"); reference existing tantivy-devirt content and use the exact
names "tracing subscriber" and "ConstScorer" so readers can correlate with the
PR objectives.

19 changes: 19 additions & 0 deletions benchmarks/tantivy-devirt/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[package]
name = "tantivy-devirt-bench"
version = "0.0.0"
edition = "2024"
publish = false

[dependencies]
devirt = { path = "../../crates/core" }

[dev-dependencies]
criterion = { version = "0.5", features = ["html_reports"] }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Criterion version mismatch with workspace.

The workspace specifies criterion = "0.8" but this benchmark uses "0.5". While this won't cause build errors (since benchmarks are excluded from the workspace), the version inconsistency could lead to confusion, API incompatibilities, or different benchmark behavior.

Consider updating to match the workspace version:

-criterion = { version = "0.5", features = ["html_reports"] }
+criterion = { version = "0.8", features = ["html_reports"] }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
criterion = { version = "0.5", features = ["html_reports"] }
criterion = { version = "0.8", features = ["html_reports"] }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@benchmarks/tantivy-devirt/Cargo.toml` at line 11, Update the benchmark
crate's dependency entry for criterion to match the workspace version: change
the current line for criterion to use version "0.8" (e.g. criterion = { version
= "0.8", features = ["html_reports"] }), then run cargo update to refresh the
lockfile and verify the benchmark builds and its HTML reports feature still
works.


[[bench]]
name = "scorer"
harness = false

[profile.bench]
lto = "thin"
codegen-units = 1
Comment on lines +17 to +19

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Profile configuration duplicates workspace settings.

These benchmark profile settings are identical to the workspace's [profile.bench] (lines 25-27 in root Cargo.toml). Since the benchmarks directory is excluded from the workspace, this duplication is necessary, but it creates a maintenance burden—changes to workspace settings must be manually synchronized here.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@benchmarks/tantivy-devirt/Cargo.toml` around lines 17 - 19, The
[profile.bench] block duplicates workspace settings (lto and codegen-units); fix
by making the benchmark crate a member of the workspace so it inherits the
workspace [profile.bench] and then remove the duplicated [profile.bench] section
(remove the lto and codegen-units keys) from this crate's Cargo.toml;
alternatively, if you must keep the crate excluded from the workspace, add a
short comment above [profile.bench] documenting why it diverges and keep only
the necessary keys to minimize future maintenance.

Loading