Skip to content

logdb: range-scan every address-less criterion and cap open-ended spans - #1694

Merged
otherview merged 2 commits into
tony/logdb-topic0-scanfrom
pedro/logdb-addressless-scan
Oct 28, 2026
Merged

otherview merged 2 commits into
tony/logdb-topic0-scanfrom
pedro/logdb-addressless-scan

Conversation

@otherview

Copy link
Copy Markdown
Member

Description

Stacked on #1693. Merge into that branch or cherry-pick.

#1693 evaluates narrow topic0-only criteria by a seq range scan. Two cases still get the chain-wide plan:

  1. Open-ended block ranges. ConvertRange maps an omitted to to MaxBlockNumber, so {from: head-10} counts as a ~268M-block span and keeps event_i1 (topic0=?) + temp b-tree. narrowRange now caps To at the newest block holding events (MAX(seq)) for the plan decision. It is only queried when the requested span exceeds the bound and a criterion has no address. The SQL still binds the requested bounds, so results cannot change.
  2. Every address-less criterion, not only topic0-only. Every event index except event_i0 has address between its topic columns and seq, so topic1, topic4 and topic0+topicN without an address read all matching entries chain-wide (event_i2 (topic1=? AND topic0=?) + temp b-tree). The rule is now c.Address == nil, and every topic term is written with the unary plus.

Synthetic DB (8M events, 70% one topic0, warm cache):

query before after
hot topic0, {last-10, MaxBlockNumber} 144 ms 216 µs
Transfer topic0 + busy topic1, single block 54 ms 54 µs

Fixes # (none)

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • TestEventQueryPlanPerCriteriaShape: adds topic1, topic4, topic0+topic1/2/3 shapes, which take the scan plan when narrow and keep their index when wide or unbounded
  • TestNarrowRange: the cap for nil, inverted and narrow ranges, open-ended ranges on both sides of the bound, a start past the newest block, and an empty DB
  • TestEventFilterScanEquivalence: adds topic0+topic1, topic1, mixed address/address-less sets and a {5, MaxBlockNumber} range. The index path is now built by buildEventFilterSubQuery(f, false) instead of a hand copy.
  • Reverting either fix fails the new tests (40 subtests without the cap, the topic0+topic1 cases with the old predicate)
  • ./logdb/... and ./api/... pass; golangci-lint v2.12.2, modernize -test and go mod tidy clean

Test Configuration:

  • Go Version: go1.26.5 darwin/arm64
  • Hardware: Apple Silicon (macOS)
  • Docker Version: N/A

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not added any vulnerable dependencies to my code

Every event index except event_i0 has address between its topic columns
and seq, so any criterion without an address (topic1-only, topic0+topic1,
topic4, ...) reads all its matching entries chain-wide, not just
topic0-only ones. Write every topic term with the unary plus when such a
criterion meets a narrow range.

A block range with no "to" reaches logdb as To = MaxBlockNumber and was
treated as wide. Cap To at the newest block holding events when deciding
the plan; the query still binds the requested bounds, so results are
unchanged.

The equivalence test now takes its index path from
buildEventFilterSubQuery with the scan disabled instead of a hand copy.
@codecov-commenter

codecov-commenter commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@libotony
libotony force-pushed the pedro/logdb-addressless-scan branch from 04165ec to 65cc4dc Compare October 27, 2026 09:00
@libotony

Copy link
Copy Markdown
Member

Measured both parts of this PR against a mainnet logs-v2.db snapshot (797M events, block 26,237,512, external SSD, sqlite3 3.51, opened read-only). Each case ran as the index form (current plan) and the scan form (unary plus on every topic of each address-less criterion), twice each; r1 is cold, r2 warm. The 100k-block window (26137512..26237512) holds 1.15M rows.

criterion range index form scan form
topic0 = Transfer single block 107 s 1 ms
topic0 = Transfer 100k blocks 106 s 5 ms
topic0 = rare (4.9k rows chain-wide) 100k blocks 5 ms 1.55 s / 61 ms
topic0 = Transfer, topic1 = hot (53.7M rows chain-wide) single block 56 s <1 ms
same 100k blocks 56 s 10 ms
same, ORDER BY seq DESC 100k blocks 57–81 s 24 ms
same no range (outside the bound, unchanged) 56 s —
topic0 = Transfer, topic1 = sparse (135 rows chain-wide) single block 1 ms 1 ms
same 100k blocks <1 ms 1.40 s / 65 ms
topic1 = hot only 100k blocks 56 s 9 ms
topic1 = sparse only 100k blocks 1 ms 1.36 s / 66 ms
(topic0 = Transfer) OR (address = X AND topic0 = Transfer) single block 106 s 1 ms
same 100k blocks 108 s 5 ms
address = X (busiest, 85M rows) 100k blocks 1 ms unchanged
address = X AND topic0 = Transfer 100k blocks 2 ms unchanged

So extending the rule from topic0-only to every address-less criterion is worth it: every hot value goes from a guaranteed timeout to milliseconds, including topic0+topic1, topic1 alone and mixed sets. The cost is bounded by the window, not the value: sparse values on the full 100k-block window pay ~1.4–1.5 s cold / ~65 ms warm (a scan of 1.15M rows), and are unchanged on narrow windows. That keeps a safe margin under the API timeout, so the 100k bound stays.

I pushed 65cc4dc on top: open-ended and oversized block ranges are now capped at best + 6 in ConvertRange, where the best block is already at hand (the time-unit branch already ends at the head). That makes the plan decision in logdb a pure function of the requested range again, so narrowRange no longer probes MAX(seq), and TestNarrowRange becomes a table test. Point 1 of the description should be updated accordingly.

Raw logs and plans: single-block and 100k-window queries for all shapes, plus the discovery queries for the hot/sparse values, are kept locally if anyone wants them.

@otherview
otherview merged commit 1881416 into tony/logdb-topic0-scan Oct 28, 2026
33 checks passed
@otherview
otherview deleted the pedro/logdb-addressless-scan branch October 28, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants