perf(decode): faster entity decoding

[fb55]

Faster named-entity decoding and a denser trie encoding:

• Sync + streaming decoders: inline jump-table descent, compact-run matching, and walk state in locals (flushed at chunk boundaries); linear scan for small dictionary nodes.
• Trie data ships as a dict+BPE-coded base-91 string — bundle size within ~25 B gzip of main.
• Fixes the streaming consumed count for legacy entities ending in a compact run.

html-entity-benchmarks (one process per variant, best of 5): decodeNormal geo mean 1.37M → 1.57M ops/s (+15%), decodeStrict +14% — entity-dense named texts +43% to +72% — decodeXML flat (+1%). Details in the trie commit message.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Expanded named-entity decoding support, including broader HTML, XML, and legacy entity coverage.
  • Added support for HTML4 entity names in the generated decode data.

• Bug Fixes

  • Improved entity decoding accuracy for streaming and chunked input, including boundary cases and attribute-mode handling.
  • Fixed several edge cases where legacy entities could decode incorrectly or stop with the wrong consumed length.

[coderabbitai]

Warning

Review limit reached

@fb55, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 57 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 44a254af-f9aa-409d-bb7d-738938668149

📥 Commits

Reviewing files that changed from the base of the PR and between 93e2b8b and fb60b35.

📒 Files selected for processing (9)

• scripts/trie/encode-dict.spec.ts
• scripts/trie/encode-dict.ts
• scripts/write-decode-map.ts
• src/decode-codepoint.ts
• src/decode-stream.spec.ts
• src/decode.spec.ts
• src/decode.ts
• src/internal/bin-trie-flags.ts
• src/internal/decode-shared.ts

📝 Walkthrough

Walkthrough

Refactors trie pointer arithmetic to end-relative addressing across encoder, decoder, and generated flags documentation, rewrites the decode data-map generation script to use base-91 slot-code encoding with delta+RLE atoms and BPE-merged ngrams, adds HTML4 entity data, and optimizes the entity decoder's numeric/named-entity streaming and non-streaming paths with packed numeric results and inlined fast paths, plus expanded regression tests.

Changes

Entity Decoder Refactoring

Layer / File(s)	Summary
HTML4 input and trie format documentation maps/html4.json, src/internal/bin-trie-flags.ts	Adds a JSON array of HTML4 named entities and expands header-word/branch-layout documentation to describe end-relative pointer semantics.
End-relative pointer arithmetic scripts/trie/decode-trie.ts, scripts/trie/encode-trie.ts, scripts/trie/encode-trie.spec.ts	Switches dictionary/jump-table/recursive branch pointer computation from position-relative to end-relative (branchEnd/dictEnd-based) offsets, refactors branch emission and jump-table conditions (<=63), adds a size assertion, and updates test assertions/comments to the new scheme.
Slot-code value encoding and decoder rewrite scripts/write-decode-map.ts, src/internal/decode-shared.ts	Replaces fixed dictionary encoding with base-91 slot codes, delta+RLE atom streams, and greedy BPE ngram merging; rewrites decodeTrieDict signature and logic to decode dict1/dict2 atoms, ngrams, and final data via slot indirection; adds per-node traffic computation and hot-node selection for HTML tries.
Entity decoder performance and streaming correctness src/decode.ts	Introduces NumericPacking for packed consumed/codepoint results, rewrites streaming named/numeric-entity state handlers with local accumulators and flush-before-emit, inlines jump-table descent/compact-run handling and adds fast paths in the non-streaming decode loop, and optimizes adjacent-entity scanning.
Regression tests for streaming and full entity maps src/decode-stream.spec.ts, src/decode.spec.ts	Adds exhaustive coverage of named/legacy/XML entity maps, legacy consumed-length and chunk-boundary tests, and attribute-mode terminator checks.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Input
  participant stateNamedEntity
  participant flushAndEmitLegacyOrReject
  participant DecoderState as Decoder Instance
  Input->>stateNamedEntity: write(chunk)
  activate stateNamedEntity
  stateNamedEntity->>stateNamedEntity: track local treeIndex/consumed/excess
  stateNamedEntity->>stateNamedEntity: inline jump-table descent
  stateNamedEntity->>stateNamedEntity: match compact-run entities
  stateNamedEntity->>stateNamedEntity: retain best legacy match
  alt chunk boundary or completion reached
    stateNamedEntity->>flushAndEmitLegacyOrReject: finalize state
    activate flushAndEmitLegacyOrReject
    flushAndEmitLegacyOrReject->>DecoderState: persist treeIndex/consumed/excess
    flushAndEmitLegacyOrReject->>DecoderState: emit entity or reject
    deactivate flushAndEmitLegacyOrReject
  end
  stateNamedEntity->>Input: return consumed offset
  deactivate stateNamedEntity

Loading

sequenceDiagram
  participant Script as write-decode-map
  participant BPE as bpeOptimize
  participant Encoder as tryEncodeWithSplit
  participant Decoder as decodeTrieDict
  Script->>BPE: tokenize atoms, count pairs
  BPE->>BPE: select profitable merges, promote ngrams
  Script->>Encoder: partition atoms/ngrams into dict1/dict2
  Encoder->>Encoder: deltaRleEncode atom streams
  Encoder->>Encoder: emit slot codes for final data
  Encoder->>Decoder: encoded stream + atomCount/dict1AtomCount/ngramCount/dictSize
  Decoder->>Decoder: readSlotCode, decode dict1/dict2 atoms
  Decoder->>Decoder: expand ngram slots
  Decoder->>Decoder: expand final Uint16Array output

Loading

Possibly related PRs

• fb55/entities#2199: Both PRs refactor the same trie encode/decode pointer math and the packed base-91/delta-RLE decodeTrieDict data format together with related decodeWithTrie hot-path changes.
• fb55/entities#2213: Both PRs modify src/decode.ts's stateNamedEntity streaming logic for attribute-mode legacy-entity emission governed by the excess state.
• fb55/entities#2262: Related streaming fix for legacy compact-run consumed-count handling in src/decode.ts with matching new tests in src/decode-stream.spec.ts.

Poem

A rabbit hops through trie and byte,
End-relative pointers now feel just right,
Slots of ninety-one, so slim and neat,
BPE merges make dictionaries sweet,
Named entities decoded fast and true —
Thump thump! 🐇 A refactor built anew.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 76.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the PR’s main goal: improving entity decoding performance.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf-trie-decode

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/trie/encode-trie.spec.ts`:
- Around line 61-67: The comment describing the shared-child single-branch
layout in the encode-trie spec is stale: update the block that lists indices
[5],[6],[7] to reflect that this case encodes as a 1-slot jump table (so result
length is 7 with max index 6) rather than a 3-slot dictionary; adjust the listed
slot indexes and the "dest" value accordingly and clarify that dict pointers are
relative to (branchIndex + packedKeySlots + branchCount = 1 + 1 + 1 = 3) for
this case; modify the comment near the shared-child single-branch example and
any references to result length/indices named "result" so they match the actual
encoded layout.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: a7d8beed-d5e7-4b6f-8321-6ac738f8a028

📥 Commits

Reviewing files that changed from the base of the PR and between 2d15d3b and b11840b.

⛔ Files ignored due to path filters (2)

• src/generated/decode-data-html.ts is excluded by !**/generated/**
• src/generated/decode-data-xml.ts is excluded by !**/generated/**

📒 Files selected for processing (10)

• maps/html4.json
• scripts/trie/decode-trie.ts
• scripts/trie/encode-trie.spec.ts
• scripts/trie/encode-trie.ts
• scripts/write-decode-map.ts
• src/decode-stream.spec.ts
• src/decode.spec.ts
• src/decode.ts
• src/internal/bin-trie-flags.ts
• src/internal/decode-shared.ts

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@maps/html4.json`:
- Around line 1-254: The HTML4 hot-set array in maps/html4.json is critical to
hot-node selection used by scripts/write-decode-map.ts and must be validated;
add a contract test that (1) loads maps/html4.json and asserts there are no
duplicate entries, and (2) loads maps/entities.json and asserts every string
from html4.json exists as a key in entities.json; implement the test near
existing map tests (or create a new test file) and fail the CI if duplicates or
missing entity names are detected so typos/omissions cannot silently change trie
layout or benchmarks.

In `@scripts/trie/encode-trie.spec.ts`:
- Around line 161-179: Add two unit tests that build a TrieNode with a
contiguous range of children of length 63 and length 64 (e.g., populate a Map
with keys 0..62 and 0..63), call encodeTrie on each, and assert they take
different branch-encoding paths: for the 63-length case assert the encoded
output matches the jump-table shape used elsewhere in tests (the branch
header/jump-table layout and non-zero slot pointers like in the existing
adjacent self-ref test), and for the 64-length case assert it falls back to the
dictionary encoding shape (i.e., does not have the jump-table header/layout).
Use the same encodeTrie entry point and the same structural checks (header byte
and pointer layout) relied on by other tests to lock the guard implemented
around lines 143–147.

In `@scripts/write-decode-map.ts`:
- Around line 653-661: The HTML-root validation must also ensure the header has
no value and is not a compact run: in scripts/write-decode-map.ts update the
existing check that inspects rootJumpOffset/rootBranchCount to also assert that
(data[0] & (BinTrieFlags.VALUE_LENGTH | BinTrieFlags.FLAG13)) === 0; if that
mask is non-zero throw the same Error (mentioning the header 0x...); this keeps
the invariant expected by decodeWithTrie and stateNamedEntity so the inline root
navigation won't be skipped.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: c2a98a66-18e8-47a0-9226-6bfe7cece1f7

📥 Commits

Reviewing files that changed from the base of the PR and between b11840b and 3f63936.

⛔ Files ignored due to path filters (2)

• src/generated/decode-data-html.ts is excluded by !**/generated/**
• src/generated/decode-data-xml.ts is excluded by !**/generated/**

📒 Files selected for processing (10)

• maps/html4.json
• scripts/trie/decode-trie.ts
• scripts/trie/encode-trie.spec.ts
• scripts/trie/encode-trie.ts
• scripts/write-decode-map.ts
• src/decode-stream.spec.ts
• src/decode.spec.ts
• src/decode.ts
• src/internal/bin-trie-flags.ts
• src/internal/decode-shared.ts

[coderabbitai]

🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Add a contract test for HTML4 hot-set integrity.

This list directly drives hot-node selection in scripts/write-decode-map.ts; a typo/duplicate/missing entry silently changes trie layout and benchmark behavior. Please add a test that asserts this array is duplicate-free and every name exists in maps/entities.json.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@maps/html4.json` around lines 1 - 254, The HTML4 hot-set array in
maps/html4.json is critical to hot-node selection used by
scripts/write-decode-map.ts and must be validated; add a contract test that (1)
loads maps/html4.json and asserts there are no duplicate entries, and (2) loads
maps/entities.json and asserts every string from html4.json exists as a key in
entities.json; implement the test near existing map tests (or create a new test
file) and fail the CI if duplicates or missing entity names are detected so
typos/omissions cannot silently change trie layout or benchmarks.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[Copilot]

Pull request overview

This PR significantly optimizes HTML/XML named-entity decoding performance by revising the binary trie format (end-relative pointers, compact runs) and introducing a denser, dictionary/BPE-based encoding for shipped trie data, alongside streaming correctness fixes at chunk boundaries.

Changes:

• Reworked named-entity traversal for both sync and streaming decoders (inline jump-table descent, compact-run handling, reduced per-character field stores).
• Switched shipped HTML trie data to a dictionary + BPE-coded base-91 string format, and updated the generator/encoder accordingly.
• Added broad regression coverage (full WHATWG/XML/legacy maps + streaming chunk-boundary/compact-run consumed-count invariants).

Reviewed changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/internal/decode-shared.ts	Replaces the old dict-header decoder with a slot-based atom/ngram/data stream decoder for the new base-91+BPE trie string format.
src/internal/bin-trie-flags.ts	Expands documentation to precisely describe trie node/branch layouts and end-relative pointer semantics.
src/generated/decode-data-xml.ts	Updates the generated XML trie array to match the new encoding/pointer scheme.
src/generated/decode-data-html.ts	Updates the generated HTML trie payload and call signature to the new decodeTrieDict format parameters.
src/decode.ts	Major performance refactor: inline jump-table descent, improved compact-run streaming behavior, faster numeric entity handling, and adjacent-entity scanning optimizations.
src/decode.spec.ts	Adds full-map regression tests for WHATWG entities, XML entities, and legacy entities (no semicolon), plus literal-prefix non-entity guards.
src/decode-stream.spec.ts	Adds focused streaming tests for legacy consumed-count correctness around compact runs and chunk-boundary legacy match recording.
scripts/write-decode-map.ts	Overhauls trie generation: hot/cold node heuristics, BPE merge optimization, base-91 slot coding, and updated file generation strategy.
scripts/trie/encode-trie.ts	Changes trie pointer encoding to end-relative modulo-uint16, adds per-node overhead control, and enforces 6-bit branch-length constraints.
scripts/trie/encode-trie.spec.ts	Updates/expands tests to validate end-relative pointers and 6-bit BRANCH_LENGTH behavior (including boundary cases).
scripts/trie/decode-trie.ts	Updates debug decoding to interpret end-relative pointers for both dictionary and jump-table layouts.
maps/html4.json	Adds the HTML4 entity-name set used to bias encoding decisions toward real-world “hot” entities.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/internal/bin-trie-flags.ts`:
- Around line 29-30: The compact-run documentation in BinTrieFlags is incorrect:
the encoder only uses compact runs when the chain length is greater than 2, so
update the spec comment in bin-trie-flags to describe compact runs as 3..63
rather than 2..63. Keep the wording aligned with the behavior in encode-trie.ts
and the compact-run/branch encoding description so the documented format matches
what the trie encoder actually emits.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1a8c82b2-b506-424a-9e1c-7f18b77e3b39

📥 Commits

Reviewing files that changed from the base of the PR and between b11840b and 93e2b8b.

⛔ Files ignored due to path filters (2)

• src/generated/decode-data-html.ts is excluded by !**/generated/**
• src/generated/decode-data-xml.ts is excluded by !**/generated/**

📒 Files selected for processing (10)

• maps/html4.json
• scripts/trie/decode-trie.ts
• scripts/trie/encode-trie.spec.ts
• scripts/trie/encode-trie.ts
• scripts/write-decode-map.ts
• src/decode-stream.spec.ts
• src/decode.spec.ts
• src/decode.ts
• src/internal/bin-trie-flags.ts
• src/internal/decode-shared.ts

[fb55]

Pushed a round of follow-up commits (merge of main plus five focused changes):

• test(maps) — addresses the two open CodeRabbit items: a contract spec asserting maps/html4.json is duplicate-free and every name exists in maps/entities.json, and the compact-run doc range corrected to 3..63 (the encoder only emits runs longer than 2).
• perf(decode): cheaper import-time trie decoding — decodeTrieDict was ~2.4x slower cold than main's decoder due to per-char alphabet arithmetic, [...a, ...b] spreads, and iterator allocations (not the format). Rewritten with a precomputed base-91 inverse table, flat typed arrays (atom values in a single table, ngram expansions in a shared pool), indexed loops, and a tiny warm-up decode at module load so V8 baseline-compiles the function before the real ~18k-char call. The decoded trie is byte-identical.
• fix(decode): consume overlong numeric entities correctly — the packed (consumed << 21) | cp return wrapped its 11-bit consumed field for entities of 2048+ chars, so decodeHTML('&#' + '1'.repeat(2048) + ';X') mis-consumed and leaked digits. Lengths ≥ 2047 now spill into a module-level side channel; output matches entities@7.0.1 for arbitrarily long bodies (verified across 1,536 boundary cases: decimal/hex, ±semicolon, all four decode flavors). Also clamps the streaming accumulators at 0x110000 so errors callbacks never see Infinity, and adds 2045/2046/2047/2048/4096-digit boundary tests, sync + streaming.
• refactor(decode) — the duplicated BMP fast-path expression is now codePointToString in decode-codepoint.ts (no perf change; V8 inlines it).
• test(trie) — the writer's base-91/BPE encoder moved to scripts/trie/encode-dict.ts (byte-identical output) so a new spec can round-trip it against the runtime decodeTrieDict (real maps + edge-case payloads + the whole dictSize grid); an exhaustive streaming spec runs every entity from both maps through EntityDecoder whole and char-by-char asserting agreement with the sync decoders; and write-decode-map.ts gained a staleness guard for BPE_RANK_OVERRIDES.

Fresh-process startup (min/median of 20 runs, Node 26, Apple Silicon):

	decodeTrieDict cold	full import
main	—	2276 / 2316 µs
branch before	1013 / 1062 µs	3143 / 3195 µs
branch now	531 / 584 µs (1.9x)	2466 / 2509 µs

Transient decode garbage dropped from ~945 KB to ~50 KB per process; retained memory unchanged. The import gap vs main shrank from +870 µs to +190 µs.

Differential validation vs origin/main: all entities from all three maps × {±semicolon, ±trailing alnum/=} × 4 decode flavors (170k comparisons), streaming with chunk sizes 1/2/3/whole × 3 modes (204k), and 200k random fuzz strings — zero divergences. The only intentional divergences are the >2047-char numeric bodies (200 cases), all matching entities@7.0.1. npm run build:trie / build:encode-trie regenerate the committed files byte-identically; a numeric-entity-heavy corpus benchmarks within 1% of the pre-fix branch.