Skip to content

perf(decode): faster entity decoding #2248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fb55 wants to merge 16 commits into
base: main
Choose a base branch
from
Open

perf(decode): faster entity decoding #2248

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
125b41b
perf(decode): faster trie dispatch, denser trie encoding
fb55 Jun 10, 2026
4c4ebe6
refactor(decode): consolidate streaming flush+reject exit paths
fb55 Jun 10, 2026
ae47019
perf(decode): adjacency scan and BMP fast path in decodeXML
fb55 Jun 10, 2026
71dd559
test(decode): chunk-boundary invariants for the resumable walk
fb55 Jun 10, 2026
a37941d
style(decode): satisfy stricter @feedic/eslint-config after rebase
fb55 Jul 1, 2026
2bb0ae2
fix(trie): tighten build-time validation of decoder invariants
fb55 Jul 1, 2026
13be75d
test(trie): assert trie-size and dict-fill invariants; pin jump-table…
fb55 Jul 1, 2026
93e2b8b
Merge branch 'main' into perf-trie-decode
fb55 Jul 1, 2026
df6d87b
Merge remote-tracking branch 'origin/main' into perf-trie-decode
fb55 Jul 1, 2026
1aa1a2d
test(maps): guard html4.json contract; correct compact-run length docs
fb55 Jul 1, 2026
cd79024
perf(decode): cheaper import-time trie decoding
fb55 Jul 1, 2026
1b8f5c9
fix(decode): consume overlong numeric entities correctly
fb55 Jul 1, 2026
63e7ef4
refactor(decode): extract codePointToString
fb55 Jul 1, 2026
59a4d1d
test(trie): pin writer-reader agreement across the trie pipeline
fb55 Jul 2, 2026
d58d60b
Merge remote-tracking branch 'origin/main' into perf-trie-decode
fb55 Jul 2, 2026
fb60b35
style: fix formatting and import order in new specs
fb55 Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 254 additions & 0 deletions maps/html4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
[
"Aacute",
"aacute",
"Acirc",
"acirc",
"acute",
"AElig",
"aelig",
"Agrave",
"agrave",
"alefsym",
"Alpha",
"alpha",
"amp",
"and",
"ang",
"Aring",
"aring",
"asymp",
"Atilde",
"atilde",
"Auml",
"auml",
"bdquo",
"Beta",
"beta",
"brvbar",
"bull",
"cap",
"Ccedil",
"ccedil",
"cedil",
"cent",
"Chi",
"chi",
"circ",
"clubs",
"cong",
"copy",
"crarr",
"cup",
"curren",
"dagger",
"Dagger",
"darr",
"dArr",
"deg",
"Delta",
"delta",
"diams",
"divide",
"Eacute",
"eacute",
"Ecirc",
"ecirc",
"Egrave",
"egrave",
"empty",
"emsp",
"ensp",
"Epsilon",
"epsilon",
"equiv",
"Eta",
"eta",
"ETH",
"eth",
"Euml",
"euml",
"euro",
"exist",
"fnof",
"forall",
"frac12",
"frac14",
"frac34",
"frasl",
"Gamma",
"gamma",
"ge",
"gt",
"harr",
"hArr",
"hearts",
"hellip",
"Iacute",
"iacute",
"Icirc",
"icirc",
"iexcl",
"Igrave",
"igrave",
"image",
"infin",
"int",
"Iota",
"iota",
"iquest",
"isin",
"Iuml",
"iuml",
"Kappa",
"kappa",
"Lambda",
"lambda",
"lang",
"laquo",
"larr",
"lArr",
"lceil",
"ldquo",
"le",
"lfloor",
"lowast",
"loz",
"lrm",
"lsaquo",
"lsquo",
"lt",
"macr",
"mdash",
"micro",
"middot",
"minus",
"Mu",
"mu",
"nabla",
"nbsp",
"ndash",
"ne",
"ni",
"not",
"notin",
"nsub",
"Ntilde",
"ntilde",
"Nu",
"nu",
"Oacute",
"oacute",
"Ocirc",
"ocirc",
"OElig",
"oelig",
"Ograve",
"ograve",
"oline",
"Omega",
"omega",
"Omicron",
"omicron",
"oplus",
"or",
"ordf",
"ordm",
"Oslash",
"oslash",
"Otilde",
"otilde",
"otimes",
"Ouml",
"ouml",
"para",
"part",
"permil",
"perp",
"Phi",
"phi",
"Pi",
"pi",
"piv",
"plusmn",
"pound",
"prime",
"Prime",
"prod",
"prop",
"Psi",
"psi",
"quot",
"radic",
"rang",
"raquo",
"rarr",
"rArr",
"rceil",
"rdquo",
"real",
"reg",
"rfloor",
"Rho",
"rho",
"rlm",
"rsaquo",
"rsquo",
"sbquo",
"Scaron",
"scaron",
"sdot",
"sect",
"shy",
"Sigma",
"sigma",
"sigmaf",
"sim",
"spades",
"sub",
"sube",
"sum",
"sup",
"sup1",
"sup2",
"sup3",
"supe",
"szlig",
"Tau",
"tau",
"there4",
"Theta",
"theta",
"thetasym",
"thinsp",
"THORN",
"thorn",
"tilde",
"times",
"trade",
"Uacute",
"uacute",
"uarr",
"uArr",
"Ucirc",
"ucirc",
"Ugrave",
"ugrave",
"uml",
"upsih",
"Upsilon",
"upsilon",
"Uuml",
"uuml",
"weierp",
"Xi",
"xi",
"Yacute",
"yacute",
"yen",
"yuml",
"Yuml",
"Zeta",
"zeta",
"zwj",
"zwnj"
]
Comment on lines +1 to +254

@coderabbitai coderabbitai Bot Jun 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Add a contract test for HTML4 hot-set integrity.

This list directly drives hot-node selection in scripts/write-decode-map.ts; a typo/duplicate/missing entry silently changes trie layout and benchmark behavior. Please add a test that asserts this array is duplicate-free and every name exists in maps/entities.json.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@maps/html4.json` around lines 1 - 254, The HTML4 hot-set array in
maps/html4.json is critical to hot-node selection used by
scripts/write-decode-map.ts and must be validated; add a contract test that (1)
loads maps/html4.json and asserts there are no duplicate entries, and (2) loads
maps/entities.json and asserts every string from html4.json exists as a key in
entities.json; implement the test near existing map tests (or create a new test
file) and fail the CI if duplicates or missing entity names are detected so
typos/omissions cannot silently change trie layout or benchmarks.

9 changes: 6 additions & 3 deletions scripts/trie/decode-trie.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,29 +90,32 @@ export function decodeNode(
* or high (odd i) byte of slot i>>1.
*/
const packedKeySlots = Math.ceil(branchLength / 2);
const branchEnd = branchIndex + packedKeySlots + branchLength;
for (let keyIndex = 0; keyIndex < branchLength; keyIndex++) {
const slot = keyIndex >> 1;
const packed = decodeMap[branchIndex + slot];
const key = (packed >> ((keyIndex & 1) * 8)) & 0xff;
const destinationIndex = branchIndex + packedKeySlots + keyIndex;
// Pointers are relative to the end of the branch data.
decodeNode(
decodeMap,
resultMap,
prefix + String.fromCharCode(key),
(destinationIndex + decodeMap[destinationIndex]) & 0xff_ff,
(branchEnd + decodeMap[destinationIndex]) & 0xff_ff,
);
}
} else {
const branchEnd = branchIndex + branchLength;
for (let index = 0; index < branchLength; index++) {
const stored = decodeMap[branchIndex + index];
if (stored !== 0) {
const code = jumpOffset + index;
const p = branchIndex + index;
// Stored offsets are end-relative, +1 (0 = no branch).
decodeNode(
decodeMap,
resultMap,
prefix + String.fromCharCode(code),
(p + stored - 1) & 0xff_ff,
(branchEnd + stored - 1) & 0xff_ff,
);
}
}
Expand Down
104 changes: 104 additions & 0 deletions scripts/trie/encode-dict.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
import { describe, expect, it } from "vitest";
import entityMap from "../../maps/entities.json" with { type: "json" };
import legacyMap from "../../maps/legacy.json" with { type: "json" };
import xmlMap from "../../maps/xml.json" with { type: "json" };
import { decodeTrieDict } from "../../src/internal/decode-shared.js";
import { encodeFullTrie, tryEncodeWithSplit } from "./encode-dict.js";
import { encodeTrie } from "./encode-trie.js";
import { getTrie } from "./trie.js";

/*
* The dict/BPE string encoder here and the runtime `decodeTrieDict` in
* src/internal/decode-shared.ts are hand-synced implementations of the same
* format. These specs round-trip real and synthetic data through both, so a
* change to either side that breaks the agreement fails loudly instead of
* corrupting the shipped trie.
*/

/**
* Encode with the writer, decode with the runtime reader, and expect the
* original data back.
* @param data Uint16 values to round-trip.
* @param dictSize Optional fixed dictSize; defaults to the full search.
*/
function roundTrip(data: Uint16Array, dictSize?: number): void {
const result =
dictSize === undefined
? encodeFullTrie(data)
: tryEncodeWithSplit(data, dictSize);
expect(result).not.toBeNull();
const decoded = decodeTrieDict(
result!.encoded,
data.length,
result!.atomCount,
result!.dict1AtomCount,
result!.ngramCount,
result!.dictSize,
);
expect(decoded).toStrictEqual(data);
}

describe("encode-dict ↔ decodeTrieDict round-trip", () => {
it("should round-trip the real HTML trie", () => {
const data = new Uint16Array(encodeTrie(getTrie(entityMap, legacyMap)));
roundTrip(data);
});

it("should round-trip the real XML trie", () => {
/*
* Shipped inline (too small for the dict), but the format must
* still round-trip; the tiny atom count needs a small dictSize.
*/
const data = new Uint16Array(encodeTrie(getTrie(xmlMap, {})));
roundTrip(data, 10);
});

it("should round-trip repeated patterns (exercises BPE ngrams)", () => {
const values: number[] = [];
for (let index = 0; index < 200; index++) {
values.push(7, 1000, 42, 7, 1000, index % 5);
}
roundTrip(new Uint16Array(values), 4);
});

it("should round-trip large deltas (escape and double-escape)", () => {
/*
* Deltas: 88 (max 1-char), 89 (escape min), 8278 (escape max),
* 8279 (double-escape min), and a jump to the uint16 max.
*/
const values = [0, 88, 177, 8455, 16_734, 65_535];
roundTrip(new Uint16Array([...values, ...values, ...values]), 3);
});

it("should round-trip long consecutive runs (RLE chunking)", () => {
/*
* A run of 200 consecutive values spans multiple RLE chunks
* (chunk cap is BASE + 1 = 92).
*/
const values: number[] = [];
for (let index = 0; index < 200; index++) values.push(index);
for (let index = 0; index < 200; index++) values.push(index);
roundTrip(new Uint16Array(values), 40);
});

it("should round-trip across the whole dictSize search grid", () => {
const values: number[] = [];
for (let index = 0; index < 500; index++) {
values.push((index * 37) % 120, index % 60);
}
const data = new Uint16Array(values);
for (let dictSize = 45; dictSize <= 75; dictSize++) {
const result = tryEncodeWithSplit(data, dictSize);
if (result === null) continue;
const decoded = decodeTrieDict(
result.encoded,
data.length,
result.atomCount,
result.dict1AtomCount,
result.ngramCount,
result.dictSize,
);
expect(decoded).toStrictEqual(data);
}
});
});
Loading