BE-607: HashQL: Refactor `ModuleRegistry` into builder/immutable split with allocator-generic stdlib construction by indietyp · Pull Request #8887 · hashintel/hash

indietyp · 2026-06-19T15:14:33Z

🌟 What is the purpose of this PR?

Replaces the StandardLibrary type (12.9KB on the stack due to nested SmallVec) with a flat ModuleCache backed by MaybeUninit slots and a FiniteBitSet, and restructures ModuleRegistry construction around a builder pattern (PartialModuleRegistry). Reduces stdlib construction cost by ~20% in instructions and ~60% in L1D cache misses.

🔍 What does this change?

Module cache and construction:

Replaced nested SmallVec<(TypeId, ModuleDef)> in StandardLibrary with a flat ModuleCache using Box<[MaybeUninit<ModuleDef>]> and FiniteBitSet<CacheId, u64> for initialization tracking
Split StandardLibrary into StandardLibrary (coordinator), StandardLibraryContext (passed to define methods), and ModuleCache (shared across modules)
Added CacheId enum with one variant per stdlib module in preorder traversal order, replacing runtime TypeId linear scan with compile-time indices
ModuleDef tracks emitted_len for exact output capacity when building item slices
Direct match in build() instead of [Option<ItemKind>; 2].flatten() for item emission
Item slices allocated directly on the heap (bypassing InternSet) since module item slices are provably unique by ModuleId ownership

Registry restructuring:

Introduced PartialModuleRegistry as a mutable builder that collects modules during construction
ModuleRegistry is now immutable after construction, backed by a contiguous IdSlice instead of InternMap
Removed RwLock from the root namespace (mutable access during construction, shared reference after)
Added ModuleRegistry::builder() test helper and PartialModuleRegistry::insert_root_module() for tests that inject custom modules

Symbol and type cleanup:

Replaced all heap.intern_symbol("...") calls in stdlib modules with pre-interned sym:: constants
Replaced all raw string .opaque("::...") calls with sym::path:: nested constants
Interned::empty() for zero-generic decl! expansions and special form types
ModuleDef parameterized by allocator S for bump-scope 2.x support

Benchmark results (Apple Silicon, bump-scope 2.x):

Counter	Before	After	Delta
Instructions	440.0K	351.2K	-20.2%
L1D cache misses	1,213	487	-59.9%
Branch mispredictions	420	347	-17.4%
Walltime	23.49 us	19.57 us	-16.7%

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

do not affect the execution graph

🐾 Next steps

Potential further optimizations identified but not implemented:

Cache primitive TypeIds (number, boolean, etc.) on TypeBuilder to avoid repeated intern round-trips
Hoist repeated monomorphic TypeDefs in modules like math and special_form
Pre-size hash maps for the finite stdlib construction
Batch generic declaration construction in the decl! macro

🛡 What tests cover this?

Existing hashql-core test suite (880 unit + 326 integration + 20 doc tests) all pass
The module::namespace and module::resolver tests exercise custom module construction through the new PartialModuleRegistry builder API

❓ How to test this?

cargo test --package hashql-core
Verify all 1226 tests pass

vercel · 2026-06-19T15:14:34Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hash	Ready	Preview, Comment	Jul 8, 2026 11:27am
petrinaut	Ready	Preview	Jul 8, 2026 11:27am

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
hashdotdesign-tokens	Ignored	Preview	Jul 8, 2026 11:27am

cursor · 2026-06-19T15:14:42Z

PR Summary

High Risk
This rewrites the global module registry and stdlib construction path used for every compile-time name lookup; regressions would break resolution, prelude imports, or stdlib registration across the compiler.

Overview
Module registry is built through PartialModuleRegistry (provision IDs, insert modules, register roots, finish) and then stored as an immutable ModuleRegistry over a contiguous IdSlice, with a plain root map instead of InternMap, item interning, and an RwLock. Module item lists are &'heap [Item] on the heap rather than interned slices; lookups and traversal use registry.modules[id] and items.iter().

Standard library registration drops the nested SmallVec / TypeId scan in favor of a fixed CacheId preorder index, a ModuleCache of MaybeUninit<ModuleDef> slots with a FiniteBitSet, and StandardLibraryContext plus allocator-generic ModuleDef. Modules request cached defs and build emits items with precomputed emitted_len. define hooks take context + cache; cross-module types use cache.request instead of manifest.

Resolution and diagnostics: root import wiring reads registry.root directly; search_by_name tracks visited modules with DenseBitSet. Item::absolute_path is removed; path formatting for suggestions uses absolute_path_rev (with reverse for :: order). Tests use ModuleRegistry::builder and insert_root_module.

Symbols: stdlib paths and names move to sym:: / sym::path::; decl! and special forms use Interned::empty() when there are no generic args.

Unrelated: darwin-kperf drops as5-1 from the M5 CPU name mapping.

^{Reviewed by Cursor Bugbot for commit 5a8147e. Bugbot is set up for automated code reviews on this repo. Configure here.}

indietyp · 2026-06-19T15:14:52Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

BE-614: HashQL: Generate synthetic closure bodies for intrinsics used as first-class values #8895
BE-615: HashQL: Introduce the concept of synthetic closures in the MIR #8894
BE-607: HashQL: Refactor ModuleRegistry into builder/immutable split with allocator-generic stdlib construction #8887 👈 (View in Graphite)
BE-538: HashQL: Permission system integration #8882
BE-71: HashQL: Consolidate AST special-form expansion stages into one #8870
BE-481: HashQL: Reify null and empty tuple as unit constants #8853
BE-537, BE-537: hgraph: REST /hashql endpoint with E2E tests #8827
BE-592: HashQL: Remove old eval graph path and core value types #8841
BE-590: hgraph: Refactor entity query handlers into entity/query submodule #8840
BE-587, BE-588: HashQL: Introduce CodeGenerationContext/CodeExecutionContext split and eval-stage Interner #8839
BE-581, BE-589: HashQL: Introduce pass::lower/pass::place pipeline facades and fix cached constructor closure bug #8838
BE-585, BE-591: HashQL: Add MIR interpreter test suite and refactor diagnostic categories #8837
BE-583, BE-584, BE-586: HashQL: Add HeapPool, improve spans, and propagate Critical severity through diagnostics #8836
BE-582: HashQL: Change opaque types from invariant to covariant #8835
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),
            _ => None,


Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 2 comments.

            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),


    pub const fn from_db_name(name: &str) -> Option<Self> {
        match name.as_bytes() {
            b"a14" => Some(Self::M1),
            b"a15" => Some(Self::M2),
            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),
            _ => None,


Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),
            _ => None,


Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),


Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),
            _ => None,


Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

            b"a14" => Some(Self::M1),
            b"a15" => Some(Self::M2),
            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),
            _ => None,


Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 2 comments.

            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),


@@ -277,27 +322,15 @@ impl<'heap> ModuleRegistry<'heap> {
    /// This method requires allocation for the traversal state:
    /// - A vector for the module exploration stack
    /// - A hash set for tracking visited modules


Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 2 comments.

            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),


    /// This method requires allocation for the traversal state:
    /// - A vector for the module exploration stack
    /// - A hash set for tracking visited modules


Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 3 comments.

            b"a14" => Some(Self::M1),
            b"a15" => Some(Self::M2),
            b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),
            b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),
-            b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),
+            b"as5" | b"as5-2" => Some(Self::M5),
            _ => None,


+    /// Looks up a root-level module by name.
+    #[must_use]
+    pub fn find_by_name(&self, name: Symbol<'heap>) -> Option<Module<'heap>> {
+        let id = self.root.get(&name).copied()?;
+
+        Some(self.modules[id])
+    }


            if next == ModuleId::ROOT {
                return None;
            }

-            let module = registry.modules.index(next);
+            let module = registry.modules[next];
            next = module.parent;

            Some(module)
        })


indietyp added 10 commits June 19, 2026 11:36

chore: checkpoint

b0fcbb3

feat: replace dynamic symbols with real ones

ae1cecc

feat: module cache

62a7ee9

feat: stdlib reorganization

ae66acb

chore: lints

958b7b5

feat: speedup

d3763e1

chore: remove any locking concerns from the module system

0ef9654

chore: remove any locking concerns from the module system

dc5e716

chore: docs

f17820c

chore: revert kperf changes

8298802

Copilot AI review requested due to automatic review settings June 19, 2026 15:14

This was referenced Jun 19, 2026

BE-581, BE-589: HashQL: Introduce pass::lower/pass::place pipeline facades and fix cached constructor closure bug #8838

Merged

BE-71: HashQL: Consolidate AST special-form expansion stages into one #8870

Merged

graphite-app Bot assigned indietyp Jun 19, 2026

This was referenced Jun 19, 2026

BE-481: HashQL: Reify null and empty tuple as unit constants #8853

Merged

BE-537, BE-537: hgraph: REST /hashql endpoint with E2E tests #8827

Merged

BE-585, BE-591: HashQL: Add MIR interpreter test suite and refactor diagnostic categories #8837

Merged

github-actions Bot added area/libs Relates to first-party libraries/crates/packages (area) type/eng > backend Owned by the @backend team labels Jun 19, 2026

This was referenced Jun 19, 2026

BE-583, BE-584, BE-586: HashQL: Add HeapPool, improve spans, and propagate Critical severity through diagnostics #8836

Merged

BE-590: hgraph: Refactor entity query handlers into entity/query submodule #8840

Merged

indietyp mentioned this pull request Jun 19, 2026

BE-538: HashQL: Permission system integration #8882

Open

6 tasks

Copilot started reviewing on behalf of indietyp June 19, 2026 15:14 View session

chore: remove deprecated code

c904161

indietyp force-pushed the bm/be-607-hashql-remove-smallvec-from-the-standardlibrary-for-module branch from 6696c2c to 5d51d92 Compare July 3, 2026 11:35

vercel Bot had a problem deploying to Preview – hash July 3, 2026 11:38 Failure

Copilot AI review requested due to automatic review settings July 3, 2026 11:39

indietyp force-pushed the bm/be-607-hashql-remove-smallvec-from-the-standardlibrary-for-module branch from 5d51d92 to f62be24 Compare July 3, 2026 11:39

Copilot started reviewing on behalf of indietyp July 3, 2026 11:39 View session

vercel Bot had a problem deploying to Preview – hash July 3, 2026 11:42 Failure

vercel Bot temporarily deployed to Preview – petrinaut July 3, 2026 11:43 Inactive

Copilot AI reviewed Jul 3, 2026

View reviewed changes

Comment thread libs/@local/hashql/core/src/module/mod.rs

indietyp force-pushed the bm/be-607-hashql-remove-smallvec-from-the-standardlibrary-for-module branch from f62be24 to 558d30c Compare July 3, 2026 14:27

Copilot AI review requested due to automatic review settings July 3, 2026 14:28

indietyp force-pushed the bm/be-607-hashql-remove-smallvec-from-the-standardlibrary-for-module branch from 558d30c to d08cc92 Compare July 3, 2026 14:28

Copilot started reviewing on behalf of indietyp July 3, 2026 14:29 View session

Copilot AI reviewed Jul 3, 2026

View reviewed changes

Comment thread libs/darwin-kperf/events/src/lib.rs

Comment on lines 60 to 63

b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),

b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),

b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),

b"as5" | b"as5-2" => Some(Self::M5),

_ => None,

vercel Bot temporarily deployed to Preview – petrinaut July 3, 2026 14:34 Inactive

indietyp force-pushed the bm/be-607-hashql-remove-smallvec-from-the-standardlibrary-for-module branch from d08cc92 to 5014ae0 Compare July 3, 2026 14:36

vercel Bot temporarily deployed to Preview – petrinaut July 3, 2026 14:41 Inactive

vercel Bot deployed to Preview – hash July 3, 2026 14:50 View deployment

Copilot AI reviewed Jul 3, 2026

View reviewed changes

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread libs/darwin-kperf/events/src/lib.rs

Comment on lines 60 to 63

b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),

b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),

b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),

b"as5" | b"as5-2" => Some(Self::M5),

_ => None,

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread libs/darwin-kperf/events/src/lib.rs

b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),

b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),

b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),

b"as5" | b"as5-2" => Some(Self::M5),

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread libs/darwin-kperf/events/src/lib.rs

Comment on lines 60 to 63

b"a16" | b"as1" | b"as2" | b"as3" => Some(Self::M3),

b"as4" | b"as4-1" | b"as4-2" => Some(Self::M4),

b"as5" | b"as5-1" | b"as5-2" => Some(Self::M5),

b"as5" | b"as5-2" => Some(Self::M5),

_ => None,

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Navigation Menu

Uh oh!

BE-607: HashQL: Refactor `ModuleRegistry` into builder/immutable split with allocator-generic stdlib construction - #8887

BE-607: HashQL: Refactor `ModuleRegistry` into builder/immutable split with allocator-generic stdlib construction#8887
indietyp wants to merge 12 commits into
bm/be-538-hashql-permission-system-integrationfrom
bm/be-607-hashql-remove-smallvec-from-the-standardlibrary-for-module

indietyp commented Jun 19, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jun 19, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Jun 19, 2026 •

edited

Loading

Uh oh!

indietyp commented Jun 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

indietyp commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌟 What is the purpose of this PR?

🔍 What does this change?

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

📜 Does this require a change to the docs?

🕸️ Does this require a change to the Turbo Graph?

🐾 Next steps

🛡 What tests cover this?

❓ How to test this?

Uh oh!

vercel Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

indietyp commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

indietyp commented Jun 19, 2026 •

edited

Loading

vercel Bot commented Jun 19, 2026 •

edited

Loading

cursor Bot commented Jun 19, 2026 •

edited

Loading

indietyp commented Jun 19, 2026 •

edited

Loading