PRD: Replace HDF5 schema backend with nwbscan metadata parser #29

Problem Statement

The current fast HDF5 schema backend in lazynwb does not fulfill the intended nwbscan migration goal. Recent work created a backend catalog architecture, source-identity cache substrate, table schema snapshots, request-budget tests, and Zarr metadata reader, but the migrated HDF5 schema path still uses h5py.File over a custom range-reader file object to interpret HDF5 metadata.

From the user's perspective, this means lazynwb has a “fast backend” shell but not the core fast HDF5 metadata scanner that motivated the work. Cold-cache schema inference remains much slower than the nwbscan proof of concept, individual table calls do not reuse parsed HDF5 metadata well, and the implementation violates the architectural intent that the migrated remote HDF5 schema path should parse HDF5 metadata directly from byte ranges.

Solution

Replace the HDF5 table schema implementation behind the existing backend catalog interface with the h5py-free metadata scanner from the nwbscan proof of concept.

Users should continue calling the same public APIs, especially get_table_schema and scan_nwb, but migrated remote HDF5 schema reads should no longer instantiate h5py.File. The HDF5 backend should parse superblocks, old-style group metadata, object headers, dataspaces, datatypes, dataset layout facts, attributes, and heap-backed metadata directly from range-read bytes. It should then map those facts into the existing accessor-free catalog snapshot model and persistent SQLite cache.

The solution should preserve the recent backend architecture where it is useful: the backend reader protocol, source identity model, table schema snapshots, public schema merge semantics, Zarr backend, and request-budget tests. The replacement should be focused on the HDF5 reader internals and on adding a parsed-HDF5-metadata cache so individual table calls on the same source get faster without requiring users to batch table requests.

User Stories

1. As a lazynwb user, I want remote HDF5 table schema inference to avoid h5py, so that schema reads use the fast metadata scanner rather than the old HDF5 object model.
2. As a lazynwb user, I want get_table_schema to keep its public API, so that existing notebooks and scripts do not need migration.
3. As a lazynwb user, I want scan_nwb to benefit from the h5py-free schema path, so that lazy scans do not spend unnecessary time constructing schemas.
4. As a lazynwb user, I want cold-cache schema inference for trials and units to approach the nwbscan proof-of-concept timings, so that cloud-hosted NWB exploration feels interactive.
5. As a lazynwb user, I want warm-cache schema inference to avoid range GETs, so that repeated schema reads are dominated by local cache lookup and schema materialization.
6. As a lazynwb user, I want a trials schema read to cache parsed source metadata that can help a later units schema read, so that individual table calls are still fast.
7. As a lazynwb user, I want individual table access to be optimized, so that I do not need to know about or use a multi-table API to get good performance.
8. As a lazynwb user, I want schema inference across multiple files to keep merging column types as before, so that mixed-version datasets still produce a single useful schema.
9. As a lazynwb user, I want first_n_files_to_infer_schema to keep limiting schema work, so that large collections remain controllable.
10. As a lazynwb user, I want exclude_array_columns to keep filtering array-like columns after inference, so that cached snapshots remain complete while public schemas can be smaller.
11. As a lazynwb user, I want missing-table behavior to stay the same, so that raise_on_missing and skipped files remain predictable.
12. As a lazynwb user, I want schema results for scalar columns, ragged columns, fixed-shape array columns, strings, references, metadata tables, and TimeSeries tables to match current behavior, so that the parser swap is not a behavior regression.
13. As a lazynwb user, I want the HDF5 scanner to avoid broad raw array reads, so that metadata operations do not accidentally download large spike, waveform, or timeseries data.
14. As a lazynwb user, I want source identity validation to remain strict, so that persistent cached metadata is not reused across changed objects.
15. As a lazynwb user, I want debug logs to explain cache hits, cache misses, parser coverage, request counts, byte counts, and unsupported HDF5 features, so that performance and parser failures are diagnosable.
16. As a maintainer, I want the HDF5 metadata parser isolated behind a small deep-module interface, so that parser behavior can be tested independently from public API orchestration.
17. As a maintainer, I want the HDF5 backend reader to continue returning accessor-free catalog snapshots, so that public code and cache payloads are not coupled to parser internals.
18. As a maintainer, I want the current Zarr backend to remain in place, so that this HDF5 work does not destabilize Zarr schema reads.
19. As a maintainer, I want a guard test that fails if the migrated HDF5 schema path calls h5py.File, so that the architectural constraint cannot regress silently.
20. As a maintainer, I want request-budget tests to track cold and warm HDF5 schema reads, so that performance regressions are visible.
21. As a maintainer, I want parsed metadata cache tests to prove follow-up table reads use fewer range requests, so that the individual-table optimization is locked down.
22. As a maintainer, I want structured parser errors for unsupported HDF5 constructs, so that failures point to missing parser coverage rather than falling back to slow behavior.
23. As a maintainer, I want the parser to reuse the existing range reader abstraction where practical, so that object-store support is not limited to S3-only code.
24. As a maintainer, I want the nwbscan proof-of-concept concepts mapped into lazynwb modules cleanly, so that the codebase does not inherit a monolithic scanner file.
25. As a maintainer, I want the existing SQLite cache design extended rather than replaced, so that source identity, table schema snapshots, and future cache kinds share one substrate.
26. As a maintainer, I want parser-derived dataset facts to include dtype, shape, ndim, chunks, layout, compression/filter facts where available, fill value where available, and relevant attributes, so that future read planning can build on the same catalog.
27. As a maintainer, I want old-style HDF5 group parsing support sufficient for the benchmark NWB files, so that the dynamic-routing trials and units cases validate the migration.
28. As a maintainer, I want unsupported newer HDF5 group/object-header cases to fail clearly, so that parser gaps are explicit and triageable.
29. As a performance-focused developer, I want cold-cache timing broken down by bootstrap, group resolution, object-header loading, heap reads, schema construction, and cache writes, so that future optimization targets are visible.
30. As a performance-focused developer, I want warm-cache timing broken down by identity validation, parsed metadata cache load, schema snapshot cache load, and schema conversion, so that local overhead is visible.
31. As a performance-focused developer, I want parsed metadata cache payloads to be versioned, so that parser model changes invalidate safely.
32. As a performance-focused developer, I want object-store construction and source identity behavior to avoid repeated expensive setup where safe, so that warm and follow-up calls do not pay unnecessary fixed cost.
33. As a user comparing lazynwb to nwbscan, I want benchmark numbers to make it clear that the scanner has been incorporated, so that the project performance story is credible.
34. As a user working in notebooks, I want repeated get_table_schema(path, "trials") and get_table_schema(path, "units") calls to naturally benefit from metadata caching, so that exploratory workflows are fast without new concepts.
35. As a user working across multiple NWB versions, I want schema inference to continue preserving version-specific column differences and type disagreements, so that dynamic-routing version comparisons remain accurate.
36. As a maintainer, I want the old h5py-backed accessor path to remain available as compatibility/reference behavior outside the migrated fast path, so that local fixture creation and non-migrated APIs do not need to be rewritten in this slice.
37. As a maintainer, I want public docs and ADR notes to align with implementation reality, so that “fast HDF5 backend” means h5py-free parser-backed metadata scanning.
38. As an agent working on the repo later, I want the parser module boundaries and cache contracts documented in tests and debug logs, so that future changes do not reintroduce broad HDF5 traversal.

Implementation Decisions

• Keep the current backend catalog architecture and replace only the HDF5 schema reader internals.
• Treat h5py-free remote HDF5 schema parsing as a hard architectural constraint for the migrated fast path.
• Preserve the existing public API contracts for schema inference, including multi-file behavior, missing-table behavior, internal-column behavior, array-column filtering, and schema override interactions through scan_nwb.
• Extract a deep HDF5 metadata parser module that parses byte ranges into source-level metadata and dataset descriptors through a small async interface.
• Port the relevant nwbscan scanner capabilities: superblock detection, old-style group traversal, symbol table parsing, local heap parsing, object header loading, dataspace parsing, datatype parsing, attribute parsing, layout/filter/fill-value parsing where supported, global heap string handling where needed, and NWB table schema construction policy.
• Keep the range I/O abstraction generic and object-store agnostic. The scanner should use the existing obstore-backed range reader or a compatible wrapper rather than introducing an S3-only path.
• Add a parsed metadata cache separate from table schema snapshots. The parsed metadata cache should store reusable HDF5 parser state such as superblock facts, group entries, object header facts, and heap-derived metadata needed for later table schema reads.
• Keep table schema snapshots as the final-result cache keyed by source identity and exact normalized table path.
• Version parsed metadata cache payloads independently from schema snapshot payloads.
• Map parser dataset descriptors into the existing catalog snapshot model rather than exposing parser-specific objects to public code.
• Keep catalog snapshots accessor-free and live-object-free.
• Preserve strict source identity validation using version ID, strong ETag, or last-modified plus content length.
• Add debug logging at parser, cache, and range-I/O boundaries with enough detail to classify cache behavior, request budgets, bytes fetched, parser coverage, and unsupported HDF5 features.
• Add a clear failure mode for unsupported HDF5 constructs rather than silently falling back to the h5py/accessor path after a source is routed to the migrated HDF5 backend.
• Retain the accessor backend as a reference/compatibility reader, but not as an automatic fallback for migrated remote HDF5 schema reads.
• Retain the Zarr schema backend and shared catalog mapping behavior.
• Consider an internal multi-table read helper only as an optimization and testing aid; the primary real-world optimization is parsed source metadata reuse across separate individual table calls.

Testing Decisions

• Good tests should assert external behavior and architectural contracts rather than incidental implementation details. For this work, external behavior includes public schemas, request counts, byte counts, cache hit behavior, failure modes, and absence of h5py from the migrated HDF5 schema path.
• Add a guard test that makes h5py.File raise during migrated remote HDF5 schema reads and verifies schema inference still succeeds for supported fixtures or remote test inputs.
• Add parity tests comparing parser-derived table schemas against the existing expected schemas for trials and units.
• Add tests for scalar columns, string columns, HDF5 reference-like columns mapped to strings where current behavior expects that, ragged/list columns, fixed-shape array columns, metadata tables, TimeSeries with explicit timestamps, TimeSeries with starting time plus rate, and unaligned TimeSeries sidecar columns.
• Add request-budget tests for cold-cache HDF5 schema reads that track range GETs, fetched bytes, elapsed time, and column counts.
• Add warm-cache tests proving schema snapshot hits avoid range GETs.
• Add parsed metadata cache tests proving a follow-up individual table schema read after a prior table/path/metadata scan uses fewer range GETs than a fully cold read.
• Add cache invalidation tests for parsed metadata payload version mismatch and source identity mismatch.
• Add structured-error tests for unsupported superblock, unsupported group style, unsupported object header feature, undecodable datatype, and missing table path.
• Add tests around public multi-file schema merge semantics so the parser replacement does not alter type conflict handling.
• Reuse prior art from the existing cache tests, catalog backend tests, HDF5 backend parity tests, range reader tests, remote schema budget tests, and the nwbscan request-budget and parsed-metadata-cache tests.
• Avoid testing parser private helper functions when an end-to-end table schema or parser-facing deep-module test can cover the behavior through a stable interface.
• Keep network tests opt-in where they depend on external S3, and use buffer/local fixtures for deterministic parser coverage where practical.

Out of Scope

• Rewriting table data materialization to avoid h5py.
• Replacing all FileAccessor behavior.
• Replacing public get_internal_paths with a full catalog-backed path-summary API, except where parsed metadata cache support makes it natural to reuse scanner primitives.
• Changing the public get_table_schema or scan_nwb API shape.
• Adding a user-facing multi-table schema API.
• Supporting every HDF5 feature in the first parser migration; unsupported features should fail explicitly with structured parser errors.
• Reworking the Zarr backend beyond any shared catalog model adjustments required by the HDF5 parser mapping.
• Optimizing raw array reads, predicate pushdown, or LazyFrame execution beyond the schema path.

Further Notes

The current local commits are valuable scaffolding but not the completed nwbscan incorporation. The backend reader seam, catalog dataclasses, source identity model, SQLite schema snapshot cache, public schema merge integration, and request-budget tests should be retained. The missing piece is the h5py-free HDF5 metadata parser and parsed metadata cache.

The existing ADR already states that the HDF5 backend should parse HDF5 metadata needed for catalog snapshots and avoid broad raw array reads. This PRD brings the implementation into alignment with that ADR.

The benchmark target discussed in context is that nwbscan achieved roughly 2 seconds cold cache and 0.44 seconds warm cache for all tested trials and units schema reads, while the current lazynwb backend measured roughly 5 seconds cold and 1.1 to 1.3 seconds warm. The performance target for this PRD is to recover the proof-of-concept performance profile without changing user-facing APIs.