Add hdf5-meta extension#2
Open
daltoncass wants to merge 1 commit into
Open
Conversation
Optional HDF5 metadata sidecar for Recordings with very large captures or annotations arrays. The sidecar stores a columnar, performance-optimized duplicate of the global/captures/annotations metadata so tools can read metadata as numpy columns instead of re-parsing JSON in tight loops. The JSON .sigmf-meta file remains the complete, authoritative source of truth; the sidecar is optional, derived, and digest-verified so a stale one is detected and ignored. Includes a Scope and Non-Goals section clarifying that this is a metadata-only cache: it is not a sample-data container, does not represent multiple channels or arrays, and leaves both multichannel mechanisms (core:num_channels and SigMF Collections) untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
|
LGTM! The whole point of this repo was to lower the bar for new extensions, and then give them time to see if anyone else uses them |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the
hdf5-metaextension: an OPTIONAL HDF5 sidecar that stores a columnar, performance-optimized duplicate of a Recording'sglobal/captures/annotationsmetadata. It enables faster loads and tighter iteration for Recordings with very largecaptures/annotationsarrays (RF survey, ML, signal-detection workflows) by letting tools read metadata as numpy columns instead of re-parsing JSON.The JSON
.sigmf-metafile remains the complete, authoritative, SigMF-Compliant source of truth. The sidecar is optional, derived, and digest-verified against the JSON, so a stale sidecar is detected and ignored — removing it always leaves a compliant Recording.Scope (see §0.1): this is a metadata-only cache. It is not a sample-data container, does not represent multiple channels or arrays, and leaves both multichannel mechanisms (
core:num_channelsand SigMF Collections) untouched — it composes with Collections per-Recording.Originally proposed in sigmf/SigMF#354 and drafted as sigmf/SigMF#355; moved here per the maintainer's guidance that extensions start in community-extensions and migrate to core if adopted by multiple entities. A reference implementation is in sigmf/sigmf-python#157.
🤖 Generated with Claude Code