Skip to content

Speed up backend schema handling - #36

Draft
bjhardcastle wants to merge 84 commits into
mainfrom
codex/fast-schema-backends
Draft

bjhardcastle wants to merge 84 commits into
mainfrom
codex/fast-schema-backends

Conversation

@bjhardcastle

Copy link
Copy Markdown
Owner

Summary

  • add fast backend catalog and SQLite snapshot caching work
  • add obstore range reading plus richer HDF5/Zarr schema backends
  • add coverage and defaults around remote/network schema behavior

Notes

  • Draft PR opened for continued iteration.
  • Local uncommitted files were left uncommitted: .vscode/launch.json, .DS_Store, benchmarks/test_schema.py.

Refs #82, #83, #84, #85, #86, #87, #88, #89.

Add remote non-consolidated Zarr metadata discovery, batched Zarr reads, bounded chunk transfer concurrency, persistent metadata payload caches, centralized process cache clearing, deeper backend reader seams, normalized scan/read-plan naming, and shared async/Zarr catalog accessor paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant