Speed up backend schema handling

.gitignore

@@ -167,6 +167,7 @@ cython_debug/
 
 .sandbox.py
 .scratch/
+.autoresearch/
 
 tests/files/
-.github/copilot-instructions.md
+.github/copilot-instructions.md

README.md

@@ -78,7 +78,8 @@ lazynwb.get_df('my_file.nwb', '/trials')
 lazynwb.get_df('s3://bucket/session.nwb', '/trials')
 
 # DANDI archive
-lf = lazynwb.scan_dandiset('000363', '/trials')
+dandi_sources = lazynwb.get_dandi_sources('000363', version='0.231012.2129')
+lf = lazynwb.scan_nwb(dandi_sources, '/trials')
 ```
 
 ### Basic benchmarks
@@ -125,6 +126,19 @@ to reproduce or run against your own files:
 python benchmarks/streaming_benchmark.py [NWB_PATH]
 ```
 
+Schema inference latency for the remote dynamic-routing workload can be measured
+with [benchmarks/schema_benchmark.py](benchmarks/schema_benchmark.py):
+```
+uv run python benchmarks/schema_benchmark.py
+LAZYNWB_SCHEMA_BENCH_JSON=metrics.json uv run python benchmarks/schema_benchmark.py
+LAZYNWB_SCHEMA_BENCH_UNITS_SOURCES_FILE=tests/paths.txt uv run python benchmarks/schema_benchmark.py
+```
+
+The schema benchmark uses an isolated temporary catalog cache, defaults to
+anonymous public object-store access, and reports cold/warm totals plus per-source
+range GET counts, fetched bytes, and timings. Useful environment variables are
+documented in the script docstring.
+
 ## Why not to use lazynwb
 
 - some convenience features of `pynwb` will not be available, for example object references in tables
@@ -145,6 +159,18 @@ df = lazynwb.get_df('my_file.nwb', '/intervals/trials')
 Use `get_internal_paths` to find available paths if you're not sure what's in a file:
 ```python
 lazynwb.get_internal_paths('my_file.nwb')
+# ['/intervals/trials', '/processing/behavior/running_speed', '/units', ...]
+```
+
+Use `get` when you want lazynwb to choose the return type from the NWB container:
+```python
+df = lazynwb.get('my_file.nwb', '/units')  # pandas DataFrame by default
+ts = lazynwb.get('my_file.nwb', '/processing/behavior/running_speed', exact_path=True)
+```
+
+Force DataFrame materialization, even for a TimeSeries container:
+```python
+df = lazynwb.get('my_file.nwb', '/processing/behavior/running_speed', exact_path=True, as_df=True)
 ```
 
 ---
@@ -285,6 +311,13 @@ Get a table as polars:
 df = nwb.get_df('/units', as_polars=True)
 ```
 
+Use the general accessor when you want tables as DataFrames and TimeSeries as
+`TimeSeries` objects:
+```python
+result = nwb.get('/processing/behavior/running_speed', exact_path=True)
+df = nwb.get('/processing/behavior/running_speed', exact_path=True, as_df=True)
+```
+
 Get a summary of everything in the file:
 ```python
 nwb.describe()
@@ -348,9 +381,16 @@ Returns columns including `identifier`, `session_id`, `session_start_time`,
 See what's inside an NWB file:
 ```python
 paths = lazynwb.get_internal_paths('my_file.nwb')
-# {'/acquisition/lick_sensor_events/data': <HDF5 dataset ...>,
-#  '/intervals/trials': <HDF5 group ...>,
-#  '/units': <HDF5 group ...>,
+# ['/acquisition/lick_sensor_events',
+#  '/intervals/trials',
+#  '/processing/behavior/running_speed',
+#  '/units',
+#  ...]
+
+path_info = lazynwb.get_internal_path_info('my_file.nwb')
+# {'/acquisition/lick_sensor_events': {'is_timeseries': True, 'is_group': True, ...},
+#  '/intervals/trials': {'is_group': True, 'attrs': {'colnames': ...}, ...},
+#  '/units': {'is_group': True, 'attrs': {'colnames': ...}, ...},
 #  ...}
 ```
 
@@ -435,52 +475,136 @@ df = lazynwb.get_df('az://my-container/data/file.nwb', '/units')
 df = lazynwb.get_df('https://example.com/data/file.nwb', '/units')
 ```
 
-Configure cloud access via `lazynwb.file_io.config`:
+Configure global defaults via `lazynwb.config`:
 ```python
-from lazynwb.file_io import config
+from lazynwb import config
 
+config.use_polars = True                          # return Polars by default from get_df/get_metadata_df
 config.use_obstore = True                         # use obstore for S3/GCS/Azure (default: False)
 config.use_remfile = False                        # use remfile for HTTP byte-range requests (default: True)
 config.anon = True                                # anonymous access across backends
-config.fsspec_storage_options = {"region": "us-west-2"}  # backend-specific extras if needed
+config.fsspec_storage_options = {"request_payer": True}  # backend-specific extras if needed
 config.disable_cache = False                      # disable FileAccessor caching (default: False)
 ```
 
+For normal AWS S3 buckets, the region belongs to the bucket, not the caller's
+current AWS session. The fast HDF5 range reader discovers and caches bucket
+regions per bucket, so avoid setting a generic `{"region": "..."}` for workflows
+that may mix buckets from different AWS regions. Keep explicit region or endpoint
+settings for S3-compatible storage such as localstack, MinIO, or R2, where they
+describe that custom service rather than an AWS bucket location.
+
 ---
 
 ## DANDI archive
 
-Scan a table across all NWB files in a DANDI dandiset:
+Use DANDI as a URI discovery step: resolve NWB asset sources with
+`get_dandi_sources`, then pass those sources to the regular `lazynwb` APIs.
+This keeps DANDI-specific version and asset resolution separate from table,
+TimeSeries, and metadata reads.
+
+Draft-only dandisets need an explicit draft version. This example uses the
+DANDI:001637 draft sample:
 ```python
-lf = lazynwb.scan_dandiset(
-    dandiset_id='000363',
-    table_path='/units',
-    version='0.231012.2129',
-    max_assets=10,  # limit number of files (useful for testing)
+import lazynwb
+
+dandi_sources = lazynwb.get_dandi_sources('001637', version='draft')
+```
+
+Known DANDI asset IDs can be resolved directly, avoiding asset listing:
+```python
+dandi_sources = lazynwb.get_dandi_sources(
+    '001637',
+    version='draft',
+    asset_ids=[
+        'ca248278-e1b2-4896-ad1c-900e4506cd04',
+        '1e37bc82-fd23-4cb5-a253-e794cea932ba',
+    ],
 )
-df = lf.collect()
 ```
 
-Filter which assets to include:
+Published versions can be pinned for reproducible analyses:
 ```python
-lf = lazynwb.scan_dandiset(
+published_sources = lazynwb.get_dandi_sources(
     '000363',
+    version='0.231012.2129',
+)
+```
+
+Use the resolved sources with `get_df` for eager table reads. Large array columns
+are excluded by default, so this reads scalar columns unless you opt in to
+specific arrays:
+```python
+units = lazynwb.get_df(
+    dandi_sources[:2],
     '/units',
-    asset_filter=lambda asset: 'probe' in asset['path'],
+    include_column_names=['id', 'firing_rate'],
+)
+```
+
+Use the same sources with `scan_nwb` when you want Polars projections and filters
+to bound the remote reads before collection:
+```python
+import polars as pl
+
+lf = lazynwb.scan_nwb(dandi_sources, '/units', infer_schema_length=2)
+units = (
+    lf
+    .filter(pl.col('firing_rate') > 1.0)
+    .select('id', 'firing_rate', 'spike_times', '_nwb_path')
+    .collect()
 )
 ```
 
-Get S3 URLs for all NWB files in a dandiset:
+Use a single resolved source for TimeSeries discovery and bounded reads. The
+returned `data` and `timestamps` objects stay lazy until you slice them:
 ```python
-urls = lazynwb.get_dandiset_s3_urls('000363')
+timeseries_by_path = lazynwb.get_timeseries(dandi_sources[0], match_all=True)
+path, ts = next(iter(timeseries_by_path.items()))
+preview = ts.data[:1000]
 ```
 
-Open a single DANDI asset:
+Use the resolved sources for metadata across the same DANDI asset set:
 ```python
-accessor = lazynwb.from_dandi_asset(
+metadata = lazynwb.get_metadata_df(dandi_sources, as_polars=True)
+```
+
+Performance expectations for DANDI workflows are the same as for other remote
+NWB files:
+
+- `get_dandi_sources` reads DANDI asset metadata and resolves object-store
+  sources; it does not read NWB table or array payloads.
+- `get_df` avoids full large array reads by default with
+  `exclude_array_columns=True`. Use `include_column_names` or
+  `exclude_column_names` to keep table reads narrow.
+- `scan_nwb` lets Polars push down `.select(...)` projections and `.filter(...)`
+  predicates before `.collect()`, which is the preferred path for large remote
+  tables.
+- TimeSeries data and timestamps are returned as backend arrays. Slice bounded
+  ranges such as `ts.data[:1000]`; avoid `ts.data[:]` unless you intend to read
+  the full remote array.
+- Metadata and internal-path discovery use bounded catalog reads where possible,
+  so they should not require broad raw data traversal on supported remote HDF5
+  sources.
+
+Limit DANDI source discovery before handing sources to table or TimeSeries APIs:
+```python
+dandi_sources = lazynwb.get_dandi_sources(
     dandiset_id='000363',
-    asset_id='21c622b7-6d8e-459b-98e8-b968a97a1585',
+    version='0.231012.2129',
+    max_assets=10,
 )
+lf = lazynwb.scan_nwb(dandi_sources, '/units', infer_schema_length=2)
+```
+
+Prefer the source-first form for new workflows because the same resolved sources can
+feed `get_df`, `scan_nwb`, `get_timeseries`, and `get_metadata_df`.
+
+The opt-in DANDI:001637 integration checks exercise the draft sample against
+tables, TimeSeries, and metadata without enabling those multi-GB remote reads in
+the default test suite:
+```bash
+uv run pytest tests/test_dandi.py tests/test_dandi_tables.py tests/test_dandi_timeseries.py tests/test_dandi_metadata.py --run-dandi-integration -m "integration and dandi_sample"
 ```
 
 ---

autoresearch.toml

@@ -0,0 +1,9 @@
+# Autoresearch configuration
+# Generated by autoresearch CLI v0.3.3
+
+branch = "codex/fast-remote-zarr-v2"
+eval_command = "uv run python benchmarks/zarr_schema_benchmark.py --max-sources 1 --json-output /tmp/lazynwb-zarr-schema-benchmark.json"
+metric_direction = "lower"
+metric_name = "cold_zarr_schema_seconds"
+target_file = "src/lazynwb/_zarr/reader.py"
+time_budget = "10m"