Conversation
|
|
||
| return attr_json | ||
|
|
||
| def getDatasetValues(self, dset_id, sel=None, dtype=None, query=None): |
There was a problem hiding this comment.
Doesn't match the signature from getDatasetValues defined in hdf5-json. That has dset_id instead of obj_id. They should be the same
| # domain doesn't exist - fall through and create it below | ||
| db.plugin = HsdsPlugin(domain, getobjs=getobjs, **kwargs) | ||
| new_domain = True | ||
| elif mode in ('r', 'r+', 'a'): |
There was a problem hiding this comment.
It looks like this grouping will result in r+ causing a new domain to be created if it didn't already exist, which isn't what the semantics imply.
There was a problem hiding this comment.
I tried this and it didn't create a new file with 'r+'. I think there' some trickery going on with the "if new_domain: db.flush()" that restricts new domain creation otherwise.
| if rsp.status_code != 200: | ||
| # file must exist | ||
| http_conn.close() | ||
| raise IOError(rsp.status_code, rsp.reason) |
There was a problem hiding this comment.
I think FileNotFoundError would be more applicable here. This is minor, but could potentially bite us if we have except FileNotFoundError blocks elsewhere.
| def run_scan(self): | ||
| MAX_WAIT = 10 | ||
| self._getVerboseInfo() | ||
| self._getStats(verbose=True) |
There was a problem hiding this comment.
_getStats() is defined to take no arguments, so this will result in an error
There was a problem hiding this comment.
oh-oh - this wasn't plumbed through to HSDS. I've taken out the whole function.
hsstat (the only thing that used it) now just prints "rescan is not supported".
|
|
||
| return attr_json | ||
|
|
||
| def getDatasetValues(self, dset_id, sel=None, dtype=None, query=None): |
There was a problem hiding this comment.
HsdsPlugin doesn't override queryDataset(), so when h5pyd calls it, it will falls back to pulling down every element in the queried range rather than using the query. Not a crucial fix for the release, but we should make a note this for later.
There was a problem hiding this comment.
Yes, I'll make an issue for this.
3 participants
Overview
This branch migrates h5pyd's storage/object-model layer to the shared h5json library instead of h5pyd's own bespoke
JSON-schema/selection/dtype code. h5json now owns the HDF5 object-model representation
(groups/datasets/datatypes/attributes as JSON), dtype (de)serialization, and selection logic, while h5pyd is reduced
to the high-level (h5py-compatible) API plus an HSDS-specific storage plugin. This lets h5pyd share that core logic
with other tools built on h5json (e.g. HSDS itself) rather than maintaining a parallel implementation.
Major changes
Core architecture
h5json (h5json.selections, h5json.hdf5dtype, h5json.hdf5db, etc.) — removed ~1,900 lines of now-redundant local
implementation (selections.py, h5type_test.py, requests_lambda.py deleted outright; h5type.py and base.py cut by more
than half).
handles object/attribute/link CRUD, chunked value reads/writes, point/fancy/hyperslab selections over HTTP, and
flush/dirty-object tracking.
reflecting their new role as shared infrastructure rather than internal _hl details.
New/improved features
per-object round trips, when available.
pytable-style query expression and returns a region reference (point selection) over matching elements.
compound-field convention for compatibility.
flush-before/flush-after semantics around query-based updates so local and server state stay consistent.
Bug fixes
absolute so it also runs standalone, not just as an installed package.
Housekeeping
from the README.
Test coverage
Extensive test suite changes alongside the migration: several existing suites were reworked to run against both real
h5py and h5pyd (test_dataset.py, test_dataset_getitem.py, test_group.py, test_vlentype.py, test_dimscale.py,
test_table.py, etc.), plus new test files (test_attribute_create.py, test_attribute_data.py, test_file_read.py) and
substantial expansions to test_dataset_pointselect.py and test_dataset_query.py.