IP Storage: use ragged list Zarrv3

[indecisiveuser]

This PR aims to change both the data format and structure of stored interest points. Currently, interest points are stored in n5 format. There is one folder per timepoint/viewid combination. All interest points for that tile are stored in an array containing the IP id and its (x,y,z) coordinates. Then correspondences are stored as an adjacency list: correspondence id in the current tile --> timepoint, setup number, label, detection number in the other setup.

We are trying to migrate to zarrv3 for all parts of the registration pipeline. But we cannot reap the benefits of sharded zarr3 files for setups consisting of a large number of small variable-length files.

Therefore, this PR introduces uncompressed ragged list zarr3 interest point storage. In particular, for each timepoint, setup number, and label, an index is kept. This shows the offset and count of each view's interest points in a flat IP list. A two-level indexing scheme occurs for correspondences: view and pair indexes, where view indexes into pair. In more detail, the view index stores (per timepoint,setup,label) the offset and count of the number of pairs in the pair index. Then, the pair index stores (for the timpoint,setup,label in question) each partner's timepoint,setup,label in addition to its offset and count in the flat correspondence array.

This becomes more interesting when considering the case of IP re-detection or re-matching. In particular, if one re-detects interest points on a single tile, the new version appends these points to the current points / correspondences lists. Then it updates the index to reflect the new location of the points. This workflow applies whenever 3/4 or more of the array would still be considered to be 'current' following an update. (Here, ¾ was an arbitrary choice; I have not benchmarked this.) If <=3/4 of the array would be 'current' following an update, the index would be rewritten.

I have benchmarked the current version against the new IP storage structure on a dataset crop (25 tiles from LICONN mirror scope data). I chose 2^16 point chunks / 2^20 point shards as this performed well in my exploratory tests. Here are the results:

task	wall (old / new) (s)	IP work (old / new) (s)
detect	57 / 35	10 / 6
match	359 / 244	11 / 10 (RANSAC not counted)
split	34 / 18	28 / 13
solve	51 / 15	26 / 1
TPS fusion	1419 / ~1407 (est)	12.4 / 0.5 (est)

Additionally, the number of IP files decreased from 10,043 to 32. Evidently, the efficiency of all parts of the registration pipeline seem to have improved (some drastically like the solver). TPS fusion was not run as part of the suite because most of it does not use interest points, and so I did not figure the extra compute cost to be necessary. Complete runs showed that the two methods produce output that is essentially the same. (There are minor differences due to the non-deterministic nature of RANSAC etc.)

P.S. In order to better extract performance gains from the new data structure, there were some changes in BigStitcher-Spark. These are located at: https://github.com/JaneliaSciComp/BigStitcher-Spark/tree/ip-store-update

[StephanPreibisch]

Great initiative - since we are touching this, interest points may have further attributes (size once we implement a full Difference of Gaussian pyramid, intensity is already supported in BigStitcher-Spark as an optional attribute stored in its own N5 per view, consenus-set for correspondences ...). I think it is fine to add additional files for different attributes since it is optional. Let us discuss it more ...

[indecisiveuser]

Hi @StephanPreibisch, thanks for your comments.

I have updated this to hopefully have a better structure. Now,

1. InterestPointsN5 has now been stripped of all instance methods. As such, it is static-only.
2. InterestPointsN5ToZarr reads unconverted entries from old N5 stores, and modified lists are resaved to the Zarr store. It also converts entire N5 stores to Zarr stores on request.
3. InterestPointZarr / InterestPointsZarrStore handle the new IP reading / writing
4. Optional attributes (e.g. intensity and size in the future) can now be easily added as an extra column in the stored IP array (default value = -1)
5. I have confirmed what occurs when extra IPs (or correspondences) are added. In particular, the metadata gets rewritten to reflect the new dimensions, the last (partial) shard is rewritten, and the index is updated. Whole-IP dataset rewriting occurs when a) a new optional attribute is introduced, or b) when <75% of the current data on disk is currently in-use