ITADN

Strategies for pipe-lining additional computation over TensorStore/KvStores

#262Closedsjperkins 创建于 2025-11-25
S
sjperkinscommented
We're writing an xarray interface over a HTTP archive containing NPY blobs that uses tensorstore to retrieve and transform data for presentation to users/developers. Additional context: - https://github.com/ratt-ru/xarray-kat - https://github.com/ratt-ru/xarray-kat/issues/2 - https://github.com/google/tensorstore/issues/253 - https://github.com/google/tensorstore/issues/258 As it stands we've used `virtual_chunked` TensorStore's to transform raw data from the HTTP archive. Is this the standard technique for transforming data via the TensorStore python API? The reason I ask is that we have three large arrays of data (Visibilities, Weights and Flags) whose final values can depend on each other. For the purposes of discussion, rather than requesting a deep understanding of the domain, I've included a diagram of these relations below (rechunk operations can be ignored): <img width="1920" height="1080" alt="Image" src="https://github.com/user-attachments/assets/c0b565a8-7eae-45e7-b2f6-5d5e45974189" /> Fortunately, these operations are performed chunkwise. Our current strategy for presenting the final TensorStores to the end user is to use a `virtual_chunked` constructed with pickleable Multitons for the dependent TensorStores. Additionally, we would use caches (or batches) to avoid repeated requests for data. This would probably involve constructing separate thread pools for each `virtual_chunked` to avoid deadlock. For example, a `scale_by_power` TensorStore would depend on weight and visibility kvstores, possibly applying the `Van Fleck` transform and `Calibration` solutions. Perhaps there is a better way of achieving the above goals?
关闭于 2025-12-10 1 条评论