Modify archiver to improve data retrieval usability
There are substantial challenges with data retrieval due to the fact that there is padding at the end of the recorded history segment.
One solution I proposed was to add the length of contents to each record: https://github.com/autonomys/subspace/issues/3318#issuecomment-2550135643
This might work, but introduces some churn during archiving.
While at it, there is trickiness in the archiver caused by variable length encoding that results in non-determinism in some cases that requires extra complexity to handle. This should be fixed, probably by switching to `u32` for length encoding everywhere (since real blocks are likely to be larger than 2^16 bytes anyway).
Another side effect of the variable length encoding is that in Subspace the number of segment items in a segment is not included at the beginning of the encoding. The reason for it is that it would cause bytes shifting, which would break incremental archiving there. There isn't incremental archiving here and knowing the number of blocks in a segment isn't essential, but still something to keep in mind.
With all these improvements it should be possible to make archiver write things strictly sequentially and in a fully deterministic way based on immediately available input. On a flip side it should be possible to make efficient data retrieval way less (see https://github.com/autonomys/subspace/issues/3318 for a small part of the discussion in Subspace).
0 条评论