`gcs_grpc` can produce corrupted reads

We ran into a situation where auth refresh resulted in corrupted reads: - we refresh every N minutes. - if a read occurs precisely at N minutes, some portion of that read can be corrupted data. - this doesn't happen with the `gcs` driver. - the reads are nothing complicated. Below is a simple repro. It's not _exactly_ the code we ran but it's very very close with open source tooling: ``` import time import numpy as np import tensorstore as ts BUCKET = "my-bucket" PATH = "credential_refresh_test" DURATION_MINUTES = 300 # Write write_spec = { "driver": "zarr3", "kvstore": {"driver": "gcs_grpc", "bucket": BUCKET, "path": PATH}, "schema": {"shape": [10, 10], "dtype": "float32"}, "create": True, "delete_existing": True, } write_store = ts.open(write_spec).result(timeout=30) write_store.write(np.random.randn(10, 10).astype(np.float32)).result(timeout=30) print("Write complete") # Read repeatedly read_spec = { "driver": "zarr3", "kvstore": {"driver": "gcs_grpc", "bucket": BUCKET, "path": PATH}, "open": True, } read_store = ts.open(read_spec).result(timeout=30) for minute in range(DURATION_MINUTES): print(f"Minute {minute + 1}/{DURATION_MINUTES}: reading...") data = read_store.read().result(timeout=30) assert data.shape == (10, 10) print(f"Minute {minute + 1}/{DURATION_MINUTES}: success") if minute < DURATION_MINUTES - 1: time.sleep(60) print("Done - all reads succeeded") ```